I almost recommended the Foley book, but it's woefully out of date (publication date of 1995, but that's just for the most recent edition; the original was published much, much earlier). The basic math background is pretty well done, but the red book discussion ought to be equally useful for this context. If you're after a good reference for CG, the Foley book used to be the gold standard because of the breadth of its coverage. Like most things computer graphics, though, it hasn't aged well. A search turns up what purports to be a PDF of the book, but it doesn't say what edition and I don't know if it's real or not.

http://www.realtimerendering.com/books.html is a good book list and there are a number of free ones in there. For good background on the underpinnings of physics, optics, etc., I heartily recommend Glassner's Principles of Digital Image Synthesis ( http://realtimerendering.com/Princip...sis_v1.0.1.pdf ), and not just for witty commentary like "This is an approximation based on using an ideal spherical cow of radius R."