Actually, our physiological perception of color bears little resemblance to the color theory we teach either artists or children. The primary colors we teach to children aren't really related to any physical properties of light. And in the art world there are actually two sets of primary colors.

Red, green and blue are the additive primaries (adding more light makes the color brighter). Mixing all three equally gives you white light, and any color of visible light can be created using these three colors. Cyan, magenta and yellow are the subtractive primaries (adding more pigment makes the color darker). Theoretically, mixing all three of these in equal measure will give you black. I say theoretically because that would require perfect pigments, which is something we cannot actually make. The reality is that mixing the colors typically gives you a very dark brown.

Furthermore, the additive primaries are the subtractive secondaries and vice versa. That is, red and cyan are complementary colors. If you subtract red from an image in Photoshop, you'll increase its cyan. Conveniently, when we talk about RGB and CMYK colors, they're arranged in a particular order that makes those relationships easy to remember. Just write them down in two lines:

RGB
CMY

Red is complementary to cyan, green to magenta, and blue to yellow.

Color perception is also a cultural artifact. There is at least one society in the south Pacific that has only one word to describe both blue and green. As a result, they cannot even tell the difference between the two colors. They can still describe the color with qualifiers, much like we can say something is "pale blue" or "a slightly purply blue," but no matter how green something is, they'll still associate it with blue. There is another culture in southeast Asia that doesn't have a way of describing hue; they can only talk about light and dark. Alas, I don't remember the names of either of those peoples; it's been far too long since my anthropology classes. It will be in my notes somewhere, but I'm not sure where those ended up.

Anyway, the consequences of a race that perceives color differently than we do might be text that is difficult to read because their rules of contrast are different. Maybe light blue text on yellow is very contrasty to them, but we just get a headache looking at it. Their notions of color harmony could be very different, resulting in art that we find ugly being thought very beautiful by them. Although, again, that's largely a cultural thing; our own Western ideas of color harmony are quite different today than they were even 50 years ago.