I believe that everybody sees the same colors as everybody else. Even with color-blindness you'll still see the same blues and yellows and pinks and browns and stuff. The notion that someone else's brain could perceive colors differently to you was one I toyed with when I was little, but I realised after studying it that the basic programming in the visual cortex is part of the DNA - it's not just random for each person. So you'll share the same color perception as everyone else in the same broad species.
I've limited evidence to back this up - the colors that people tend to like and dislike are found to be consistent across different races and societies. This means that there is basic programming in our brains that relies on a consistent interpretation of colors for survival. Perhaps that's why people don't like brown very much. Did the early humans survive better when they lived in green areas than barren muddy areas? Did they survive better when they didn't try to eat animal faeces? Of course they did, so maybe color helped them to make the right choices.