Right.
Now imagine that you decide that natives are beneath you and consciously keep away from them. You marry only your kind. Your group gets together and avoids outsiders; punishes any one who breaks the rules by kicking them out. Concepts of pure blood and supremacy reign. Who are you really hurting?
Yeah, but what if I secretly bang hot natives? It worked for Thomas Jefferson!
So it could be different cues:
See a brown person, not want to share.
See a lusty, brown babe, of childbearing stature, and your competitive thoughts turn off, and other thoughts are, uh, turned on.
Edit
Marriage?
Wtf?
What sort of advantage does MARRIAGE have?
Actually, we're arguing sort of black and white, but I'm certain the truth is in the middle. Yes, people are hard wired for group behavior; that doesn't mean we are incapable of behaving otherwise, or have contradictory wiring to tell us to behave differently to maximize fitness. Organisms aren't ideologues, after all.
But that brings me back to marriage. In many human societies, monogamy is absolutized, moralized, scandalized. But if you look at monogamy in the animal world (or apparent monogamy), many times it's rife with cheating. Why? Because having a mate look after offspring and sharing resource gathering mitigates individual cost BUT sleeping around also increases fitness. Both behaviors must increase fitness, otherwise we wouldn't see such finely evolved behaviors in the same organism.
I did some genetics work for a woman who was studying this exact thing in the supposedly monogamous black capped chick-a-dee. She got some veerrry interesting results. Big project, though.