The reason for the assymetry is yet to be resolved.In short, by counting photons, we find that for every 10 billion matter/anti-matter annihilations, early on, there was one matter particle left over, so this is but a very slight break of symmetry, but enough that most or all of the anti-matter is gone. It is not necessarily what shouldn’t have happened, though, for during expansion, and especially during inflation, there is no equilibrium, as a lot of stuff is separated very quickly in a really short time.
In fact, it is one of the most important unsolved features of the universe. The idea you give that there is one matter particle left over for every 10 billion matter/anti-matter collisions is just speculation, but still doesn't explain why that asymmetry should occur.
Baryon Asymmetry
Also, if the universe should equal nothing then there should be no assymetry at all (not even one per 10 billion) as nothing is completely perfect and cannot contain any asymmetry - as then it would not be nothing as it could be defined through the asymmetry.
Observation tells us that there are asymmetry between matter/anti-matter, so it may not be a fact that symmetry absolutely has to be. I'm not saying that it is a fact that it doesn't have to be either, there could be a symmetrical explanation why there isn't a equal amount of matter/anti-matter too, such as anti-matter possessing anti-gravity (which could mean that it is completely seperated from us, and perhaps even have formed another universe). All we know is that the observable universe has that asymmetry, so we can't conclude that symmetry must be true in all circumstances.As such, too, the virtual particles of the pairs were separated so quickly that they couldn’t get back together to ‘go back in’. Could be something like that for gravity, too.
Symmetry is an inescapable conclusion, and if super-partners are found, then nature will have used all possible symmetries, as all the rest are already employed.
Matter and anti-matter, space versus time, polarity of charge, mass versus gravity, and others all play a part, as exposing symmetry; so, the ultimate answer is to be found in why symmetry absolutely has to be, no other option possible, and, of course, a causeless prime mover can’t have any, unless it has all of them, as equal, and this is Symmetry.
They haven't been able to measure if anti-matter have anti-gravity as they have so small amounts of it (it would still follow ordinary space-curvature as it couldn't do much to bend it the other way).