Two points I'd like to make:
1. Several people have mentioned that life appeared pretty quickly on Earth, but it took it a long time to achieve complexity and intelligence. As I just spent a long time arguing in another thread, you can't really extract any meaningful information from this, because our sample size here is one. It might have been an incredible stroke of luck that it "only" took a few billion years for intelligent life to appear, or it might have been incredible bad luck that it took so long. We have no way to know at the moment.
2. As for the Fermi paradox, remember that we really have no way of knowing whether or not aliens might have visited us in the past, or what their motivations in settling various planets might be. Maybe some alien civilization has colonized every planet in the galaxy that they found hospitable, but they didn't like any of the planets in our system. Perhaps they visited Earth a million years ago (or a thousand years ago), and were disappointed to see that we have an oxidizing atmosphere rather than a reducing atmosphere, so they just moved on. Although I'm not one of the nuts who suspect that aliens are constantly visiting us and abducting people etc, I think it's interesting that if aliens had visited our solar system even a few hundred years ago, we probably wouldn't have known about it. Heck, some alien probe the size of a bus could be coasting through the solar system and beaming data back home with a directional antenna/laser/whatever right now and we probably wouldn't see it.