The correct answer is yes. Yes, you should. So don't tell anybody that I sometimes use the following tricks to speed up my world-building process.
Have you ever noticed how fractal geography is? You can zoom in on any part, and it generally looks like any other bit of land. To use this to your advantage, go to Google Maps, find a relatively obscure bit of geography (i.e. don't use Long Island or the SF Bay Area, or anything) and zoom in until it looks like something you can use. Take a screenshot, change the scale, and voila! One fantasy continent.
If you can tell me where in the world these maps are from, you win a custom sketch. Not even joking.
It turns out history is just as fractal as geography: zoom in on any point, and you can find something to use for your world. Need a war? Take your pick. How about a revolution or a realistic-but-obscure form of government? Wikipedia (and the internet in general) is full of stuff like this. Just change the names and dates, perhaps a few key details here and there, and you've got your own semi-original history.
The same thing applies to creating a civilization. Do a little research on some unknown people group, then mix and match their traditions and values with some other culture you're into. Choose a technology level, flesh it out by asking how, why, and what result, and pretty soon you have a viable society with relatively little work.
Does this sound unoriginal? Like plagiarism? It's not, really. Stealing from our own world's history, geography, and cultures is no different than creating characters by mixing and matching attributes from yourself and the people you know. Just like making up fantasy languages, the trick is obscuring your sources.
And I'm not suggesting you build your entire world this way. Use this as the foundation, then tweak and twist things as you go. Ultimately the parts you care most about will be the parts that are most originally you, while the rest of the world still feels fleshed out because it has a strong, realistic base behind it.
Anyone have any other quick-and-dirty tips on building a world? I'd love to hear them.