For me, a scary aspect of worldbuilding is the potential that the story world can hold. I mean, I'm making stuff up, acting as a kind of a god, divine being. That's an ego trip in of itself, but there's so much to put into that world too. What happens if, what happens where, what happens to whom, what do they look like, how do they do what they do, and why does it all happen? So many potential choices, so much potential to go wrong, pull the blankets over my head and hide.
As tempting as hiding might be, Stories don't shut up--they want to be told, and told nao, damnit. So worldbuilding needs to happen. Some people do minimal work; they know the place and vague hand-wave'ems, and move on with the story. Some storytellers make stuff up as they go along, and create a story bible out of necessity. I know others who go through elaborate mapping, but leave the small details up for grabs as they go. Some writers build brand new planets with wonky things stuck in them. Then there's people who go to the extreme of planning the physics of the place, and how that affects the whole universe they're creating.
All of these ways of worldbuilding are legitimate techniques. YMMV, of course. And my mileage definitely does vary. I start with Story for a very good reason--I've got a lot of choices already made, so I don't have unlimited potential.
Will there be travel? Yes, and I know I've gotta create a map that's larger. No, and I just need a map of the place this Story happens. Also, I'll need to deal with transportation if they're travelling.
What tech level? Story tells me if it's the Bronze Age, the Space Age, or the Far, Far Future, or a Romen-esque feel. Lots of information shakes out of that one decision--travel, food, drink, agriculture, building materials, morals, ethics, crime, communication, just to name a few.
Can I use an Earth-based place as a model for this Story world? Yes, and I've got all my plants, animals, water resources, fish, metal and material resources, agriculture, birds, and a whole bunch of things pre-planned. All I have to do is look things up and decide what I want and don't want to use. If the answer is no, I get to do research! Yay, Research!
What do I not want in this world? I love leaving expected things out. In my current Migraine-in-Progress, I'm leaving out *redacted* and *cough* because everyone expects those *sniffles* to be in every world, and I don't wanna. So many more types of *sniffles* out there to bring forward, and I can have FUN with my Easter eggs in a particular story.
Once I've got these four questions at least wrangled in a specific direction, I can move on to the more intricate parts of worldbuilding. But answering these four questions help me get the big picture of what I'm creating.
Then the real fun begins. *grin*