The core of horror is the emotion of "I have no idea what is going on and I am unequipped to handle it". Something is wrong. It's a confrontation with what you don't know and don't understand in a liminal intersection between the perceived rules of reality and that reality itself.
If you ask people what they are frightened of you'll get seven billion different answers and change. Everything comes back to "something is wrong", and further categorization is an exercise in missing the forest.
The horrific element can be as mundane or fantastic, banal or supernatural as you want, but it all comes down to the great violation.
"There is a tiger in the room", "the government is feeding people to tigers" and "the tiger stands up and begins to speak in slurred and broken words" all roost in the fact that there is a tiger not where (you think) it is supposed to be e.g. very far away from you.
Now then: how to use this in games.
My experience: ratchet up that imagery. Burn the wrongness of it all into your players heads. You can make anything scary with a good description, even if it's a single goblin with a kitchen knife. Names and boundaries make people comfortable - get rid of them. Turn them on their head. Don't play by the narrative rules your players expect. Get grotesque. Let things build up over time instead of cashing all your chips at once. Let the players' imaginations do the legwork for you.
Prime example, the man in the car from when I played Esoteric Enterprises. Ordinary enough start followed by the ramp-up. The audible "oh shit" when it was revealed that he had shot himself in the head three times was music to my ears. But, having now used that trick, I won't be able to get the same effect if I try it again because it will no longer be a violation of the normal-predictable-knowable world.
I've written a bit about how horror can evolve into less-horrible things over time before and there's a huge flaw in that essay: sometimes shit just refuses to be understood. The mythic underworld is the mythic underworld and no amount of self-assured Enlightenment era skepticism changes that. Sometimes there are no answers. This is best to keep it horrible. Do not explain.
(Spoilers for US, presented in Rot-13 cipher)
Vg znxrf zr zvssrq gb urne crbcyr pbzcynva nobhg ubj gur ghaaryf va HF qba'g znxr nal ybtvpny frafr - bs pbhefr gurl qba'g znxr ybtvpny frafr! Gung'f jung znxrf gurz ubeevslvat! Gurl erwrpg lbhe ernyvgl! Erq jnf whfg bssrevat gur orfg rkcynangvba fur pbhyq guvax bs naq vg vf pyrneyl jebat!
Now then: How to use this for Mothership in particular.
"Xenomorph" and "gigantic fleshblob" are classics for good reason, but they are known quantities. It's still possible to get people creeped out by them, but you really need to pull out the stops. Kinda like using Mythos monsters when running Call of Cthulhu - players expect certain things. But, that can be used to one's advantage.
Somewhat counter-productively, I'd say to aim for something that doesn't immediately get people thinking about Alien or Event Horizon. Even if you are using xenomorphs and gigantic fleshblobs. Use horror that doesn't normally "belong" in space (it belongs wherever it pleases to). Lean into your own fears.
- You wake up from hibernation and there's a live tiger in the cryobay.
- You return home to find that everything swerved to some sort of Brazil meets Repeairer of Reputations setup and everyone talks like it's always been like this.
- The colony you were supposed to land on is filled with farming drones doing a really bad LARP of the Wizard of Oz and you can't find any of the colonists.
- A ship emerges with hyperspace, every available cubic meter filled with asphyxiated corpses.
- You've missed three generations of your family; you can't understand them and they don't like or understand you.
This post brought to you by Great Lakes Brewing Co. Holy Moses Raspberry White Ale.