Quote: "but there is only so far you can go without making the game look unrealistic or breaking out of genre"
Well, this isn't true.
Take a look at "Lifeless" for example, the Scenery looks much more realistic than the most Fpsc "Horror Projects" I saw before, but it still has this frightening touch, which comes from the realistic Level Design (the Maps are abandoned, empty and all that).
I think that you're view of horror is a bit...onesided.
There a different kinds of Horror.
1) The 'In your Face' Horror, like the most american Movies.
2) The Horror Movies like the original "Ringu", which Horror is based on the Sounds, the overall Atmosphere, for me it's kinda frightening on a constant Level.
The list could probably go on for plenty more numbers, but you should get what I mean.
For me I must say that your current Mapping IS unrealistic, and not really frightening, because it's repetitive and everything has to be "filled with Entitys"...
You should use your good Level Design Skills, and experiment with Scenarios that are based on real Photos... just to see how it comes out.