Feeling like something impossible or paranormal is happening
Solution 1:
As Dan Bron said, the proper term is lovecraftian, which actually refers to a horror sub-genre which deals with the terror of the unknown and sometimes unknowable. While it technically means related to the work of the American horror writer, H.P. Lovecraft, it usually refers to that style of horror.
Lovecraftian horror is a sub-genre of horror fiction that emphasizes the cosmic horror of the unknown (and in some cases, unknowable) more than gore or other elements of shock, though these may still be present. It is named after American author H. P. Lovecraft (1890–1937). From Wikipedia
However, some also use it to extreme tentacle-y, misplaced limbs and appendage style horror as well, so might not always work.
If not, paranoia might work. It sounds good enough for a genre, and it means:
par·a·noi·a : suspicion and mistrust (of people or their actions) without evidence or justification.
Solution 2:
dissonance (Merriam-Webster)
- lack of agreement; especially : inconsistency between the beliefs one holds or between one's actions and one's beliefs
- an instance of such inconsistency or disagreement
cognitive dissonance - psychological conflict resulting from incongruous beliefs and attitudes held simultaneously
Isn't that genre usually called psychological horror, though? Most of Hitchcock's films fit that description. And even though they were sci-fi, a lot of Twilight Zone and Outer Limits stories were mostly psychological horror as well.
Solution 3:
Uncanny, perhaps. Vague and undefined feelings about a situation.
Solution 4:
This feeling can be called eeriness. (also spelled as eariness)
n. an undefined sense of fear; superstitious dread. [OED]
Feeling the sensation of eariness as twilight came on.
- T. De Quincey
As for the genre, you can consider supernatural horror or supernatural horror fiction.
Supernatural-Horror film is a sub-genre that includes ghosts, demons, or other depictions of supernatural occurrences.
Examples: The Exorcist, The Ring, The Omen.
http://thescriptlab.com/screenplay/genre/supernatural#
In its broadest definition, supernatural fiction includes examples of weird fiction, horror fiction, fantasy fiction, and such sub-genres as vampire literature and the ghost story.
http://en.wikipedia.org/wiki/Supernatural_fiction
Wikipedia has a list of supernatural horror films. Some notable examples:
- Paranormal activity
- The Amityville Horror
- The Blair Witch Project
- The Conjuring
- The Exorcist
- The Grudge
- Insidious
- It
- The Shining
- Suspiria
You can check weird fiction also.