What were nightmares called before "nightmare" was used in that sense?

One possibility could be "wodan dreame".

Before getting to this point I'd like to add that whereas the spelling "nightmare" is indeed recent (see the corresponding Google ngram), more archaic spellings are reported in the OED; viz "niȝt-mare" (1290), "nytmare" (1340) and "Nyghte Mare" (1440).

Leaving aside older spelling of "nightmare", I also came across an article on Google books suggesting that when preceded by "wod" or "wodan", then the dream turned out to be a nightmare...

This is in line with Wodan's main domains of death, war and afterlife. The article oldest quote is circa 890 and there are also quotes from Ælfric of Eynsham. Unfortunately the pages after 161 are not available from Google books.

Update
Then, starting with the 18th century the Latin world "incubus" was in use. But that's not Old English any more I'm afraid. It's still used in Italian (under the form incubo) actually.

As another contender for what nightmares were called before it got its current meaning I proffer:

Ephialtes

Etymonline's entry for ephialtes says: "nightmare or demon that causes nightmares, c.1600, from Gk. Ephialtes, name of a demon supposed to cause nightmares; the ancient explanation is that it was from ephallesthai "to leap upon," but OED finds "considerable" phonological difficulties with this."

This entry leaves some ambiguity whether or not 'ephialtes' was used to describe a bad dream or was specifically the name the demon believed to be responsible.

Websters Online dictionary has information from the 1913 and 1828 editions. From the 1913 edition we read: "1. The nightmare. Date 'Ephialtes' was first used in popular English literature: sometime before 1321." From the 1828 edition we get: "[Noun] The night-mar."

By using the definite articles before 'nightmare' and 'nightmar', Webster's Dictionary implies that 'Ephialtes' refers to a specific thing, namely the demon believed to cause bad dreams.

"A Dictionary of Hallucinations", by Jan Dirk Blom, has considerably more information about terms used to describe nightmares and other human perceptions; excerpts of some entries are provided. On page 128 we find a definition for 'Daymare': "Daymare-Also known as 'ephialtes vigilantium'. The term daymare is indebted to the Old English noun mare, which means hag or goblin."

The definitions of 'mar' and 'mare' (male and female love-phantom) are found on page 317. Both of these entries reference a book by William Heinrich Roscher (1845-1923) entitled, "Ephialtes, A pathological-mythological treatise on the nightmare in classical antiquity."

Finally, on page 357, there is an extended entry for 'Nightmare': "Nightmare-Also known as 'ephialtes nocturnus', dream anxiety attack, REM anxiety dream, REM-nightmare, and D-nightmare... ...In 1830 the Scottish physician Robert MacNish (1802-1837) qualified the nightmare as follows: 'Nightmare may be defined [as] a painful dream, accompanied with difficult respiratory action, and a torpor in the powers of volition... The affection, the Ephialtes of the Greeks, and Incubus of the Romans, is one of the most distressing to which human nature is subject.' "

From these entries, it appears that 'ephialtes' is used in a clinical sense, where it is differentiated between 'ephialtes vigilantium' and 'ephialtes nocturnus'. Also, there is evidence that 'nightmare' was used in a clinical sense; it does not say if 'nightmare' was used informally to describe a bad dream.

Etymonline's entry for nightmare says: "late 13c., 'an evil female spirit afflicting sleepers with a feeling of suffocation,' compounded from night + mare 'goblin that causes nightmares, incubus,'... ...Meaning shifted mid-16c. from the incubus to the suffocating sensation it causes. Sense of "any bad dream" first recorded 1829; that of "very distressing experience" is from 1831." [emphasis mine]

Although, as the original question reports, the first recorded use of 'nightmare' for 'any bad dream' was in 1829, the meaning shifted in the mid 1500s from the Incubus to the distress 'it caused'.

From ngrams.googlelabs.com we can compare the usage of 'ephialtes' with 'nightmar' between 1800 and 2000:

It is clear from the following two images that 'nightmare' is, and has been, in much wider use than 'ephialtes' and 'nightmar':

A little late, but nonetheless:

The word "dream" was used for both pleasant and unpleasant dreams. It is so in Middle English as well:

drēm

1. (a) A vision experienced in sleep, a dream; a nightmare; a prophetic dream;

"Ephialtes" was used as nightmare as well, but it also meant the demon that caused it, so I don't think it classifies to your query.

Old English drēm, as mentioned before, didn't seem to have the same meaning as today, or it had a second meaning that hasn't been recorded. The words swefn and gesihþ seem to have meant "dream". Swefn originally meant "sleep", as other etymologically cognate nouns in other languages, Lithuanian sapnas, Old Church Slavonic sunu, and the Romanic words (French songe, Spanish sueño, Italian sogno all from Latin somnium (from PIE *swep-no-; cognate with Greek hypnos;).

Old English had ælfādl (ælf "elf" + ādl "disease, sickness") which meant "nightmare", since they thought elves caused them.

Etymonline.com on Elf(n.)