Why does "prayer" end with -er?
Solution 1:
It ends in ‑er because it mostly started out that way, and that aspect of it did not change.
The OED says that this word was initially a direct borrowing from French, and that its etymon was the Norman French word praere. They attest an incredible number of variant spellings for this word in Middle English. Just take a gander at how many different historical approaches there have been! As Greg Lee mentioned in comments, the spelling we now use today ended up being the luck of whatever of those super-many spellings in manuscript form that the early printers decided on for printing.
ME praeyer, ME praior, ME praiyer, ME prayeer, ME prayeere, ME prayȝer, ME prayur, ME preere, ME preȝeer, ME preȝere, ME preieere, ME preier, ME preiere, ME preiȝer, ME preiȝere, ME preire, ME prer, ME preyeer, ME preyer, ME preyere, ME preyȝer, ME preyȝere, ME preyor, ME preyour, ME preyowr, ME preyur, ME priere, ME–15 praer, ME–15 praiere, ME–15 praire, ME–15 prayere, ME–15 prayor, ME–15 prayour, ME–16 praier, ME–16 prayr, ME– prayer, 15 prayar, 15 preyare, 15–16 prair, 15–16 prayre, 16 praijer, 16 prai’r, 16–17 pray’r, 18– prar (English regional (Shropshire)); U.S. regional (southern and south Midland) 18 pra’r, 18– prar, 19– praar; Scottish pre-17 praer, pre-17 praier, pre-17 praiere, pre-17 praire, pre-17 prayair, pre-17 prayar, pre-17 prayare, pre-17 prayeir, pre-17 prayere, pre-17 prayier, pre-17 prayour, pre-17 prear, pre-17 preir, pre-17 preyer, pre-17 preyere, pre-17 preyir, pre-17 priar, pre-17 pryer, pre-17 17– prayer. N.E.D. (1907) also records forms ME prair, ME preir.
Sure was convenient when people actually spelled things the way they said them, now wasn’t it? So, do you like any of those better than the one we use today? :) You can be sure that whichever one you pick, the fellow in the next village will have chosen another one. Happy days.
They list as its etymology the following:
Anglo-Norman praere, praiere, preire, Anglo-Norman and Old French preere, Anglo-Norman and Old French, Middle French priere, Old French preiere entreaty, petition, request, act of praying, words uttered when praying (first half of the 12th cent.; French prière) < post-classical Latin precaria (see precary n.); the sense development to ‘entreaty, request in general’ may already have occurred in post-classical Latin; compare post-classical Latin precaria call to prayer (1305 in a French source). Compare Old Occitan preguiera (c1200), pregueira (13th cent. in an isolated attestation), pregaria, Italian preghiera (a1250), Spanish plegaria (13th cent.), Catalan pregària (13th cent.). Compare precary n.
And give the following etymological note:
Monosyllabic pronunciations are suggested by metre from c1600, although disyllabic pronunciations are occasionally recorded in verse until at least the mid 17th cent.
So we’ve had the one-syllable pronunciation in poetry since the end of the 15th century, but the two-syllable version could nonetheless still be found through at least the middle of the 17th century, if not later.
That means that the two-syllable pronunciation was still being used when the printed form got frozen forever.
Never Forget
Remember that English spelling today represents only Middle English pronunciation from half a millennium past, not that of current English. Spellings were locked down long ago, just as soon as we starting setting words in lead — meaning, in movable type. That froze everything. Pronunciations all changed dramatically after that, and often very soon afterwards during the Great Vowel Shift, but by then it was much too late to change those frozen spellings.
Plus you can indeed still have a two-syllable prayer, meaning one who prays.
Solution 2:
French built infinitives with -er and derived participles and nominalized nouns from the infinitive. This was borrowed as is into English to form a noun. The English verb must have been backformed from the noun; Middle English simplified verbs anyway. It helps that -er is a regular nominal suffix, notwithstanding the further etymology.