What’s the rule for adding “-er” vs. “-or” when forming an agent noun from a verb?

This has been previously touched on here: “Commentor” vs “Commentator”. I'll try to expand on that.

Both -er and -or are so-called "agent noun suffixes", and Wiktionary has entries for both:

• -er
• -or

Generally speaking, -er is much more common in English (which should come as no surprise since it has deep Germanic roots, see the link) and can be easily attached to any English verb to form the corresponding noun (drive — driver, run — runner, drink — drinker, etc.).

The suffix -or, on the other hand, comes from Latin, and is used much more seldom, basically where Latin would do it. Just try building the words drivor, runnor or drinkor, and see for yourself. In fact, Wiktionary lists only a handful of terms that were derived using this suffix, such as actor, author and sculptor, and goes on to provide the following usage notes:

English generally appends this suffix where Latin would do it—to the root of a Latin-type perfect passive participle. For other words, English tends to use the suffix -er. Occasionally both are used (computer vs. computor).

Etymonline has additional info on the origin and usage of -er and -or.

Edit: courtesy of Martha, here's a link to a post on the "Separated by a common language" blog that provides further details and addresses the differences between US and UK English:

The -or suffix is primarily found in words derived from Latin, whereas -er can be put on the end of just about any verb that involves an agent (a 'doer' of the 'action'). But Latin-derived words differ in how strongly they are associated with the -or suffix. Latin-derived verbs that end in -ate, for example, almost always take the -or suffix. So we have dictator, but not a variant dictater, alternator but not alternater.

Things are less clear-cut with other Latin-derived verbs. For example, in my job, I advise students and convene courses, and when I spell out those roles, I'm an advisor and a convenor, but when my UK university spells them, I'm often an adviser (which just looks wrong to me) and a convener. [...] The -or form is stronger in the US than the UK, though there's considerable variation within each country.

The best rule I've found?

If you can change the word to have "ion" at the end, it is OR. If you can't, it's ER.

TeachER (can not be teachion) ConductOR (conduction) ProfessOR (profession)

I can't really think of any ER's sorry!

Latinate agent nouns end in -tor (or -sor)

As RegDwigнt's answer mentions, verbs that form agent nouns in -or usually have the form of Latin perfect passive participle stems.

Latin perfect passive participle stems characteristically end in t or s. So in practice, verbs that form agent nouns in -or almost always end in -t(e) or -s(e), and form agent nouns in -tor or -sor specifically. In the context of Latin, the suffix is often given as -tor rather than as -or.

Other -or words tend to be from French

There are agent nouns in English that end in -or preceded by a letter other than t or s. The main origin of this type of -or agent noun is French: in Old French, the Latin ending -atorem (as well as some other similar endings) was phonetically reduced to -or/-our/-ur (Modern French -eur), which was taken into English as an agent-noun suffix -or.

• A large proportion of -or agent nouns only occur in legal English, where -or is used more than in regular English (often as a counterpart to -ee). For example, "deliveror" and "settlor" are basically only found in legal contexts; in other contexts, the spellings "deliverer" and "settler" are used.

• Some -or agent nouns that are not restricted to legal English are conquer-or, purvey-or, survey-or, counsel-(l)-or, and vend-or (the last has a less common but accepted variant vend-er).

• sail-or is an example of the French suffix being used on an English base, the verb sail.

There are other -or words that don't end in -tor or -sor, but most of the rest aren't built directly on an English verb as a base. For example, donor, emperor, tailor, juror don't refer to people who "done", "emper", "tail" or "jure".

There seems to be sporadic use of the form -or instead of -er simply as a means of forming words with a specialized meaning, sometimes as part of technical terminology or jargon.

• The Oxford English Dictionary entry for the word "sailor" describes it as "An altered spelling of sailer n., probably assimilated to tailor, in order to distinguish the designation of a regular calling from the unspecialized agent-noun. The differentiation, however, does not appear in our early examples, and was not fully established before the 19th cent."

• In electronics, the form "expandor", as well as a derived term "compandor" (from compressor and expandor), has apparently been used.

It's hard to rule out -er as a potential spelling for an agent noun

Ending in -t(e) or -s(e) is (mostly) necessary for a verb to form an agent noun in -or, but it certainly isn't sufficient. Sven Yargs left a comment mentioning "disrupter" as an example of an agent noun that is commonly spelled with -ter.

The suffix -er is highly productive and is used to form agent nouns with a range of meanings. Words ending in -er can be animate (runner, worker, speaker, reader) or inanimate (washer, dryer, circuit breaker, holder). They can be names of professions (teacher, writer, baker, publisher, typographer, undertaker).