Why is it "astrologer" and "geologist", but not "astrologist" or "geologer"?
Firstly the etymology of the two suffixes.
-ist
word-forming element meaning "one who does or makes," also used to indicate adherence to a certain doctrine or custom, from French -iste and directly from Latin -ista (source also of Spanish, Portuguese, Italian -ista), from Greek agent-noun ending -istes, which is from -is-, ending of the stem of verbs in -izein, + agential suffix -tes. - Etymonline
-er (1)
English agent noun ending, corresponding to Latin -or. In native words it represents Old English -ere (Old Northumbrian also -are) "man who has to do with," from Proto-Germanic *-ari (cognates: German -er, Swedish -are, Danish -ere), from Proto-Germanic *-arjoz. Some believe this root is identical with, and perhaps a borrowing of, Latin -arius (see -ary).
Generally used with native Germanic words. In words of Latin origin, verbs derived from past participle stems of Latin ones (including most verbs in -ate) usually take the Latin ending -or, as do Latin verbs that passed through French (such as governor); but there are many exceptions (eraser, laborer, promoter, deserter; sailor, bachelor), some of which were conformed from Latin to English in late Middle English. - Etymonline
Thus, the -er suffix is from Germanic origin, while -ist is from Latin.
In the specific case of the astrologer there might be some more information from the following excerpt to consider.
astrologer (n.)
late 14c., from astrology + -er (1). Drove out French import astrologein, which, had it survived, probably would have yielded *astrologian, as in Chaucer's "The wise Astrologen." Earliest recorded reference is to roosters as announcers of sunrise. - Etymonline
The first astrologers were animals, if we consider the two dictionary entries from ODO regarding the two suffixes, -er and -ist, we can see that -ist is constricted to people, while -er explicitly applies to people or things. Since -ist is reserved for humans the astrologist had no chance while the job was done by an animal.