Are there rules to determine whether a musician's title will end with “-er” or “-ist”?
There are drummers, buglers, fifers, whistlers, and fiddlers. Folks who play all the other instruments use the -ist suffix -- pianist, violinist, cellist, tympanist, guitarist, flautist, etc, etc, ad nauseum. What determines which suffix should be used?
Solution 1:
What strikes me is that the -er ones look like they are derived from verbs: a drummer drums, a fiddler fiddles, a whistler whistles.
A guitarist plays guitar, a pianist plays piano.
So if the instrument is also (used) as a verb, we seem to prefer deriving the name for the musician from that verb, rather than from the instrument.
Solution 2:
Reference The New Fowler's Modern English Usage.
Compare doer and perpetrator. (-or is the Latin agent-noun ending corresponding to English -er)
English verbs derived from Latin —such as act, credit, invent, oppress, possess, prosecute, protect—usually prefer this Latin ending to the English one in -er.
Some other verbs, e.g. conquer, govern, and purvey, not corresponding to the above description have agent-nouns in -or owing to their passage through French or through some other circumstance.
A select list of differences may be of interest: corrupter and corrector, deserter and abductor; dispenser and distributor, eraser and ejector.
Some verbs generate alternative forms, generally preferring -er for the personal and -or for the mechanical agent (e.g. adapt, convey, distribute, resist)
There are several -ist types:
(a) Forming a simple agent-noun and usually having an accompanying verb in -ize (antagonist, apologist, evangelist, etc.) and an accompanying abstract noun in -ism (antagonism,etc.);
(b) designating a person devoted to some art, science, etc., e.g. archaeologist, economist, artist↔instrumentalist, dramatist, philologist;
(c) designating an adherent of some creed, doctrine, etc., e.g. atheist, Buddhist, Calvinist, hedonist;
(d) modern formations of various kinds, e.g. balloonist, cyclist, fetishist, finalist
Solution 3:
I am tempted to explain the -er musicians' titles as being military-related. Buglers, fifers, drummers...all are associated with military events, ceremonies, or marching. Even penny-whistlers have accompanied marching men...and anyway, "whistlist" would sound like a roster of card-players.
...But there's an exception. The odd-man-out is the fiddler...not associated with military matters, to the best of my knowledge. Perhaps we can attribute the use of "fiddler" to the informal type of music generally played on a fiddle -- folk and country music -- which condemns the musician to the lowly -er. If one plays more formal music, one is therefore recognized by all cognoscenti to be a violinist.
But if you'd like a little giggle, try to imagine a contestant at a local banjo and fiddle festival who advertises himself as "Jim-Bob Yokum, Fiddlist".