What do we call 'Shakespearean trash-talk'?

If it's mild, but witty, it's simply word-play.

If it's humorously argumentative, it's repartee. or perhaps verbal tennis (I made that one up). They are trading ripostes.

if they try to outdo one another with their words, they are having a battle of wits or a battle of words.

if they insult one another, they are trading barbs or trading insults. They are sharp-tongued.

If they are quick to come up with a retort, they could be said to have a rapier wit.

How about flyting? It was a fairly commonly-practised activity in Shakespeare's time. Essentially it was the equivalent of a rap battle, in which it was not unusual for participants to insult the virility of their opponent, or suggest that their mothers were... promiscuous.

As a formal/literary alternative to trash-talk I suggest :

altercation:

• a heated or angry dispute; noisy argument or controversy.

or, with a stronger connotation, a confrontation:

• a situation in which people, groups, etc., fight, oppose, or challenge each other in an angry way

(TFD)

The Clown in As You Like It (act 5, scene 6) identifies seven forms of Shakespearean trash talk, each tied to a level of escalation in a quarrel:

the Retort courteous,

the Quip modest,

the Reply churlish,

the Reproof valiant,

the Countercheck quarrelsome,

the Lye circumstantial,

and the Lye direct.

—all of which may be obviated by a well-timed if.

A footnote to the linked (1766) edition of As You Like It points out that Shakespeare took the Lye circumstantial (or conditional) and the Lye direct (or certain) directly from a book on disputation and dueling by Vincentio Saviello called Of Honour and Honourable Quarrels (1594)—"A discourse most necessary for all gentlemen that have in regard their honors, touching the giving and receiving the lye, whereupon the Duello and the Combat in divers forms doth ensue ; and many other inconveniences, for lack only of true knowledge of honor, and, the RIGHT UNDERSTANDING OF WORDS, which here is set down." Prescriptivism at its best!

The term I would use is invective, which refers specifically to this kind of verbal sparring and is often used in Shakespearean scholarship.