How did "beest" become replaced by "are"?

Solution 1:

Definitely by the time "thou" had got pushed out of the language

As other people have said, the -(e)st conjugated verb forms like "beest" were generally only used in sentences that had the old second-person singular pronoun thou as the subject, so your question may reduce to the question of when, how and why thou fell out of use.

Wikipedia says

Fairly suddenly in the 17th century, thou began to decline in the standard language (that is, particularly in and around London), often regarded as impolite or ambiguous in terms of politeness. It persisted, sometimes in an altered form, particularly in regional dialects of England and Scotland farther from London, as well as in the language of such religious groups as the Society of Friends. [...]

In the 18th century, Samuel Johnson, in A Grammar of the English Tongue, wrote: "in the language of ceremony ... the second person plural is used for the second person singular", implying that thou was still in everyday familiar use for the second-person singular, while you could be used for the same grammatical person, but only for formal contexts. However, Samuel Johnson himself was born and raised not in the south of England, but in the West Midlands (specifically, Lichfield, Staffordshire), where the usage of thou persists until the present day, see below, so it is not surprising that he would consider it entirely ordinary and describe it as such. By contrast, for most speakers of southern British English, thou had already fallen out of everyday use, even in familiar speech, by sometime around 1650.

"Thou beest" vs. "Thou art" (in the indicative)

That said, before thou had fallen completely out of use, art was in competition with beest as the 2nd-person singular present tense indicative form, and it seems that in the most well-known dialects, and in the standard language, "thou art" was either winning or had largely won the competition.

Goold Brown's "Grammar of English Grammars" from 1882, quotes Robert Lowth's grammar of English, published in 1762, as saying that

"Be, in the singular number of this time and mode, especially in the third person, is obsolete; and is become somewhat antiquated in the plural."—Lowth's Gram., p. 36.

The connection of forms like "beest" with the "subjunctive" seems to be complicated and unstraightforward

The issue of defining when forms "be" and "beest" are "subjunctive" is actually fairly complicated. There have been differences in the use of the subjunctive mood across the course of English, but there also have been differences in the use of "be" forms as indicative vs. subjunctive forms. As described in Quuxplusone's answer to the question "Why “the powers that be”?", the word "be" could be used as an indicative form (especially in the plural) in the time of Shakespeare.

The OED says of forms like "beest":

Historically, these continue indicative forms, but apparently at least some early modern English texts prefer beest (rather than art) in syntactic environments that otherwise favour use of the subjunctive such as conditional clauses

As far as I can tell, the original 2nd-person singular subjunctive form of the verb be would have been be: Wiktionary's conjugation tables say that Old English used bēo for the singular of all persons in the present subjuctive, and that Middle English used be (of course, Wiktionary conjugation tables don't show the variety of forms that were actually used in a time period, and there are sometimes coding errors that cause them to have incorrect forms, but in general I find Wiktionary a useful reference for information about old languages).

In the past tense, the history of the forms wert and wast seems to be similarly complicated. The OED says

In Old English the 2nd singular past subjunctive of this verb (in common with all strong verbs) is not formally distinct from either the 2nd singular past indicative (as it would be for a weak verb) or from the 1st and 3rd singular past subjunctive; in its later history it continues to be problematic as a formal category. This section illustrates use of the 2nd singular in syntactic environments that typically trigger use of the past subjunctive; it is not entirely certain to what extent these should be regarded as subjunctive forms.

You cna see more complicated notes about "beest" and "wert" if you use Google Books to look at the linked page from Brown.