Why do courts use "What say you?"

... instead of "What do you say?"

I am not sure if "What say you?" is even grammatically correct.

The way modern English formulates questions by using "do" as an auxiliary is rather recent in the story of English (~1400-1500).
It is termed "periphrastic do" and a growing number of linguists actually ascribe this evolution to the influence of Celtic languages. The Germanic word order instead is termed V2 syntax (which means that the verb is always in second position).

In Modern German for instance you would say:

"Was sagst du?"

In Old English (before the Norman invasion) you would say something very similar.

"Hwæt sægst ðū"

The UK legal and judicial system is deeply rooted in the Anglo Norman historical period (1066-1485) and therefore predates the "periphrastic do" evolution (~1400-1500).
This is yet another manifestation of the marked conservatism of this institution.

Just because it's archaic, doesn't make What say you ungrammatical.

And it's not just a stock phrase used in court, either. Here's What think you, which wouldn't often be said in a courtroom. But as the chart clearly shows, definitely an expression from yesteryear...

Over at the "Phrase of the Week" website, there's a question about the origin: http://www.phrases.org.uk/bulletin_board/58/messages/1572.html

With that in mind, for an exchange of information as critical as asking for a plea in court, the language should be formalized. If the question "what say you" evolved into "what is your plea?" or maybe "what would you like to plead?"... then maybe later the defendant might say "well you asked me what I would like to plead, not what I actually do plead." and then an unnecessary argument follows.

So for an important exchange, formal language helps with clarity.

As for why it hasn't been formalized as "what do you say" ... well, I think it's human nature to want assurances of structure. The awkward phrases sounds more formal, so helps us trust that the court proceedings are structured and formal.

According to Urban dictionary it's a variation of what do you think. Here is the full description.

1. What say you

A question asked mostly by radio/TV personality/talk-show host Bill O'Reilly. It means, "What do you think?" or "What are your feelings on the subject?"
Variation: What say ye?

Bill O'Reilly to caller: "What say you?"
Caller: "I say you're full of shit, Bill!"

And here is what Learners dictionary has to say:

2. "What say you?"

"What say you?" is an odd construction.

It's used to ask someone "what do you think about this?" It is certainly not as >common as "What do you say?" or "What do you think?" or "What is your opinion?" - but it is idiomatic English. It is old-fashioned and appears mostly in spoken English these days.

It has a slightly urgent but familiar tone:

What say you, Mr. Brown? Will we have an early frost this year?

I think she should take a semester off before returning to college. What say you?

==

The expression can also be a slightly aggressive way to ask a question. In this context, "what say you?" means "what do you say in response?" Here's a couple of examples:

You made your choice, but what say you to young people who struggle with that dilemma even as we speak?

You've heard all the evidence. What say you to that?

==

It is also part of the old-fashioned and more formal language of courts of law, and is used to ask about decisions or to ask a defendant to issue an official plea of "guilty" or "not guilty":

To the charge of murder in the first degree, what say you?

What say you, foreperson? Is the defendant guilty or not guilty?