Elementary proof of the Prime Number Theorem - Need?

Although I am very much new to "Analytic Number Theory", there are some non mathematical questions which puzzle me. First of all, why was G.H.Hardy so keen to have an elementary proof of the Prime Number Theorem. He also stated that producing such a proof, will change the complexion of Mathematics, but nothing like that has happened. What was on Hardy's mind?

Although the elementary proof has some intricate tricks involved, I am curious to know whether the methodology can be applied for attacking more complex problems. I have seen that the analytic proof has a continuation and is not over and discuss some more interesting properties regarding the $\zeta$ function.

I also saw this thread. Are there any such theorem's which piques peoples curiosity in getting an elementary proof. (While writing this FLT comes to my mind!!).

As KCd explains in a comment, the proof of the PNT in Hardy's time seemed to be intimately connected to the complex analytic theory of the $\zeta$-function. In fact, it was known to be equivalent to the statement that $\zeta(s)$ has no zeroes on the half-plane $\Re s \geq 1$. Although this equivalence may seem strange to someone unfamiliar with the subject, it is in fact more or less straightforward.

In other words, the equivalence of PNT and the zero-freeness of $\zeta(s)$ in the region $\Re s \geq 1$ does not lie as deep as the fact that these results are actually true.

The possibility that one could then prove PNT in a direct way, avoiding complex analysis, seemed unlikely to Hardy, since then one would also be giving a proof, avoiding complex analysis, of the fact that the complex analytic function $\zeta(s)$ has no zero in the region $\Re s \geq 1$, which would be a peculiar state of affairs.

What added to the air of mystery surrounding the idea of an elementary proof was the possibility of accessing the Riemann hypothesis this way. After all, if one could prove in an elementary way that $\zeta(s)$ was zero free when $\Re s \geq 1$, perhaps the insights gained that way might lead to a proof that $\zeta(s)$ is zero free in the region $\Re s > 1/2$ (i.e. the Riemann hypothesis), a statement which had resisted (and continues to resist) attack from the complex analytic perspective.

In fact, when the elementary proof of PNT was finally found, it didn't have the ramifications that Hardy anticipated (as KCd pointed out in his comment).

For a modern perspective on the elementary proof, and a comparison with the complex analytic proof, I strongly recommend Terry Tao's exposition of the prime number theorem. In this exposition, Tao is not really concerned with elementary vs. complex analytic techniques as such, but rather with explaining what the mathematical content of the two arguments is, in a way which makes it easy to compare them. Studying Tao's article should help you develop a deeper sense of the mathematical content of the two arguments, and their similarities and differences.

As Tao's note explains, both arguments involve a kind of "harmonic analysis" of the primes, but the elementary proof works primarily in "physical space" (i.e. one works directly with the prime counting function and its relatives), while the complex analytic proof works much more in "Fourier space" (i.e. one works much more with the Fourier transforms of the prime counting function and its relatives).

My understanding (derived in part from Tao's note) is that Bombieri's sieve is (at least in part) an outgrowth of the elementary proof, and it is in sieve methods that one can look to find modern versions of the type of arguments that appear in the elementary proof. (As one example, see Kevin Ford's paper On Bombieri's asymptotic sieve, which in its first two pages includes a discussion of the relationship between certain sieving problems and the elementary proof.) But I should note that modern analytic number theorists don't pursue sieve methods out of some desire for having "elementary" proofs. Rather, some results can be proved by $\zeta$- or $L$-function methods, and others by sieving methods; each has their strengths and weaknesses. They can be combined, or played off, one against the other. (The Bombieri--Vinogradov theorem is an example of a result proved by sieve methods which, as far as I understand, is stronger than what could be proved by current $L$-function methods; indeed, it an averaged form of the Generalized Riemann Hypothesis.)

To see how this mixing of methods is possible, I again recommend Tao's note. Looking at it should give you a sense of how, in modern number theory, the methods of the two proofs of PNT (elementary and complex analytic) are not living in different, unrelated worlds, but are just two different, but related, methods for approaching the "harmonic analysis of the primes".

It's definitely late in the game to this question, but Martin kindly pointed me here, and I think there's something else to be said.

The search for the elementary proof has two points of view I think are related to the soul of mathematics as it stood during the time when the preoccupation was great.

The First

Further Advancement of the subject through new techniques.

The idea of an elementary proof meant that we would necessarily need to come up with new ideas that we had not before in order to produce the proof that would prior only yield to techniques using analytic methods. It is very common in mathematics to have several proofs of the same results. Oftentimes new proofs either simplify old ones--allowing easier transmission of the ideas and oftentimes bringing in new viewpoints which allow the theory to take a large step forward.

Tate's thesis allows one to prove the functional equation for the $\zeta$ function. This is something we've known for some time, but the ideas present in the new proof, allowed for the roots of incredibly important new mathematics to be developed because of how he did it. One can see very slick, elucidating proofs which somehow seem to be the "right" ones. Erdös would probably say they were "proofs from the book" which is the namesake and inspiring spirt of this book. The proof of the PNT equivalence with the non-vanishing statement mentioned in the previous answer comes from a proof of the PNT through the Wiener-Ikehara theorem, another idea that has more widespread applications than just the purpose to which it is put with the PNT.

And this is not a phenomenon unique to number theory. We have a proof of the classification of surfaces via Ricci flow, decades after the original proof, which motivated the idea that techniques using Ricci flow might give the classification of $3$ manifolds (Perelman's celebrated proof of the Poincaré conjecture proved this and more to be true.) Many old proofs in modular arithmetic are much easier to prove using the language of groups, such as the fact that $\left(\Bbb Z/p\Bbb Z\right)^*$ is a cyclic group or Wilson's theorem. The proof that $\Bbb R^1\not\cong\Bbb R^n$ for $n>1$ is easy and uses only that the image of a connected set is connected, however that method doesn't generalize nicely. Compare with the homology proof, and we can easily demonstrate $\Bbb R^n\not\cong\Bbb R^m$ (as topological spaces) for any $n\ne m$. The idea of "uniform distribution modulo $1$" gave way to topological group dynamics, whose modern techniques have proven very powerful indeed.

In short there are some techniques that are only able to go so far, it often takes a genuinely new idea to allow for a great surge forward in the theory, and in this spirit the elementary proof presented an opportunity for such advancement. If it proved to exist, as Hardy noted it might give us greater insight in how to understand the theorem.

It is of course true that we do not "need" that proof for the purposes of establishing the PNT, but that's too simplistic a way to think about the elementary proof, or indeed of any proof which uses different approaches from the original proofs. New proofs have almost inherent merit if they are qualitatively different, in that they encourage use to look in those directions for proving other results which might not yield to the standard techniques to which we are accustomed.

The second

A little bit of superstition

Historically new discoveries in all of the sciences, mathematics included, have been intertwined with politics and superstition. In the ancient days the Pythagoreans had an entire cult dedicated to numbers, which--to the Greeks--meant rational numbers. Indeed, there is some historical evidence to suggest the man who discovered irrational numbers may have been killed over the fact! Other notable examples are complex numbers, which were shunned or ignored for the longest time.

The biggest example I can think of is the axiom of choice. Once upon a time it was of a much more central focus in mathematics. Proofs using it were sometimes rejected by some sectors for not being constructivistic. Brouwer even renounced his own fixed-point theorem proof for not being constructive. I imagine it must just not have sat well with a greater percentage of the mathematical population than it does today (I still know of a good handful who will argue vehemently on the subject). I think especially after the long time where we had a lot less precise formulations of theorems, and less air-tight proofs than we do today, this was a more valid concern, but experience seems to show we really shouldn't worry too much about such things.

That is all to say that the relative importance attached to an elementary proof seems historically similar to desires for constructive proofs of things like fixed points, and to an extent I acknowledge they are useful to have for the purposes of illustration, and sometimes in practice where such things are useful objects. Cantor's work proves there are a lot of transcendental numbers, but Liouville, Lindemann, Baker, et al are those that give us our best examples. At the same time, the community's interest moves on to bigger and ultimately more important things.

In short

I think Hardy was much like any other mathematician of his time: still not too far away from the original proof to wonder about a simpler (or at least more elementary) proof of the PNT, an interest which has greatly faded away with all the many proofs we have today. Even today there is a bit of mysticism with our subject, some things that feel like they ought to be true, even if we cannot prove them. I think Hardy was--mistakenly--in the camp that thought somehow an elementary proof would reveal some deep secret about primes in the way it was proved. It could certainly have seemed that way based on what was known and believed in his time, so it was not unreasonable, it just happened to be incorrect.

In despite of what some may think, a lot of what is studied comes from where the general interest lies--Kronecker vs Dedekind turned a lot if those in the latter's school to push down those in the former's, though that effect has naturally lessened over time. The personalities of the day played a large role in what was deemed "important" to discover. Hilbert's famous problems probably gave direction to thousands upon thousands of careers. Ultimately the field continues to evolve, and things like the elementary proof do have their place in the history and in the practice, it just depends on your personal views on whether "requiring complex analysis" really makes a result "fundamentally deeper" than one that does not.