Is it not effective to learn math top-down?

By top-down I mean finding a paper that interests you which is obviously way over your head, then at a snail's pace, looking up definitions and learning just what you need and occasionally proving basic results. Eventually you'll get there but is this a bad idea? Is learning each required math area by textbook the better way?

I think this sort of question, and pursuant discussion/answers/comments, can be very useful for people who're new in the math biz. Namely, in my considered opinion, neither a "strict" bottom-up approach, nor "strict" top-down approach is optimal (except, in both cases, for a few extreme personality types). This is plausible on general principles, since, after all, the smart money says we should do a good bit of both, as in "hedging". And this is true, and for more than those general principles, in my opinion.

The way that "do both" is the only sensible route would seem to be that the extremes have lent themselves to highly stylized, almost caricatured, and editorially-pressured extremes. Research papers very often are written in the first place not to inform and help beginners, but to impress experts, etc. Sometimes journals' editorial pressures push in this direction. Peoples' understandable professional insecurities push in this direction. Experts' boredom with beginners' issues helps drift. And, at the other end, publish-for-profit situations push ... for profitability, which in real "textbook markets" means that "new" textbooks will mostly resemble old ones. We are fortunate that some people (Joe Silverman, as Matt E. noted) manage to move things forward within that strangely constrained milieu.

An important further point, in my experience, is exactly that of the question's "follow the branching graph of references backward to reach the ground..." 's hidden, unknown fallacy. That is, (having tried this in all good faith, very many times in my life) peculiar conclusions are reached when/if one does not give up, but pursues things to their actual ends. Namely, some significant fraction of the time, no one ever proved the thing that gradually was back-attributed to them... though it is true, and by now people have figured out how to prove such things. Another is that the "standard reference" is nearly incomprehensible, and only if one knows that subsequent explications were life-savers can a beginner find a readable thing.

And, of course, these graphs going backward branch so rapidly that literally reading everything referred-to is ... well, physically impossible for most of us... and even if one tries to approximate it, the effort is ... let's say... "not repaid in kind". At various moments I did estimates of how many pages I'd need to read to correctly honor all the background. When it hit "more than 10 pages a second, for the next 20 years", I knew both that I couldn't do it and that either it was crazy-impossible or ... not the real issue. :)

Yet, at the same time, exaggerated reliance on "popular" textbooks (excepting some like Silverman's... hard for the novice to know who to trust, yes, ...) really doesn't lead one forward/upward.

After all the above blather, my operational advice would approximately be that one should try to figure out _what_to_do_. Thus, any source which cannot be fairly interpreted as giving help is instantly secondary. More tricky are the sources that do well-describe the Mount Fuji at a great distance... To see Mt Fuji is a great thing. What should one do?

At this date, it seems to me that on-line notes on topics that otherwise seem to be "standard", but have become cliched, are far better than most "textbooks". But, yes, there is some volatility in this, because, as above, Joe S.'s books on elliptic curves are excellent, even while being entirely within all conventions and such.

Summary: run many threads...

I learn mathematics this way, to a certain extent. I began my studies just a few years before Wiles's paper appeared, so I didn't have that to read when I started, but from Silverman's book on elliptic curves I learned about Mazur's theorem on rational torsion points on elliptic curves over $\mathbb Q$, and started trying to read Mazur's paper (the famous Eisenstein ideal paper, discussed here in more detail; let me just note here that it is the paper which initiated the research direction in number theory of which Wiles's paper is one of the highlights).

However, I didn't try to read this paper in isolation. Using the bibliography as a guide, I tried to figure out what other material I had to learn to understand what he was doing, and went on to learn this material, from a mixture of text books (such as Serre's Course in arithemtic, for modular forms) and other papers (too many to list here!). In the end, it wasn't until much later in my career that I developed anything approaching a complete understanding of what Mazur does in that paper.

If your goal is to become a research mathematician, then you need to develop a knowledge of mathematics, as well as intuition and a sense of the big picture. You can't be dogmatic about how your learn this. Sometimes papers help; sometimes textbooks help. (E.g. if you want to learn elliptic curves, which is certainly necessary for understanding Wiles, it is silly not to use Silverman's book; it is simply a great text-book, which will teach you a lot much more efficiently than any other approach that I know of.) Having a mentor, or just other, perhaps slightly more experienced, students around, to give advice on what is good to read and what to avoid, helps a lot.

If your goal is something else, then I don't really have any advice. All my experience is based on learning mathematics with the ultimate goal of doing mathematics.

It really depends what kind of person you are. Most of the people I know hate this. I love it.

The most important thing to keep in mind, though, is that you'll almost surely end up somewhere entirely different. You'll start with a complicated topic, take a few steps back, and then try to move forward. But by the time you've made any significant progress, you'll probably have diverged and started studying something entirely different.

This isn't a bad thing. Self-study is governed by student interest much more than traditional teaching. As such, you'll inevitably go off on tangents and become re-directed. I think this is a feature, more than a bug. You'll naturally gravitate towards subjects you find interesting. (Just make sure that you actually make progress. It's quite easy to skim through basic results and lose interest before getting to the meat of a topic.)

In "Surely You're Joking Mr. Feynman", Richard Feynman describes being challenged by his sister to do just this, at an early point in his graduate studies when he was feeling overwhelmed by some of the papers. (And by the apparent ease with which his contemporaries appeared to absorb them.)

As I recall the anecdote, he describes staying up all night to work through one paper, which he finally grasps. He goes on to describe this as a seminal moment in his career, when he might have abandoned theoretical physics without his sister's goading.