How are more difficult proofs discovered?

I think this is learning proof skills, practicing, challenging yourself and sometimes allowing the art of proof to flow like a master creates a painting.

You should do problems, practice, explore.

For example, see my response here and you should learn all the proofing approaches in those books and then step it up in the areas that interest you.

Expanding problem solving skill

As an example, Wiles spent seven years working on the proof for Fermat's Last Theorem (and it had a correctable error). When Wiles was asked about this error in a Nova (www.pbs.org/wgbh/nova/physics/andrew-wiles-fermat.html) special, he literally starting shaking. In other words, just like an artist cutting off an ear, it can drive you mad!

So, I am not sure if anyone can truly answer your question, but in simple terms, one must learn and practice the art and maybe there is a masterpiece in your future!

Explore!

-A

This book is quite old, but it may answer your question. I found it enlightening.

"An Essay on the Psychology of Invention in the Mathematical Field", by Jacques Hadamard.

It's not too technical neither in the mathematics, nor in the psychology.

http://books.google.be/books/about/An_Essay_on_the_Psychology_of_Invention.html?id=VxUHJmRpSgAC&redir_esc=y

I know exactly what you mean, when you wonder how some results were discovered. By trial and error, and a lot of playing and guesses, you can prove them. But where did they get the result?

How a new result makes its way through your mind is a fascinating question. Sometimes I myself come up with some new mathematical intuition. It starts as a blurry idea, and when it finally gets a concrete form, that can be proved or disproved, it feels as if I had had that idea for ages.

I find helpful, sometimes, to read from the very mathematicians that made the discoveries. You can often catch a glimpse of what they were thinking, and why. This is true especially if the mathematician in question was highly intuitive (like Riemann or Cartan). For example, what was Riemann thinking when he came up with his famous conjecture?