When to give up on a hard math problem?

My opinion is that you should sleep on it.

Move onto another problem, or just take a break, and come back to it later.

Also, if you've hit a brick wall you may just need to approach from another angle to find the way through.

EDIT: To be fair, I never directly answered the question when I gave the four following points. I think there is a progression, though. Both (1) and (2) describe feelings you may have that suggest you should quit. My point is that neither by themselves point to giving up on a problem. For (2), I suggest you may need to think about why you want to solve a problem in the first place (note the philosophical nature of my answer to your question). But then, problem solving requires techniques, and asking when to give up is sort of like asking for more techniques. I suggest in (3) and (4) some techniques for progressing the process along that may help if you feel very stuck. Ultimately, you decide how hard you try and when to give up.

(1) That feeling of being stuck and/or somewhat frustrated is typical and part of what you sign up for when you try to solve hard problems. If you are training to be a mathematician you need to get used to this feeling. It's like being a soldier--it's just as much as a lifestyle change as knowing how to shoot guns. You need to cope with it and use it to your advantage. That you are asking this question shows you are starting to do this. I can't say I have it all figured out, but I will say that one of the most important things I learned in grad school is how to be at peace with this addicted/stubborn/frustrated feeling when working on a problem and not getting anywhere.

(2) Do not allow negative or self-derogatory thoughts. You will often think, "I am so stupid, other people smarter than me would have solved this problem ages ago." Or you might think, "Math is worthless, there's too much work and no reward." It is true, other people are smarter than you and will solve the problem faster than you--possibly. So what? Are you doing mathematics to impress other people? Or are you doing it for a job or for your own satisfaction or for your own education? Now what about math being not worth the effort? Maybe it's true. But there are different difficulties of problems. The Riemann hypothesis is probably too hard. For you, Putnam problems are probably just fine. So try to decide what difficulty is worth it to you. Don't avoid difficulty at all, since then there is no reward at all.

In any of these cases, they reveal a fundamental bias you have about math that should be addressed. Maybe you can't really convinced yourself that math is worth doing. Okay then, pick up something else instead. But just ask plenty of people here and they will have something convincing to say.

If you are truly becoming agitated, and can't beat these thoughts, the best thing to do is to distract yourself with something else. See (4).

(3) Find something of interest in the problem or tangentially related to the problem. When you are stuck, you often become bored of the problem. There is just nothing new you can see! If you could see something new, you wouldn't feel stuck. But have you considered changing the premises of the problem? Have you tried searching for examples or counter examples of the hypothesis? Do any mathematical techniques come to mind, even tangentially, when working on the problem? Think about these things instead. The idea is to find something easy to explore about the problem. Be creative and don't have the goal of solving the problem completely in mind. Have your enjoyment in mind.

(4) Sleep on it, eat lunch, go for a walk, or crumple up the problem statement and dig it out from the trash in a month. Often the ability to solve a certain problem is based on what you have already learned. If you haven't done the hard work of, say, learning a particular inequality derived from Holder's inequality, then it will be impossible to use it in any problem that requires it. Maybe you have learned it but not very well or a long time ago. In that case, you need to give your mind some time to bring it up to the front.