The math problem is too difficult for `X` to work out

Solution 1:

As FF has pointed out, only everybody and anybody are acceptable
(everyone and anyone also work). They work, and the others don't, for the same reasons:

1. Too is a Negative -- too Adj to VP means 'so Adj that Not VP';
   therefore NPIs like anyone work within its scope,
   and so anybody is used here, instead of somebody.

   That's also why nobody is ungrammatical here, because that would mean
   *The math problem is so difficult that nobody can't work it out.
   which is ungrammatical (that's what the "*" means).

2. Everybody, somebody, and anybody are all Quantifiers -- logically, ∀ and ∃.
   By DeMorgan's Laws, propositions with both a negative and a quantifier are ambiguous.

   Note the opposite orders of not (logically ¬),
   compared to some/any (∃) and every (∀) below.

   I.e, Everybody doesn't like it
   can mean either
   It is true that there is somebody who does not like it.
   ¬(∀x) φ(x) ≡ (∃x) ¬φ(x)
   or
   It's not true that there is somebody who likes it.
   ¬(∃x) φ(x) ≡ ∀(x) ¬φ(x)

So the two choices that are grammatical do not mean the same thing.

• The test is too difficult for everybody to figure out means
  'The test is so difficult that there are some people who can not figure it out'
• The test is too difficult for anybody to figure out means
  'The test is so difficult that there are no people who can figure it out'

Solution 2:

Which of everybody, somebody, anybody and nobody should be used depends on the meaning to be conveyed. If the meaning is that the problem is so difficult that nobody can work it out, then (as other answers indicate) the best fit is:

The math problem is too difficult for anybody to work out.

If, however, the meaning is: Nobody finds the problem too difficult to work out, then nobody fits:

The math problem is too difficult for nobody.

Admittedly, adding back the (superfluous) to work out makes the sentence now sound a little awkward:

The math problem is too difficult for nobody to work out.

So I prefer:

The math problem is too difficult to work out for nobody

with appropriate prosody.