"Both of" vs. just "both" with mathematical symbols

Solution 1:

Because I and I' are nouns in this sentence, your second example is the correct usage.

Both I and I' are irreducible ideals.

To see how this works, compare with:

Both of cat and dog are animals

vs

Both cat and dog are animals