"Both of" vs. just "both" with mathematical symbols
Solution 1:
Because I and I' are nouns in this sentence, your second example is the correct usage.
Both I and I' are irreducible ideals.
To see how this works, compare with:
Both of cat and dog are animals
vs
Both cat and dog are animals