Should "index" be countable here?

It depends on the underlying data. Also, I prefer "less" rather than "smaller" for mathematical descriptions; "less" is the better term to pair with "greater", and "smaller" would pair best with "bigger".

Is there one index for general high school and higher education? Then your sentence is correct because the index in the second comparison is easily inferred from the stated index in the first comparison. If there are two different indices and they are each greater than one, then you have to use the plural and you have to state you're referring to an index (not several indices):

In urban areas, the graduation indices of general high school and higher education were each greater than 1, while the index of compulsory education and secondary vocational education was less than 1.

Unless, of course, there are actually two indices for "compulsory education" and "secondary vocational education", in which case you need to make the same change in the second part of the sentence.

Should I change "index" to "indexes", and thus "was" to "were", and "that of" to "those of"?

That doesn't really get rid of the ambiguity, it just changes it to a different ambiguity.

A reduplication of the preposition can help here to make explicit how many there really are.

In urban areas, the graduation index of (or for) general high school and [of or for] higher education [was?] were greater than one, while that of (or for) compulsory education and [of or for] secondary vocational education [was?] were smaller than one.

This may be the best way to get the one-index-per-item-listed point across.

The changes you suggested may imply that there are multiple graduation indices associated with each one. Perhaps context handles this problem, but as a stand alone statement, English is severely challenged when trying to handle the distribution of a plural subject across multiple PPs. Grammar doesn't handle this. If you want to be precise, you just have to explain it in words.