What does 'ten of six' mean in regard to time?

As others have said, "ten of six", though not used in many English speaking areas, would be understood as 5:50.

Nowhere in the English-speaking world, as far as I know, would it be understood as 5:10. I can imagine that Russian speakers, for example, might hear it that way, as it might be taken as a translation of the Russian "десять шестого" (/d'es'at' ʃestovo/ = "ten of the sixth") which does mean 5:10.

(A similar 'false friend' is "half six", which in British English means 6:30, but looks like a translation of German "halb sechs", which means 5:30.)

In American English, using "of" when telling the time denotes the number minutes before the upcoming hour. Thus, "ten of six" would mean 5:50 p.m. As another example, "quarter of three" would be 2:45 p.m.

In the British, "to" is used instead of "of". Thus, 5:50 p.m. would be "ten to six" and 2:45 p.m. would be "quarter to three". Americans also use "to" when telling the time.)

To indicate the minutes following the hour, "after" or "past" is used. Thus, 6:10 p.m. would be expressed as "ten past six" or "ten after six". I'm of the opinion that "past" is more commonly used in British strains, while "after" is mostly American.

While most people just say the numbers these days, e.g. "six-ten", "five-fifty", etc, "ten of--" is still quite popular, as well as "five of--", though to a lesser degree. You probably would not hear "twenty of--" or "twenty-five of--" too often.

For the origins of "ten of six", searching Google books shortly after 1800, which is when this expression seems to have originated, I came across quite a few uses of expressions such as

It wants ten minutes of six.

which makes more sense than just "ten of six", and is a cumbersome enough expression that one can see how it might be shortened to "ten of six". Such expressions seem to have been used both in the U.K. and the U.S., but (if this was indeed the origin of the phrase) were only shortened in the U.S.

"Ten of six" probably means 5:50 but I have not heard it before.

"Ten [shy] of six" would necessarily mean "Ten less than six"

The other time prepositions,

after, past, to
such as

0-30 minutes

(6:05) 5 after 6

(6:25) 25 past 6

31-59 minutes

(6:50) 10 to 7

(8:55) 5 'til 9 or 5 until 9

I'm an American and I have not heard "of" used before in conversation (ever), but I would never associate it with the 0-30 minutes range. To me it must mean "until"