What was the original sound of rh?

The subject more or less says it all. I would like to know how rh (as in rhythm) was originally pronounced. It is listed as being something which was originally present in Latin, but, in Latin, "h" is used to harden a vowel. I can't think of a soft pronunciation for "r" in Latin or any potential intermediate steps.

In English, as far as I know, "rh" was never pronounced any differently from plain "r". This spelling, as @third-idiot wrote, is basically how the Ancient Greek "ῥ" was transliterated in Latin characters.

In Old Greek using polytonic orthography, an initial "ρ" was always written with a rough breathing, indicating the Greek /r/ (whatever its actual phonetic value) was probably aspirated or voiceless at the start of a word. Greek lost its aspirates quite early on though, and by the 4th century AD this rough breathing didn't mark anything anymore. Still, it kept on being written until the 1982 reform, which abolished the polytonic orthography along with the Puristic language and introduced the monotonic system and the Demotic language as official Modern Greek language (though it is still used by, for instance, the Greek Orthodox Church, which refused to acknowledge the reform).

People who made learned borrowings from Old Greek, like "hymn", "hypnosis" or "helium", transliterated the rough breathing on vowels as "h", and did that also for words where the rough breathing was on "ῥ", like "rhythm" or "rhapsody", even though they probably didn't pronounce that initial "r" as voiceless or aspirated themselves. It was just an orthographic convention.

The Greek rho, transliterated into Latin as rh and retained in English words like rhythm and rhetorical would have been pronounced more like a modern Italian or Greek r than how we pronounce it in English, although aspirated at the start of words.

More info at wikipedia, although the best resource for the pronunciation of Ancient Greek is W. Sidney Allen's Vox Graeca.