Why is X used when we pronounce it Z?

I've seen a lot of words use an X but be pronounced with a Z. Mitch Hedberg put it best:

Xylophone is spelled with an X, that's wrong, xylophone's zzzz, X? I don't $%(@#& see it. It should be a Z up front, next time you have to spell xylophone, use a Z. When someone says, "Hey that's wrong," say, "No it ain't. If you think that's wrong, you need to get your head Z-rayed." It's like X wasn't given enough to do, so they had to promise it more. Okay, you don't start a lot of words, but we'll give you a co-starring role in tic-tac-toe. And you will be associated with hugs and kisses. And you will mark the spot. And you will make writing Christmas easier. And incidentally, you will start xylophone. Are you happy, you $%(@#&' X!?!

So why is X used like this and where did it start?

Xylophone is from the Greek xylon, which is (or was—I'm not up on my modern Greek) pronounced with an initial [ks]. Many words borrowed from Greek via French developed a [gz] pronunciation along the way, which was reduced to [z] word-initially when adopted into English.

English, like some other languages, systematically reduces certain consonant clusters; but it has a conservative spelling system (which incidentally often reminds us where a word comes from).

The clusters we reduce are mostly in borrowings from other languages: particularly initial clusters /ks/ ('x'), /pt/, /pn/ which are generally from Greek, but also for example /hr/ (and in fact /h/ anywhere but initial) as in "Tahrir square".

This reduction happens in some native words as well: initial /kn/ as in 'knave' is pronounced /n/ in modern English, though the related word 'Knabe' in German still has the /k/ pronounced; and words like 'night', old English 'niht' (with the /h/ pronounced), cf German 'Nacht'.