Why “left me of” in this sentence?
Solution 1:
The grouping "left me of" is not what you should consider. Instead consider "left me" as Hellion did, and separately consider "of"...
Of what had been a passion, all that was left me was that.
Solution 2:
Here, the "me" is an indirect object, as in "he left me a $1 tip" -> "he left a $1 tip for me." So, something had been a passion for someone, and they presumably created a lot of items as a result of the passion, but this particular "work of a craftsman" is the only thing that was left for "me".