What is the meaning of 'too fond of fast horses'?

There are a variety of examples in literature where the concept of fast horses relates to high living, or excess. An example is of Kentucky being home to "beautiful women, fast horses and good whiskey": (Source)

Or characters referring to people spending more on fast horses (which could be considered somewhat frivolous) than they do on more important things: (Trojan Horses)

The complete phrase appears to be listing faults that the governors had, so considering these examples, if he was "too fond of fast horses" it suggests he put things of style and excitement before things of importance, which would not have made him a good governor. Although without further clarification, the sentence alone is open to interpretation as it could either be literal, or metaphorical.