The use of the word "closet"

On ODO, one of *closet 8's definitions is:

Used to refer to a state of secrecy or concealment, especially about one's homosexuality.

Though by memory, I remember that it may have been used in other examples. For example, would the following sentence make sense:

He is a closet gardener.

Does it mean that this person enjoys gardening, but doesn't tell anybody about his enjoyment of gardening?

If this usage of the word closet is correct, is it possible that it would offend anybody, or does it seem alright to use?

especially about one's homosexuality.

The use of especially here means it's at least sometimes used about something other than homosexuality. (One example that is actually the opposite is that in the bisexual community the pressure to pretend to be either gay or straight is often called "picking a closet", with pretending to be gay seen as being as much "in the closet" as pretending to be straight).

is it possible that it would offend anybody

Only if either the statement without "closet" would offend, or if the suggestion of secrecy expressed another way would offend.