What does a "man of leisure" do exactly? What is the definition and the connotation?
Solution 1:
A man of leisure is a man who has a source of income that does not require him to do any work.
Running Mr Dorrit's boarding school presumably did not require much of his time. Perhaps he employed a headmaster and would only be present a few times each year.
British National Corpus
The British National Corpus finds this use of the phrase in The Economist 1991:
"Mr Hoi returned home in August 1990, [...] A severance payment of DM3,000 ($1,740) has enabled Mr Hoi to buy a house and spend the past nine months as a man of leisure. He is thinking of opening a small shop."
(Mr Hoi worked in Germany and returned to his native Vietnam). It should be clear that Mr Hoi lives on his savings and does not (yet) work for a living.
It seems plausible that English usage of this phrase has been influenced by its appearance in commentaries on, and translations of, the works of classical philosophers:
Aristotle
“Leisure”, Aristotle observes, “is necessary both for the development of excellence and the performance of political duties.” (1329a1–2) The man of leisure, as he says in the Nicomachean Ethics, stands a better chance of obtaining excellence and happiness than does one constantly consumed by daily cares and woes. (NE 1177b4–27)
(from http://www.analyse-und-kritik.net/2008-1/AK_Nederman_2008.pdf)
Cicero
Cicero contrasts the life of the man of leisure with that of those who have dedicated themselves to politics and the conduct of affairs (i.e. business).
(See http://fds.oup.com/www.oup.co.uk/pdf/0-19-924018-3.pdf)
Solution 2:
Perhaps I inhabit less exalted circles than RedGrittyBrick, but I chiefly hear (and use) man of leisure to mean someone who is unemployed. There is possibly some convergence in that a man of leisure will usually be able to claim unemployment benefit (or whatever the current euphemism is). Whether unemployment benefit constitutes a "source of income that does not require him to do any work" is a question of politics and economics that is probably best left to the reader.
Solution 3:
Man of leisure, in my opinion, does not have to work to support his or others' lifestyle. This may connote a wealthy man who inherited his money, or, alternatively, it could also depict a man who is "kept" by someone who has the means to provide him with such a lifestyle.
In other words, in its pure sense, it describes a man who is independently wealthy. And in a more modern or, dare I say, more "impure" sense, it is sometimes used to describe a high-end gigolo who himself has few financial concerns as the result of another man or woman's generosity. This interpretation is more commonly used in the gay community.