Is this 'off' also an adverb?

The online OED has a separate entry for off-duty, which can function as an adverb, a noun and an adjective. It is made up of the preposition off and the noun duty. In your example, it is perhaps best seen as an an adjective postmodifying time.

You've come across a much-debated, and probably still-unresolved, part of English semanto-syntactic (meaning + grammar) analysis here, Listenever.

The authors of CGEL would doubtless say that the usages of off in

He needs more time off.

and

He needs more time off work / school / duty. ...

are so similar that they should not be put into different classes. They use the traditional term preposition for the second usage, where off fulfils the syntactic role of relating the occupation 'he' needs to be resting from to the first part of the sentence (He needs more time), as well as meaning 'away from'. They use the term 'intransitive preposition' (not their coinage) for the usage which assumes the hearer can easily fill in the unstated occupation mentally.

However, Cappelle, in this paper, argues in my opinion convincingly that this particular (no pun intended) lumping of word-classes is erroneous.