Catenative verbs without "to"

It makes sense (obviously people understood) and is possibly grammatical (maybe - see references below, may just be a lot of sports commentator / web-speak misuse of English).

As far as what's going on, there is a to-infinitival relative clause containing a complex catenative construction where the object (of the complex catenative) is relativized.

Complex Catenatives take the form of:

verb - object - non-finite clause

Where the object of the verb (or NP in the middle if you don't want to call it an object) above is understood as the subject of the non-finite clause (assuming it doesn't include one).

Structure of the noun phrase

an absolute joy - to watch do literally anything (relative clause)

Structure of relative clause

to watch (relativized object: this absolute joy referred to = this man) literally do anything

This is a similar catenative structure to:

We watched the sun go down.

Which could be put into a similar larger structure like:

That sun was a joy to watch go down.

As far as understanding the construction or rephrasing it to something unquestionably grammatical, it would probably best be rearranged as:

To watch this man literally do anything is an absolute joy.

OR

It is an absolute joy to watch this man literally do anything.

I'd move the 'literally' to a position just after 'do' to make it even more natural, but that might change the original too much.

Here are some examples of similar; many more exist, but mostly in sports reporting or blogs.

It was a slightly bizarre scene to watch unfold but, in fairness, it's cool when teams tap into their local culture to craft these retirement gifts. (Fox Sports)

the man is an unstoppable machine, and an inspiring machine to watch compete (Garage Gyms)