Could downloading Tumblr videos damage/infect Mac?

Short answer yes but probably -- no.

You should definitely keep your videos. Don't delete them because you're concerned they could contain malicious data that could trigger errors in the video codec implementations that will lead to code execution on your computer. In fact waiting for some time will lower the likelihood of getting infected since any attack vectors will be invalidated in newer versions of codecs/applications. Keep your video players up to date. Don't use unsupported software which is not receiving updates.

If you like to still download and watch those cute movies and have a better conscience I would propose you install VirtualBox and a simple linux distribution for watching and downloading and have backups from your movies. Should that virtual system be infected by whatsoever thingie, just delete that virtual system and install a new one. Infecting a virtual system AND breaching the Hypervisor (the technique that drives virtual box, vmware et al.) is not very likely.

And remember to have BACKUPS! Then you are quite on the safe side. Let there be cat videos.

Adding to this, it may be also possible to share your media on your local network and watch it with guarded devices / OSs like an iPad running a non jailbroken recent system.

Some notes about downloads:

Downloading a file will not infect your system unless it is run or interpreted (like displaying a JPEG oder a Video or running javascript code). So browsing a site may infect your system, since the web-browser will interpret all kind of files not just the desired video. You could use a web scraper like jDownloader or something else to just download stuff from a given URL. My opinion is they do less interpretation. But who knows.