Can anyone access Time Machine backups from a stolen HDD?

Short answer - assume YES, and immediately change all your account details, Apple and email ID, cloud access etc. I would also inform your bank, online stores and everyone in your contact book (that they might receive a false comm purporting to be you).

Even with an encrypted drive, forensic level software and hacks (as used by the Police / Military and available to download), can circumvent most protection. For example, I would start by putting the HDD in any Mac, interrupt boot and attempt to change the encryption / Apple ID at root level. This will circumvent an Admin account password / provide access to the keychain on a Mac and it might work for the physical drive if I got lucky. Or I would try replacing the HDD's circuit board with one from an identical model - if the encryption is managed through memory on the board, its now gone.

Having said this, criminals generally fall into too categories, dumb / impatient in which case they wont bother with an encrypted device or capable in which case they will probably be into the data within 24 hours.

When we remove a drive (at work) with any sensitive client data, we physically destroy the internals and spread the pieces or incinerate them. You should bear this in mind if selling your Mac also. Wiped does not mean unrecoverable. Always remove the drive.