Troubleshooting: 90 seconds to eject external Time Machine drive?

I have a 5TB USB-C drive attached to my 16" MacBook Pro running Big Sur 11.2.3. I use this drive exclusively for Time Machine. When ejecting the drive, it takes anywhere from 90 - 120 seconds before the drive is ejected. This happens even when Time Machine is not actively backing up to the drive. A couple OSs back, I remember thinking 15 seconds was a long time to wait for a drive to eject, but now, 90+ seconds feels like forever.

What steps can I take to troubleshoot and shorten the ejection time?

Update

I plugged the drive in this morning. After letting Time Machine run, I waited for it to finish and ran the command from @jaume :

Anthony@Vincent ~ % sudo lsof +d /Volumes/LaCie.Backup
COMMAND PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
mds      90 root   21r   DIR   1,16      510    2 /Volumes/LaCie.Backup
mds      90 root   25r   DIR   1,16      510    2 /Volumes/LaCie.Backup
mds      90 root   27r   DIR   1,16      510    2 /Volumes/LaCie.Backup
mds      90 root   31r   DIR   1,16      510    2 /Volumes/LaCie.Backup
mds      90 root   34r   DIR   1,16      510    2 /Volumes/LaCie.Backup

I waited another 10 minutes ran the command a second time to see if mds was still using the drive and the results were the same.

While ejecting the drive, I ran the command and the output was:

Anthony@Vincent ~ % sudo lsof +d /Volumes/LaCie.Backup
lsof: WARNING -- child process 72811 may be hung.
Anthony@Vincent ~ % 

Update #2

I added the drive to Spotlight exceptions and reran the lsof command. I noticed that there are now 3 mds processes rather than 5:

Anthony@Vincent ~ % sudo lsof +d /Volumes/LaCie.Backup
COMMAND PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
mds      90 root   21r   DIR   1,16      510    2 /Volumes/LaCie.Backup
mds      90 root   24r   DIR   1,16      510    2 /Volumes/LaCie.Backup
mds      90 root   28r   DIR   1,16      510    2 /Volumes/LaCie.Backup

Right-clicking and selecting Eject from the context menu still takes quite a while to eject the drive. During that time, I ran the lsof command and received the following output:

Anthony@Vincent ~ % sudo lsof +d /Volumes/LaCie.Backup
lsof: WARNING -- child process 73283 may be hung.
COMMAND PID    USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
Finder  431 Anthony   15r   DIR    1,6      160 23740 /Volumes
Finder  431 Anthony   16r   DIR    1,6      160 23740 /Volumes

The Finder commands appeared in the results right before the drive disappeared from the desktop.

However, running diskutil unmount /dev/disk3 from Terminal ejects the drive almost immediately.

Why would Finder take so long to eject the drive? I would've thought Finder was using diskutil in the background.

Update #3 (final update)

After a reboot, Finder, diskutil, and Alfred -- the 3 ways I use to eject this drive -- all seem to work fine. My Mac also restarts and shuts down much faster.

Thank you @jaume for the constructive comments. Can a comment be marked as an answer?

Update #4 (Final, FINAL, update, post accepted answer)

A month or so on from this, I was still having slow eject times. My tests ejecting the drive were done after a reboot but before Time Machine had a chance to run.

The answer from Jaume was helpful but I don't think it solved the problem.

The drive was purchased when I was running Catalina and I don't think Catalina supported Time Machine disks formatted with APFS. Now that Big Sur does support the format, I updated the drive to APFS and the eject times are now in the 5 second range.

After numerous restarts and Time Machine backups, the disk is quickly ejecting.

Also, in Big Sur, Time Machine drives are no longer able to be added to Spotlight Privacy:

Some process (or processes) seem to be accessing the drive and preventing it from ejecting. You can list those processes with this command:

sudo lsof +d /Volumes/<Time Machine hard drive name>

which displays all running processes that are accessing the folder /Volumes/<Time Machine hard drive name>

From the information in your post, the culprit is mds:

sudo lsof +d /Volumes/LaCie.Backup
COMMAND PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
mds      90 root   21r   DIR   1,16      510    2 /Volumes/LaCie.Backup
mds      90 root   25r   DIR   1,16      510    2 /Volumes/LaCie.Backup
mds      90 root   27r   DIR   1,16      510    2 /Volumes/LaCie.Backup
mds      90 root   31r   DIR   1,16      510    2 /Volumes/LaCie.Backup
mds      90 root   34r   DIR   1,16      510    2 /Volumes/LaCie.Backup

which is the metadata server used by Spotlight.

To prevent this from happening, add the Time Machine drive to the Spotlight exceptions:

1. Open System Preferences > Spotlight.
2. Select the Privacy tab.
3. Press the + button, select the Time Machine drive and select "Choose".

You may need to reboot your Mac for changes to take effect.