Does "Stopping" USB drives actually affect the drive?

I have always just unplugged my USB drives from my computer for years and nothing bad has happened; but I heard that you are supposed to use "Safely Remove Hardware" before removing a drive. What does the "Safely Remove Hardware" program do (besides flush the output buffer)?
Is this recommended just to ensure that all data transmission to the device has stopped, or is there another reason?

This depends on what settings you have set for the drive. For instance, if you look in the device manager for the disk drive itself, you'll see something like this:

If you have the first option (quick removal) selected, then ejecting the drive merely unmounts the partition, and no longer allows programs to access the drive.

If you choose the second option (better performance), then ejecting the drive flushes any caches the OS has in memory, unmounts the filesystem (which may also flush a few buffers), and flushes on-device buffers.

The new data is not instantly saved to USB drive, it's stored in RAM for some time. The primary motivation of that appoarch is to gain some performance on pages repeatedly rewritten in place.

So, by unplugging unsafely you have some risk that recently written pages temporarily kept in RAM will never reach the disk.

This may affect not only data, but metadata too - entire directories may disappear, show garbage etc. It's less likely to happen on NTFS, as NTFS has transaction log for the metadata. So on NTFS you just corrupt user data in files and have metadata changes rolled back if you're unlucky on unsafe unplug.

Exactly, this is all this does. It flushes all buffers and ensures nothing can access the USB drive anymore by disconnecting it from the system.