What is the difference between "safely remove" and "eject"?

Solution 1:

Eject removes the media from the device (e.g., ejects the CD tray) but doesn't remove the device itself.

Safely Remove flushes any pending writes from the cache and removes the entire device.

Solution 2:

During the time of Windows 98 there was only one option Eject. Any USB device like pen drive required that the driver for that device be manually installed.

To remove the device you had to select Eject from the context menu.

Since Windows XP the driver gets automatically installed and a new improved feature is provided to remove the USB device which is Safely Remove Hardware.

And as Matt Jenkins said

Eject removes the media from the device (e.g., ejects the CD tray) but doesn't remove the device itself.

Safely Remove flushes any pending writes from the cache and removes the entire device.

Try both the features with a memory card reader.
When you select Safely Remove Hardware the Card Reader device is disconnected from the system while when you select Eject the device is still connected to the computer but you can safely remove the card from the card reader.

Solution 3:

They both make sure all the pending writes to the device are executed, and they both unmount the device afterwards. So I'd say they're the same.

@Matt Jenkins is right, the difference lies with cd-drives where the Eject option only removes de cd, but doesn't remove the cd-drive device. Where as with usb-disks, the behavior is identical.

Solution 4:

@Majenko already answered the Question
Here is some added info

Windows itself tells you that you don’t need to use the Safely Remove Hardware option if you use certain settings – the default settings.

Quick Removal vs. Better Performance

Windows allows you to optimize your USB device for quick removal or improved performance. By default, Windows optimizes USB devices for quick removal. You can access this setting from the device manager – open the Start menu, type Device Manager, and press Enter to launch it.

Expand the Disk drives section in the Device Manager, right-click your device, and select Properties.

Select the Policies tab in the Properties window. You’ll notice that Windows says you can disconnect your USB device safely without using the Safely Remove Hardware notification icon, so this means you can unplug your USB device without ever safely removing it, right? Not so fast.

Data Corruption Danger

The Windows dialog shown above is misleading. If you unplug your USB device while data is being written to it – for example, while you’re moving files to it or while you’re saving a file to it – this can result in data corruption. No matter which option you use, you should ensure that your USB device isn’t in-use before unplugging it – some USB sticks may have lights on them that blink while they’re being used.

However, even if the USB device doesn’t appear to be in-use, it may still be in-use. A program in the background may be writing to the drive – so data corruption could result if you unplugged the drive. If your USB stick doesn’t appear to be in-use, you can probably unplug it without any data corruption occurring – however, to be safe, it’s still a good idea to use the Safely Remove Hardware option. When you eject a device, Windows will tell you when it’s safe to remove – ensuring all programs are done with it. Write Caching

If you select the Better Performance option, Windows will cache data instead of writing it to the USB device immediately. This will improve your device’s performance – however, data corruption is much more likely to occur if you unplug the USB device without using the Safely Remove Hardware option. If caching is enabled, Windows won’t write the data to your USB device immediately – even if the data appears to have been written to the device and all file progress dialogs are closed, the data may just be cached on your system.

EJECT

When you eject a device, Windows will flush the write cache to the disk, ensuring all necessary changes are made before notifying you when it’s safe to remove the drive. Ultimately, no matter which option you use, you should use the Safely Remove Hardware icon and eject your device before unplugging it. You can also right-click it in the Computer window and select Eject. Windows will tell you when it’s safe to remove the device, eliminating any changes of data corruption.*

Note This advice doesn’t just apply to Windows – if you’re using Linux, you should use the Eject option in your file manager before unplugging a USB device, too. The same goes for Mac OS X