When is it safe to remove my installation USB?

Solution 1:

Keep the USB Stick plugged in until the computer is shut down and restarts. Then you can unplug it to start from your HDD and not from the USB stick again.

Pull it out when the computer starts (BIOS Post; black screen, white letters) or shut the computer down and pull it out -- this way you're safe.