Getting UEFI BIOS boot order consistency when attaching/removing USB?

Unfortunately, there's no standardization in how EFIs handle this sort of thing, but the behavior you describe is fairly common, in my experience. The trouble is that EFIs are much "smarter" about identifying boot devices than are BIOSes, and many EFIs remove invalid boot entries when they're identified as such. Thus, when you unplug a USB drive and start the computer up again, it will notice that there's an invalid boot entry for the USB drive and delete it from the boot list. I know of no in-EFI solution to this sort of problem.

That's not to say that there aren't workarounds, though. Specifically, you could install an EFI boot manager on your internal disk that does what you want -- that is, to boot Windows when the USB flash drive is not plugged in and boot Ubuntu from the USB drive when it is plugged in. You'd then leave this boot manager set as the default boot program.

Unfortunately, AFAIK Ubuntu's GRUB can't be configured in this way, so you'll need to look elsewhere. The rule you want to follow is simple enough that a really simple program could be written to do the job, but the one I know can do it is much more complex: My own rEFInd boot manager. I recommend you try installing and configuring it as follows:

1. Boot to Ubuntu.
2. Check to see what (if anything) is mounted at /boot/efi. (Typing df -h will let you see the currently-mounted partitions.)
3. If /boot/efi corresponds to a partition on the USB drive, unmount it and mount the EFI System Partition (ESP) from your hard disk. (As noted on the link, the ESP is a FAT partition with a particular type code. You may have to track it down with gdisk, parted, or GParted.)
4. Install the rEFInd Debian package from the rEFInd downloads page. Do not install the PPA; install the Debian package (or using the .zip file and running refind-install manually).
5. Edit /boot/efi/EFI/refind/refind.conf and make the following changes:
   • Optionally set timeout -1. If this is set, the rEFInd menu will not appear unless you hold down a key as rEFInd starts. You might want to hold off on setting this until after you've tested the installation.
   • Set default_selection "vmlinuz,Microsoft". This causes rEFInd to boot Linux if it finds a Linux kernel and Windows if a Linux kernel is not present -- that is, if the Linux USB drive is unplugged. Replacing vmlinuz with grub should also work, and is worth trying if vmlinuz doesn't work.
6. Reboot and test the system.

If you try it without setting the timeout value, you should see the rEFInd menu appear and highlight the Windows option (which may be the only option) if the USB drive is unplugged; but the Ubuntu option should be highlighted by default if the USB drive is plugged in. If this works, setting timeout -1 will cause a straight boot to the desired OS.

If rEFInd doesn't detect the Ubuntu installation with the USB drive plugged in, try hitting Esc. If an Ubuntu option then appears, try setting scan_delay 1 (or conceivably some higher value).

The biggest problem with using rEFInd in this way is that there will be a delay as it starts up, scans for OSes, and so on. This should be just a second or two when the USB drive is not plugged in, and maybe another two or three seconds when the USB drive is plugged in. A more specialized boot manager to handle this specific task might be able to do the job with less of a delay, but I don't know of any such program.