How do I secure an USB device to a computer so it can't be stolen?

There are internal USB ports, get something like an internal USB adapter and remove it from the bracket or use an internal USB expansion module. Either mount the USB adaptor internally, or permanently fix it to a USB extension cable (cable tied to the inside of the case), and route it through a suitable office. If a headset dies, you're just out a usb extension cable and the dead headset

Have you tried superglue?

Heck, there are even ways to remove superglue: How to Remove Super Glue

How about the NZXT Bunker Secure USB 2.0 Drive Bay? $24.99

http://www.legitreviews.com/article/1537/3/

The above image doesn't look like this would do anything for you, but the below image reveals how this could be helpful.