Control loading of kernel module automatically

When i do rmmod usb_storage and then dmesg i can see the output saying usb_storage deregistered. But when i attach a flash drive and then dmesg , it says

Initializing USB Mass Storage driver...
[16565.129239] scsi41 : usb-storage 1-1:1.0
[16565.130134] usbcore: registered new interface driver usb-storage
[16565.130172] USB Mass Storage support registered.

How do i manually insert/remove the module without it automatically getting loaded?

Just blacklist the kernel driver:

echo "blacklist usb-storage" | sudo tee -a /etc/modprobe.d/blacklist.conf

After that, nobody can use a USB flash drive in your system.

You'll have to manually "insmod" your driver with sudo to use USB devices again or if you want to restore the initial behaviour remove the module from the blacklist file.