Is there a way to change IIS bindings without recycling the application pool?

Nope. Any changes will cause an app pool recycle. I'd look at reworking the site so it doesn't have to load so much data at startup.