How can I turn off responses to Calendar invites?
This is not possible using Calendar.app, including both macOS (OS X) and iOS.
Your best bet is to submit feedback on Apple's website: https://www.apple.com/feedback/macos.html
If enough of us do it hopefully they'll add the functionality.