Nginx: How can I re-use this proxy pass config in multiple locations?

You're already aware of include. This is the way it's normally done, especially when you have to reuse the configuration in different servers.