How can one re-create Apple’s defunct ‘Back To My Mac’ service with specific open-source tools?

Your question is broad and contains numerous elements, therefore I'll need to enumerate my answer accordingly:

Starting point 1:

I cannot see how any of the involved technology could be patented, nor that any technology as such is licensed here. Apple could ofcourse be licensing an implementation, but they would be able to reimplement themselves. The BTTM system can be re-implemented in a variety of ways using various protocols and technologies and still end up with similar functionality - so patents is not a problem in recreating similar functionality.

Note also that modern versions of similar systems exist in the market place now. For example you could just install Tailscale and have similar access to all of your devices. As long as you're a single user and have less than 100 devices, it is free of charge.

Starting point 2:

There are open-source versions of almost everything included in Apple's BTTM system. You would however need to integrate those yourself and make it scalable. This would involve things like automatizing a CA, having a specialized DNS-server, etc. It is not rocket science, but it's not something you do in a Sunday afternoon either.

Note that there's a big difference between making a service like Apple's BTTM that is supposed to support thousands (if not millions) of users and even more devices - and making something that will just work for your own purposes. If you're just looking for some way to create a BTTM-like experience for yourself, it can be done much simpler than trying to replicate Apple's BTTM system.

Starting point 3:

For a technical reference you can just look up RFC6281. It holds information enough for a professional (or motivated hobbyist) to create a similar system. However if the goal is to make it interoperable with Apple's BTTM service (i.e. used with the clients included in older versions of macOS) - that is much harder (bordering on impossible) and not worth doing. Some of the issues here is that you wouldn't have access to Apple's secret keys, their me.com domain and the custom protocols for AppleID authentication.