Self-hosted vs outsourced domain name for NS record

What are the things to look out for when implementing self-hosted or outsourced?

If you are serious about your domain name you should not rely on any single DNS provider but have 2 of them. You can not pick them arbitrarily and hope it will work, it needs to be fully coordinated between the two, but it is possible, even to have full DNSSEC support.

No matter which DNS provider you choose, you will have problems one day. If your domain is really important (and the services on it) you should use multiple DNS providers.

You are not used the correct terminology for what you describe. I welcome you to read RFC 8499 about DNS Terminology. You will see that what you describe is in-bailiwick nameservers (ns.example.com being nameserver for example.com) or fully external nameservers.

You seem to be more concerned with the naming, or at least that is how I read your question, than really where the service is provided, because this is almost orthogonal: no matter if your nameservers are in-bailiwick or not, technically they can be under your control and maintenance or not.

You won't find any sole piece of advice for any case, both have advantages. I would however strongly suggest not to go in the "in-bailiwick" case, until you fully understand the DNS and how it works and specially when it intersects with the registration plane, because for in-bailiwick nameservers you need to maintain glues at the registry, through the registrar of the domain, and this unfortunately is often a pain point.

If you use external nameservers, regarding naming (there are other considerations: they should not be hosted in the same datacentre, not all be behind the same AS - except if anycast is into play - or the same IP block, etc.), you should make sure to have nameservers using names in multiple registries (so not only multiple TLDs, if you take com and net both TLDs are at the same registry).

All big serious DNS providers give that option to their clients and on top of that the set of nameservers may differ from one zone to another or one client to another for better isolation and possibly different level of services.

Also, once you do that, you create a transitive dependency. The level of security of your domain is tied to the level of security of the domain name used for the naming of the nameserver authoritative on your domain name.

For example, if you want to do DNSSEC, it is fine in your zone, but then if the authoritative nameservers of your zone are themselves in zone NOT DNSSEC enabled, it lowers the real security of your zone.