How are child nameservers resolved?

When specifying DNS information for a domain (e.g. in the registrar control panel) one can use nameservers which are subdomains of the domain. For example, if I have a domain example.com, I can set nameservers "ns1.example.com 1.1.1.1" and "ns2.example.com 1.1.1.2" (IP-addresses are mandatory). What is the process of resolving the name ns1.example.com?

You're looking for glue records:

Name servers in delegations are identified by name, rather than by IP address. This means that a resolving name server must issue another DNS request to find out the IP address of the server to which it has been referred. If the name given in the delegation is a subdomain of the domain for which the delegation is being provided, there is a circular dependency. In this case the name server providing the delegation must also provide one or more IP addresses for the authoritative name server mentioned in the delegation. This information is called glue. The delegating name server provides this glue in the form of records in the additional section of the DNS response, and provides the delegation in the answer section of the response.

In the zone file for example.com you must have a glue record such as:

ns1.example.com 604800 IN A 1.1.1.1
ns2.example.com 604800 IN A 1.1.1.2