How domain provider's DNS name server entries work?

What you are after are called "glue records".

A glue record is the IP address of a name server held at the Domain Name registry.

Glue records are required when you wish to set the name servers of a domain name to a hostname under the domain name itself.

For example if you wished to set the name servers of example.com to ns1.example.com and ns2.example.com you would need to also provide the glue records (i.e. the IP addresses) for ns1.example.com and ns2.example.com.

If you did not provide the glue records for these name servers then your domain name would not work as anyone requiring DNS information for it would get stuck in a loop:

What is the name server for example.com? -> ns1.example.com What is the IP address of ns1.example.com? -> don't know, try looking at name server for example.com What is the name server for example.com? -> ns1.example.com

...and so on.

With the glue record in place the registry will hold the IP address and the loop will not occur:

What is the name server for example.com? -> ns1.example.com What is the IP address of ns1.example.com? -> [IP Address]

http://faq.domainmonster.com/dns/glue_record/