How does a registrar update the gtld name servers?

I don't know the exact details or implementations, but my understanding is that it revolves around EPP - Extensible Provisioning Protocol

The motivation for the creation of EPP was to create a robust and flexible protocol that could provide communication between domain name registries and domain name registrars. These transactions are required whenever a domain name is registered or renewed, thereby also preventing domain hijacking. Prior to its introduction, registries had no uniform approach, and many different proprietary interfaces existed.

http://en.wikipedia.org/wiki/Extensible_Provisioning_Protocol