Why does nsupdate fail with "operation canceled"?
Solution 1:
The target server's firewall may be blocking TCP port 53.
Even though DNS usually uses UDP, nsupdate sometimes needs TCP; see man nsupdate:
By default, nsupdate uses UDP to send update requests to the name server unless they are too large to fit in a UDP request, in which case TCP is used.