custom domain resolves through google but not company servers
If ns1.company.com is an authoritative-only server (which the name suggests it might be), it's normal that it doesn't answer for names that are not in any of its own zones.
If it's a name in a zone delegated from one of it's zones it will respond with a referral (like what the question seems to indicate), if it's just an entirely unrelated name the typical handling is to respond with status REFUSED.