I dominated a planet/moon/station but their ships are still hostile to me. How to fix it?

If they are paying you every day tribute, then you have dominated them (to confirm, use your hyperdrive or land on a planet/moon/station/wormhole). Relinquishing them will not make the ships friendly to you (relinquishing is not really worth it until the developer makes relinquishing increasing your reputation). To properly make them again friendly to you, you have to increase your reputation with whatever government the planet/moon/station is part of. For example, if it is a Republic planet, kill a lot of pirates. So yes, you can still be friends with that government while dominating them all. "Dominating" a planet is not the same as "capturing", "dominating a planet" is more like a "protection racket".