How to do a clean installation of a software in Mac

There is no registry on macOS so the developer can choose from many ways to license the software. Some store files and licenses in your user keychain, others in your user library. If the software is licensed per computer (say three people have user accounts on your mac but all can use the software) then the license key would be stored in the system area.

I would start with the vendor to get the uninstall / unlicense steps needed for however they choose to implement their licensing scheme. It could be a preference that is cryptographically signed or something as trivial as writing a hidden (or obscurely located) file on the filesystem.

Things like this are why App Store and buying apps through Apple is so compelling - all this confusion on licensing is cut through with a simple to understand model. Unfortunately - many vendors still try the old ways or think they can save money by not supporting their customers. This vendor is doubly unfair since the new install should clean up after the old if the old uninstall process left behind detritus or they don't have a clear path to get support.

I'm glad to see they did support you and resolve this issue. Feel free to add an answer if you learned where this vendor stores their licenses or if it was just re-running their license tool that fixed it.