"This Connection Is Not Private" → "Visit The Website" asks for admin password to "Update Settings". Are permanent system changes made?

When you enter your password and press Update Settings, what really happens is that the certificate presented by that web site is added to the Certificates set in your login keychain. Your password is needed in order for Safari to have permission to add something to your keychain.

If you open up Keychain Access.app, select the login keychain and then the Certificates category, you'll find the specific certificate in the list on the right. The certificate is marked with a blue circle with a white plus inside it. This means it is marked as "trusted".

I.e. you're essentially overriding standard system behavior and stating that you trust a specific certificate even though it is not otherwise trusted (for example because it is expired). This is what will allow you to browse the site without further warnings.

Does it make your computer permanently less secure? - In some ways yes, because you're now trusting a certificate that is not really valid. If some malicious person used that certifcate to trick you then the system will trust that certificate. As you're manually stating that you do indeed trust the certificate, you're already aware that something is out of the ordinary, so it's not a huge problem. You just need to be aware that the trust doesn't automatically "expire" and disappear the next day or something like that.

If you want to manually revoke your trust, you can use Keychain Access.app to right-click on the certificate to Delete it. If you visit the same web site again with Safari, you'll get the warning again.

Note that some browsers (including Safari in Incognito Mode) solve the problem with overriding invalid certificates by allowing the user to override the warning temporarily. In that case nothing is added to the keychain, thus you do not need to enter a password. You can then browse the web site, but after you have closed down your browser, you'll get warnings again the next time you visit the site.

Also note that in these cases, you're only overriding the warning for that specific browser session. I.e. if you're using some other application (i.e. not the browser) to communicate with the site, such as a temporary override won't work. In contrast, doing what Safari does means that other programs (using the system trust store) will also respect your trust in that site.

When you allow Safari to "update settings", the certificate for that web site is added to a list of trusted certificates in your login keychain for web-based TLS (SSL) traffic, but the exception will be limited in scope since the certificate is not trusted for other uses, like S/MIME, code signing, IPsec encryption, etc.

In addition to using the Keychain Access app (as described by @jksoegaard), you can also view (and manipulate) "trusted certificates" from the shell using the security command.

security dump-trust-settings

will show certificates that have been trusted by the user, including those added when you click the "Visit this website" button in Safari. They will have a "Policy OID" of "SSL" and show the "Allowed Error", such as host name mismatch, expired certificate, etc.

In addition,

security dump-trust-settings -d

will show certificates that have been trusted by the administrator (added by a corporate MDM profile, for example). And

security dump-trust-settings -s

will show the certs trusted by macOS by default.

(Trusted certificates can be removed with security remove-trusted-cert and new ones added via security add-trusted-cert, but require the certificate in DER or PEM format. )