Is it possible to restrict the use of a root certificate to a domain

My customer uses a self signed certificate for an application to work. To be able to work, I have to install the root certificate they used to sign the certificate.

Is it possible to configure a root certificate so it only validates towards one domain ?

As a rule of thumb:

No, implied in trusting the customer's CA certificate is the trust in every certificate signed by that CA.

I don't know of any applications/libraries that have an easy option that allows you as the end-user to select that you'll trust your customers or any other CA certificate only for certain (sub-) domains i.e. only for *.example.com and *.example.org and nothing else.

Mozilla has a similar concern about currently trusted government sponsored CA's as an open attention point and for instance Chrome has extra checks built in for accessing Google sites, which was how the rogue *.google.com certificate and the compromise of the Diginotar CA became public.

But even if you don't trust the CA, you can still import/trust a specific server certificate signed by that CA, which will prevent SSL warnings for the hostnames in that certificate. That should make your application work without errors or complaints.

Exceptions:

A very underused option of the X.509v3 PKI standard is the Name Constraints extension, which allows a CA certificate to contain white- and blacklists of domain name patterns it is authorized to issue certificates for.

You might be lucky and your customer has restrained themselves when they set up their PKI infrastructure and included that Name constraint in their CA certificate. Then you can import their CA certificate directly and know that it can only validate a limited range of domain names.

@CryptoGuy had a pretty good answer here, but I wanted to expand on it.

To paraphrase:

You can restrict 3rd party CA to trust certificates (from that CA) issued to a name list you wish. Even if 3rd party CA do not have Name Constraints extension, it is possible to apply them by using your own internal CA server via cross-certification. The trick is that you sign 3rd party CA by using your internal CA.

leaf SSL cert -> cross-certificate -> your CA certificate -> your internal root certificate.

And here's how you make that work (using OpenSSL command line CA)

Create a simple CA

openssl req -new -x509 -days 3650 -newkey rsa:2048 -sha256 -out root-ca.crt -keyout root-ca.key -subj "/CN=My Root CA"

You may skip creating an intermediate CA

Create an intermediate CA request, with Name Constraints.

openssl req -new -days 3650 -newkey rsa:2048 -out domain-ca.req -sha256 -keyout domain-ca.key -config ossl_domain_com.cfg

With this in the ossl_domain_com.cfg file:

[ req ]
prompt=no
distinguished_name=req_distinguished_name
req_extensions=domain_ca

[ req_distinguished_name ]
CN=somedomain.com trust CA

[ domain_ca ]
basicConstraints=critical,CA:true,pathlen:1
nameConstraints=critical,permitted;DNS:.somedomain.com

Then, sign that Intermediate domain CA with your CA.

openssl x509 -req -in domain-ca.req -CA root-ca.crt -CAkey root-ca.key -sha256 -set_serial 1 -out domain-ca.crt -extensions domain_ca -extfile ossl_domain_com.cfg

If you skipped creating the intermediate, use your root CA to sign

Now re-sign the original domain's CA under your authority, using their certificate. You can add the CA extensions here.

openssl x509 -in third_party_ca.crt -CA domain-ca.crt -CAkey domain-ca.key -set_serial 47 -sha256 -extensions domain_ca -extfile ossl_domain_com.cfg -out domain-cross-ca.crt

You may need to use openssl x509 -x509toreq to create a request, which you would sign exactly the same way as the intermediate above.

Now, add your root CA, intermediate CA, and the domain-cross-ca to your browser's trust database.