Can I send encrypted e-mail using Mail and self-signed certificates?

I can only answer the Mac Mail part of this question as I do not want to test the Thunderbird part.

I am just writing this part for the people who are not familiar with asymmetric encryption:

In this form of encryption you need two keys, a private one and a public one. A certificate in that sense is nothing more than the public key but it has some more information than just a key, it tells who you are, your organisation, etc. The private must never ever be lost and given to others. If it is you have to start over and the person who stole it can decrypt everything, so keep it save and password protected. Usually certificates are signed by so called Certificate Authorities and they most often take money for their service. This is why some people create their own CA and self-signed certificates, which is cheaper.

The way e-mail encryption works is that you send someone your public key (certificate) by signing your e-mail with it. You can sign any e-mail, because no harm is done by giving away your public key. Then this other person has your public key and encrypts an e-mail to you with this key. From this point on nobody, other than the NSA with a possible Quantum computer, can decrypt this message in a reasonable amount of time given proper key length. Or some people infiltrate the actual encryption libraries like the HEARTBLEED (heartbeat) bug, which might have been intentional and who knows how many more of these backdoors are there. Certificates usually have a limited lifetime and need to be renewed every now and then.

So things to take away: You need public and private key and you only give away the public one.

Here is what I have done:

1. Created a Root CA and a Signing CA (you do not need that, but it is what I did) and I used this very good tutorial for it: Multi-level CAs in http://pages.cs.wisc.edu/~zmiller/ca-howto/

2. Then I changed my openssl.conf in such a way that Mac Mail allows me to use the certificated for e-mail encryption as well.

   The important bit is to have the keyUsage and extKeyUsage

   [ v3_req ]
   
   # Extensions to add to a certificate request
   
   basicConstraints = CA:FALSE
   keyUsage = nonRepudiation, digitalSignature, keyEncipherment,       dataEncipherment
   extKeyUsage = emailProtection, Apple .Mac email signing, Apple .mac email encryption
   

   For the people who like specs, read this https://www.ietf.org/rfc/rfc2459, but you will not find the MacOS specific parts in it.

3. Created the private key and the certificate:

   openssl req -newkey rsa:4096 -keyout <e-mail>.key -config openssl.cnf -out <e-mail>.req -days 3650
   
   openssl ca -config openssl.cnf -out <e-mail>.crt -infiles <e-mail>.req
   

   You might argue that ten years is too long as a certificate life time, but I wanted to have something that is working now and in 10 years time or even less this way of encryption will not work anyways. I did not want to renew the certs all the time as I created them for all my family.

   If somehting goes wrong in this process, you can revoke the cert by:

   openssl ca -config openssl.cnf -revoke <e-mail>.crt
   

   One should also create crls, but I did not do that either.

4. Converted keys to a p12 format

   openssl pkcs12 -export -in <e-mail>.crt -inkey <e-mail>.key -out <e-mail>.p12
   

5. Imported p12 to the KeyChain on OS X 10.9.4 (13E28)

6. Marked it as trusted

7. Associated Certificate with mail account in the MacOS Mail preferences under Account Information in the field TLS Certificate. Send mail to other mail address that also has a cert and key and signed the message with my public key, well MacOS Mail did this for me.

8. Received the mail and send back my first encrypted mail.

What happened to me when I tried the procedure with my wife was that the lock symbol in my reply mail to the first signed mail was greyed out and I was puzzled. This meant I could not encrypt mails. The reason was that I received the signed mail that my wife had sent to me, but I did not mark here signature as trusted in the MacOS keychain, as all signatures that one receives also end up there. After marking it trusted and restarting Mail everything worked fine.

Maybe someone or even the person asking can add the Thunderbird part to this answer.

Is there a reason you're trying this particular approach?

If your real question is how to send signed and encrypted email, I use GPG (through GPGTools). There is a plug-in that works with Thunderbird but I use Apple's Mail.