How to encrypt a large file in openssl using public key

Public-key crypto is not for encrypting arbitrarily long files. One uses a symmetric cipher (say AES) to do the normal encryption. Each time a new random symmetric key is generated, used, and then encrypted with the RSA cipher (public key). The ciphertext together with the encrypted symmetric key is transferred to the recipient. The recipient decrypts the symmetric key using his private key, and then uses the symmetric key to decrypt the message.

The private key is never shared, only the public key is used to encrypt the random symmetric cipher.

Solution for safe and high secured encode anyone file in OpenSSL and command-line:

You should have ready some X.509 certificate for encrypt files in PEM format.

Encrypt file:

openssl smime -encrypt -binary -aes-256-cbc -in plainfile.zip -out encrypted.zip.enc -outform DER yourSslCertificate.pem

What is what:

• smime - ssl command for S/MIME utility (smime(1))
• -encrypt - chosen method for file process
• -binary - use safe file process. Normally the input message is converted to "canonical" format as required by the S/MIME specification, this switch disable it. It is necessary for all binary files (like a images, sounds, ZIP archives).
• -aes-256-cbc - chosen cipher AES in 256 bit for encryption (strong). If not specified 40 bit RC2 is used (very weak). (Supported ciphers)
• -in plainfile.zip - input file name
• -out encrypted.zip.enc - output file name
• -outform DER - encode output file as binary. If is not specified, file is encoded by base64 and file size will be increased by 30%.
• yourSslCertificate.pem - file name of your certificate's. That should be in PEM format.

That command can very effectively a strongly encrypt big files regardless of its format.
Known issue: Something wrong happens when you try encrypt huge file (>600MB). No error thrown, but encrypted file will be corrupted. Always verify each file! (or use PGP - that has bigger support for files encryption with public key)

Decrypt file:

openssl smime -decrypt -binary -in encrypted.zip.enc -inform DER -out decrypted.zip -inkey private.key -passin pass:your_password

What is what:

• -inform DER - same as -outform above
• -inkey private.key - file name of your private key. That should be in PEM format and can be encrypted by password.
• -passin pass:your_password - your password for private key encrypt. (passphrase arguments)

I found the instructions at http://www.czeskis.com/random/openssl-encrypt-file.html useful.

To paraphrase the linked site with filenames from your example:

Generate a symmetric key because you can encrypt large files with it

openssl rand -base64 32 > key.bin

Encrypt the large file using the symmetric key

openssl enc -aes-256-cbc -salt -in myLargeFile.xml \
  -out myLargeFile.xml.enc -pass file:./key.bin

Encrypt the symmetric key so you can safely send it to the other person

openssl rsautl -encrypt -inkey public.pem -pubin -in key.bin -out key.bin.enc

Destroy the un-encrypted symmetric key so nobody finds it

shred -u key.bin

At this point, you send the encrypted symmetric key (key.bin.enc) and the encrypted large file (myLargeFile.xml.enc) to the other person

The other person can then decrypt the symmetric key with their private key using

openssl rsautl -decrypt -inkey private.pem -in key.bin.enc -out key.bin

Now they can use the symmetric key to decrypt the file

openssl enc -d -aes-256-cbc -in myLargeFile.xml.enc \
  -out myLargeFile.xml -pass file:./key.bin

And you're done. The other person has the decrypted file and it was safely sent.