How does a public key verify a signature?

Your understanding of "public keys encrypt, private keys decrypt" is correct... for data/message ENCRYPTION. For digital signatures, it is the reverse. With a digital signature, you are trying to prove that the document signed by you came from you. To do that, you need to use something that only YOU have: your private key.

A digital signature in its simplest description is a hash (SHA1, MD5, etc.) of the data (file, message, etc.) that is subsequently encrypted with the signer's private key. Since that is something only the signer has (or should have) that is where the trust comes from. EVERYONE has (or should have) access to the signer's public key.

So, to validate a digital signature, the recipient

1. Calculates a hash of the same data (file, message, etc.),
2. Decrypts the digital signature using the sender's PUBLIC key, and
3. Compares the 2 hash values.

If they match, the signature is considered valid. If they don't match, it either means that a different key was used to sign it, or that the data has been altered (either intentionally or unintentionally).

Hope that helps!

The keys work inversely:

Public key encrypts, private key decrypts (encrypting):

openssl rsautl -encrypt -inkey public.pem -pubin -in message.txt -out message.ssl
openssl rsautl -decrypt -inkey private.pem       -in message.ssl -out message.txt

Private key encrypts, public key decrypts (signing):

openssl rsautl -sign -inkey private.pem       -in message.txt -out message.ssl
openssl rsautl       -inkey public.pem -pubin -in message.ssl -out message.txt

Below is an example script to test this whole flow with openssl.

#!/bin/sh
# Create message to be encrypted
echo "Creating message file"
echo "---------------------"
echo "My secret message" > message.txt
echo "done\n"

# Create asymmetric keypair
echo "Creating asymmetric key pair"
echo "----------------------------"
openssl genrsa -out private.pem 1024
openssl rsa -in private.pem -out public.pem -pubout
echo "done\n"

# Encrypt with public & decrypt with private
echo "Public key encrypts and private key decrypts"
echo "--------------------------------------------"
openssl rsautl -encrypt -inkey public.pem -pubin -in message.txt         -out message_enc_pub.ssl
openssl rsautl -decrypt -inkey private.pem       -in message_enc_pub.ssl -out message_pub.txt
xxd message_enc_pub.ssl # Print the binary contents of the encrypted message
cat message_pub.txt # Print the decrypted message
echo "done\n"

# Encrypt with private & decrypt with public
echo "Private key encrypts and public key decrypts"
echo "--------------------------------------------"
openssl rsautl -sign    -inkey private.pem -in message.txt          -out message_enc_priv.ssl
openssl rsautl -inkey public.pem -pubin    -in message_enc_priv.ssl -out message_priv.txt
xxd message_enc_priv.ssl
cat message_priv.txt
echo "done\n"

This script outputs the following:

Creating message file
---------------------
done

Creating asymmetric key pair
----------------------------
Generating RSA private key, 1024 bit long modulus
...........++++++
....++++++
e is 65537 (0x10001)
writing RSA key
done

Public key encrypts and private key decrypts
--------------------------------------------
00000000: 31c0 f70d 7ed2 088d 9675 801c fb9b 4f95  1...~....u....O.
00000010: c936 8cd0 0cc4 9159 33c4 9625 d752 5b77  .6.....Y3..%.R[w
00000020: 5bfc 988d 19fe d790 b633 191f 50cf 1bf7  [........3..P...
00000030: 34c0 7788 efa2 4967 848f 99e2 a442 91b9  4.w...Ig.....B..
00000040: 5fc7 6c79 40ea d0bc 6cd4 3c9a 488e 9913  [email protected].<.H...
00000050: 387f f7d6 b8e6 5eba 0771 371c c4f0 8c7f  8.....^..q7.....
00000060: 8c87 39a9 0c4c 22ab 13ed c117 c718 92e6  ..9..L".........
00000070: 3d5b 8534 7187 cc2d 2f94 0743 1fcb d890  =[.4q..-/..C....
My secret message
done

Private key encrypts and public key decrypts
--------------------------------------------
00000000: 6955 cdd0 66e4 3696 76e1 a328 ac67 4ca3  iU..f.6.v..(.gL.
00000010: d6bb 5896 b6fe 68f1 55f1 437a 831c fee9  ..X...h.U.Cz....
00000020: 133a a7e9 005b 3fc5 88f7 5210 cdbb 2cba  .:...[?...R...,.
00000030: 29f1 d52d 3131 a88b 78e5 333e 90cf 3531  )..-11..x.3>..51
00000040: 08c3 3df8 b76e 41f2 a84a c7fb 0c5b c3b2  ..=..nA..J...[..
00000050: 9d3b ed4a b6ad 89bc 9ebc 9154 da48 6f2d  .;.J.......T.Ho-
00000060: 5d8e b686 635f b6a4 8774 a621 5558 7172  ]...c_...t.!UXqr
00000070: fbd3 0c35 df0f 6a16 aa84 f5da 5d5e 5336  ...5..j.....]^S6
My secret message
done

If I had to re-phrase your question from how I understand it, you are asking the following:

If public key cryptography ensures that a public key can be derived from a private key, but a private key cannot be derived from a public key, then you might wonder, how can a public key decrypt a message signed with a private key without the sender exposing the private key within the signed message to the recipient? (re-read that a few times until it makes sense)

Other answers have already explained how asymmetric cryptography means that you can either:

1. Encrypt with public key, decrypt with matching private key (pseudocode below)

var msg = 'secret message';

var encryptedMessage = encrypt(pub_key, msg);

var decryptedMessage = decrypt(priv_key, encryptedMessage);

print(msg == decryptedMessage == 'secret message'); // True

2. Encrypt with private key, decrypt with matching public key (pseudocode below)

var msg = 'secret message';

var encryptedMessage = encrypt(priv_key, msg);

var decryptedMessage = decrypt(pub_key, encryptedMessage); // HOW DOES THIS WORK???

print(msg == decryptedMessage == 'secret message'); // True

We know that both example #1 and #2 work. Example #1 makes intuitive sense, while example #2 begs the original question.

Turns out, elliptic curve cryptography (also called "elliptic curve multiplication") is the answer to the original question. Elliptic curve cryptography is the mathematical relationship that makes the following conditions possible:

1. A public key can be mathematically generated from a private key
2. A private key cannot be mathematically generated from a public key (i.e. "trapdoor function")
3. A private key can be verified by a public key

To most, conditions #1 and #2 make sense, but what about #3?

You have two choices here:

1. You can go down a rabbit-hole and spend hours upon hours learning how elliptic curve cryptography works (here is a great starting point)... OR...
2. You can accept the properties above--just like you accept Newton's 3 laws of motion without needing to derive them yourself.

In conclusion, a public/private keypair is created using elliptic curve cryptography, which by nature, creates a public and private key that are mathematically linked in both directions, but not mathematically derived in both directions. This is what makes it possible for you to use someone's public key to verify that they signed a specific message, without them exposing their private key to you.

The public key encrypts and only the private key can decrypt it, and the reverse is true. They both encrypt to different hashes but each key can decrypt the other's encryption.

There are a few different ways to verify that a message came from some expected sender. For example:

The sender sends:

1. The message

2. The hash of the message encrypted with their private key

The receiver:

3. Decrypts the signature (2) with the public key to obtain a message, supposedly the same message as (1) but we don't know yet. We now have two messages that we need to verify are identical. So to do this, we will encrypt them both with our public key and compare the two hashes. So we will ....
4. Encrypt the original message (1) with the public key to obtain a hash
5. Encrypt the decrypted message (3) to get a second hash and compare to (4) to verify that they are identical.

If they aren't identical it means either the message was tampered with or it was signed with some other key and not the one we thought...

Another example would be for the sender to use a common hash that the receiver might know to use as well. For example:

The sender sends:

1. A message
2. Takes a known hash of the message, then encrypts the hash with the private key

The receiver:

3. Decrypts (2) and gets a hash value
4. Hashes the message (1) with the same hash used by the sender
5. Compares the two hashes to make sure they match

This again ensures the message wasn't tampered with and it is from the expected sender.