Why are the RSA-SHA256 signatures I generate with OpenSSL and Java different?

Solution 1:

openssl dgst -sha256 < data.txt

produces something like:

(stdin)= b39eaeb437e33087132f01c2abc60c6a16904ee3771cd7b0d622d01061b40729

notice the (stdin)='? you don't want that to be part of your hash, if you need to create a digest, use the -binary option.

try using this to sign your data:

openssl sha -sha256 -sign private.pem < data.txt

This does everything you need.

edit - a little more explanations:

let's create a digest and show it

$ openssl dgst -sha256 -binary < data.txt > digest
$ hd digest
00000000  26 3b 0a a1 2e b9 32 db  b8 dc d3 6f 37 94 0b 05  |&;....2....o7...|
00000010  71 9c ba 79 46 34 28 9f  5c 5b 98 9a 64 61 c9 ec  |q..yF4(.\[..da..|

now we take this digest and sign int using rsautl:

$ openssl rsautl -sign -inkey private.pem < digest > sign1
$ hd sign1
00000000  1b 7a cf a4 8d 41 8e 04  ed 3a bb ba 86 f1 f8 e0  |.z...A...:......|
00000010  df f7 47 3e d7 a7 f4 90  7a 05 f8 7f 45 e5 29 e7  |..G>....z...E.).|
00000020  9f f4 2c 91 97 2f e7 26  69 9f 6a 07 a3 48 1b 85  |..,../.&i.j..H..|
00000030  2e f8 ee 44 4d 25 9f ae  05 95 81 c9 e3 07 68 ad  |...DM%........h.|

now let's sign the same file using dgst directly:

$ openssl dgst -sha256 -sign private.pem < data.txt > sign2
$ hd sign2
00000000  15 c2 94 87 eb e6 cb 45  c8 63 0c 97 60 d3 07 f3  |.......E.c..`...|
00000010  dc 65 32 ad 44 1c c2 2a  7f a3 e1 fc dd 84 27 8c  |.e2.D..*......'.|
00000020  77 a6 97 2b 33 6b c6 d7  7d e1 1d 39 5c 48 b6 48  |w..+3k..}..9\H.H|
00000030  cb 18 be bf 6a 66 90 d3  88 89 52 6c dd d1 b9 99  |....jf....Rl....|

So what's different here? To see that, we can verify the signature and show the raw output. Both files do contain the digest, but the metadata and padding is different:

$ openssl rsautl -raw -verify -inkey private.pem < sign1 | hd
00000000  00 01 ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
00000010  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff 00  |................|
00000020  26 3b 0a a1 2e b9 32 db  b8 dc d3 6f 37 94 0b 05  |&;....2....o7...|
00000030  71 9c ba 79 46 34 28 9f  5c 5b 98 9a 64 61 c9 ec  |q..yF4(.\[..da..|

$ openssl rsautl -raw -verify -inkey private.pem < sign2 | hd
00000000  00 01 ff ff ff ff ff ff  ff ff ff ff 00 30 31 30  |.............010|
00000010  0d 06 09 60 86 48 01 65  03 04 02 01 05 00 04 20  |...`.H.e....... |
00000020  26 3b 0a a1 2e b9 32 db  b8 dc d3 6f 37 94 0b 05  |&;....2....o7...|
00000030  71 9c ba 79 46 34 28 9f  5c 5b 98 9a 64 61 c9 ec  |q..yF4(.\[..da..|

To see this more clearly, we can try to use the -asn1parse flag, which won't work for the first signature, but for the second it shows the correct structure of the signature:

$ openssl rsautl -verify -inkey private.pem -asn1parse < sign1
Error in encoding
139931349546656:error:0D07209B:asn1 encoding routines:ASN1_get_object:too long:asn1_lib.c:142:

$ openssl rsautl -verify -inkey private.pem -asn1parse < sign2
    0:d=0  hl=2 l=  49 cons: SEQUENCE          
    2:d=1  hl=2 l=  13 cons:  SEQUENCE          
    4:d=2  hl=2 l=   9 prim:   OBJECT            :sha256
   15:d=2  hl=2 l=   0 prim:   NULL              
   17:d=1  hl=2 l=  32 prim:  OCTET STRING      
      0000 - 26 3b 0a a1 2e b9 32 db-b8 dc d3 6f 37 94 0b 05   &;....2....o7...
      0010 - 71 9c ba 79 46 34 28 9f-5c 5b 98 9a 64 61 c9 ec   q..yF4(.\[..da..