Decrypt SSL traffic with the openssl command line tool - continued

I got an answer to my original question here hopefully it is ok to ask a follow up question by posting another question. From my original question above I was able to decrypt the client pre master secret/key using my private key as it was explained to me using

openssl rsautl -in cpre.key -inkey key.pem -decrypt -out spre.key

This seems pretty straight forward and it created a 48 byte file. I do realize what I am doing is not going to actually decrypt or encrypt the https traffic I am working on this in small steps. The next step from my reading is going to be to generate the master secret/key and from the answer on my previous post it seems I should be able to also do this with openssl which I am struggling a bit here to do this. I've come up with a variation of openssl dgst

echo -n 'value' | openssl dgst -sha256 -mac hmac -macopt hexkey:$(hexkey)

where value is "master secret"+client.random+server.random and hexkey is my decrypted pre master secret from the previous step. Am I doing this step correctly? From the way I understand the RFC since this first time through it produces 32 bytes and the RFC explains it will be 48 bytes. The way I am interpreting the RFC is that I need to take that result and pass it through again which generates an additional 32 bytes of which I take the first 16 and concatenate that on the end of the first 32 to get my 48 bytes for the master secret/key. Am I completely off on this next step after I have decrypted and have the server side pre master key? Thanks David B

Solution 1:

Preliminarily, I hope you didn't actually do what you showed. Except for the labels, all the data used in TLS key derivation is binary data that cannot accurately be given as argument to a shell echo command or any other command, except possibly the builtin version in zsh if you disable escapes, and certainly cannot be 'typed' (even cut&pasted) in a literal argument with singlequotes.

If you actually have the data in files, those can be used. Binary data can be read from, and written to, files -- at least on the platforms OpenSSL supports. If you don't have files (but do have pipes) you can work-around this by passing the data in hex, and except for -macopt hexkey: which already takes hex, using either a program like xxd -r, or printf with the hex formed into escapes, or echo with the hex formed into non-portable escapes, or hacks like GNU sed s///e, or a more general program awk or perl, to convert the hex to binary for input to openssl. And when necessary similar hacks to convert binary output to hex, but dgst -mac hmac can output in hex as long as you remove the blahblah=(sp) from the front.

More substantively, running the first HMAC output back through itself only gives you the 'A' blocks, which you then run through another layer of HMACs to get the actual output 'P_hash'. See https://crypto.stackexchange.com/questions/46549/tls-1-2-prf-with-hmac-sha256 which is effectively the same question using a test vector published at https://www.ietf.org/mail-archive/web/tls/current/msg03416.html except I answered in Java. Here is the OpenSSL commandline equivalent:

$ echo $key; echo $lbsd # start from hex
9bbe436ba940f017b17652849a71db35
74657374206c6162656c a0ba9f936cda311827a6f796ffd5198c
$ echo $lbsd | xxd -r -p >a0
$ openssl dgst -sha256 -mac hmac -macopt hexkey:$key <a0 -binary >a1
$ openssl dgst -sha256 -mac hmac -macopt hexkey:$key <a1 -binary >a2
$ openssl dgst -sha256 -mac hmac -macopt hexkey:$key <a2 -binary >a3
$ openssl dgst -sha256 -mac hmac -macopt hexkey:$key <a3 -binary >a4
$ cat a1 a0 | openssl dgst -sha256 -mac hmac -macopt hexkey:$key -binary >k1
$ cat a2 a0 | openssl dgst -sha256 -mac hmac -macopt hexkey:$key -binary >k2
$ cat a3 a0 | openssl dgst -sha256 -mac hmac -macopt hexkey:$key -binary >k3
$ cat a4 a0 | openssl dgst -sha256 -mac hmac -macopt hexkey:$key -binary >k4
$ cat k1 k2 k3 k4 | head -c100 | xxd
0000000: e3f2 29ba 727b e17b 8d12 2620 557c d453  ..).r{.{..& U|.S
0000010: c2aa b21d 07c3 d495 329b 52d4 e61e db5a  ........2.R....Z
0000020: 6b30 1791 e90d 35c9 c9a4 6b4e 14ba f9af  k0....5...kN....
0000030: 0fa0 22f7 077d ef17 abfd 3797 c056 4bab  .."..}....7..VK.
0000040: 4fbc 9166 6e9d ef9b 97fc e34f 7967 89ba  O..fn......Oyg..
0000050: a480 82d1 22ee 42c5 a72e 5a51 10ff f701  ....".B...ZQ....
0000060: 8734 7b66                                .4{f

If you can get your code to replicate this, it should work for the actual TLS1.2 handshake also, adjusted for the correct lengths: 48 for the master secret, and depending on the ciphersuite for the working keys.