How can I determine this string value based on the C disassembly?
So I am working on a "find the password" type binary disassembly problem and cannot quite figure it out.
The assembly is as follows:
function checkpw
**************************************************************
* *
* FUNCTION *
**************************************************************
undefined8 __stdcall checkpw(void)
undefined8 RAX:8 <RETURN>
checkpw XREF[2]: Entry Point(*), main:001012a1(c)
00101179 48 89 c8 MOV RAX,RCX
0010117c 48 31 c1 XOR RCX,RAX
0010117f 8a 08 MOV CL,byte ptr [RAX]
00101181 80 f1 52 XOR CL,0x52
00101184 80 f9 11 CMP CL,0x11
00101187 75 5e JNZ LAB_001011e7
00101189 8a 48 07 MOV CL,byte ptr [RAX + 0x7]
0010118c 80 e9 16 SUB CL,0x16
0010118f 80 f9 0d CMP CL,0xd
00101192 75 53 JNZ LAB_001011e7
00101194 8a 48 01 MOV CL,byte ptr [RAX + 0x1]
00101197 48 31 d2 XOR RDX,RDX
0010119a fe c2 INC DL
0010119c 48 d1 e2 SHL RDX,1
0010119f 40 8a 3c 10 MOV DIL,byte ptr [RAX + RDX*0x1]
001011a3 40 30 cf XOR DIL,CL
001011a6 40 80 ff 40 CMP DIL,0x40
001011aa 75 3b JNZ LAB_001011e7
001011ac 80 c1 63 ADD CL,0x63
001011af 80 f9 d6 CMP CL,0xd6
001011b2 75 33 JNZ LAB_001011e7
001011b4 8a 4c 10 01 MOV CL,byte ptr [RAX + RDX*0x1 + 0x1]
001011b8 80 f9 23 CMP CL,0x23
001011bb 7e 2a JLE LAB_001011e7
001011bd 80 c1 5b ADD CL,0x5b
001011c0 70 25 JO LAB_001011e7
001011c2 48 8d 0c 50 LEA RCX,[RAX + RDX*0x2]
001011c6 8a 09 MOV CL,byte ptr [RCX]
001011c8 80 f1 f3 XOR CL,0xf3
001011cb 80 f9 c7 CMP CL,0xc7
001011ce 75 17 JNZ LAB_001011e7
001011d0 8a 48 05 MOV CL,byte ptr [RAX + 0x5]
001011d3 8a 68 06 MOV CH,byte ptr [RAX + 0x6]
001011d6 66 81 f1 XOR CX,0x4c47
47 4c
001011db 66 81 f9 CMP CX,0x1234
34 12
001011e0 75 05 JNZ LAB_001011e7
001011e2 48 31 c0 XOR RAX,RAX
001011e5 eb 05 JMP LAB_001011ec
LAB_001011e7 XREF[8]: 00101187(j), 00101192(j),
001011aa(j), 001011b2(j),
001011bb(j), 001011c0(j),
001011ce(j), 001011e0(j)
001011e7 b8 01 00 MOV EAX,0x1
00 00
LAB_001011ec XREF[1]: 001011e5(j)
001011ec c3 RET
According to Ghidra, the decompiled function is:
undefined8 checkpw(void)
{
undefined8 uVar1;
char *in_RCX;
if (((((*in_RCX != 'C') || (in_RCX[7] != '#')) || ((byte)(in_RCX[2] ^ in_RCX[1]) != 0x40)) ||
((in_RCX[1] != 0x73 || (in_RCX[3] < '$')))) ||
((SCARRY1(in_RCX[3],'[') || ((in_RCX[4] != '4' || (*(short *)(in_RCX + 5) != 0x5e73)))))) {
uVar1 = 1;
}
else {
uVar1 = 0;
}
return uVar1;
}
It decompiles main function as :
void main(int param_1,long param_2)
{
long lVar1;
size_t sVar2;
undefined8 uVar3;
lVar1 = ptrace(PTRACE_TRACEME,0,1,0);
if (lVar1 < 0) {
/* WARNING: Subroutine does not return */
exit(1);
}
if (param_1 != 2) {
/* WARNING: Subroutine does not return */
exit(2);
}
sVar2 = strlen(*(char **)(param_2 + 8));
if (sVar2 != 8) {
/* WARNING: Subroutine does not return */
exit(3);
}
uVar3 = checkpw();
if ((int)uVar3 != 0) {
puts("Invalid Password!");
/* WARNING: Subroutine does not return */
exit(4);
}
puts("Correct Password!");
/* WARNING: Subroutine does not return */
exit(0);
}
At this point, I can tell the password must be 8 characters.
Also based on the checkpw decompilation, I believe the following is true (assuming variable passwd is a character array containing a valid password):
passwd[0] = 'C'
passwd[1] = 's'
passwd[2] = '3'
passwd[3] = '$'
passwd[4] = '4'
passwd[7] = '#'
Though I'm not entirely confident on a few of them, I really have am having trouble with those in position 5 and 6.
The decompiled function doesn't seem to reference the seventh character so I'm assuming it can be anything, but not sure what this means as it relates to this problem:
(*(short *)(in_RCX + 5) != 0x5e73)
This:
(*(short *)(in_RCX + 5) != 0x5e73)
Is comparing two characters at once. This statement is calculating in_RCX + 5 and then casting it to short * i.e. a pointer to a 16bit signed integer (the register used in the function body is actually RAX despite the name of this variable in the decompiled code). It is then dereferencing said pointer to get two bytes at once, and it compares them to 0x5e73. Of course, this is just decompiled code, so it doesn't mean that the program is actually doing a 16-bit MOV (indeed it is doing two 8-bit MOVs), it is only the C version of what the decompiler thinks is going on.
You can see this a lot more clearly in the disassembly:
001011d0 8a 48 05 MOV CL,byte ptr [RAX + 0x5]
001011d3 8a 68 06 MOV CH,byte ptr [RAX + 0x6]
001011d6 66 81 f1 XOR CX,0x4c47
47 4c
001011db 66 81 f9 CMP CX,0x1234
34 12
001011e0 75 05 JNZ LAB_001011e7
The check in the disassembly is done with XOR + CMP + JNZ, so Ghidra already xored 0x4c47 and 0x1234 together for you, which is 0x5e73. This means that in order for the check to pass, passwd[5] must be 0x73 ('s') and passwd[6] must be 0x5e ('^').