Trying to compare two strings in x86-64

Solution 1:

a cmp instruction compares two values that fit into a register (8, 16, 32, or 64 bit values), not two strings. You need a cmpsb to compare each character of the strings, and can use a rep with it to compare the whole string in one go:

;compare input to the password
    mov rsi, attempt    ; first string
    mov rdi, password   ; second string
    mov rcx, 5          ; length of the password
    repe cmpsb          ; compare
    jne L2              ; no match
    cmp byte ptr [rsi], 10  ; \n to end the input
    jne L2              ; extra stuff in attempt
    jmp L1

Note that you could simplify this a bit by adding the '\n' character to the expected password and using a length of 6. Then you wouldn't need the extra cmp/jne checking for the end of the input.