GNU assembler on Windows: WriteFile returns ERROR_INVALID_HANDLE

The main issue with my code was that I violated Windows' ABI, which included

• not aligning the stack properly before a function call
• not adding a 32B padding (shadow space) onto the stack, meaning that the 5th parameter was incorrectly provided

Here's a working version of this code:

.intel_syntax noprefix

.extern GetStdHandle
.extern GetLastError
.extern WriteFile

.section .rodata
.Lchar: .ascii "F"

.section .data
.Lbytes_written: .long 0      # surprisingly just needs to be a dword, not qword in win64

.section .text

.global main

main:
    sub   rsp, 40                      # allocate shadow space + re-align the stack to RSP%16 == 0
    mov   rcx, -11                     # Magic number for stdout
    call  GetStdHandle
    mov   rcx, rax                     # hFile = return value
    lea   rdx, [rip + .Lchar]          # lpBuffer
    mov   r8, 1                        # 1 byte to write
    lea   r9, [rip + .Lbytes_written]  # output arg pointer
    mov   QWORD PTR [rsp + 32], 0      # lpOverlapped=NULL  in the stack space that was padding for the first call
    call  WriteFile
    add   rsp, 40
    ret

Thank you to Peter Cordes, Joshua and RbMm for helping me on this one!

main:
    sub rsp, 8
    mov rcx, -11
    call GetStdHandle

Memory trashed at this point. The top four slots (32 bytes) in the stack need to be free. The correct line seems to be

    sub rsp, 40

Moving right along:

    call WriteFile
    call GetLastError

Mistake here. Check rax for error code. If no error, GetLastError returns the previous error whatever it was.

If something seems absolutely nuts, check code further along. As far as I can determine from trying to make this stuff work, Windows disassembles your code in some places and code that isn't reached can still crash if it would misalign the stack. I had a case where the crash was due to the difference between not commenting code and commenting code that was unconditionally jumped around.