64-bit syscall documentation for MacOS assembly

I'm having trouble finding the good documentation for writing 64-bit assembly on MacOS.

The 64-bit SysV ABI says the following in section A.2.1 and this SO post quotes it:

• A system-call is done via the syscall instruction. The kernel destroys registers %rcx and %r11.

• Returning from the syscall, register %rax contains the result of the system-call. A value in the range between -4095 and -1 indicates an error, it is -errno.

Those two sentences are ok on Linux but are wrong on macOS Sierra with the following code:

global _start
extern _exit

section .text
_start:

; Align stack to 16 bytes for libc
and rsp, 0xFFFFFFFFFFFFFFF0

; Call write
mov rdx, 12             ; size
mov rsi, hello          ; buf
mov edi, 1              ; fd
mov rax, 0x2000004      ; write ; replace to mov rax, 0x1 on linux
syscall

jc .err                 ; Jumps on error on macOS, but why?
jnc .ok

.err:
mov rdi, -1
call _exit              ; exit(-1)

.ok:
; Expect rdx to be 12, but it isn't on macOS!
mov rdi, rdx
call _exit              ; exit(rdx)

; String for write
section .data
hello:
.str db `Hello world\n`
.len equ $-hello.str

Compile with NASM:

; MacOS: nasm -f macho64 syscall.asm && ld syscall.o -lc -macosx_version_min 10.12 -e _start -o syscall
; Linux: nasm -f elf64 syscall.asm -o syscall.o && ld syscall.o -lc -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o syscall

Run on macOS:

./syscall      # Return value 0
./syscall >&-  # Return value 255 (-1)

I found out that:

• A syscall return errno an sets the carry flag on error, instead of returning -errno in rax
• rdx register is clobbered by syscall
• On Linux, everything works as expected

Why is rdx clobbered? Why doesn't a syscall return -errno? Where can I find the real documentation?

The only place I found where someone talks about the carry flag for syscall errors is here

Solution 1:

I used this:

# as hello.asm -o hello.o
# ld hello.o -macosx_version_min 10.13 -e _main -o hello  -lSystem
.section __DATA,__data
str:
  .asciz "Hello world!\n"

.section __TEXT,__text
.globl _main
_main:
  movl $0x2000004, %eax           # preparing system call 4
  movl $1, %edi                   # STDOUT file descriptor is 1
  movq str@GOTPCREL(%rip), %rsi   # The value to print
  movq $13, %rdx                  # the size of the value to print
  syscall

  movl %eax, %edi
  movl $0x2000001, %eax           # exit (return value of the call to write())
  syscall

and was able to catch return value into eax. Here return value is the number of bytes actually written by write system call. And yes MacOS being a BSD variant it is the carry flag that tells you if the syscall was wrong or not (errno is just an external linkage variable).

# hello_asm.s
# as hello_asm.s -o hello_asm.o
# ld hello_asm.o -e _main -o hello_asm
.section __DATA,__data
str:
        .asciz "Hello world!\n"
good:
        .asciz "OK\n"

.section __TEXT,__text
.globl _main
_main:
        movl $0x2000004, %eax           # preparing system call 4
        movl $5, %edi                   # STDOUT file descriptor is 5
        movq str@GOTPCREL(%rip), %rsi   # The value to print
        movq $13, %rdx                  # the size of the value to print
        syscall

        jc err

        movl $0x2000004, %eax           # preparing system call 4
        movl $1, %edi                   # STDOUT file descriptor is 1
        movq good@GOTPCREL(%rip), %rsi  # The value to print
        movq $3, %rdx                   # the size of the value to print
        syscall
        movl $0, %edi
        movl $0x2000001, %eax           # exit 0
        syscall
err:    
        movl $1, %edi
        movl $0x2000001, %eax           # exit 1
        syscall

This will exits with error code one because descriptor 5 was used, if you try descriptor 1 then it will work printing another message and exiting with 0.