systemd ignores return code while starting service

For systemd to detect if the process was started successfully you have to use Type=forking, then fork your process in a helper script, and check in that script if the process was started successfully. With forking systemd will wait for the ExecStart command to finish and it will check its exit status.

You should change your unit file like this:

[Unit]
Description=test service
After=syslog.target

[Service]
Type=forking
User=testuser
Group=testuser
ExecStart=/usr/local/bin/fork_service

[Install]
WantedBy=multi-user.target

and in /usr/local/bin/fork_service you should have something like this:

#!/bin/bash

# Run your process in background
/path/to/your_service &

# Check if the services started successfully 
if ! kill -0 $! 2>/dev/null; then
    # Return 1 so that systemd knows the service failed to start
    exit 1
fi

I'm here just checking if the background process PID is still active, but you can have any check you want. The only important thing is that this script exits with 0 if the process started successfully, or a positive non zero value if it failed.

Also you don't have to use Bash to fork a process, you can use any language you want.

So what you want is for systemd to wait until the process is initalized before returning. Which is a reasonable feature. However since process initialization can take an arbitrary amount of time, there is no sane way for systemd to know how long to wait without help from the process being started.

The traditional way to do this is to use a forking daemon. The daemon does the initialization in one process, and then forks off the actual process once it has been establish that it can initalize successfully. The fork happening and the original process exiting successfully is then the signal to systemd that the daemon is initialized. From the systemd documentation:

If set to forking, it is expected that the process configured with ExecStart= will call fork() as part of its start-up. The parent process is expected to exit when start-up is complete and all communication channels are set up. The child continues to run as the main service process, and the service manager will consider the unit started when the parent process exits. This is the behavior of traditional UNIX services. If this setting is used, it is recommended to also use the PIDFile= option, so that systemd can reliably identify the main process of the service. systemd will proceed with starting follow-up units as soon as the parent process exits.

Systemd offers another solutions which seems more elegant to me. There is no reason to fork, just make the daemon tell systemd directly when it is initialized; this is the Type=notify. (see the documentation I linked above)

So to fix your specific example, you would modify your service file to say Type=notify and /usr/local/bin/return1 to be

#!/bin/bash                                                                     
exit 1
systemd-notify READY=1

Where obviously the exit 1 would normally happen conditionally if the initialization fails. And where the READY=1 is sent once initalization is finished.

This will give you the expected error message on the command line when you try to start it. To see that systemd actually waits for the READY=1, you can try it with the following /usr/local/bin/return1:

#!/bin/bash                                                                     
sleep 3
systemd-notify READY=1
sleep 1000000