How can I debug a docker container initialization?

Docker events command may help and Docker logs command can fetch logs even after the image failed to start.

First start docker events in the background to see whats going on.

docker events&

Then run your failing docker run ... command. Then you should see something like the following on screen:

2015-12-22T15:13:05.503402713+02:00 xxxxxxxacd8ca86df9eac5fd5466884c0b42a06293ccff0b5101b5987f5da07d: (from xxx/xxx:latest) die

Then you can get the startup hex id from previous message or the output of the run command. Then you can use it with the logs command:

docker logs <copy the instance id from docker events messages on screen>

You should now see some output from the failed image startup.

As @alexkb suggested in a comment: docker events& can be troublesome if your container is being constantly restarted from something like AWS ECS service. In this scenario it may be easier to get the container hex id out of the logs in /var/log/ecs/ecs-agent.log.<DATE>. Then use docker logs <hex id>.

Well the best I have found out so far is:

#stop the current demon and start it in debug modus
sudo service docker stop
dockerd -D # --debug

The just start the client from a new shell. The misconception was to think the client actually does anything at all... well it's just communicating with the daemon, so you don't want to debug the client but the daemon itself (normally).

In my case, the -a (attach to STDOUT/STDERR) flag was enough:

user@machine:~$ docker start -a server_name
Error: The directory named as part of the path /log/log_path/app.log does not exist.
For help, use /usr/bin/supervisord -h

It showed the startup error (in our case, a missing log path used by supervisord). I assume most container startup errors would show up here as well.

I can't answer your question on how to make docker output more complete but I can tell you that in-place regex replacing a string in a .so file is a bit insane: the string only has so much space allocated to it, and if you change the file offsets of other entries, the elf file becomes corrupted. Try running objdump or readelf on your .so file after running the perl command (before LD_LIBRARY_PATH change) outside of a container -- dollars to donuts it is now corrupt.

The reason it works in this sadly necessary hack is because "tmp" and "etc" are the same string length so no offsets change. Consider the directory /dkr or similar if you prefer not to use /tmp.

If you MUST take this approach and your desired paths are unchangeable, rebuild the library and change the default path for /etc/hosts in the source. Or better, when building your modified libnss_files.so rename it to something like libnss_altfiles.so and change nsswitch.conf to use hosts: altfiles when starting your docker container (unless docker has bind mounted nsswitch.conf as well, then you can't change it). This will let you have the libnss_altfiles.so in parallel with your normal libraries in the base system. If docker does bind-mount nsswitch.conf, leave a copy of your rebuilt libnss_files.so in your /lib-override directory ready to be loaded by LD_LIBRARY_PATH.

As a heads up, suid/sgid binaries ignore LD_LIBRARY_PATH and LD_PRELOAD, so some stuff is going to break (read: go back to using the default /etc/hosts) if you use those variables.