Running DOSBox completely headless
DOSBox uses SDL, which can be told to use a different display driver by setting the environment variable SDL_VIDEODRIVER. The dummy driver will display nothing at all, not even a window.
On Unix, use the export shell command:
export SDL_VIDEODRIVER=dummy
and on Windows, use set.
set SDL_VIDEODRIVER=dummy
Note that on Windows, you will need to use -noconsole to prevent DOSBox from generating the status terminal window.
Source: this forum post.