How to run multiple commands synchronously from one subprocess.Popen command?

If you want to execute many commands one after the other in the same session/shell, you must start a shell and feed it with all the commands, one at a time followed by a new line, and close the pipe at the end. It makes sense if some commands are not true processes but shell commands that could for example change the shell environment.

Example using Python 2.7 under Windows:

encoding = 'latin1'
p = subprocess.Popen('cmd.exe', stdin=subprocess.PIPE,
             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for cmd in cmds:
    p.stdin.write(cmd + "\n")
p.stdin.close()
print p.stdout.read()

To have this code run under Linux, you would have to replace cmd.exe with /bin/bash and probably change the encoding to utf8.

For Python 3, you would have to encode the commands and probably decode their output, and to use parentheses with print.

Beware: this can only work for little output. If there was enough output to fill the pipe buffer before closing the stdin pipe, this code would deadlock. A more robust way would be to have a second thread to read the output of the commands to avoid that problem.

This is similar to the answer posted by Serge Ballesta, but not quite. Use his for asynchronous execution, where you don't care about the results. Use mine for synchronous processing and result gathering. Like his answer, I'm showing the Windows solution here - run a bash process in Linux rather than cmd in Windows.

from subprocess import Popen, PIPE
process = Popen( "cmd.exe", shell=False, universal_newlines=True,
                  stdin=PIPE, stdout=PIPE, stderr=PIPE )                             
out, err = process.communicate( commands ) 

USAGE DETAILS: The commands argument being passed here to the process.communicate method is a newline delimited string. If, for example you just read a batch file contents into a string, you could run it this way because it would already have the newlines.

Important: your string must end in a newline "\n". If it does not, that final command will fail to execute. Just like if you typed it into your command prompt but didn't hit enter at the end. You will however see a mysterious More? line in the end of the stdout returned. (that's the cause if you encounter this).

process.communicate runs synchronously by definition, and returns the stdout and stderr messages (if you directed them to subprocess.PIPE in your Popen constructor).

When you create a cmd.exe process in this way, and pass a string to it, the results will be exactly like if you were to open a command prompt window and entered commands into. And I mean that quite literally. If you test this, you will see that the stdout which is returned contains your commands. (It does NOT matter if you include an @echo off like if executing a batch file).

Tips for those who care about "clean" stdout results:

• @echo off will not suppress your commands from appearing in this returned string, but it does remove extra newlines that find their way in there otherwise. (universal_newlines=True strips an another set of those)

• Including an @ symbol prefix to your commands allows them to still execute. In a "normal" batch process that's the line-by-line way to "hide" your commands. In this context, it's a safe an easy marker by which you can find stdout lines you want to remove. (if one were so inclined)

• The cmd.exe "header" will appear in your output (which says the version of Windows etc.). Since you probably want to start your set of commands with @echo off, to cut out the extra newlines, that is also a great way to find where the header lines stopped and your commands/results began.

Finally, to address concerns about "large" output filling the pipes and causing you problems - first I think you need a HUGE amount of data coming back for that to be an issue - more than most people will encounter in their use cases. Second, if it really is a concern just open a file for writing and pass that file handle (the reference to the file object) to stdout/err instead of PIPE. Then, do whatever you want with the file you've created.