subprocess.call() arguments ignored when using shell=True w/ list [duplicate]

I am trying to get python's subprocess.call method to accept some args commands through a list (consisting of a sequence of strings) as advised in the python documentation. To explore this behavior before putting it into my actual script, I opened up IPython, ran some commands involving different combinations of shell settings and args commands and got the following behavior:

In [3]: subprocess.call(['ls', '-%sl' %'a'])
total 320
drwxr-xr-x  20 Kohaugustine  staff   680 Oct 15 16:55 .
drwxr-xr-x   5 Kohaugustine  staff   170 Sep 12 17:16 ..
-rwxr-xr-x   1 Kohaugustine  staff  8544 Oct 15 16:55 a.out
-rwxr-xr-x   1 Kohaugustine  staff  8544 Oct  3 10:28 ex1-6
-rw-r--r--@  1 Kohaugustine  staff   204 Oct  3 10:28 ex1-6.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Oct  3 10:15 ex1-7
-rw-r--r--@  1 Kohaugustine  staff    71 Oct  3 10:15 ex1-7.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:22 hello
-rw-r--r--@  1 Kohaugustine  staff    58 Sep 12 16:27 hello.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:24 hello.o
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:24 hello_1.o
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:27 hello_2.o
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:27 hello_3.o
-rwxr-xr-x   1 Kohaugustine  staff  8544 Oct 15 16:55 lesson_1-5
-rw-r--r--@  1 Kohaugustine  staff   185 Sep 28 10:35 lesson_1-5.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 21 10:06 temperature.o
-rw-r--r--@  1 Kohaugustine  staff   406 Sep 21 09:54 temperature_ex1-3.c
-rw-r--r--@  1 Kohaugustine  staff   582 Sep 21 10:06 temperature_ex1-4.c
-rw-r--r--@  1 Kohaugustine  staff   178 Sep 23 17:21 temperature_ex1-5.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 23 17:21 temperature_ex1-5.o
Out[3]: 0

In [4]: subprocess.call(['ls', '-%sl' %'a'], shell=True)
a.out           ex1-7           hello.c         hello_2.o       lesson_1-5.c            temperature_ex1-4.c
ex1-6           ex1-7.c         hello.o         hello_3.o       temperature.o           temperature_ex1-5.c
ex1-6.c         hello           hello_1.o       lesson_1-5      temperature_ex1-3.c     temperature_ex1-5.o
Out[4]: 0

In [6]: subprocess.call(['ls', '-al'])    
total 320
drwxr-xr-x  20 Kohaugustine  staff   680 Oct 15 16:55 .
drwxr-xr-x   5 Kohaugustine  staff   170 Sep 12 17:16 ..
-rwxr-xr-x   1 Kohaugustine  staff  8544 Oct 15 16:55 a.out
-rwxr-xr-x   1 Kohaugustine  staff  8544 Oct  3 10:28 ex1-6
-rw-r--r--@  1 Kohaugustine  staff   204 Oct  3 10:28 ex1-6.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Oct  3 10:15 ex1-7
-rw-r--r--@  1 Kohaugustine  staff    71 Oct  3 10:15 ex1-7.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:22 hello
-rw-r--r--@  1 Kohaugustine  staff    58 Sep 12 16:27 hello.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:24 hello.o
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:24 hello_1.o
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:27 hello_2.o
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 12 16:27 hello_3.o
-rwxr-xr-x   1 Kohaugustine  staff  8544 Oct 15 16:55 lesson_1-5
-rw-r--r--@  1 Kohaugustine  staff   185 Sep 28 10:35 lesson_1-5.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 21 10:06 temperature.o
-rw-r--r--@  1 Kohaugustine  staff   406 Sep 21 09:54 temperature_ex1-3.c
-rw-r--r--@  1 Kohaugustine  staff   582 Sep 21 10:06 temperature_ex1-4.c
-rw-r--r--@  1 Kohaugustine  staff   178 Sep 23 17:21 temperature_ex1-5.c
-rwxr-xr-x   1 Kohaugustine  staff  8496 Sep 23 17:21 temperature_ex1-5.o
Out[6]: 0

In [7]: subprocess.call(['ls', '-al'], shell = True)
a.out           ex1-7           hello.c         hello_2.o       lesson_1-5.c            temperature_ex1-4.c
ex1-6           ex1-7.c         hello.o         hello_3.o       temperature.o           temperature_ex1-5.c
ex1-6.c         hello           hello_1.o       lesson_1-5      temperature_ex1-3.c     temperature_ex1-5.o
Out[7]: 0

It seems like whenever shell=True, the output seems to be the same as:

In [9]: subprocess.call(['ls'])
a.out           ex1-7           hello.c         hello_2.o       lesson_1-5.c            temperature_ex1-4.c
ex1-6           ex1-7.c         hello.o         hello_3.o       temperature.o           temperature_ex1-5.c
ex1-6.c         hello           hello_1.o       lesson_1-5      temperature_ex1-3.c     temperature_ex1-5.o
Out[9]: 0

I'm puzzled; what happened to the '-a' option when I set shell=True? Didn't the shell read it? I've read the Docs and that it says that when shell=True, it means that my specified command will be executed through the shell, so it should mean that ls -a was fed to the shell and acted upon by the shell. Then why the behavior in [4] and [7] ? Also the pydocs doesn't explain it directly (although it does say what subpprocess will NOT DO when we set shell=False); what does it mean when we let shell=False? Is a new process spawned in the OS without having the shell actually control it?

Also, in case it might seem really awkward that I'm using a format string in [3] and [4], its because in my actual script where I'll be using subprocess.call, I will have to rely on these format strings to substitute in the appropriate command options. I cannot hardcode some of the command line options. Using a pure string for args is out of the question too because in my script there will be a method that has to do list operations on the commands. I don't know if there might be a better way to go about this though, so if will really help if anyone can suggest something different.

Thank you very much!

When shell is True, the first argument is appended to ["/bin/sh", "-c"]. If that argument is a list, the resulting list is

["/bin/sh", "-c", "ls", "-al"]

That is, only ls, not ls -al is used as the argument to the -c option. -al is used as the first argument the shell itself, not ls.

When using shell=True, you generally just want to pass a single string and let the shell split it according the shell's normal word-splitting rules.

# Produces ["/bin/sh", "-c", "ls -al"]
subprocess.call("ls -al", shell=True)

In your case, it doesn't see like you need to use shell=True at all.

When you use shell=True with a list, extra arguments are passed to the shell itself, not to the command running in the shell. They can then be referred to from within the shell script (passed as argv[0]) as $0, $1, etc.

The easiest answer is "don't do that": If you want to pass a list, don't use shell=True; if you want to pass a string, always use shell=True.

That said, it is possible to form your command in such a way as to read those arguments. The below is an example that violates my above rule -- a command you couldn't implement[*] without shell=True (and /bin/sh provided by bash), because it relies on bash's built-in version of printf (which supports %q as an extension):

subprocess.call([
    "printf '%q\\n' \"$0\" \"$@\"",
    'these strings are\r\n',
    '"shell escaped" in the output from this command',
    "so that they can *safely* be run through eval"
], shell=True)

[*] - Not really true; one could use /bin/bash as argv[0] with shell=False to implement this more reliably, as it would then no longer depend on your /bin/sh being bash.