python subprocess.call() not working as expected

I started down this rabbit hole as a means to familiarize myself with how one would go about creating a setup script in python. The choice of python was simply rooted in my familiarity with it while I am sure there would be better alternatives than python for this task.

The goal of this script was to install ROS onto the machine running the script and also setup the catkin environment. Directions can be found here and here, respectively.

The script as it currently sits is as follows:

subprocess.call(["sudo", "sh", "-c", "'echo \"deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main\" > /etc/apt/sources.list.d/ros-latest.list'"])
subprocess.call(["sudo", "apt-key", "adv", "--keyserver", "hkp://ha.pool.sks-keyserver.net:80", "--recv-key", "0xB01FA116"])
subprocess.call(["sudo", "apt-get", "update"])
subprocess.call(["sudo", "apt-get", "install", "ros-kinetic-desktop-full", "-y"])
subprocess.call(["sudo", "rosdep", "init"])
subprocess.call(["rosdep", "update"])
subprocess.call(["echo", '"source /opt/ros/kinetic/setup.bash"', ">>", "~/.bashrc", "source", "~/.bashrc"])
subprocess.call(["sudo", "apt-get", "install", "python-rosinstall", "-y"])
mkdir_p(os.path.expanduser('~') + "/catkin_ws/src")
subprocess.call(["(cd "+ os.path.expanduser('~') + "/catkin_ws/src)"])
subprocess.call(["(cd "+ os.path.expanduser('~') + "/catkin_ws && catkin_make)"])
subprocess.call(["(cd "+ os.path.expanduser('~') + "/catkin_ws && source devel/setup.bash"])

When the script is currently runs it errors out with the error:

Traceback (most recent call last):
  File "setup.py", line 46, in <module>
    subprocess.call(["(cd "+ os.path.expanduser('~') + "/catkin_ws/src)"])
  File "/usr/lib/python2.7/subprocess.py", line 523, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

I have verified that the command does correctly work when manually executed from a terminal window, and as such I believe this is a fundamental misunderstanding about how this script and its scope are handled within the OS. The part that is causing me a lot of confusion is why it complains that it is unable to locate the provided directory, while I have verified that this directory exists. When the command is rather printed from python and pasted into a terminal window no errors are encountered.

By default subprocess.call doesn't use a shell to run our commands you so can't shell commands like cd.

To use a shell to run your commands use shell=True as parameter. In that case it is recommended to pass your commands as a single string rather than as a list. And as it's run by a shell you can use ~/ in your path, too:

subprocess.call("(cd ~/catkin_ws/src && catkin_make)", shell=True)

subprocess.call() expects a list, with first item obviously being a legitimate shell command. Compare this for instance:

>>> subprocess.call(['echo hello'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/subprocess.py", line 523, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
>>> subprocess.call(['echo', 'hello'])
hello
0

In your case , subprocess.call(["(cd "+ os.path.expanduser('~') + "/catkin_ws/src)"]) will expect to find binary that looks like so(note backslash designating space charater):

 cd\ /home/user/catkin_ws/src

That is treated as one single name that is to be expected to live somewhere on your system. What you really would wanna do is:

 subprocess.call(["cd", os.path.expanduser('~') + "/catkin_ws/src"])

Note that I have removed parenthesis around the comma, as there is no reason to use subshell.

EDIT:

But it has been already mentioned by progo in the comments that using cd in this case is redundant. Florian's answer also properly mentions that subprocess.call() doesn't use shell. You could approach that in two ways. One , you could use subprocess.call("command string",shell=True)

The other way , is to call specific shell explicitly. This is especially useful if you want to run a script that requires specific shell. Thus you could do:

subprocess.call(['bash' , os.path.expanduser('~')  + "/catkin_ws/src"  ) ] )

Use os.chdir() instead.

Apart from the issues, mentioned in the existing answers, I wouldn't prefer using shell=True, nor subprocess.call() here to change directory.

Python has its own way of changing directory in os.chdir() (don't forget to import os). ~ ("home") can be defined in several ways, a.o. os.environ["HOME"].

Reasons to prefer that over shell=True can be read a.o. here