Why does Popen.communicate() return b'hi\n' instead of 'hi'?

Can someone explain why the result I want, "hi", is preceded with a letter 'b' and followed with a newline?

I am using Python 3.3

>>> import subprocess
>>> print(subprocess.Popen("echo hi", shell=True,
                           stdout=subprocess.PIPE).communicate()[0])
b'hi\n'

This extra 'b' does not appear if I run it with python 2.7

Solution 1:

The b indicates that what you have is bytes, which is a binary sequence of bytes rather than a string of Unicode characters. Subprocesses output bytes, not characters, so that's what communicate() is returning.

The bytes type is not directly print()able, so you're being shown the repr of the bytes you have. If you know the encoding of the bytes you received from the subprocess, you can use decode() to convert them into a printable str:

>>> print(b'hi\n'.decode('ascii'))
hi

Of course, this specific example only works if you actually are receiving ASCII from the subprocess. If it's not ASCII, you'll get an exception:

>>> print(b'\xff'.decode('ascii'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0…

The newline is part of what echo hi has output. echo's job is to output the parameters you pass it, followed by a newline. If you're not interested in whitespace surrounding the process output, you can use strip() like so:

>>> b'hi\n'.strip()
b'hi'

Solution 2:

As mentioned before, echo hi actually does return hi\n, which it is an expected behavior.

But you probably want to just get the data in a "right" format and not deal with encoding. All you need to do is pass universal_newlines=True option to subprocess.Popen() like so:

>>> import subprocess
>>> print(subprocess.Popen("echo hi",
                           shell=True,
                           stdout=subprocess.PIPE,
                           universal_newlines=True).communicate()[0])
hi

This way Popen() will replace these unwanted symbols by itself.