Last unbuffered line can't be read

I'm trying to read the last line from a command like 'apt-get download firefox'. Normally the output will be like

Get:1 http://archive.ubuntu.com/ubuntu/ utopic/main firefox amd64 32.0+build1-0ubuntu2 [34.9 MB]
2% [1 firefox 646 kB/34.9 MB 2%]

with the last line continuously updating (it is not writing a newline until it reaches 100%). My goal is now to read the progress in realtime. Here is my current example code:

#!/usr/bin/python3 -u
# coding=utf-8

import subprocess, sys

pipe = subprocess.Popen(['apt-get', 'download', 'firefox'], 0, stderr = subprocess.PIPE, stdout = subprocess.PIPE)
while True:
    content = pipe.stdout.read(1).decode()
    if content == '':
        break
    sys.stdout.write(content)
    sys.stdout.flush()
pipe.wait()

I have disabled the output buffering for the subprocess call and also for binary output for the Python process (with the -u argument). But I'm only getting the first line but not the progress of the second line. Does somebody know how I can achieve this?

Solution 1:

If stdout of apt-get is redirected to a pipe e.g.,

$ apt-get download firefox | cat

then it doesn't report progress (the last line e.g., 2% [1 firefox 646 kB/34.9 MB 2%] will not be in the output). stdout=subprocess.PIPE naturally creates a pipe; therefore apt-get doesn't print the download progress in your case.

If you want both to capture apt-get output and to see it on the screen in real-time with the last line (progress report) present then you could use pexpect module to trick the child process into thinking that it runs in a terminal:

import sys
import pexpect # $ pip install pexpect

output, exitstatus = pexpect.runu('apt-get download firefox',
                                  logfile=sys.stdout,
                                  withexitstatus=1)

You could do the same using only stdlib pty module:

#!/usr/bin/env python3
import os
import pty

output = []
def read(fd):
    data = os.read(fd, 1024)
    output.append(data)
    return data
status = pty.spawn(['apt-get', 'download', 'firefox'], read)

@eryksun on Python Issue tracker suggested apt-get's --quiet option:

#!/usr/bin/env python3
import shlex
from io import TextIOWrapper
from subprocess import Popen, PIPE

output = []
with Popen(shlex.split("apt-get --quiet=0 download firefox"),
           stdout=PIPE, bufsize=1) as p:
    # recognize '\r' as newline but don't convert it to '\n'
    for line in TextIOWrapper(p.stdout, newline=''):
        print(line, end='', flush=True) # print to terminal
        output.append(line) # save for later
print(p.returncode)

Last unbuffered line can't be read

Solution 1:

Related

Recent Posts