Why does cURL return error "(23) Failed writing body"?

It works ok as a single tool:

curl "someURL"
curl -o - "someURL"

but it doesn't work in a pipeline:

curl "someURL" | tr -d '\n'
curl -o - "someURL" | tr -d '\n'

it returns:

(23) Failed writing body

What is the problem with piping the cURL output? How to buffer the whole cURL output and then handle it?

Solution 1:

This happens when a piped program (e.g. grep) closes the read pipe before the previous program is finished writing the whole page.

In curl "url" | grep -qs foo, as soon as grep has what it wants it will close the read stream from curl. cURL doesn't expect this and emits the "Failed writing body" error.

A workaround is to pipe the stream through an intermediary program that always reads the whole page before feeding it to the next program.

E.g.

curl "url" | tac | tac | grep -qs foo

tac is a simple Unix program that reads the entire input page and reverses the line order (hence we run it twice). Because it has to read the whole input to find the last line, it will not output anything to grep until cURL is finished. Grep will still close the read stream when it has what it's looking for, but it will only affect tac, which doesn't emit an error.

Solution 2:

For completeness and future searches:

It's a matter of how cURL manages the buffer, the buffer disables the output stream with the -N option.

Example: curl -s -N "URL" | grep -q Welcome

Solution 3:

Another possibility, if using the -o (output file) option - the destination directory does not exist.

eg. if you have -o /tmp/download/abc.txt and /tmp/download does not exist.

Hence, ensure any required directories are created/exist beforehand, use the --create-dirs option as well as -o if necessary