How to get a progress bar for a file upload with Apache HttpClient 4?

I've got the following code for a file upload with Apache's HTTP-Client (org.apache.http.client):

  public static void main(String[] args) throws Exception
  {
    String fileName = "test.avi";
    File file = new File(fileName);

    String serverResponse = null;
    HttpParams params = new BasicHttpParams();
    params.setParameter(HttpProtocolParams.USE_EXPECT_CONTINUE, true);
    HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
    HttpClient client = new DefaultHttpClient(params);
    HttpPut put = new HttpPut("http://localhost:8080/" + fileName);

    FileEntity fileEntity = new FileEntity(file, "binary/octet-stream");
    put.setEntity(fileEntity);   

    HttpResponse response = client.execute(put);
    HttpEntity entity = response.getEntity();
    if (entity != null)
    {
      serverResponse = EntityUtils.toString(entity);
      System.out.println(serverResponse);
    }
  }

It work's quite well but now I want to have a progress bar which shows the progress of the file upload. How can this be made? I found a code snippet at File Upload with Java (with progress bar) but it is designed for Apache HTTP Client 3 (org.apache.commons.httpclient) and the RequestEntity class does not exist in Apache HTTP Client 4. ;(

Maybe someone of you has an approach?

Many greetings

Benny

Solution 1:

I introduced a derived FileEntity that just counts the written bytes. It uses OutputStreamProgress that does the actual counting (kind of a decorator to the actual OutputStream).

The advantage of this (and decoration in general) is that I do not need to copy the actual implementation, like the the actual copying from the file stream to the output stream. I can also change to use a different (newer) implementation, like the NFileEntity.

Enjoy...

FileEntity.java

public class FileEntity extends org.apache.http.entity.FileEntity {

    private OutputStreamProgress outstream;

    public FileEntity(File file, String contentType) {
        super(file, contentType);
    }

    @Override
    public void writeTo(OutputStream outstream) throws IOException {
        this.outstream = new OutputStreamProgress(outstream);
        super.writeTo(this.outstream);
    }

    /**
     * Progress: 0-100
     */
    public int getProgress() {
        if (outstream == null) {
            return 0;
        }
        long contentLength = getContentLength();
        if (contentLength <= 0) { // Prevent division by zero and negative values
            return 0;
        }
        long writtenLength = outstream.getWrittenLength();
        return (int) (100*writtenLength/contentLength);
    }
}

OutputStreamProgress.java

public class OutputStreamProgress extends OutputStream {

    private final OutputStream outstream;
    private volatile long bytesWritten=0;

    public OutputStreamProgress(OutputStream outstream) {
        this.outstream = outstream;
    }

    @Override
    public void write(int b) throws IOException {
        outstream.write(b);
        bytesWritten++;
    }

    @Override
    public void write(byte[] b) throws IOException {
        outstream.write(b);
        bytesWritten += b.length;
    }

    @Override
    public void write(byte[] b, int off, int len) throws IOException {
        outstream.write(b, off, len);
        bytesWritten += len;
    }

    @Override
    public void flush() throws IOException {
        outstream.flush();
    }

    @Override
    public void close() throws IOException {
        outstream.close();
    }

    public long getWrittenLength() {
        return bytesWritten;
    }
}

Solution 2:

A new version using the package org.apache.commons.io.output from commons-io (2.4) and its class CountingOutputStream.

I changed the initial code to reflect my project needs to use a multipart form as input and the post method (this dues to the requirements imposed by the server side).

Consider that the delta of large file correspond in my tests to 4096 bytes. This means that the listener method counterChanged() is called every 4096 bytes of transfered data, what is acceptable for my use case.

The method looks like:

public void post(String url, File sendFile) {
    HttpParams params = new BasicHttpParams();
    params.setParameter(HttpProtocolParams.USE_EXPECT_CONTINUE, true);
    HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
    HttpClient client = new DefaultHttpClient(params);
    HttpPost post = new HttpPost(url + "/" + sendFile.getName());
    MultipartEntity multiEntity = new MultipartEntity(); 
    MyFileBody fileBody = new MyFileBody(sendFile);

    fileBody.setListener(new IStreamListener(){

        @Override
        public void counterChanged(int delta) {
            // do something
            System.out.println(delta);
        }});

    multiEntity.addPart("file", fileBody);
    StringBody stringBody = new StringBody(sendFile.getName());
    multiEntity.addPart("fileName", stringBody);
    post.setEntity(multiEntity);   
    HttpResponse response = client.execute(post);
}

The class MyFileBody becomes:

public class MyFileBody extends FileBody {

    private IStreamListener listener;

    public MyFileBody(File file) {
        super(file);
    }

    @Override
    public void writeTo(OutputStream out) throws IOException {
        CountingOutputStream output = new CountingOutputStream(out) {
            @Override
            protected void beforeWrite(int n) {
                if (listener != null && n != 0)
                    listener.counterChanged(n);
                super.beforeWrite(n);
            }
        };
        super.writeTo(output);

    }

    public void setListener(IStreamListener listener) {
        this.listener = listener;
    }

    public IStreamListener getListener() {
        return listener;
    }

}

Finally, the listener interface looks like:

public interface IStreamListener {

    void counterChanged(int delta);

}

Solution 3:

This answer extends kilaka's answer by adding a simple listener to the OutputStreamProgress.java class instead of having the public getProgress() method (I'm honestly not sure how you are suppose to call the getProgress() method since the thread will be executing inside of httpclient's code the entire time you might want to call getProgress()!).

Please note you'll need to extend the entity class for each entity type you want to use, and when you write your HttpClient code, you'll need to create the entity of that new type.

I wrote a very basic write listener that implements the WriteListener interface. This is where you'll add your logic to do something with the write reports from the OutputStreamProgress, something like updating a progress bar :)

Big thanks to kilaka for using the decorator idea to sneak in a counting outstream.

WriteLisener.java

public interface WriteListener {
    void registerWrite(long amountOfBytesWritten);
}

OutputStreamProgress.java

import java.io.IOException;
import java.io.OutputStream;

public class OutputStreamProgress extends OutputStream {

    private final OutputStream outstream;
    private long bytesWritten=0;
    private final WriteListener writeListener;
    public OutputStreamProgress(OutputStream outstream, WriteListener writeListener) {
        this.outstream = outstream;
        this.writeListener = writeListener;
    }

    @Override
    public void write(int b) throws IOException {
        outstream.write(b);
        bytesWritten++;
        writeListener.registerWrite(bytesWritten);
    }

    @Override
    public void write(byte[] b) throws IOException {
        outstream.write(b);
        bytesWritten += b.length;
        writeListener.registerWrite(bytesWritten);
    }

    @Override
    public void write(byte[] b, int off, int len) throws IOException {
        outstream.write(b, off, len);
        bytesWritten += len;
        writeListener.registerWrite(bytesWritten);
    }

    @Override
    public void flush() throws IOException {
        outstream.flush();
    }

    @Override
    public void close() throws IOException {
        outstream.close();
    }
}

BasicWriteListener

public class BasicWriteListener implements WriteListener {

public BasicWriteListener() {
    // TODO Auto-generated constructor stub
}

public void registerWrite(long amountOfBytesWritten) {
    System.out.println(amountOfBytesWritten);
}

}

MultipartEntityWithProgressBar

import java.io.IOException;
import java.io.OutputStream;
import java.nio.charset.Charset;

import org.apache.http.entity.mime.HttpMultipartMode;
import org.apache.http.entity.mime.MultipartEntity;

public class MultipartEntityWithProgressBar extends MultipartEntity {
    private OutputStreamProgress outstream;
    private WriteListener writeListener;

    @Override
    public void writeTo(OutputStream outstream) throws IOException {
        this.outstream = new OutputStreamProgress(outstream, writeListener);
        super.writeTo(this.outstream);
    }

    public MultipartEntityWithProgressBar(WriteListener writeListener)
    {
        super();
        this.writeListener = writeListener;
    }
    public MultipartEntityWithProgressBar(HttpMultipartMode mode, WriteListener writeListener)
    {
        super(mode);
        this.writeListener = writeListener;
    }
    public MultipartEntityWithProgressBar(HttpMultipartMode mode, String boundary, Charset charset, WriteListener writeListener)
    {
        super(mode, boundary, charset);
        this.writeListener = writeListener;
    }

    // Left in for clarity to show where I took from kilaka's answer
//  /**
//   * Progress: 0-100
//   */
//  public int getProgress() {
//      if (outstream == null) {
//          return 0;
//      }
//      long contentLength = getContentLength();
//      if (contentLength <= 0) { // Prevent division by zero and negative values
//          return 0;
//      }
//      long writtenLength = outstream.getWrittenLength();
//      return (int) (100*writtenLength/contentLength);
//  }

}