Spring - How to stream large multipart file uploads to database without storing on local file system [duplicate]

Spring boot's default MultiPartResolver interface handles the uploading of multipart files by storing them on the local file system. Before the controller method is entered, the entire multipart file must finish uploading to the server.

We are storing all of our uploaded files directly to a database and our servers have a very small disk quota, so if a large file is uploaded, we are seeing an IOExeption - Disk quota exceeded.

Is there a way to get the stream directly from the client's incoming request before Spring's MultiPartResolver stores the file on the local filesystem so the we can stream directly to our db?

Solution 1:

You could use apache directly, as described here https://commons.apache.org/proper/commons-fileupload/streaming.html.

@Controller
public class UploadController {

    @RequestMapping("/upload")
    public String upload(HttpServletRequest request) throws IOException, FileUploadException {

        ServletFileUpload upload = new ServletFileUpload();

        FileItemIterator iterator = upload.getItemIterator(request);
        while (iterator.hasNext()) {
            FileItemStream item = iterator.next();

            if (!item.isFormField()) {
                InputStream inputStream = item.openStream();
                //...
            }
        }
    }
}

Make sure to disable springs multipart resolving mechanism.

application.yml:

spring:
   http:
      multipart:
         enabled: false

Solution 2:

Actually it is not trivial task. If you would like to write stream from client right to the database, you have to process request manually. There are some libraries, that can make this task simpler. One of them is "Apache Commons FileUpload". Below very simple example, how can you process incoming multipart/form-data request by this library.

@Controller
public class Controller{

    @RequestMapping("/upload")
    public String upload(HttpServletRequest request){
        
        String boundary = extractBoundary(request);

        try {
            MultipartStream multipartStream = new MultipartStream(request.getInputStream(), 
                boundary.getBytes(), 1024, null);
            boolean nextPart = multipartStream.skipPreamble();
            while(nextPart) {
                String header = multipartStream.readHeaders();

                if(header.contains("filename")){
                    //if input is file
                    OutputStream output = createDbOutputStream();
                    multipartStream.readBodyData(output);
                    output.flush();
                    output.close();
                } else {
                    //if input is not file (text, checkbox etc)
                    ByteArrayOutputStream output = new ByteArrayOutputStream();
                    multipartStream.readBodyData(output);
                    String value = output.toString("utf-8");
                    //... do something with extracted value
                }
                nextPart = multipartStream.readBoundary();
            }
        } catch (IOException e) {
            throw new RuntimeException(e);
        }    
    }

    private String extractBoundary(HttpServletRequest request) {
        String boundaryHeader = "boundary=";
        int i = request.getContentType().indexOf(boundaryHeader)+
            boundaryHeader.length();
        return request.getContentType().substring(i);
    }    
}

Header for file field will looks like:

Content-Disposition: form-data; name="fieldName"; filename="fileName.jpg"
Content-Type: image/jpeg

Header for simple field will looks like:

Content-Disposition: form-data; name="fieldName";

Note, that this snippet is just simplified example to show you direction. There is no some details like: extract field name from header, create database output stream etc. You can implement all of this stuff by your own. Examples of multipart request's field headers you can find in RFC1867. Information about multipart/form-data RFC2388.