How to transfer data that does not fit in ram from a microcontroller to AWS using direct s3 upload?

I need to transfer some data files from an stm32 microcontroller to an AWS S3 bucket. My microcontroller is connected via UART to an esp32-s2. I was planning to use s3 direct upload to do this that used HTTP to upload a file. However, I only have limited RAM (some kbs) and I need to transfer files around 100MM. I have looked into HTTP stream, that would allow me to send my files in small chunks but I do not know how to create a request that would work as I need to transfer the file content but also a set a form fields (e.g. 'x-amz-security-token', 'signature', ...) for my request to be accepted by AWS. Is it possible to achieve this using HTTP so that I can use the direct upload mechanism from S3? or should I look into other ways, for example sending pieces of my files through a websocket and reconstructing them on the AWS side in a lambda to then store them in S3.

Thank you in advance for your help.

Solution 1:

You need to send in chunks. Espressif has a rather useful HTTPS client library for the ESP32 which permits data transfers in chunks of arbitrary size and also adding any custom headers. You create the client, set up any headers, connect and send the data in pieces. You also need to figure out a protocol for requesting the transmitted data from the STM32.