Architectural and design question about uploading photos from iPhone app and S3

I want to allow users of an iPhone app to upload photos and use Amazon S3. There are 2 ways I see going about this:

1. Upload from iPhone to my server, which proxies it then to Amazon S3.
2. Upload from iPhone direct to S3

For option 1, the security is straightforward. I don't ever have to tell the iPhone my S3 secret. Downside is that everything is proxied through our server for uploads which sort of defeats the purpose of going to S3.

For option 2, in theory it's better but the issue is how do you enable the iPhone (or any mobile app on a different platform) to write into my S3 bucket without giving it my secret? Additionally, I'm not sure if this is a good design or not because the flow would be: iphone uploads to S3, gets the URL, then tells the server what the URL is so it can add it to our database to reference in the future. However, since the communication is separated into 2 legs (iphone->S3 vs iPhone->My-Server) it leaves it fragile as a non-atomic operation.

I've found some older info that references using Browser-based Uploads using POST but unsure if that is still the recommended approach. I'm hoping for a better solution where we can just use the REST APIs rather than relying on POST. I've also see the AWS iOS Beta SDK, but their docs didn't help much and I found an Amazon article that was equally unhelpful as it cautions you on what not to do, but doesn't tell you an alternative approach:

The mobile AWS SDKs sign the API requests sent to Amazon Web Services (AWS) in order to validate the identity of the AWS account making the request. Otherwise, a malicious developer could easily make requests to another developer's infrastructure. The requests are signed using an AWS Access Key ID and a Secret Access Key that AWS provides. The Secret Access Key is similar to a password, and it is extremely important to keep secret.

Tip: You can view all your AWS security credentials, including Access Key ID and Secret Access Key, on the AWS web site at http://aws.amazon.com/security-credentials.

Embedding credentials in source code is problematic for software, including mobile applications, because malicious users can de-compile the software or view the source code to retrieve the Secret Access Key.

Does anyone have any advice on the best architectural design and flow for this?

Update: The more I dig into this, it seems that a bunch of pople recommend using the HTTP POST method with the json policy file as described here: http://docs.amazonwebservices.com/AmazonS3/2006-03-01/dev/index.html?UsingHTTPPOST.html.

With this, the flow would be something like (1) iPhone makes request to my server, asking for policy file (2) server generates json policy file and gives back to client (3) iPhone does HTTP POST of photo + json policy to S3. I hate that I'm using HTTP POST in an apparently kludgy way but it appears to be better because it removes the need for my server to store the photo at all.

Solution 1:

I've discussed this issue on the AWS forums before. As I say there, the proper solution for accessing AWS from a mobile device is to use the AWS Identity and Access Management service to generate temporary, limited-privilege access keys for each user. The service is great, but it's still in beta for now and it's not part of the mobile SDK yet. I have a feeling once this thing is released for good, you'll see it out on the mobile SDK immediately afterwards.

Until then, generate presigned URLs for your users, or proxy through your own server like some others have suggested. The presigned URL will allow your users to temporarily GET or PUT to an S3 object in one of your buckets without actually having your credentials (they are hashed into the signature). You can read about the details here.

EDIT: I've implemented a proper solution for this problem, using the preview beta of IAM. It's available on GitHub, and you can read about it here.

Solution 2:

Upload to your server and then post to S3. From an architecture standpoint you will want to do this from your server. There are many things that could go wrong during the data transfer you can handle better on the server and if you want to store any data about the image you are sending to S3 you are probably going to have a server side call anyway.

Plus, your Secret Access Key is stored in a more secure environment. Your own.

If you are worried about scalability and you are going to be doing a considerable number of S3 transfers I would consider hosting your server on and EC2 instance. Transferring data between the two is free (given you store you data in following data centers).

There is no Data Transfer charge for data transferred between Amazon EC2 and Amazon S3 within the same Region or for data transferred between the Amazon EC2 Northern Virginia Region and the Amazon S3 US Standard Region." Amazon Simple Storage Service (Amazon S3)

There are many post here on SO Amazon - EC2 cost? (example) about the pros and cons of using EC2.

Solution 3:

You can upload directly from your iPhone to S3 using the REST API, and having the server be responsible for generating the part of the Authorization header value that requires the secret key. This way, you don't risk exposing the access key to anyone with a jailbroken iPhone, while you don't put the burden of uploading the file on the server. The exact details of the request to make can be found under "Example Object PUT" of "Signing and Authenticating REST Requests". I strongly recommend reading that document before proceeding any further.

The following code, written in Python, generates the part of the Authorization header value that is derived from your S3 secret access key. You should substitute your own secret access key and bucket name in virtual host form for _S3_SECRET and _S3_BUCKET_NAME below, respectively:

import base64
from datetime import datetime
import hmac
import sha

# Replace these values.
_S3_SECRET = "my-s3-secret"
_S3_BUCKET_NAME = "my-bucket-name"

def get_upload_header_values(content_type, filename): 
  now = datetime.utcnow()
  date_string = now.strftime("%a, %d %b %Y %H:%M:%S +0000")
  full_pathname = '/%s/%s' % (_S3_BUCKET_NAME, filename)
  string_to_sign = "PUT\n\n%s\n%s\n%s" % (
      content_type, date_string, full_pathname)
  h = hmac.new(_S3_SECRET, string_to_sign, sha)
  auth_string = base64.encodestring(h.digest()).strip()
  return (date_string, auth_string)

Calling this with the filename foo.txt and content-type text/plain yields:

>>> get_upload_header_values('text/plain', 'foo.txt')
('Wed, 06 Feb 2013 00:57:45 +0000', 'EUSj3g70aEsEqSyPT/GojZmY8eI=')

Note that if you run this code, the time returned will be different, and so the encoded HMAC digest will be different.

Now the iPhone client just has to issue a PUT request to S3 using the returned date and HMAC digest. Assuming that

• the server returns date_string and auth_string above in some JSON object named serverJson
• your S3 Access Key (not your secret, which is only on the server) is named kS3AccessKey
• your S3 bucket name (set to my-bucket-name above) is named kS3BucketName
• the file contents are marshalled in an NSData object named data
• the filename that was sent to the server is a string named filename
• the content-type that was sent to the server is a string named contentType

Then you can do the following to create the NSURLRequest:

NSString *serverDate = [serverJson objectForKey:@"date"]
NSString *serverHmacDigest = [serverJson objectForKey:@"hmacDigest"]

// Create the headers.
NSMutableDictionary *headers = [[NSMutableDictionary alloc] init];
[headers setObject:contentType forKey:@"Content-Type"];
NSString *host = [NSString stringWithFormat:@"%@.s3.amazonaws.com", kS3BucketName]
[headers setObject:host forKey:@"Host"];
[headers setObject:serverDate forKey:@"Date"];
NSString *authorization = [NSString stringWithFormat:@"AWS %@:%@", kS3AccessKey, serverHmacDigest];
[headers setObject:authorization forKey:@"Authorization"];

// Create the request.
NSMutableURLRequest *request = [[NSMutableURLRequest alloc] init];
[request setAllHTTPHeaderFields:headers];
[request setHTTPBody:data];
[request setHTTPMethod:@"PUT"];
NSString *postUrl = [NSString stringWithFormat:@"http://%@.s3.amazonaws.com/%@", kS3BucketName, filename];
[request setURL:[NSURL URLWithString:postUrl]];

Next you can issue the request. If you're using the excellent AFNetworking library, then you can wrap request in an AFXMLRequestOperation object using XMLDocumentRequestOperationWithRequest:success:failure:, and then invoking its start method. Don't forget to release headers and request when done.

Note that the client got the value of the Date header from the server. This is because, as Amazon describes under "Time Stamp Requirement":

"A valid time stamp (using either the HTTP Date header or an x-amz-date alternative) is mandatory for authenticated requests. Furthermore, the client time-stamp included with an authenticated request must be within 15 minutes of the Amazon S3 system time when the request is received. If not, the request will fail with the RequestTimeTooSkewed error status code. "

So instead of relying on the client having the correct time in order for the request to succeed, rely on the server, which should be using NTP (and a daemon like ntpd).