Storing and serving files securely for multiple clients
We are working on a web app, where (among other features) our users can upload their files. However we can't store these files on our VPS because storage space is limited, so we decided to go with S3.
The main problem is that we must make sure users can access only their own data. So we keep the list of files in our database, and the list of users having access to them. Our server can easily decide if a user has, or not, access to a file. But how to actually serve the files to users?
There are some possibilities I've already considered, however none of them actually seems to be the best one.
1. Generating (expiring) signed urls with PHP
This is a really simple approach, it is also fast but results in very very ugly and long urls.
Here's how to do it.
2. Obfuscated URLs
This means, that we keep the files public for read on S3, but all the files are stored in hard to guess named folders like: 24fa0b8ef0ebb6e99c64be8092d3ede20000. However, maybe this is not the safest way to go. Even if you can never guess a folder name, after you know it (because you actually have access to it), you can share that link with anybody (with any not authorized person).
3. Download the files through our server
This means that the files are not served directly by S3, but first our server reads it securely and serves it. We really don't want this :)
4. Checking referrer
The Obfuscated URLs solution can be improved by "making sure" the request comes from our server (you can set up S3 to check the referrer). However this would be a very unreliable solution, because not all browsers send the referrer data, and it can also be faked.
What is a good way to serve files from Amazon S3 securely for different clients?
Solution 1:
This is really bordering on "Do my system architecture" for you, but your four ideas are interesting case-studies in variable security, so let's run your options and see how they fare:
4. Checking referrer
The referrer is provided by the client. Trusting the client-provided authentication/authorization data pretty much voids security (I can just claim to have been sent from where you expect me to come from).
Verdict: TERRIBAD idea - trivial to bypass.
3. Download the files through our server
Not a bad idea, as long as you're willing to spend the bandwidth to make it happen, and your server is reliable.
Going on the assumption that you've already solved the security problem for your normal server/app, this is the most secure of the options you've presented.
Verdict: Good solution. Very secure, but possibly suboptimal if bandwidth is a factor.
2. Obfuscated URLs
Security Through Obscurity? Really? No.
I'm not even going to analyze it. Just no.
Verdict: If #4 was TERRIBAD this is TERRIWORSE because people don't even have to go through the effort of forging a referrer header. Guess the string and win a prizeall the data!
1. Generating (expiring) signed urls with PHP
This option has a pretty low suck quotient.
Anyone can click on the URL and snarf the data, which is a security no-no, but you mitigate this by making the link expire (as long as the link life is short enough the vulnerability window is small).
The URL expiring may inconvenience some users who want to hang on to the download link for a long time, or who don't get the link in a timely manner -- that's a bit of a User Experience suck, but it may be worth it.
Verdict: Not as good as #3, but if bandwidth is a major concern it's certainly better than #4 or #2.
What would I do?
Given these options, I would go with #3 -- Pass the files through your own front-end server, and authenticate the way your app normally does. Assuming your normal security is pretty decent this is the best option from a security standpoint.
Yes, this means more bandwidth use on your server, and more resources playing middleman -- but you can always just charge a tiny bit more for that.
Solution 2:
Use Amazon S3 pre-signed queries to serve the S3 objects directly to users after doing whatever user validation you wish. This method creates a time-limited URL to which you can redirect users.