Deployment to thousands of Amazon S3 accounts?

Solution 1:

That's a really good question.

Last time I checked even COPY between different regions did not work. I know Cloudberry's Explorer app has a feature to copy data between S3 accounts, can you do a test with it? I haven't tried it, it's Windows-only.

I guess if it works, it's a matter of trying the API.

Are all your customers in the same region? Because if COPY between accounts didn't work, I'd boot an instance (or multiple) to speed up the process. If everyone is in the same region, no bandwidth charges should apply.

This is not ideal, but I guess with multiple instances you could get a lot of work done for less than 10 bucks. And it should be possible to automate the setup too.

Update

So to elaborate on EC2. An EC2 instance is just like another server. I suggested it originally so you can download the file once and upload it to other S3 accounts within AWS so you save money on bandwidth (it's free if the bucket and the instance are in the same region).

Anyway, an EC2 instance being like a server, it would require a little bit setup to bootstrap it. E.g. a custom AMI, or any AMI and some user-data (shell script passed to the instance and executed on first boot). You'd probably need to install PHP, the Amazon SDK etc. -- all of which can be automated.

The thing is, I'm not entirely sure if this is necessary in your case.

Check out the following example code: http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?CopyingObjectUsingPHP.html

It shows how to copy data from one bucket to another. Since bucket names are unique across all S3 this shouldn't be a problem. I think all you'd need to do is give read on the files to everyone on your own AWS account (at least temporarily) and then loop through and copy the files to your customers' AWS accounts.

I think you can issue that code anywhere and don't have to worry about bandwidth charges since COPY should be all internal. No download of the actual file required.

Not sure if you reviewed the documentation, but it seems that AWS requires some sort of origin header (x-amz-copy-source) and then it takes care of the rest.

HTH