S3-compatible Object Storage systems [closed]

Solution 1:

Swift is OpenStack's object storage engine, and as of the Bexar release in Feb 2011, it claims an experimental S3-compatible middleware. As OpenStack is getting a lot of attention (Canonical are shifting Ubuntu to it over Eucalyptus this October, for example) it might be worth a look.

Solution 2:

In the end, I had to move forward with Eucalyptus Walrus. Eventhough I ended up finding some fundamental issues with it, they all had possible work-arounds.

Thanks everyone for your valuable input!

Solution 3:

[DISCLAIMER: I work for Scality]

Amazon stays very active on their AWS products and makes additions and changes/improvements to their API very frequently. It is then hard to find a near-100% compatible product (especially in commercial products, probably OSS products would be more reactive to any changes). Many object/cloud storage vendors have been toying around with their own protocols for some time (Atmos, DDN even OpenStack's swift..) but they've mostly all abandoned the idea of their own protocol becoming as widely used as S3. They are all (almost?) announcing an S3-compatible interface now or in the months to come. It is after all the industry de facto standard.The only contender left might be CDMI just because of its Standardized nature and the fact it comes from SNIA, a non-profit, non-vendor organization. But it's clearly not being adopted or talked about as much as S3.

There are a few options as far as S3-compatible interface goes, like Mezeo, the ones you've mentioned (Eucalyptus and Cumulus), OpenStack (although it's not out yet i believe). Scality is another one of those.

Scality's S3 implementation (called RS2 - REST Storage Service) is very close to S3 in that our requests/replies are consistent with what you'd get from S3. Some of the functionalities tied to S3's infrastructure are not present for obvious reasons (Region selection for instance). But the necessary commands are there: GET/PUT/DELETE an object, bucket creation and listing ; and react just like S3 would.

I would just add, to answer your second problem, that Scality's product, RING, is a software object storage platform that solves the data integrity issues you're referring to by checksumming and doing background integrity checks of all objects stored on the RING. Storing an object is basically a transactional query, and once an object has been claimed as "successfully stored", it will forever be, with constant integrity checks (on every read for example).

I won't go into details. you can go to our website to understand how we alleviate storage admins from the common data management nightmares and let them sleep at night knowing their data is available. Always. :)

Best, Marc Villemade @mastachand