s3 per object expiry

Solution 1:

It doesn't look like per-object expiration is supported, but rather a per-bucket lifecycle configuration with up to 100 rules per configuration, as you have found.

A bucket has one lifecycle configuration. A lifecycle configuration can have up to 100 rules.

The lifetime value must be a nonzero positive integer. Amazon S3 calculates expiration time by adding the expiration period specified in the rule to the object creation time and rounding the resulting time to the next day midnight UTC.

If per file expiration is not supported, is it possible to exclude a file that matches an expiration prefix from being expired ?

It doesn't look like you can overlap rules, either.

Take care to ensure the rules don't overlap. For example, the following lifecycle configuration has a rule that sets objects with the prefix "documents" to expire after 30 days. The configuration also has another rule that sets objects with the prefix "documents/2011" to expire after 365 days. In this case, Amazon S3 returns an error message.

Solution 2:

An alternative way to accomplish this task is to make use of object tags.

Set a unique tag for each object of the objects you want to set lifetimes for, Then create a life cycle configuration rule for each object and in the filter element you can use the object specific tag.

This tag can be the object name, Also you have the ability to make a rule for a subset of objects that don't have common prefix.

You can find more info about life cycle configuration here http://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html

Solution 3:

Here's an alternative approach that might work for some situations:

You could set the bucket lifecycle to expire objects after a short number of days (1, 7, whatever). Then, before each object expires, programmatically extend its expiration if you don't want that object to expire yet.

You can reset the timestamp on an S3 object by "copying" the object to itself with an instruction to change the storage class to the same class it currently is. It's basically a "noop" that acts like a Linux/Unix "touch" command.

I wrote an article that walks through example aws-cli commands demonstrating how to do this: http://alestic.com/2013/09/s3-lifecycle-extend

Solution 4:

Similarly to Eric's answer you could set a short bucket lifecycle policy and then use retention period locks (see the docs) to set expiration dates on a per-object basis.

Solution 5:

I extended the expiration time by copying the object into itself using s3cmd command. It basically worked like 'touch'.

s3cmd cp s3://bucket-name/file.txt s3://bucket-name/file.txt