Add multiple S3 lifecycle rules to S3 bucket with boto3

I want to add multiple lifecycle rules to a S3 bucket using lambda and boto3. However, with boto3, it only seems to allow you to add 1 lifecycle rule, which also overwrites the already existing ones.

For example, doing the following simply overwrites any pre-existing rules that I have as well as the newly written ones, and only leaves the last one in the list:

bucket_name = "test-bucket"
folder_paths = ["test_folder","test_folder1", "test_folder2"]
expiration = 1

for folder_path in folder_paths:
    client = boto3.client('s3')
    response = client.put_bucket_lifecycle_configuration(
        Bucket=bucket_name,
        LifecycleConfiguration={
            'Rules': [
                {
                    'Expiration': {
                        'Days': expiration
                    },
                    'ID': folder_path,
                    'Filter': {
                        'Prefix': folder_path
                    },
                    'Status': 'Enabled'
                }
            ]
        }
    )

Of course using the AWS console it is possible to add multiple separate S3 lifecycle configurations on a bucket.

Similarly to put_bucket_lifecycle_configuration, I have also tried put_bucket_lifecycle, which gives me the same result.

Is there any way to use boto3 to add multiple S3 lifecycle configurations on a bucket? Am I missing something obvious?

Any help is appreciated and thanks in advance!

Solution 1:

Of course using the AWS console it is possible to add multiple separate S3 lifecycle configurations on a bucket.

Every bucket has 1 lifecycle configuration, which can have up to 1000 rules. Your console may show something similar:

These are not different lifecycle configurations, they are different rules part of the same lifecycle configuration.

In the input for the put_bucket_lifecycle_configuration we can see that we can pass a list of rules, which can contain 1 ore many more (up to 1000) rules.

for folder_path in folder_paths:
    client = boto3.client('s3')
    response = client.put_bucket_lifecycle_configuration(
        Bucket=bucket_name,
        LifecycleConfiguration={
            'Rules': [
                {
                    'Expiration': {
                        'Days': expiration
                    },
                    'ID': id_rule_1,
                    'Filter': {
                        'Prefix': folder_path
                    },
                    'Status': 'Enabled'
                },
                {
                    'Expiration': {
                        'Days': expiration2
                    },
                    'ID': id_rule_2,
                    'Filter': {
                        'Prefix': folder_path2
                    },
                    'Status': 'Enabled'
                },
             ...
            ]
        }
    )

As the docs says, put_bucket_lifecycle_configuration "creates a new lifecycle configuration for the bucket or replaces an existing lifecycle configuration." If you want to update the lifecycle configuration, you have to use get_bucket_lifecycle_configuration to retrieve the existing rules, modify them and then use put_bucket_lifecycle_configuration to overwrite the existing configuration.