Object Storage - Smart storage management with lifecycle rules

Introduction

What is lifecycle?

OVHcloud Object Storage bucket lifecycle is a feature that allows you to optimize your storage costs by managing your objects throughout their lifecycle. By uploading a lifecycle configuration to a bucket, you define a set of rules that the object storage solution applies to the objects of the said bucket to perform specific actions.

There are two types of actions that OVHcloud Object Storage performs on your objects:

• expiration: These actions determine when your objects expire. Expired objects are then automatically deleted.
• transition: These actions determine when your objects are transitioned to a lower-cost storage tier. For instance, you might want to transition your objects stored in the High Performance tier to the Standard tier after 30 days.

Examples of use cases

By leveraging the lifecycle configuration feature, you can tell OVHcloud Object Storage to:

• clean incomplete multi-part uploads: Suppose you have uploaded a large number of large (>5GB) objects using multi-part uploads, but for some reason, for many objects, the multi-part upload did not complete successfully. In this scenario, even if you haven't fully uploaded all the parts of an object, you still have to pay for the storage cost of the uploaded parts. In that case, you might want to clean the parts of all the incomplete multi-part uploads to save money.
• clean old unused data: Suppose you have an application that stores its logs in a bucket. Your organization might define a log retention policy of 30 days. After that, the logs are no longer needed, and you might want to delete them to save money.
• optimize storage costs by transitioning infrequently accessed data to a less expensive storage tier: Suppose you have certain files that are often used for a brief duration before they are hardly used again. You might not require immediate access, but your organization or laws may require that you keep them for a certain period. After that, you can delete them to save money.

Special considerations

When an object reaches the end of its lifetime based on its lifecycle configuration, the result of transition or expiration actions performed by OVHcloud Object Storage varies according to the versioning state of the bucket:

• non-versioned: There is only one existing version of the object, which is the current version, and it is deleted permanently.
• versioned: A delete marker is created and becomes the current version. You can also choose how many old versions you want to keep. If the current object version is the only object version, and it is also a delete marker, that delete marker will be removed.
• versioning-suspended: We currently do not allow for the suspension of versioning if you have a lifecycle configuration in effect and vice versa. We currently do not allow for the upload of a lifecycle configuration if versioning is suspended on the bucket.

Expiration

Lifecycle rules are processed asynchronously and on a best-effort basis. Most rules are applied within 24 hours, but for very large buckets or when processing many objects, it might take longer. During this delay, you continue to be billed for the object's current storage tier, even if the rule (e.g., expiration or transition) has already been triggered but not yet completed. For example, if an object is set to be deleted on day 30, but is only processed on day 32, you may be billed for an additional two days.

Conflicting expiration dates

Typically, the lifecycle feature is designed to help you optimize your storage costs. For instance, if two expiration rules overlap, i.e, they target the same set of objects but with different expiration dates, the rule with the shorter duration is applied, ensuring that data is not retained beyond the anticipated timeframe: OVHcloud Object Storage always tries to choose the most cost-effective path for you.

Generally speaking, when you have multiple rules in a bucket lifecycle configuration that apply to the same set of objects:

• Permanent deletion takes precedence over transition.
• Transition takes precedence over the creation of delete markers.
• The shorter expiration/transition date takes precedence over the longer one.

Current vs noncurrent versions

In a versioning-enabled bucket, each object has one current version and zero or more noncurrent versions. When you delete an object:

• If you don't specify a version ID in your delete request, a delete marker is created instead of deleting the object. The current object version becomes noncurrent, and the delete marker becomes the current version.
• If you specify a version ID in your delete request, the version of the object that is specified is permanently deleted (a delete marker is not created).
• A delete marker with zero noncurrent versions is called an expired object delete marker.

Configuration

{
  "Rules": [
    {
      "ID": "string",
      "Status": "Enabled"|"Disabled",
      "Filter": {
        "Prefix": "string",
        "Tag": {
          "Key": "string",
          "Value": "string"
        },
        "ObjectSizeGreaterThan": long,
        "ObjectSizeLessThan": long,
        "And": {
          "Prefix": "string",
          "Tags": [
            {
              "Key": "string",
              "Value": "string"
            }
          ],
          "ObjectSizeGreaterThan": long,
          "ObjectSizeLessThan": long
        }
      },
      "Expiration": {
        "Date": timestamp,
        "Days": integer,
        "ExpiredObjectDeleteMarker": "true"|"false"
      },
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": integer,
        "NewerNoncurrentVersions": integer
      },
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": integer
      }
    }
  ]
}

Understanding the NoncurrentDays parameter

The NoncurrentDays parameter defines the minimum number of days since a version is no longer current. This parameter must not be confused with the age of the object, but rather indicates the minimum age of a noncurrent version.

Example 1:

Suppose you have an object A with 10 versions:

• A v10 (current version, creation date: 2024-10-23).
• A v9 (noncurrent version, creation date: 2024-10-22).
• A v8 (noncurrent version, creation date: 2024-10-21).
• A v7 (noncurrent version, creation date: 2024-10-20).
• A v6 (noncurrent version, creation date: 2024-10-19).
• A v5 (noncurrent version, creation date: 2024-10-18).
• A v4 (noncurrent version, creation date: 2024-10-17).
• A v3 (noncurrent version, creation date: 2024-10-16).
• A v2 (noncurrent version, creation date: 2024-10-15).
• A v1 (noncurrent version, creation date: 2024-10-14).

If current date is 2024-10-23 and NoncurrentDays=5, the lifecycle rule will delete the noncurrent versions older than 5 days, i.e, v1, v2, v3, and v4 because:

• A v1 is noncurrent since 2024-10-15 (when A v2 was created), i.e, its age as a noncurrent version is 8 days.
• A v2 is noncurrent since 2024-10-16 (when A v3 was created), i.e, its age as a noncurrent version is 7 days.
• A v3 is noncurrent since 2024-10-17 (when A v4 was created), i.e, its age as a noncurrent version is 6 days.
• A v4 is noncurrent since 2024-10-18 (when A v5 was created), i.e its age as a noncurrent version is 5 days.

Example 2:

Suppose you have an object B with 5 versions:

• B v5 (current version, creation date: 2024-10-28).
• B v4 (noncurrent version, creation date: 2024-10-27).
• B v3 (noncurrent version, creation date: 2024-10-20).
• B v2 (noncurrent version, creation date: 2024-10-15).
• B v1 (noncurrent version, creation date: 2024-10-14).

If current date is 2024-10-29 and NoncurrentDays=5, the lifecycle rule will delete the noncurrent versions older than 5 days, i.e, only v1 and v2 because:

• B v1 is noncurrent since 2024-10-15 (when B v2 was created), i.e, its age as a noncurrent version is 14 days.
• B v2 is noncurrent since 2024-10-20 (when B v3 was created), i.e, its age as a noncurrent version is 9 days.

Get the scheduled expiration date

If an object is scheduled to be deleted, a HEAD-OBJECT call will return a special http response header x-amz-expiration that contains a timestamp indicating its expiry date and an ID of the lifecycle rule that has been applied.

The header format is: x-amz-expiration: expiry-date=<timestamp>, rule-id=<rule_id>

• expire-date: is obtained by adding the creation date and the delay from expiration
• rule-id: the matched rule id triggering the deletion

Example: Get expiration date via direct http API call

HEAD /{key}
Host: {bucket}.s3.{region}.io.cloud.ovh.us
...

HTTP/1.1 200 OK
...
x-amz-expiration: expiry-date="Fri, 21 Dec 2024 00:00:00 GMT", rule-id="123456789"

Example: Get expiration date via the cli

~aws s3api head-object --bucket <bucket_name> --key <object_key>
{
  ...
  "Expiration" : "expiry-date=\"Fri, 21 Dec 2024 00:00:00 GMT\", rule-id=\"123456789\"",
  ...
}

Examples of expiration configurations

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": { },
      "Expiration": {
        "Days": 1
      }
    }
  ]
}

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "/mpus"
      },
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": 7
      }
    }
  ]
}

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "/old"
      },
      "Expiration": {
        "Days": 30
      }
    },
    {
      "ID": "456789",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "/old/logs"
      },
      "Expiration": {
        "Days": 65
      }
    }
  ]
}

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": {
        "Tag": {
           "Key": "age",
           "Value": "old"
        }
      },
      "Expiration": {
        "Days": 30
      }
    },
    {
      "ID": "456789",
      "Status": "Enabled",
      "Filter": {        
        "Tag": {
           "Key": "type",
           "Value": "logs"
        }
      },
      "Expiration": {
        "Days": 65
      }
    }
  ]
}

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "old/"
      },
      "Expiration": {
        "Days": 45
      },
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 15,
        "NewerNoncurrentVersions": 3
      }
    }
  ]
}

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": { },
      "Expiration": {
        "Days": 1
      },
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 1
      }
    },
    {
      "ID": "654789",
      "Status": "Enabled",
      "Filter": { },
      "Expiration": {
        "ExpiredObjectDeleteMarker": true
      }
    },
    {
      "ID": "963852",
      "Status": "Enabled",
      "Filter": { },
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": 1
      }
    }
  ]
}

Transition

Supported transitions

The following are the currently supported transitions:

Minimum object size

OVHcloud Object Storage will prevent any transition to any storage tier for objects smaller than 128KB. However, you can allow transitions and expirations for smaller objects by using ObjectSizeGreaterThan or ObjectSizeLessThan parameters to filter with a minimum size or maximum size. In the following example, objects smaller than 128KB are also allowed to transition to Standard class.

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": {
        "And":{
            "ObjectSizeGreaterThan": 1
        }
      },      
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD"
        }
      ]
     }
  ]
}

Minimum transition delay

The minimum duration for transition rules is 1 day i.e your lifecycle configuration will not be valid and you will get an error if the number of days for your transition rule is less than 1 day. In practice, with multiple transitions, the minimum duration for the nth transition must be at least n days.

For example:

• with a single transition from High Performance to Standard, the minimum duration is 1 day.
• you have multiple transitions: 1) from High Performance to Standard and 2) from Standard to Infrequent Access. The minimum durations are 1 day (1st transition) and 2 days (2nd transition); a third transition would be 3 days, a fourth transition would be 4 days, and so on.

Expiration vs transitions

As already mentioned before, when you have multiple rules in a bucket lifecycle configuration that apply to the same set of objects:

• Permanent deletion takes precedence over transition
• Transition takes precedence over creation of delete markers
• The shorter expiration/transition date takes precedence over the longer one

Configuration

{
  "Rules": [
    {   
       ...

       "Transitions": [
           {
             "Date": timestamp,
             "Days": integer,
             "StorageClass": "STANDARD"
           }
        ],
       "NoncurrentVersionTransitions": [
          {
              "NoncurrentDays": integer,
              "StorageClass": "STANDARD",
              "NewerNoncurrentVersions": integer
          }
       ]
    }
  ]
}

Examples of transition configurations

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "old/"
      },      
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD"
        }
      ]
     }
  ]
}

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": { },      
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD"
        }
      ],
      "NoncurrentVersionTransitions": [
        {
          "NoncurrentDays": 30,
          "StorageClass": "STANDARD"
        }
       ]
     }
  ]
}

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": {
        "And":{
            "Prefix": "old/",
            "Tag": {
              "Key": "type",
              "Value": "logs"
            }
        }
      },
      "Expiration": {
        "Days": 90,
      },      
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD"
        }
      ]
    }
  ]
}

{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": {
        "And":{
            "Prefix": "old/",
            "Tag": {
              "Key": "type",
              "Value": "logs"
            }
        }
      },
      "Expiration": {
        "Days": 90,
      }
    },
    {
      "ID": "456789",
      "Status": "Enabled",
      "Filter": {
        "And":{
            "Prefix": "old/",
            "Tag": {
              "Key": "type",
              "Value": "logs"
            }
        }
      },
      "Transitions": [
        {
          "Days": 90,
          "StorageClass": "STANDARD"
        }
      ]
    }
  ]
}

Instructions

Using the CLI

As a prerequisite, you must have a bucket containing data on which you want to apply the lifecycle configuration and have the necessary permissions (by default the bucket owner or the s3:putLifecycleConfiguration permission given via a user access policy) to do so.

cat lifecycle.json
{
  "Rules": [
    {
      "ID": "123456",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "old/"
      },
      "Expiration": {
        "Days": 45
      },
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 15,
        "NewerNoncurrentVersions": 3
      }
    }
  ]
}

aws s3api put-bucket-lifecycle-configuration --bucket <bucket_name> --lifecycle-configuration file://lifecycle.json

Using the OVHcloud Control Panel (coming soon)

FAQ

Go further

For more information and tutorials, please see our other Public Cloud support guides or explore the guides for other OVHcloud products and services.

If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.

*S3 is a trademark filed by Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.