Skip to main content

Archiving Your Logs - Cold Storage

Stephanie R.
Updated

Learn about the long-term storage feature for Logs Data Platform that allows you to generate a daily archive of any stream with some simple configuration steps.

The Logs Data Platform gives you a custom log retention system, you can adjust it when you create your stream. But, in some cases, you may want to keep your logs beyond the provided duration. It can be for legal reasons, for analytic purposes, or maybe only for historical ones.

 

Requirements

As implied in the title, you will need a stream. If you don't know what a stream is or if you don't have any, you can follow this quick start tutorial. You can edit the stream configuration from the OVHcloud Control Panel:

  1. Choose Bare Metal Cloud from the top navigation bar.
  2. In the left-hand menu, click Logs Data Platform and select your LDP account.
  3. Select the Data streams tab.
  4. Click more options ... to the right of your stream and choose Edit.

operate_LDP_security_coldStorage01.png

 

Instructions

Activating cold storage on a stream

On the stream configuration page, you will find the Enable long-term storage checkbox ☑. Once enabled, you will be able to choose different options:

Note that OVHcloud Object Storage is more expensive than OVHcloud Public Archive but allows you to immediately download your archive whereas there is a delay (from 10 minutes to 4 hours) before being able to download your files on Public Archive. Depending on the urgency of your future logs retrieval, you must choose your backend accordingly.

The content of your archive is flexible. By default, you get the full log content in GELF format. But you can choose to have an archive containing only the value of the custom LDP field X-OVH-TO-FREEZE. This field can, for example, be used to keep your logs in a human-readable or original format. You can also choose to have two archives simultaneously: the original GELF and the X-OVH-TO-FREEZE archives.

operate_LDP_security_coldStorage02**.png

As soon as you click on Save, the cold storage is activated. Below are some more things you need to know about this feature

NOTE: As soon as the feature is activated, your logs will be stored for the specified duration. The effect is immediate so the billing of this feature will be also immediate.

  • Deactivating the cold storage on a stream will prevent the production of new archives but it won't delete the already produced archives. These archives will be kept for the duration configured.
  • Changing the retention duration will delete any archive exceeding the new retention (Ex: choosing a one-year retention will implicitly delete all archives older than one year).
  • We push a daily archive of the 2-day-old data you pushed. So every day you will get the archive of the day before yesterday.
  • When you activate the feature for the first time we can't create an archive for data older than two days before the activation.
  • Deleting the stream will delete any archive associated. The stream must be alive to be able to keep its archive.

 

Retrieving the archives

Using the OVHcloud Manager

On a cold storage-enabled stream (you can quickly see if they are with the archive checkbox), you have a new Manage archives option in the more options ... menu of your stream. Click on it to navigate to the archives pages.

operate_LDP_security_coldStorage03*.png

On this page, you have a list of the archives produced. Each archive is named after its date, so you can quickly retrieve an archive of a particular day.

From this page, you can launch the "unfreezing" process of your archive and make it available for download. This delay varies between 10 minutes to 4 hours depending on multiple factors like the size of the archive. There is no delay in the case you choose the OVHcloud Object Storage backend for your archives. Once available, its status changes and a new Download action appears in the more options ... menu.

operate_LDP_security_coldStorage04**.png

 

Using the API

If you want to download your logs using the API (to use them in a Big Data analysis platform for example), you can do all these steps by using the OVHcloud API.

You will need your OVHcloud service name associated with your account. Your service name (e.g. ldp-ab-12345) is shown in the left-hand menu of the OVHcloud Control Panel.

 

Retrieve your streams

  • Return the list of Graylog streams:

GET /dbaas/logs/{serviceName}/output/graylog/stream

Parameters:

  • serviceName: The internal ID of your Logs Data Platform service (string)

 

Retrieve the list of your archives

  • Return details of specified archive:

GET /dbaas/logs/{serviceName}/output/graylog/stream/{streamId}/archive

Parameters:

  • serviceName: The internal ID of your Logs Data Platform service (string)
  • streamId: The stream you want archives from

 

Retrieve the list of your archives and their details with the corresponding endpoints

  • Return details of specified archive:

GET /dbaas/logs/{serviceName}/output/graylog/stream/{streamId}/archive/{archiveId}

Parameters:

  • serviceName: The internal ID of your Logs Data Platform service (string)
  • streamId: The stream you want archives from
  • archiveId: The archive you want details from

 

You can generate a temporary URL download by using the following endpoint

  • Get a public temporary URL to access the archive:

POST /dbaas/logs/{serviceName}/output/graylog/stream/{streamId}/archive/{archiveId}/url

Parameters:

  • serviceName: The internal ID of your Logs Data Platform service (string)
  • streamId: The stream you want archives from.
  • archiveId: The archive you want details from.

Example result:

{
  "expirationDate": "2024-07-31T12:27:22.797Z",
  "url": "string"
}

It will take some time (depending on the size of your archive file) for your archive to unfreeze. Once it has, you will need to use the API call again. If your archive is available, you will see a result like this:

{
  "expirationDate": "2024-08-01T15:16:21.914370+00:00",
  "url": "https://archive-<region>.logs.ovh.com/eyJub25jZSI6ICJQbFFmRVpELzJ3TVRhVDM2RHZUSUNsYkZMR0Q0SGlaMiIsICJkYXRhIjogIkxVRmd6czQzeUhiaTZWSnp6d1JuQ1M5UzJrbllYWUp0U1FBOHY0cS9BUlo4cVpIeldZTjJDakN4N1I3Y0FsVi94SUhqTUx4dUNBZmhwQUdpYnpoQ1pWUlhMc2dQTkVHWm9ZZTNLOFlueUFCMFMwNWdmaXJ5YkZyZmpuekZrYkZ4am5EYjE5YlR2ejg0NUlCQW9Ralg2T3JPUFMveXg2UjRiY1lLM0tETlVMZHRtcXRMaG53d2VoZGg3bkZZQzRXbVdmL0lUbjE4czYzQm9Zc1oxNW1VSkNFdVd1MmhlTFM1WWF0WnJzWWVrU0pRVmkwZVFYbi9zd3AwSG1yVzRTTFY5Z21oaG50NEI5akM5eXBzVWFUTlRRSFFkOGpVZk9rWFd2S1pKTDlQQTBkWHJ1OFlFNmNJaCtTSUVONWhybE5adUE2QmxYOUt6NmQrVnBTZGdlS3ByT0hRVFdJalBFZmpQMjhoUkVXKzF5eDY4cjJURXNZeUhMR3FwR3lUa0ZqYkd3PT0ifQ==/<archive-name>.gz"
}

 

Using ldp-archive-mirror

To allow you to get a local copy of all your cold stored archives on Logs Data Platform, we have developed an open-source tool that will do this passively: ldp-archive-mirror. The installation and configuration procedure is described on this GitHub page.

 

Content of the archive

The data you retrieve in the archive is by default in GELF format. It is ordered by the field timestamp and retains all additional fields that you would have added (with your Logstash collector, for example). Since this format is fully compatible with JSON, you can use it right away in any other system.

This is an example of what your logs may look like. It varies greatly depending on your setup and how you choose to view it.

 {"_facility":"gelf-rb","_id":11,"_monitoring":"cb1068c485e738655cfe10df5df3a9a185aa8e301b5c8d0747b3502e8fdcc157","_type":"direct","full_message":"monitoring message (11) at 2017-05-17 09:58:08 +0000","host":"shinken","level":1,"short_message":"monitoring msg (11)","timestamp":1.4950150886486998E9}
 {"_facility":"gelf-rb","_id":23,"_monitoring":"490de2b690eaab7aaf647cfaddb95e0fe51617985efad3afcce45ae931c41c97","_type":"direct","full_message":"monitoring message (23) at 2017-05-17 09:58:08 +0000","host":"shinken","level":1,"short_message":"monitoring msg (23)","timestamp":1.4950150888951592E9}
 {"_facility":"gelf-rb","_id":10,"_monitoring":"b6e4749eb577d8534e237f930b4fc9612a74012aab50fc01b2ef317db1125df1","_type":"direct","full_message":"monitoring message (10) at 2017-05-17 10:03:09 +0000","host":"shinken","level":1,"short_message":"monitoring msg (10)","timestamp":1.4950153892895508E9}
 {"_facility":"gelf-rb","_id":22,"_monitoring":"8b08f448859698e35cbcfeaac3a4c2eb21f08230292cbbd7a1dfce625ff1e633","_type":"direct","full_message":"monitoring message (22) at 2017-05-17 10:03:09 +0000","host":"shinken","level":1,"short_message":"monitoring msg (22)","timestamp":1.4950153895357378E9}
 {"_facility":"gelf-rb","_id":9,"_monitoring":"fe7f7c6f36a6a8de0333a8bee7576181448d96a79390bf7018b66dd96582c3ba","_type":"direct","full_message":"monitoring message (9) at 2017-05-17 10:08:09 +0000","host":"shinken","level":1,"short_message":"monitoring msg (9)","timestamp":1.49501568940605E9}
 {"_facility":"gelf-rb","_id":21,"_monitoring":"cc4f1dd33a01800ea95f88807053ad9d0ba6aecfddf5d9a4499230ea598adfb1","_type":"direct","full_message":"monitoring message (21) at 2017-05-17 10:08:09 +0000","host":"shinken","level":1,"short_message":"monitoring msg (21)","timestamp":1.495015689660263E9}
 {"_facility":"gelf-rb","_id":8,"_monitoring":"25c435285c4e5feb23a8997a33b3115e43f3979e17c1d89c5eb7aa209bf6fddb","_type":"direct","full_message":"monitoring message (8) at 2017-05-17 10:13:09 +0000","host":"shinken","level":1,"short_message":"monitoring msg (8)","timestamp":1.4950159899833767E9}
 {"_facility":"gelf-rb","_id":20,"_monitoring":"59d4615fed4a68661a3a154377a938f91cdb621df6b3d700a166f20540ceda31","_type":"direct","full_message":"monitoring message (20) at 2017-05-17 10:13:10 +0000","host":"shinken","level":1,"short_message":"monitoring msg (20)","timestamp":1.495015990229378E9}
 {"_facility":"gelf-rb","_id":7,"_monitoring":"ff558f06ab12e03cd9c5ff23ba0f8bebbdf939d00e5b8c2faaf3f7a03be8a6e0","_type":"direct","full_message":"monitoring message (7) at 2017-05-17 10:18:09 +0000","host":"shinken","level":1,"short_message":"monitoring msg (7)","timestamp":1.495016289332681E9}

Remember, that you can also use a special field X-OVH-TO-FREEZE on your logs to craft an additional archive with only the value of this specific field at each line (along with the usual gelf archive). This file can be used for example to restore a common human readable log file.

 

Go further

For more information and tutorials, please see our other Logs Data Platform support guides or explore the guides for other OVHcloud products and services.

Related articles