Quick Start for Logs Data Platform

Welcome to the quick start tutorial of the Logs Data Platform! Learn about the core concepts behind the Logs Data Platform and how to send your first logs to the engine.

Instructions

Welcome to Logs Data Platform

First, you will have to create a new account on our Logs Data Platform page. Creating an account is free. With the pay-as-you-go pricing model of Logs Data Platform, you only pay for what you use.

From the OVHcloud Control Panel:

1. Select Bare Metal Cloud from the top navigation bar.
2. In the left-hand menu, click Logs Data Platform.
3. Click Order.

   

4. Choose your plan and region, then click Order.

   

5. From the main LDP page, select your LDP account (the Status will say "Configuration required").

   

6. Choose a password and click Save.

   

Once you have created your credentials, the main interface will appear:

Your Logs Data Platform username is located on the bottom left area of the page, in the Configuration panel. In this section, you can also:

1. Identify your assigned access point for this account and your LDP username.
2. Reset the password for your LDP account.
3. Edit your LDP access tokens (learn more about this functionality here).

The main page allows you also to consult the SSL certificates used on your access point in the SSL configuration panel. This panel lets you also review the available ports and formats supported. The Subscription panel allows you to change the displayed name of your account which can be useful if you have multiple accounts tied to the same OVHcloud customer ID.

Across the top of the page, you can see tabs for the following:

1. The Data stream is the recipient of your logs. When you send a log with the right stream token, it arrives automatically to your stream in an awesome software named Graylog. When you access your stream in Graylog you will be able to search your logs and immediately analyze them.
2. The Dashboards is the global view of your logs. It is an efficient way to access your logs and view global information like metrics and trends about your data without being overwhelmed by the logs' details.
3. The Data-gathering tools tab allows requesting OVHcloud to host your own dedicated collector like Logstash or Flowgger.
4. The Index tab allows you to create a dedicated OpenSearch Index or retrieve the ones used to store your OpenSearch Dashboards' settings.
5. The Alias tab provides access to your data directly from your OpenSearch Dashboards or using an OpenSearch query.
6. The OpenSearch Dashboards tab creates your personal OpenSearch Dashboards instance, to exploit the aliases and index from the powerful OpenSearch Dashboards interface.
7. The Roles tab allows access rights modification of your content.

Let's send some logs!

Logs Data Platform comes with a managed Graylog platform that you can access as you wish with the credentials from your Logs Data Platform account. If you are not familiar with it, Graylog is a web-based UI that allows you to query your logs and build dashboards to have a graphical representation of your logs. The Graylog API is also exposed.

To begin, select the Data stream tab and click the + Add a data stream button. You will be redirected to the page where you can add a name and a description to the stream.

On the same page, you can enable the live-tail WebSocket feature. Doing this allows the indexation of your logs to explore them in Graylog, choose the retention of the data in this stream, and limit the amount of logs stored in this stream to control your budget.

Click Save when finished.

You will be redirected back to the Data stream page where you will be able to copy the X-OVH-TOKEN token. This value is the only token you need to route logs to your stream. Under this token, you will have a list of your created streams.

To retrieve the token, click the more options ... button to the right of the data stream you just created and select Copy the write token.

The more options ... button to the right gives you several features:

• Edit allows you to edit the name and the description of your Stream.
• Graylog access gives you direct access to your stream and its logs.
• Copy the write token allows you to retrieve your token and use it in your different logs collector.
• Monitor in real-time allows you to see the logs incoming into your stream in real-time. Note that Graylog also provides this functionality. On this page, you can also Test different log formats from your computer to your stream (more information in this guide).
• Manage alerts allows you to define your alert conditions on the logs routed to the stream (more information in this guide)
• Manage archives allows you to download the cold stored archives (more information in this guide).
• Delete removes your stream from the system and all related content.

Logs Data Platform supports several logs formats, each with its own advantages and disadvantages. Here are the different formats available:

• GELF: This is the native format of logs used by Graylog. This JSON format will allow you to send logs easily. For more information, click here. The GELF input only accepts a null (\0) delimiter.
• LTSV: This simple format is very efficient and is still human-readable. You can learn more about it here. LTSV has two inputs that accept a line delimiter or a null delimiter.
• RFC 5424: This format is commonly used by logs utilities such as syslog. It is extensible enough to allow you to send all your data. More information about it can be found at this link: RFC 5424.
• Cap'n'Proto: The most efficient log format. This is a binary format that allows you to maintain a low footprint and high-speed performance. For more information, check out their official website: Cap'n'Proto.
• Beats: A secure and reliable protocol used by the -beats family in the Elasticsearch ecosystem (Ex: Filebeat, Metricbeat, Winlogbeat).

Here are the ports you can use on your cluster to send your logs. You can either use the secured ones with SSL Enabled (TLS >= 1.2) or use the plain unsecured ones if you can't use an SSL transport.

As said before, you can retrieve the ports and the address of your cluster on the Home tab (in the SSL Configuration panel).

To send your logs to Logs Data Platform you can easily test your stream by doing, for example, a simple echo followed by an openssl command. Here are three examples, choose the format you like the most with your preferred terminal. Note that each format has its own timestamp format: GELF uses seconds from epoch, RFC 5424 and LTSV use RFC 3339. Don't forget to change the timestamp to your current time to see your logs. By default Graylog only displays recent logs; you can change the scope of the search by using the top left time picker in the Graylog web interface. Also please ensure to change the token to your own.

GELF:

This format's time is in seconds.

echo -e '{"version":"1.1",  "_X-OVH-TOKEN":"d93eee2a-697f-4bac-a452-705416b98a04", "host": "example.org", "short_message": "A short message that helps you identify what is going on", "full_message": "Backtrace here\n\nmore stuff", "timestamp": 1385053862, "level": 1, "_user_id": 9001, "_some_info": "foo", "some_metric_num": 42.0 }\0' | \
openssl s_client -quiet -no_ign_eof -connect <your_cluster>.logs.ovh.com:12202

LTSV:

For this format the time is in the RFC 3339 format.

echo -e 'X-OVH-TOKEN:d93eee2a-697f-4bac-a452-705416b98a04\thost:example.org\ttime:2016-03-08T14:44:01+01:00\tmessage:A short message that helps you identify what is going on\tfull_message:Backtrace here\n\nmore stuff\tlevel:1\tuser_id:9001\tsome_info:foo\tsome_metric_num:42.0\0'| \
openssl s_client -quiet -no_ign_eof -connect <your_cluster>.logs.ovh.com:12200

RFC 5424:

For this format the time is in the RFC 3339 format.

echo -e '<6>1 2016-03-08T14:44:01+01:00 149.202.165.20 example.org - - [exampleSDID@8485 X-OVH-TOKEN="d93eee2a-697f-4bac-a452-705416b98a04" user_id="9001"  some_info="foo" some_metric_num="42.0" ] A short message that helps you identify what is going on\n' | \
openssl s_client -quiet -no_ign_eof -connect <your_cluster>.logs.ovh.com:6514

To see your logs in Graylog, click on the more options ... button located at the right of your stream in the stream list. Click on the Graylog access link to jump straight to Graylog. You have to use the Logs Data Platform credentials that were sent by email when you ordered your account logs-XX-XXXXX/your_password.

The Graylog login page looks like this:

Once logged, you will be redirected to this page:

On this page you can already search for the different values present in the logs by using the search bar (at the top of the page). You can also select the time range of your search by playing with time picker at the top left of the page. For example to search all the logs in the last five minutes that contain 42 for the value some_metric_num you can enter in the search bar after you have select Last 5 minutes in the time picker combobox at the top:

some_metric_num:42

Press Enter or click on the green button to launch the search and wait for your results.

It's possible to search some part of your message by entering:

helps going

Giving you all the messages that contains the terms helps& and going.

Graylog allows you to extensively search through your logs without compromising usability. For more information about how to craft relevant searches on Graylog, please see the Graylog documentation.

Send several logs with different values for user_id, for example. On the left side of the page you will see the fields present in your stream, you can click user_id to display all the values for this field along the logs.

Let's create a Dashboard

Back in the OVHcloud Control Panel, we will now create a Dashboard that will allow you to explore your data in a graphical manner. Click the Dashboards tab and then the + Add a dashboard button.

Add a Name and a Description, then click Save.

Once created, you can use the more options ... button to access it immediately.

From here, you can begin to add widgets to your Dashboard. To do that, go back to your stream in Graylog. Let's say you want all the user IDs for which the value some_metric is above 30. The process will go like this:

1. Click the clock icon 🕓 and select Relative.
2. Then, select a time frame for your search, e.g. Search in the last Hour.
3. On the left panel, click on the button that looks like an "X" to display the fields menu.
4. Click user_id and select Show top values. It will then display a widget with the distribution of the most frequent user_ids.

You can edit the widget by using its top right menu arrow. For example you can change the visualization type by choosing Pie Chart in the top left Visualization Type option. Click Save to confirm your changes.

This widget gives you the most frequent user_id in the logs of the last hour that have a some_metric_num value above 30.

To add this really critical information to your dashboard, click on the 💾 Save button and select your newly created Dashboard. After that, you will be redirected to the Dashboard with your newly created widget in it.

Mixing various widgets on the same dashboard gives a better overall view of your logs data. To add a widget in this Dashboard you can also use the + button on the left panel. This button will ask you the type of widget you want to create (Aggregation, Message Count, or Message Table). The Aggregation option allows you to create various visualization types for your data.

In this screenshot, you can see that we added a widget that represents the mean values for some_metric_num (by using generate chart instead of quick values for the field some_metric in the stream tab).

Note that dashboards are interactive and you can use the top search bar and the date picker to display the widgets for one particular type of event.

Go further

We have only scratched the surface of what Logs Data Platform can do for you!

For more information and tutorials, please see our other Logs Data Platform support guides or explore the guides for other OVHcloud products and services.