1. Articles
  2. Manage and Operate
  3. Logs Data Platform
  4. Visualizing, Querying, and Exploiting Your Logs

Articles in this section

CLI - ldp-tail, a Tool to Follow Your Logs in Real Time

Avatar
Stephanie R.
  • Updated

Learn how to stream your logs in real time using the Logs Data Platform.

The Logs Data Platform allows you to connect different applications or servers to one unique endpoint and make all of them appear in one stream if needed. Ldp-tail can follow one of your streams in real-time with sub-second latency by using one of the exclusive features of the platform: the WebSocket endpoint.

 

Requirements

  • Ldp-tail, which can be run on Linux, Mac, or Windows, locally or remotely
  • an internet connection
  • an LDP account with an active stream and WebSocket option enabled

 

Instructions

Download and test ldp-tail in two minutes

Ldp-tail is derived from an internal tool used by OVHcloud engineers to follow in real-time hundreds of applications and server logs. It is written in Go and is completely open-source, available here if you are interested. You can also download binary releases from this website. Click here to download the release for your platform. 64-bit versions of Linux, Windows, and Mac OS X are currently supported. Decompress the archive obtained and you will get the ldp-tail binary.

You can test it immediately on our demo stream by using this command in a terminal.

$ ldp@ubuntu:~$ ./ldp-tail --address "wss://gra1.logs.ovh.com/tail/?tk=demo" --pattern "{{ .short_message }}"

There are only two options here: the address and the pattern.

  • The address is the WebSocket address of your stream depending on your cluster and on your stream.
  • The pattern is the display format of your logs. Note that this option is not mandatory. You will find more information about this option later.

To test ldp-tail with one of your stream, you have first to retrieve your WebSocket address.

 

Retrieve your WebSocket address

Let's retrieve the WebSocket address that will allow you to follow your logs. From the OVHcloud Control Panel:

  1. Choose Bare Metal Cloud from the top navigation bar.
  2. In the left-hand menu, select Logs Data Platform and choose your LDP account.
  3. Select the Data streams tab.
  4. To right of your chosen stream, click the more options ... button and select Monitor in real time.

operate_LDP_vqe_ldp-tail01.png

You will land on a new page where you will see all your logs in real-time as soon as they arrive. On this page click on the Actions button and select Copy WebSocket address to copy the WebSocket address in your clipboard. Just use the address as in the example and you will see your logs flowing in your terminal right away.

operate_LDP_vqe_ldp-tail02*.png

You will also find on this page a link to the ldp-tail release page and three ways to test your stream with commands.

 

Formatting and Filtering

Ldp-tail is not just a plain tail (as its name suggest). It comes with advanced formatting and filtering capabilities. The full documentation of these capabilities are all available on GitHub. Here are the two main options that you can use to enhance your output.

 

The pattern option

This option allows you to format the output and to select which fields you want to display. For example, with the demo stream:

$ ldp@ubuntu:~$ ./ldp-tail --address "wss://gra1.logs.ovh.com/tail/?tk=demo" --pattern "My Title: {{ ._title }} , The Joke: {{ .short_message }}"
2017/06/23 17:23:13 Connecting to gra1.logs.ovh.com...
2017/06/23 17:23:14 Connected!
My Title: Why did the Ancient Egyptians build Great Pyramids? , The Joke: Because their Great Igloos melted.
My Title: Success , The Joke: Success is relative. The more success, the more relatives!
My Title: Freeway , The Joke: When everything is coming your way, you're on the wrong side of the freeway.

Please note that in this example we use the GELF field naming convention of, which means that your extra fields must all have an underscore. This is because the WebSocket endpoint sends messages fully compatible with the GELF format so you can use them after in any GELF-compatible tool.

The pattern option allows you also to customize colors, background, and text colors are customizable.

$ldp@ubuntu:~$ ./ldp-tail --address "wss://gra1.logs.ovh.com/tail/?tk=demo" --pattern "My Title: {{color \"red\"}} {{ ._title }} , {{ noColor }} The Joke: {{ color \"blue\" }} {{ .short_message }} {{ noColor }}"

In this example, the title field will be colored in red, and the body will be colored in blue. You can use the bColor attribute to color the background instead. The pattern option supports many different operations like date formatting, concatenation, human readable duration displaying, etc. Ldp-tail also supports conditional formatting. This can be useful to sort important information in your stream. For example this kind of rule can be implemented: if a value is greater than a threshold, display the message in red, otherwise display it in green.

With our demo stream, we can use this kind of filter and the rating_num numeric attribute to display in yellow every joke rated above 100.

$ ldp@ubuntu:~$ ./ldp-tail --address "wss://gra1.logs.ovh.com/tail/?tk=demo" --pattern "My Title: {{color \"red\"}} {{ ._title }} , {{noColor }} The Joke: {{if (lt (._rating_num) 100.0)}}{{ color \"blue\" }} {{.short_message}} {{else}}{{color \"yellow\"}} {{ .short_message }} {{ noColor }} {{ end }}"

 

The match option

As the name implies, the match option is able to choose which messages you want or don't want to display in your ldp-tail. The option contains several operators, all described on this GitHub page. You can easily display messages beginning with some values or display only message that have a certain field or whose a field is higher or lower than a value.

Here is how you can display only logs that have a title beginning with the word another.

$ ldp@ubuntu:~$ ./ldp-tail --address "wss://gra1.logs.ovh.com/tail/?tk=demo" --match "_title.begin=What's" --pattern "{{ ._title}} The Joke: {{ .short_message }}"

You can of course combine multiple matches by issuing ldp-tail --match --match --pattern .... A pattern and match combination can be really difficult to read, thats why ldp-tail supports a TOML configuration file.

 

TOML Configuration file

Here is an example of a TOML configuration file for ldp-tail. You can save it into a file named myfile.toml.

Address = "wss://gra1.logs.ovh.com/tail/?tk=demo"
Pattern = "{{ ._title}} The Joke: {{ .short_message }}"

[[Match]]
Key="_rating_num"
Operator="gt"
Value=100.0
Not=false

[[Match]]
Key="_title"
Operator="contain"
Value="the"
Not=false

If you are not familiar with TOML, here are some explanations. Address and pattern are similar to the options you have in the command line. Match option is a little bit different. Since you can have multiple conditions, Match is an array of tables where every object is a full condition with the field, the operator, the value, and the negation of this condition if needed. Note that the Value attribute can be a string or a number. The Not attribute must be the boolean true or false.

Once your file is ready, you can launch ldp-tail with it:

$ ldp@ubuntu:~$ ./ldp-tail --address "wss://gra1.logs.ovh.com/tail/?tk=demo" --config "myfile.toml"

 

Combining matches and filters

If you combine matches and filters you can, for example, print the call to an API and format the output depending on the status or the duration of the call.

operate_LDP_vqe_ldp-tail03.gif

This example has been generated using the following configuration:

Address = "wss://gra1.logs.ovh.com/tail?tk=demo"
Pattern = "{{if (lt (int ._duration_ms_num) 200)}}{{color \"green\"}}{{else if (lt (int ._duration_ms_num) 500)}}{{color \"yellow\"}}{{else}}{{color \"red\"}}{{end}}{{date .timestamp}} | {{ printf \"%-80s\" (join \" \" ._method ._path )}} | {{ ._httpStatus_int }} | {{ duration ._duration_ms_num 1000000 }}{{noColor}}"

[[Match]]
Key="_path"
Operator="begin"
Value="/dbaas/logs"
Not=false

 

Replay tail

It's also possible to replay a given time window in the past.

To proceed, you need a "begin" time as a URI query parameter, which must be in Unix timestamp format.

To find your current timestamp:

date +%s

Sample:

$ ldp@ubuntu:~$ ldp-tail --address "wss://gra1.logs.ovh.com/tail/?tk=demo&begin=<your_time>" --pattern "{{date .timestamp}}: {{ ._category }}"

To make the code above work, replace begin value with a timestamp from the last few weeks.

 

Go further

For more information and tutorials, please see our other Logs Data Platform support guides or explore the guides for other OVHcloud products and services.

Related articles