Learn how to deploy a personalized Logstash having a specific configuration and send logs from any source to your stream directly on the Logs Data Platform.
Logstash is an open-source software developed by Elastic. Based on its features, it is possible to send messages from several inputs to different types of output using a variety of codecs, while processing them and transforming them in the process. You can learn a lot more about it on the official website.
Topics
- What is Logstash?
- How do I configure Logstash Collector on Logs Data Platform?
- What is special about our hosted Logstash? Available plugins, SSL Certificates, and more.
Requirements
If you are new to Logs Data Platform, completion of the Quick Start Tutorial is highly recommended.
Instructions
What is Logstash
Logstash is a data processing pipeline that allows you to receive information from several sources, transforming it if necessary, before sending it to a variety of software. In the configuration file, there are three main parts to configure:
- The Input part: details where your logs come from. For instance, some inputs may wait for logs and in some cases, inputs can fetch them from a database.
- The Filter part: details how Logstash should parse and transform your logs messages.
- The Output part: specifies the format and their destination.
If you are hosting Logstash on your server, here are some sample configuration files.
Below is a small configuration file example:
This is one of many configuration examples you can use in your own Logstash to send syslog type logs to the Logs Data Platform. There are a lot of filters and plugins available in Logstash, and the list is growing! For example, by sending these kinds of lines into this Logstash:
You will have the following result in your stream in Graylog:
So, let's suppose you have a lot of hosts and you want to send all your syslog outputs to Logstash. One way to do it is to put one Logstash on one of your machines and send all the logs to this hosts. This will work but what happens if your hosts go down? Or is not fast enough? Or if it isn't compatible with Java? Don't worry, the best thing about the Logs Data Platform is that we can host your Logstash for you. We also allow you to configure the input and the filter as you wish and you can even define custom Groks. The next section will tell you how to do it!
Host a Logstash collector on Logs Data Platform
The Logstash Collector comes with many advantages:
- The output part of the Logstash is automatically configured by us.
- You have your own certificate to enable SSL. This certificate can be used for example to trust only the TCP Inputs of your Logstash. This certificate and its associated key can be found at the following paths: /etc/ssl/private/server.crt for the cert and /etc/ssl/private/server.key for the key. The CA used to create these inputs is at the following location /etc/ssl/private/ca.crt. You will find this SSL CA for reference at the end of this document. For the http input with SSL enabled, a keystore with the key+cert+CA in PKCS12 format is present at the following location /etc/ssl/private/keystore.jks with keystore password "changeit".
- And finally, we ensure that your collector is always up and running 24/7.
To host this collector in the Logs Data Platform, you will need to configure it in the Logs Data Platform manager:
- Check the Data-gathering tools tab and click on the button
Add a new data-gathering tool. - The Manager will then ask you to provide an elegant name and a wonderful description.
- Select your Software: Select Logstash 8.x.
- Fill the main port of your input. We need to know which port you will use in your input configuration. We will put 6514 for now. This is also a setting you can change later.
- If you want to restrict the Logstash to specific IP Addresses, you can use the Trusted Networks field to firewall your instances. Add every IP Address or subnet you want to allow, separated by comma. Leave it empty to allow any IP to contact it. Your collector will have a public IP Address, you can use this feature to prevent people to send you false information in it.
- Select one of the streams you have to attached this collector to the stream you have. That means that every message that goes out of this collector is automatically routed to the stream.
- Finally, select the number of instances you would need to handle your load. We recommend to have at least two instances to ensure the high availability of your Logstash. However some uses cases need you to limit the number of instance at one (like the Twitter use case).
Note that you can also choose to automatically scale the number of instances. In this case, you need to select a minimum and maximum number of instances. Your input will then be started with the selected minimum number of instances. If these instances face a heavy load for a certain amount of time (more than 1 minute), your input will be automatically scaled up — one new instance at the time — until it reaches the selected maximum number of instances. On the other hand, if the load decreases, this number of instances will be scaled down until it reaches the minimum number of instances.
Click on Next button to display the Logstash Configuration Page.
Logstash Collector configuration
There are three sections in this page:
- The Input section where you put the Input part of your configuration file.
- The Filter section where you put the filter part.
- The grok patterns sections where you can create your own grok patterns.
All the inputs have preconfigured SSL certificates at the fixed locations used in the configuration below. If we enable the SSL encryption, the configuration needs five extra parameters:
-
ssl_enable: that allows to enable SSL. Must be set to
true. - ssl_verify: this deactivates the client certificate verification process that needs a trusted certificate on client side.
- ssl_cert: the location of the auto-generated server certificate.
- ssl_key: the location of the associated key.
- ssl_extra_chain_certs: the array contains the path to the CA certificate.
Note that the input grammar configuration is already on the screen.
input {
'Input' section
}
For the filter, you would use the same configuration in the example at the beginning of this guide.
filter {
'Filter' section
}
The last section of the configuration is dedicated to custom Grok Filters. If you know about Grok, you can use this text area to create your own grok that you can use in the filter section above. It is an advanced usage of Logstash. Head to the end of the document to get some reference links that can help you use this functionality.
To validate the settings, you will have to click on Test the configuration button. After some calculations you should see on the screen a green confirmation.
You can then click on Finish the configuration button to save your configuration!
Starting the input
Starting the input is done by clicking the more options ... button to the right of your input and selecting Start. This will deploy your collector on Logs Data Platform.
Collector useful information
Once your collector is started, you can copy several pieces of information to your clipboard.
- The hostname
- The public IP
- The SSL Certificate
The hostname is in the following form:
<your_cluster>-XXXXXXXXXXXXXXXXXXXXXXX.<your_cluster>.logs.ovh.com
This is an address of your collector for the cluster on Logs Data Platform.
Send your logs to your address to have them processed by your collector using the following:
If you head to Graylog, you will find your Logs in the attached Stream just as before.
Additional information
Logstash Version
The version hosted by Logs Data Platform is the latest Logstash 8. Of course we will update to the new versions as soon as they become available.
Logstash Plugins
Below is a list of Logstash plugins we support.
Inputs plugins
logstash-input-azure_event_hubs logstash-input-beats logstash-input-couchdb_changes logstash-input-dead_letter_queue logstash-input-elasticsearch logstash-input-ganglia logstash-input-gelf logstash-input-generator logstash-input-graphite logstash-input-heartbeat logstash-input-http logstash-input-http_poller logstash-input-imap logstash-input-jdbc logstash-input-jms logstash-input-rabbitmq logstash-input-redis logstash-input-s3 logstash-input-snmp logstash-input-snmptrap logstash-input-sqs logstash-input-stdin logstash-input-syslog logstash-input-tcp logstash-input-twitter
Input GELF plugin
We patched the GELF input to support TCP+TLS. Example Input Section:
filter plugins
logstash-filter-aggregate logstash-filter-anonymize logstash-filter-cidr logstash-filter-clone logstash-filter-csv logstash-filter-date logstash-filter-de_dot logstash-filter-dissect logstash-filter-dns logstash-filter-drop logstash-filter-elasticsearch logstash-filter-fingerprint logstash-filter-geoip logstash-filter-grok logstash-filter-hex logstash-filter-http logstash-filter-jdbc_static logstash-filter-jdbc_streaming logstash-filter-json logstash-filter-kv logstash-filter-math logstash-filter-memcached logstash-filter-metrics logstash-filter-mutate logstash-filter-prune logstash-filter-sleep logstash-filter-split logstash-filter-syslog_pri logstash-filter-throttle logstash-filter-translate logstash-filter-truncate logstash-filter-urldecode logstash-filter-useragent logstash-filter-uuid logstash-filter-xml
codec plugins
logstash-codec-avro logstash-codec-cef logstash-codec-collectd logstash-codec-dots logstash-codec-edn logstash-codec-edn_lines logstash-codec-es_bulk logstash-codec-fluent logstash-codec-gelf logstash-codec-graphite logstash-codec-json logstash-codec-json_lines logstash-codec-line logstash-codec-msgpack logstash-codec-multiline logstash-codec-netflow logstash-codec-plain logstash-codec-rubydebug
The following plugins are disabled for security reasons:
logstash-input-exec logstash-input-file logstash-input-kafka logstash-input-pipe logstash-input-unix logstash-filter-ruby
SSL CA certificate
The CA certificate you can use to verify the authority presented by our hosted collectors can be copied from the homepage of the Logs Data Platform manager.
Console output
Last but not least, if you have concerns about the way your logstash is handling your logs, you may be interested to follow the messages coming from your collector. To do this, please go to the dedicated page by clicking on the Console output button.
Useful Resources
Here are some links to help you go further with Logstash:
- Logstash official documentation
- Grok filters documentation
- Pushing Logs with a Forwarder - Filebeat (Linux)
- Grok Constructor
- A Ruby regular expression editor
Go further
For more information and tutorials, please see our other Logs Data Platform support guides or explore the guides for other OVHcloud products and services.