1. Articles
  2. Manage and Operate
  3. Logs Data Platform
  4. Ingesting Your Logs

Articles in this section

See more

Pushing Logs with a Logging Library - Python 3.x (logging-ldp)

Avatar
Stephanie R.
  • Updated

Learn how to push your logs to Logs Data Platform using Python 3.x.

Logging-ldp is intended to be a high-performance logging formatter and handler to send log entries into Logs Data Platform.

This package includes:

  • a TCP/TLS handler to send log entries over TCP with TLS support.
  • a formatter to convert logging records into GELF(1.1).
  • a facility to ensure fields suit the LDP naming conventions.

 

Requirements

 

Instructions

Install

Using pip

You can use pip to install logging-ldp, make sure you have the latest version:

$ pip3 install --upgrade pip 
[...]
Successfully installed pip-<version>
$ pip3 install --upgrade logging-ldp 
[...]
Successfully installed logging-ldp-<version> setuptools-18.3.1

 

Using sources

Instructions for installing logging-ldp are available on the OVH GitHub repository.

 

How to send logs

The following example shows how to send log in Graylog TCP input:

import logging
from logging_ldp.formatters import LDPGELFFormatter
from logging_ldp.handlers import LDPGELFTCPSocketHandler

def setup_logging():
    handler = LDPGELFTCPSocketHandler(hostname="<your_cluster>.logs.ovh.com")
    handler.setFormatter(LDPGELFFormatter(token="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"))
    logging.getLogger().addHandler(handler)
    logging.getLogger().setLevel(logging.INFO)

if __name__ == '__main__':
    setup_logging()

    logging.info("Test !")

 

Send additional static meta data

To automatically append meta data on all your logs, you can implement an alternate Schema:

import logging
from marshmallow import fields
from logging_ldp.formatters import LDPGELFFormatter
from logging_ldp.handlers import LDPGELFTCPSocketHandler
from logging_ldp.schemas import LDPSchema
def setup_logging():
    # Load your config there
    config = dict(name="myapp", version="0.0.1")
    # Define a custom Schema
    class MyApp(LDPSchema):
        app_name = fields.Constant(config['name'])
        app_version = fields.Constant(config['version'])

    handler = LDPGELFTCPSocketHandler(hostname="<your_cluster>.logs.ovh.com")
    handler.setFormatter(LDPGELFFormatter(token="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX", schema=MyApp))
    logging.getLogger().addHandler(handler)
    logging.getLogger().setLevel(logging.INFO)

if __name__ == '__main__':
    setup_logging()
    logging.info("Test !")

The log entry sent to Graylog will be something like this:

{
  "_app_name": "myapp",
  "_app_version": "0.0.1",
  "_X-OVH-TOKEN": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  "file": "test_thread.py",
  "host": "cdumay-desk",
  "level": 6,
  "line": 36,
  "short_message": "Test !",
  "timestamp": 1556036745.6493497,
  "version": "1.1"
}

 

Send additional intermittent metadata

To define occasional metadata, you can define a Schema with Nested sub-items:

import logging
from marshmallow import Schema, fields
from logging_ldp.formatters import LDPGELFFormatter
from logging_ldp.handlers import LDPGELFTCPSocketHandler
from logging_ldp.schemas import LDPSchema

# Define a sub-schema
class User(Schema):
    name = fields.String(required=True)
    age = fields.Integer()

# Define a custom schema to apply on log entries
class AccountInfo(LDPSchema):
    user = fields.Nested(User, required=True)
    manager = fields.Nested(User)

def setup_logging():
    handler = LDPGELFTCPSocketHandler(hostname="<your_cluster>.logs.ovh.com")
    handler.setFormatter(LDPGELFFormatter(
        token="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
        schema=AccountInfo
    ))
    logging.getLogger().addHandler(handler)
    logging.getLogger().setLevel(logging.INFO)

if __name__ == '__main__':
    setup_logging()

    current_user = dict(name="John Doe")
    boss = dict(name="Roger Smith", age=51)
    logging.info("User has logged", extra=dict(user=current_user, manager=boss))

The log entry sent will be:

{
  "_manager_age_int": 51,
  "_manager_name": "Roger Smith",
  "_user_name": "John Doe",
  "_X-OVH-TOKEN": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  "file": "test_thread.py",
  "host": "cdumay-desk",
  "level": 6,
  "line": 40,
  "short_message": "User has logged",
 "timestamp": 1556037587.4444475,
  "version": "1.1"
}

As we can see:

  • Objects are transformed to flatten dictionaries: manager.name is renamed manager_name.
  • Fields are types using the LDP naming convention: manager.age is renamed manager_age_int
  • Null values are not sent: user.age.
  • You can set required=True to force meta data value or default=xxx to add data automatically.

 

Go further

For more information and tutorials, please see our other Logs Data Platform support guides or explore the guides for other OVHcloud products and services.

Related articles