Configuring Splinter Daemon Logging

The default log configuration is sufficient for many applications but sometimes a more targeted approach is needed. The logging system built into the Splinter Daemon is highly user configurable and can separate the proverbial wheat from the chaff. This document contains a high level description of how logging works in the context of Splinter, examples of logging configurations, and troubleshooting tips for common use cases.

Prerequisites

  • A working Splinter installation.
  • Edit access to the splinterd.toml file for the above Splinter installation.

High Level Logging Overview

There are two main abstractions built into our logging solution.

  • Loggers
  • Appenders

Loggers are the interface between Splinter code and appenders. Appenders are the interface between Loggers and where logs are actually written.

There is a many to many relationship between the two concepts, each logger can relate to any number of appenders and each appender can relate to any number of loggers.

The life cycle of log flows as follows. Log statement is called in Splinter, the message and associated metadata is directed to a logger. The logger forwards the log to all of its configured appenders. Each of those appenders write the log to their resource.

Besides their forwarding and writing duties loggers and appenders also filter the logs they receive by level.

Log Levels

  • Error
  • Warn
  • Info
  • Debug
  • Trace

Listed above in decreasing order of importance and volume. Each level includes all the higher levels i.e. Info includes messages with levels Warn, Error, and Info.

Loggers in Depth

Loggers form a tree structure. The root logger is the root of the logger tree with all other loggers being its descendants. As such any log message that is not directed to any other logger will be handled by the root logger.

Log messages are directed to loggers based on the loggers name and where the log message originated. So log messages from the splinter package will match to any loggers named splinter. Log messages match to the most specific package/logger available so a message from splinter::config would match to a logger named splinter::config if its available before matching to a splinter logger.

Loggers have two other fields appenders and level. Appenders is an array of appender names and serves as the connection between loggers and appenders. The level field controls the filtering functionality of the logger as discussed above.

For non-root loggers the appenders and level fields are optional. The level will be set to the root logger’s level. Appenders are a little more complicated, each logger inherits its parents appenders. So if you had two loggers, root and splinter and root has both stdout and stderr appenders configured. Then the splinter logger would also have the same stdout and stderr appenders plus any specifically called out in its configuration.

Appenders in Depth

There are several types of appenders denoted by the kind field.

Appender Kinds

  • stdout
  • stderr
  • file
  • rolling_file

The stdout and stderr types log to stdout and stderr respectively as you would expect. The file and rolling_file types both log to files and have another required field filename that is the path to the desired log file. The rolling_file type also has a required field size that controls when the file is rolled up. When the log file reaches the size specified the file is deleted and restarted. The files do not need to exist at Splinter startup but the containing directory does.

Example Configurations

The default config

This is the configuration provided with all defaults written out. Note that while the stdout appender does not have a level specified by default it can be filled in at runtime with one of Info, Debug, Trace for one, two or three plus occurrences of the -v verbosity flag.

[appenders.stdout]
kind = "stdout"
encoder = "[{d(%Y-%m-%d %H:%M:%S%.3f)}] T[{T}] {l} [{M}] {m}\n"

[loggers.root]
appenders = ["stdout"]
level = "Warn"

[loggers.splinter]
appenders = ["stdout"]
level = "Info"

[loggers.splinterd]
appenders = ["stdout"]
level = "Info"

Redirect Splinter Daemon logs to file

This example will redirect the logs from libsplinter and the splinter daemon to a single log file. This is the most generally useful configuration besides the default configuration because it will seperate the logs related to splinters functionality from the logs generated by underlying libraries.

[appenders.splinter_file]
kind = "file"
filename = "/var/log/splinter/splinterd.log"

[loggers.splinter]
appenders = ["splinter_file"]
level = "Info"

[loggers.splinterd]
appenders = ["splinter_file"]
level = "Info"

Redirect Splinter Daemon logs to a Rolling File

This example is the same as the one above except the log file will be rolled when it reaches a size of 50 megabytes.

[appenders.splinter_file]
kind = "rolling_file"
size = "50M"
filename = "/var/log/splinter/splinterd.log"

[loggers.splinter]
appenders = ["splinter_file"]
level = "Info"

[loggers.splinterd]
appenders = ["splinter_file"]
level = "Info"

All Logs All the Time

This example has turned off all log filtering, every log statement will be logged to stdout. This is not recommended and may cause performance issues.

[loggers.root]
appenders = ["stdout"]
level = "Trace"

[loggers.splinter]
level = "Trace"

[loggers.splinterd]
level = "Trace"

Procedure

  1. Set up Appenders.

    a. Set appender name [appenders.<appender_name_here>].

    b. Select appender kind to match your use case. kind = "<appender_kind>"

    c. Fill other required fields dependant on the appender kind. Filename is required for file and rolling_file kinds while size is only required for rolling_file appenders.

    d. Repeat steps a-c for any other appenders needed.

  2. Set up Loggers.

    a. Set logger name [loggers.<logger_name_here>], remembering that the name determines which Rust modules will write to this logger.

    b. Set appenders field to list of appender targets. appenders = ["<appender_names_configured above>"]

    c. Optionally set the Level field to filter the log messages forwarded to the appenders. level = "Info"

    d. Repeat steps a-c for each module that needs logging configuration.

  3. (Optional) Override default loggers. Any logger named root will override the default included root logger.

[loggers.root]
appenders = ["splinter_file"]
level = "Info"

The above example will set the root logger to log all messages with level Info and above to the splinter_file appender.

The tokio, tokio_reactor, and hyper loggers are included by default and will log to the stdout appender. All of them can be overridden as shown in the Example Configurations section above.

Troubleshooting

Appender names have to exactly match between the logger appenders field and the appender definition.

Setting up loggers to capture logs from sub modules requires that the names be quoted. For example [loggers."splinterd::config"] to select for the splinterd::config module.