JSON Template Layout

JsonTemplateLayout is a customizable, efficient, and garbage-free JSON generating layout. It encodes LogEvents according to the structure described by the JSON template provided. In a nutshell, it shines with its

Usage

Adding log4j-layout-template-json artifact to your list of dependencies is enough to enable access to JsonTemplateLayout in your Log4j configuration:

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-layout-template-json</artifactId>
    <version>2.19.0</version>
</dependency>

For instance, given the following JSON template modelling the Elastic Common Schema (ECS) specification (accessible via classpath:EcsLayout.json)

{
  "@timestamp": {
    "$resolver": "timestamp",
    "pattern": {
      "format": "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",
      "timeZone": "UTC"
    }
  },
  "ecs.version": "1.2.0",
  "log.level": {
    "$resolver": "level",
    "field": "name"
  },
  "message": {
    "$resolver": "message",
    "stringified": true
  },
  "process.thread.name": {
    "$resolver": "thread",
    "field": "name"
  },
  "log.logger": {
    "$resolver": "logger",
    "field": "name"
  },
  "labels": {
    "$resolver": "mdc",
    "flatten": true,
    "stringified": true
  },
  "tags": {
    "$resolver": "ndc"
  },
  "error.type": {
    "$resolver": "exception",
    "field": "className"
  },
  "error.message": {
    "$resolver": "exception",
    "field": "message"
  },
  "error.stack_trace": {
    "$resolver": "exception",
    "field": "stackTrace",
    "stackTrace": {
      "stringified": true
    }
  }
}

in combination with the below log4j2.xml configuration:

<JsonTemplateLayout eventTemplateUri="classpath:EcsLayout.json"/>

or with the below log4j2.properties configuration:

appender.console.json.type = JsonTemplateLayout
appender.console.json.eventTemplateUri = classpath:EcsLayout.json

JsonTemplateLayout generates JSON as follows:

{
  "@timestamp": "2017-05-25T19:56:23.370Z",
  "ecs.version": "1.2.0",
  "log.level": "ERROR",
  "message": "Hello, error!",
  "process.thread.name": "main",
  "log.logger": "org.apache.logging.log4j.JsonTemplateLayoutDemo",
  "error.type": "java.lang.RuntimeException",
  "error.message": "test",
  "error.stack_trace": "java.lang.RuntimeException: test\n\tat org.apache.logging.log4j.JsonTemplateLayoutDemo.main(JsonTemplateLayoutDemo.java:11)\n"
}

Layout Configuration

JsonTemplateLayout is configured with the following parameters:

Table 1. JsonTemplateLayout parameters

Parameter Name

Type

Description

charset

Charset

Charset used for String encoding

locationInfoEnabled

boolean

toggles access to the LogEvent source; file name, line number, etc. (defaults to false set by log4j.layout.jsonTemplate.locationInfoEnabled property)

stackTraceEnabled

boolean

toggles access to the stack traces (defaults to true set by log4j.layout.jsonTemplate.stackTraceEnabled property)

eventTemplate

String

inline JSON template for rendering LogEvents (has priority over eventTemplateUri, defaults to null set by log4j.layout.jsonTemplate.eventTemplate property)

eventTemplateUri

String

URI pointing to the JSON template for rendering LogEvents (defaults to classpath:EcsLayout.json set by log4j.layout.jsonTemplate.eventTemplateUri property)

eventTemplateRootObjectKey

String

if present, the event template is put into a JSON object composed of a single member with the provided key (defaults to null set by log4j.layout.jsonTemplate.eventTemplateRootObjectKey property)

eventTemplateAdditionalField

EventTemplateAdditionalField[]

additional key-value pairs appended to the root of the event template

stackTraceElementTemplate

String

inline JSON template for rendering StackTraceElements (has priority over stackTraceElementTemplateUri, defaults to null set by log4j.layout.jsonTemplate.stackTraceElementTemplate property)

stackTraceElementTemplateUri

String

URI pointing to the JSON template for rendering StackTraceElements (defaults to classpath:StackTraceElementLayout.json set by log4j.layout.jsonTemplate.stackTraceElementTemplateUri property)

eventDelimiter

String

delimiter used for separating rendered LogEvents (defaults to System.lineSeparator() set by log4j.layout.jsonTemplate.eventDelimiter property)

nullEventDelimiterEnabled

boolean

append \0 (null) character to the end of every eventDelimiter separating rendered LogEvents (defaults to false set by log4j.layout.jsonTemplate.nullEventDelimiterEnabled property)

maxStringLength

int

truncate string values longer than the specified limit (defaults to 16384 set by log4j.layout.jsonTemplate.maxStringLength property)

truncatedStringSuffix

String

suffix to append to strings truncated due to exceeding maxStringLength (defaults to set by log4j.layout.jsonTemplate.truncatedStringSuffix property)

recyclerFactory

RecyclerFactory

recycling strategy that can either be dummy, threadLocal, or queue (set by log4j.layout.jsonTemplate.recyclerFactory property)

Additional event template fields

Additional event template fields are a convenient short-cut to add custom fields to a template or override the existing ones. Following configuration overrides the host field of the GelfLayout.json template and adds two new custom fields:

XML configuration with additional fields
<JsonTemplateLayout eventTemplateUri="classpath:GelfLayout.json">
  <EventTemplateAdditionalField key="host" value="www.apache.org"/>
  <EventTemplateAdditionalField key="_serviceName" value="auth-service"/>
  <EventTemplateAdditionalField key="_containerId" value="6ede3f0ca7d9"/>
</JsonTemplateLayout>

The default format for the added new fields are String. One can also provide JSON-formatted additional fields:

XML-formatted configuration with JSON-formatted additional fields
<JsonTemplateLayout eventTemplateUri="classpath:GelfLayout.json">
  <EventTemplateAdditionalField
       key="marker"
       format="JSON"
       value='{"$resolver": "marker", "field": "name"}'/>
  <EventTemplateAdditionalField
       key="aNumber"
       format="JSON"
       value="1"/>
  <EventTemplateAdditionalField
       key="aList"
       format="JSON"
       value='[1, 2, "three"]'/>
</JsonTemplateLayout>

Additional event template fields can very well be introduced using properties-, YAML-, and JSON-formatted configurations:

Properties-formatted configuration with JSON-formatted additional fields
appender.console.layout.type = JsonTemplateLayout
appender.console.layout.eventTemplateUri = classpath:GelfLayout.json
appender.console.layout.eventTemplateAdditionalField[0].type = EventTemplateAdditionalField
appender.console.layout.eventTemplateAdditionalField[0].key = marker
appender.console.layout.eventTemplateAdditionalField[0].value = {"$resolver": "marker", "field": "name"}
appender.console.layout.eventTemplateAdditionalField[0].format = JSON
appender.console.layout.eventTemplateAdditionalField[1].type = EventTemplateAdditionalField
appender.console.layout.eventTemplateAdditionalField[1].key = aNumber
appender.console.layout.eventTemplateAdditionalField[1].value = 1
appender.console.layout.eventTemplateAdditionalField[1].format = JSON
appender.console.layout.eventTemplateAdditionalField[2].type = EventTemplateAdditionalField
appender.console.layout.eventTemplateAdditionalField[2].key = aList
appender.console.layout.eventTemplateAdditionalField[2].value = [1, 2, "three"]
appender.console.layout.eventTemplateAdditionalField[2].format = JSON
YAML-formatted configuration with JSON-formatted additional fields
JsonTemplateLayout:
  eventTemplateAdditionalField:
    - key: "marker"
      value: '{"$resolver": "marker", "field": "name"}'
      format: "JSON"
    - key: "aNumber"
      value: "1"
      format: "JSON"
    - key: "aList"
      value: '[1, 2, "three"]'
      format: "JSON"
JSON-formatted configuration with JSON-formatted additional fields
{
  "JsonTemplateLayout": {
    "eventTemplateAdditionalField": [
      {
        "key": "marker",
        "value": "{\"$resolver\": \"marker\", \"field\": \"name\"}",
        "format": "JSON"
      },
      {
        "key": "aNumber",
        "value": "1",
        "format": "JSON"
      },
      {
        "key": "aList",
        "value": "[1, 2, \"three\"]",
        "format": "JSON"
      }
    ]
  }
}

Recycling strategy

RecyclerFactory plays a crucial role for determining the memory footprint of the layout. Template resolvers employ it to create recyclers for objects that they can reuse. The behavior of each RecyclerFactory and when one should prefer one over another is explained below:

  • dummy performs no recycling, hence each recycling attempt will result in a new instance. This will obviously create a load on the garbage-collector. It is a good choice for applications with low and medium log rate.

  • threadLocal performs the best, since every instance is stored in ThreadLocals and accessed without any synchronization cost. Though this might not be a desirable option for applications running with hundreds of threads or more, e.g., a web servlet.

  • queue is the best of both worlds. It allows recycling of objects up to a certain number (capacity). When this limit is exceeded due to excessive concurrent load (e.g., capacity is 50 but there are 51 threads concurrently trying to log), it starts allocating. queue is a good strategy where threadLocal is not desirable.

    queue also accepts optional supplier (of type java.util.Queue, defaults to org.jctools.queues.MpmcArrayQueue.new if JCTools is in the classpath; otherwise java.util.concurrent.ArrayBlockingQueue.new) and capacity (of type int, defaults to max(8,2*cpuCount+1)) parameters:

    Example configurations of queue recycling strategy
    queue:supplier=org.jctools.queues.MpmcArrayQueue.new
    queue:capacity=10
    queue:supplier=java.util.concurrent.ArrayBlockingQueue.new,capacity=50

The default RecyclerFactory is threadLocal, if log4j2.enable.threadlocals=true; otherwise, queue.

See Extending Recycler Factories for details on how to introduce custom RecyclerFactory implementations.

Template Configuration

Templates are configured by means of the following JsonTemplateLayout parameters:

  • eventTemplate[Uri] (for serializing LogEvents)

  • stackTraceElementTemplate[Uri] (for serializing StackStraceElements)

  • eventTemplateAdditionalField (for extending the used event template)

Event Templates

eventTemplate[Uri] describes the JSON structure JsonTemplateLayout uses to serialize LogEvents. The default configuration (accessible by log4j.layout.jsonTemplate.eventTemplate[Uri] property) is set to classpath:EcsLayout.json provided by the log4j-layout-template-json artifact, which contains the following predefined event templates:

Event Template Resolvers

Event template resolvers consume a LogEvent and render a certain property of it at the point of the JSON where they are declared. For instance, marker resolver renders the marker of the event, level resolver renders the level, and so on. An event template resolver is denoted with a special object containing a`$resolver` key:

Example event template demonstrating the usage of level resolver
{
  "version": "1.0",
  "level": {
    "$resolver": "level",
    "field": "name"
  }
}

Here version field will be rendered as is, while level field will be populated by the level resolver. That is, this template will generate JSON similar to the following:

Example JSON generated from the demonstrated event template
{
  "version": "1.0",
  "level": "INFO"
}

The complete list of available event template resolvers are provided below in detail.

counter
config      = [ start ] , [ overflowing ] , [ stringified ]
start       = "start" -> number
overflowing = "overflowing" -> boolean
stringified = "stringified" -> boolean

Resolves a number from an internal counter.

Unless provided, start and overflowing are respectively set to zero and true by default.

When stringified is enabled, which is set to `false by default, the resolved number will be converted to a string.

Warning

When overflowing is set to true, the internal counter is created using a long, which is subject to overflow while incrementing, though garbage-free. Otherwise, a BigInteger is used, which does not overflow, but incurs allocation costs.

Examples

Resolves a sequence of numbers starting from 0. Once Long.MAX_VALUE is reached, counter overflows to Long.MIN_VALUE.

{
  "$resolver": "counter"
}

Resolves a sequence of numbers starting from 1000. Once Long.MAX_VALUE is reached, counter overflows to Long.MIN_VALUE.

{
  "$resolver": "counter",
  "start": 1000
}

Resolves a sequence of numbers starting from 0 and keeps on doing as long as JVM heap allows.

{
  "$resolver": "counter",
  "overflowing": false
}
caseConverter
config                = case , input , [ locale ] , [ errorHandlingStrategy ]
input                 = JSON
case                  = "case" -> ( "upper" | "lower" )
locale                = "locale" -> (
                            language                                   |
                          ( language , "_" , country )                 |
                          ( language , "_" , country , "_" , variant )
                        )
errorHandlingStrategy = "errorHandlingStrategy" -> (
                          "fail"    |
                          "pass"    |
                          "replace"
                        )
replacement           = "replacement" -> JSON

Converts the case of string values.

input can be any available template value; e.g., a JSON literal, a lookup string, an object pointing to another resolver.

Unless provided, locale points to the one returned by JsonTemplateLayoutDefaults.getLocale(), which is configured by log4j.layout.jsonTemplate.locale system property and by default set to the default system locale.

errorHandlingStrategy determines the behavior when either the input doesn’t resolve to a string value or case conversion throws an exception:

  • fail propagates the failure

  • pass causes the resolved value to be passed as is

  • replace suppresses the failure and replaces it with the replacement, which is set to null by default

errorHandlingStrategy is set to replace by default.

Most of the time JSON logs are persisted to a storage solution (e.g., Elasticsearch) that keeps a statically-typed index on fields. Hence, if a field is always expected to be of type string, using non-string replacements or pass in errorHandlingStrategy might result in type incompatibility issues at the storage level.

Warning

Unless the input value is passed intact or replaced, case conversion is not garbage-free.

Examples

Convert the resolved log level strings to upper-case:

{
  "$resolver": "caseConverter",
  "case": "upper",
  "input": {
    "$resolver": "level",
    "field": "name"
  }
}

Convert the resolved USER environment variable to lower-case using nl_NL locale:

{
  "$resolver": "caseConverter",
  "case": "lower",
  "locale": "nl_NL",
  "input": "${env:USER}"
}

Convert the resolved sessionId thread context data (MDC) to lower-case:

{
  "$resolver": "caseConverter",
  "case": "lower",
  "input": {
    "$resolver": "mdc",
    "key": "sessionId"
  }
}

Above, if sessionId MDC resolves to a, say, number, case conversion will fail. Since errorHandlingStrategy is set to replace and replacement is set to null by default, the resolved value will be null. One can suppress this behavior and let the resolved sessionId number be left as is:

{
  "$resolver": "caseConverter",
  "case": "lower",
  "input": {
    "$resolver": "mdc",
    "key": "sessionId"
  },
  "errorHandlingStrategy": "pass"
}

or replace it with a custom string:

{
  "$resolver": "caseConverter",
  "case": "lower",
  "input": {
    "$resolver": "mdc",
    "key": "sessionId"
  },
  "errorHandlingStrategy": "replace",
  "replacement": "unknown"
}
endOfBatch
{
  "$resolver": "endOfBatch"
}

Resolves logEvent.isEndOfBatch() boolean flag.

exception
config              = field , [ stringified ] , [ stackTrace ]
field               = "field" -> ( "className" | "message" | "stackTrace" )

stackTrace          = "stackTrace" -> (
                        [ stringified ]
                      , [ elementTemplate ]
                      )

stringified         = "stringified" -> ( boolean | truncation )
truncation          = "truncation" -> (
                        [ suffix ]
                      , [ pointMatcherStrings ]
                      , [ pointMatcherRegexes ]
                      )
suffix              = "suffix" -> string
pointMatcherStrings = "pointMatcherStrings" -> string[]
pointMatcherRegexes = "pointMatcherRegexes" -> string[]

elementTemplate     = "elementTemplate" -> object

Resolves fields of the Throwable returned by logEvent.getThrown().

stringified is set to false by default. stringified at the root level is deprecated in favor of stackTrace.stringified, which has precedence if both are provided.

pointMatcherStrings and pointMatcherRegexes enable the truncation of stringified stack traces after the given matching point. If both parameters are provided, pointMatcherStrings will be checked first.

If a stringified stack trace truncation takes place, it will be indicated with a suffix, which by default is set to the configured truncatedStringSuffix in the layout, unless explicitly provided. Every truncation suffix is prefixed with a newline.

Stringified stack trace truncation operates in Caused by: and Suppressed: label blocks. That is, matchers are executed against each label in isolation.

elementTemplate is an object describing the template to be used while resolving the StackTraceElement array. If stringified is set to true, elementTemplate will be discarded. By default, elementTemplate is set to null and rather populated from the layout configuration. That is, the stack trace element template can also be provided using stackTraceElementTemplate[Uri] layout configuration parameters. The template to be employed is determined in the following order:

  1. elementTemplate provided in the resolver configuration

  2. stackTraceElementTemplate parameter from layout configuration (the default is populated from log4j.layout.jsonTemplate.stackTraceElementTemplate system property)

  3. stackTraceElementTemplateUri parameter from layout configuration (the default is populated from log4j.layout.jsonTemplate.stackTraceElementTemplateUri system property)

See Stack Trace Element Templates for the list of available resolvers in a stack trace element template.

Note that this resolver is toggled by log4j.layout.jsonTemplate.stackTraceEnabled property.

Warning

Since Throwable#getStackTrace() clones the original StackTraceElement[], access to (and hence rendering of) stack traces are not garbage-free.

Each pointMatcherRegexes item triggers a Pattern#matcher() call, which is not garbage-free either.

Resolve logEvent.getThrown().getClass().getCanonicalName():

{
  "$resolver": "exception",
  "field": "className"
}

Resolve the stack trace into a list of StackTraceElement objects:

{
  "$resolver": "exception",
  "field": "stackTrace"
}

Resolve the stack trace into a string field:

{
  "$resolver": "exception",
  "field": "stackTrace",
  "stackTrace": {
    "stringified": true
  }
}

Resolve the stack trace into a string field such that the content will be truncated after the given point matcher:

{
  "$resolver": "exception",
  "field": "stackTrace",
  "stackTrace": {
    "stringified": {
      "truncation": {
        "suffix": "... [truncated]",
        "pointMatcherStrings": ["at javax.servlet.http.HttpServlet.service"]
      }
    }
  }
}

Resolve the stack trace into an object described by the provided stack trace element template:

{
  "$resolver": "exception",
  "field": "stackTrace",
  "stackTrace": {
    "elementTemplate": {
      "class": {
       "$resolver": "stackTraceElement",
       "field": "className"
      },
      "method": {
       "$resolver": "stackTraceElement",
       "field": "methodName"
      },
      "file": {
       "$resolver": "stackTraceElement",
       "field": "fileName"
      },
      "line": {
       "$resolver": "stackTraceElement",
       "field": "lineNumber"
      }
    }
  }
}

See Stack Trace Element Templates for further details on resolvers available for StackTraceElement templates.

exceptionRootCause

Resolves the fields of the innermost Throwable returned by logEvent.getThrown(). Its syntax and garbage-footprint are identical to the exception resolver.

level
config         = field , [ severity ]
field          = "field" -> ( "name" | "severity" )
severity       = severity-field
severity-field = "field" -> ( "keyword" | "code" )

Resolves the fields of the logEvent.getLevel().

Resolve the level name:

{
  "$resolver": "level",
  "field": "name"
}

Resolve the Syslog severity keyword:

{
  "$resolver": "level",
  "field": "severity",
  "severity": {
    "field": "keyword"
  }
}

Resolve the Syslog severity code:

{
  "$resolver": "level",
  "field": "severity",
  "severity": {
    "field": "code"
  }
}
logger
config = "field" -> ( "name" | "fqcn" )

Resolves logEvent.getLoggerFqcn() and logEvent.getLoggerName().

Resolve the logger name:

{
  "$resolver": "logger",
  "field": "name"
}

Resolve the logger’s fully qualified class name:

{
  "$resolver": "logger",
  "field": "fqcn"
}
main
config = ( index | key )
index  = "index" -> number
key    = "key" -> string

Performs Main Argument Lookup for the given index or key.

Resolve the 1st main() method argument:

{
  "$resolver": "main",
  "index": 0
}

Resolve the argument coming right after --userId:

{
  "$resolver": "main",
  "key": "--userId"
}
map

Resolves MapMessages. See Map Resolver Template for details.

mdc

Resolves Mapped Diagnostic Context (MDC), aka. Thread Context Data. See Map Resolver Template for details.

Warning

log4j2.garbagefreeThreadContextMap flag needs to be turned on to iterate the map without allocations.

message
config      = [ stringified ] , [ fallbackKey ]
stringified = "stringified" -> boolean
fallbackKey = "fallbackKey" -> string

Resolves logEvent.getMessage().

Warning

For simple string messages, the resolution is performed without allocations. For ObjectMessages and MultiformatMessages, it depends.

Resolve the message into a string:

{
  "$resolver": "message",
  "stringified": true
}

Resolve the message such that if it is an ObjectMessage or a MultiformatMessage with JSON support, its type (string, list, object, etc.) will be retained:

{
  "$resolver": "message"
}

Given the above configuration, a SimpleMessage will generate a "sample log message", whereas a MapMessage will generate a {"action": "login", "sessionId": "87asd97a"}. Certain indexed log storage systems (e.g., Elasticsearch) will not allow both values to coexist due to type mismatch: one is a string while the other is an object. Here one can use a fallbackKey to work around the problem:

{
  "$resolver": "message",
  "fallbackKey": "formattedMessage"
}

Using this configuration, a SimpleMessage will generate a {"formattedMessage": "sample log message"} and a MapMessage will generate a {"action": "login", "sessionId": "87asd97a"}. Note that both emitted JSONs are of type object and have no type-conflicting fields.

messageParameter
config      = [ stringified ] , [ index ]
stringified = "stringified" -> boolean
index       = "index" -> number

Resolves logEvent.getMessage().getParameters().

Warning

Regarding garbage footprint, stringified flag translates to String.valueOf(value), hence mind not-String-typed values. Further, logEvent.getMessage() is expected to implement ParameterVisitable interface, which is the case if log4j2.enableThreadLocals property set to true.

Resolve the message parameters into an array:

{
  "$resolver": "messageParameter"
}

Resolve the string representation of all message parameters into an array:

{
  "$resolver": "messageParameter",
  "stringified": true
}

Resolve the first message parameter:

{
  "$resolver": "messageParameter",
  "index": 0
}

Resolve the string representation of the first message parameter:

{
  "$resolver": "messageParameter",
  "index": 0,
  "stringified": true
}
ndc
config  = [ pattern ]
pattern = "pattern" -> string

Resolves the Nested Diagnostic Context (NDC), aka. Thread Context Stack, String[] returned by logEvent.getContextStack().

Resolve all NDC values into a list:

{
  "$resolver": "ndc"
}

Resolve all NDC values matching with the pattern regex:

{
  "$resolver": "ndc",
  "pattern": "user(Role|Rank):\\w+"
}
pattern
config            = pattern , [ stackTraceEnabled ]
pattern           = "pattern" -> string
stackTraceEnabled = "stackTraceEnabled" -> boolean

Resolver delegating to PatternLayout.

The default value of stackTraceEnabled is inherited from the parent JsonTemplateLayout.

Resolve the string produced by %p %c{1.} [%t] %X{userId} %X %m%ex pattern:

{
  "$resolver": "pattern",
  "pattern": "%p %c{1.} [%t] %X{userId} %X %m%ex"
}
source
config = "field" -> (
           "className"  |
           "fileName"   |
           "methodName" |
           "lineNumber" )

Resolves the fields of the StackTraceElement returned by logEvent.getSource().

Note that this resolver is toggled by log4j.layout.jsonTemplate.locationInfoEnabled property.

Resolve the line number:

{
  "$resolver": "source",
  "field": "lineNumber"
}
thread
config = "field" -> ( "name" | "id" | "priority" )

Resolves logEvent.getThreadId(), logEvent.getThreadName(), logEvent.getThreadPriority().

Resolve the thread name:

{
  "$resolver": "thread",
  "field": "name"
}
timestamp
config        = [ patternConfig | epochConfig ]

patternConfig = "pattern" -> ( [ format ] , [ timeZone ] , [ locale ] )
format        = "format" -> string
timeZone      = "timeZone" -> string
locale        = "locale" -> (
                   language                                   |
                 ( language , "_" , country )                 |
                 ( language , "_" , country , "_" , variant )
                )

epochConfig   = "epoch" -> ( unit , [ rounded ] )
unit          = "unit" -> (
                   "nanos"         |
                   "millis"        |
                   "secs"          |
                   "millis.nanos"  |
                   "secs.nanos"    |
                )
rounded       = "rounded" -> boolean

Resolves logEvent.getInstant() in various forms.

Table 2. timestamp template resolver examples

Configuration

Output

{
  "$resolver": "timestamp"
}

2020-02-07T13:38:47.098+02:00

{
  "$resolver": "timestamp",
  "pattern": {
    "format": "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",
    "timeZone": "UTC",
    "locale": "en_US"
  }
}

2020-02-07T13:38:47.098Z

{
  "$resolver": "timestamp",
  "epoch": {
    "unit": "secs"
  }
}

1581082727.982123456

{
  "$resolver": "timestamp",
  "epoch": {
    "unit": "secs",
    "rounded": true
  }
}

1581082727

{
  "$resolver": "timestamp",
  "epoch": {
    "unit": "secs.nanos"
  }
}

982123456

{
  "$resolver": "timestamp",
  "epoch": {
    "unit": "millis"
  }
}

1581082727982.123456

{
  "$resolver": "timestamp",
  "epoch": {
    "unit": "millis",
    "rounded": true
  }
}

1581082727982

{
  "$resolver": "timestamp",
  "epoch": {
    "unit": "millis.nanos"
  }
}

123456

{
  "$resolver": "timestamp",
  "epoch": {
    "unit": "nanos"
  }
}

1581082727982123456

Map Resolver Template

ReadOnlyStringMap is Log4j’s Map<String, Object> equivalent with garbage-free accessors and heavily employed throughout the code base. It is the data structure backing both Mapped Diagnostic Context (MDC), aka. Thread Context Data and MapMessage implementations. Hence template resolvers for both of these are provided by a single backend: ReadOnlyStringMapResolver. Put another way, both mdc and map resolvers support identical configuration, behaviour, and garbage footprint, which are detailed below.

config        = singleAccess | multiAccess

singleAccess  = key , [ stringified ]
key           = "key" -> string
stringified   = "stringified" -> boolean

multiAccess   = [ pattern ] , [ replacement ] , [ flatten ] , [ stringified ]
pattern       = "pattern" -> string
replacement   = "replacement" -> string
flatten       = "flatten" -> ( boolean | flattenConfig )
flattenConfig = [ flattenPrefix ]
flattenPrefix = "prefix" -> string

singleAccess resolves a single field, whilst multiAccess resolves a multitude of fields. If flatten is provided, multiAccess merges the fields with the parent, otherwise creates a new JSON object containing the values.

Enabling stringified flag converts each value to its string representation.

Regex provided in the pattern is used to match against the keys. If provided, replacement will be used to replace the matched keys. These two are effectively equivalent to Pattern.compile(pattern).matcher(key).matches() and Pattern.compile(pattern).matcher(key).replaceAll(replacement) calls.

Warning

Regarding garbage footprint, stringified flag translates to String.valueOf(value), hence mind not-String-typed values.

pattern and replacement incur pattern matcher allocation costs.

Writing certain non-primitive values (e.g., BigDecimal, Set, etc.) to JSON generates garbage, though most (e.g., int, long, String, List, boolean[], etc.) don’t.

"$resolver" is left out in the following examples, since it is to be defined by the actual resolver, e.g., map, mdc.

Resolve the value of the field keyed with user:role:

{
  "$resolver": "…",
  "key": "user:role"
}

Resolve the string representation of the user:rank field value:

{
  "$resolver": "…",
  "key": "user:rank",
  "stringified": true
}

Resolve all fields into an object:

{
  "$resolver": "…"
}

Resolve all fields into an object such that values are converted to string:

{
  "$resolver": "…",
  "stringified": true
}

Resolve all fields whose keys match with the user:(role|rank) regex into an object:

{
  "$resolver": "…",
  "pattern": "user:(role|rank)"
}

Resolve all fields whose keys match with the user:(role|rank) regex into an object after removing the user: prefix in the key:

{
  "$resolver": "…",
  "pattern": "user:(role|rank)",
  "replacement": "$1"
}

Merge all fields whose keys are matching with the user:(role|rank) regex into the parent:

{
  "$resolver": "…",
  "flatten": true,
  "pattern": "user:(role|rank)"
}

After converting the corresponding field values to string, merge all fields to parent such that keys are prefixed with _:

{
  "$resolver": "…",
  "stringified": true,
  "flatten": {
    "prefix": "_"
  }
}

Stack Trace Element Templates

exception and exceptionRootCause event template resolvers can serialize an exception stack trace (i.e., StackTraceElement[] returned by Throwable#getStackTrace()) into a JSON array. While doing so, JSON templating infrastructure is used again.

stackTraceElement[Uri] describes the JSON structure JsonTemplateLayout uses to format StackTraceElements. The default configuration (accessible by log4j.layout.jsonTemplate.stackTraceElementTemplate[Uri] property) is set to classpath:StackTraceElementLayout.json provided by the log4j-layout-template-json artifact:

{
  "class": {
    "$resolver": "stackTraceElement",
    "field": "className"
  },
  "method": {
    "$resolver": "stackTraceElement",
    "field": "methodName"
  },
  "file": {
    "$resolver": "stackTraceElement",
    "field": "fileName"
  },
  "line": {
    "$resolver": "stackTraceElement",
    "field": "lineNumber"
  }
}

The allowed template configuration syntax is as follows:

config = "field" -> (
           "className"  |
           "fileName"   |
           "methodName" |
           "lineNumber" )

All above accesses to StackTraceElement is garbage-free.

Extending

JsonTemplateLayout relies on Log4j plugin system to build up the features it provides. This enables feature customization a breeze for users. As of this moment, following features are implemented by means of plugins:

  • Event template resolvers (e.g., exception, message, level event template resolvers)

  • Event template interceptors (e.g., injection of eventTemplateAdditionalField)

  • Recycler factories

Following sections cover these in detail.

Plugin Preliminaries

Log4j plugin system is the de facto extension mechanism embraced by various Log4j components, including JsonTemplateLayout. Plugins make it possible for extensible components receive feature implementations without any explicit links in between. It is analogous to a dependency injection framework, but curated for Log4j-specific needs.

In a nutshell, you annotate your classes with @Plugin and their (static) creator methods with @PluginFactory. Last, you inform the Log4j plugin system to discover these custom classes. This can be done either using packages declared in your Log4j configuration or by various other ways described in plugin system documentation.

Extending Event Resolvers

All available event template resolvers are simple plugins employed by JsonTemplateLayout. To add new ones, one just needs to create their own EventResolver and instruct its injection via a @Plugin-annotated EventResolverFactory class.

For demonstration purposes, below we will create a randomNumber event resolver. Let’s start with the actual resolver:

Custom random number event resolver
package com.acme.logging.log4j.layout.template.json;

import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.layout.template.json.resolver.EventResolver;
import org.apache.logging.log4j.layout.template.json.util.JsonWriter;

/**
 * Resolves a random floating point number.
 *
 * <h3>Configuration</h3>
 *
 * <pre>
 * config = ( [ range ] )
 * range  = number[]
 * </pre>
 *
 * {@code range} is a number array with two elements, where the first number
 * denotes the start (inclusive) and the second denotes the end (exclusive).
 * {@code range} is optional and by default set to {@code [0, 1]}.
 *
 * <h3>Examples</h3>
 *
 * Resolve a random number between 0 and 1:
 *
 * <pre>
 * {
 *   "$resolver": "randomNumber"
 * }
 * </pre>
 *
 * Resolve a random number between -0.123 and 0.123:
 *
 * <pre>
 * {
 *   "$resolver": "randomNumber",
 *   "range": [-0.123, 0.123]
 * }
 * </pre>
 */
public final class RandomNumberResolver implements EventResolver {

    private final double loIncLimit;

    private final double hiExcLimit;

    RandomNumberResolver(final TemplateResolverConfig config) {
        final List<Number> rangeArray = config.getList("range", Number.class);
        if (rangeArray == null) {
            this.loIncLimit = 0D;
            this.hiExcLimit = 1D;
        } else if (rangeArray.size() != 2) {
            throw new IllegalArgumentException(
                    "range array must be of size two: " + config);
        } else {
            this.loIncLimit = rangeArray.get(0).doubleValue();
            this.hiExcLimit = rangeArray.get(1).doubleValue();
            if (loIncLimit > hiExcLimit) {
                throw new IllegalArgumentException("invalid range: " + config);
            }
        }
    }

    static String getName() {
        return "randomNumber";
    }

    @Override
    public void resolve(
            final LogEvent value,
            final JsonWriter jsonWriter) {
        final double randomNumber =
                loIncLimit + (hiExcLimit - loIncLimit) * Math.random();
        jsonWriter.writeNumber(randomNumber);
    }

}

Next create a EventResolverFactory class to register RandomNumberResolver into the Log4j plugin system.

Resolver factory class to register RandomNumberResolver into the Log4j plugin system
package com.acme.logging.log4j.layout.template.json;

import org.apache.logging.log4j.core.config.plugins.Plugin;
import org.apache.logging.log4j.core.config.plugins.PluginFactory;
import org.apache.logging.log4j.layout.template.json.resolver.EventResolverContext;
import org.apache.logging.log4j.layout.template.json.resolver.EventResolverFactory;
import org.apache.logging.log4j.layout.template.json.resolver.TemplateResolver;
import org.apache.logging.log4j.layout.template.json.resolver.TemplateResolverConfig;
import org.apache.logging.log4j.layout.template.json.resolver.TemplateResolverFactory;

/**
 * {@link RandomNumberResolver} factory.
 */
@Plugin(name = "RandomNumberResolverFactory", category = TemplateResolverFactory.CATEGORY)
public final class RandomNumberResolverFactory implements EventResolverFactory {

    private static final RandomNumberResolverFactory INSTANCE =
            new RandomNumberResolverFactory();

    private RandomNumberResolverFactory() {}

    @PluginFactory
    public static RandomNumberResolverFactory getInstance() {
        return INSTANCE;
    }

    @Override
    public String getName() {
        return RandomNumberResolver.getName();
    }

    @Override
    public RandomNumberResolver create(
            final EventResolverContext context,
            final TemplateResolverConfig config) {
        return new RandomNumberResolver(config);
    }

}

Almost complete. Last, we need to inform the Log4j plugin system to discover these custom classes:

Log4j configuration employing custom randomNumber resolver
<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="com.acme.logging.log4j.layout.template.json">
  <!-- ... -->
  <JsonTemplateLayout>
    <EventTemplateAdditionalField
        key="id"
        format="JSON"
        value='{"$resolver": "randomNumber", "range": [0, 1000000]}'/>
  </JsonTemplateLayout>
  <!-- ... -->
</Configuration>

All available event template resolvers are located in org.apache.logging.log4j.layout.template.json.resolver package. It is a fairly rich resource for inspiration while implementing new resolvers.

Intercepting the Template Resolver Compiler

JsonTemplateLayout allows interception of the template resolver compilation, which is the process converting a template into a Java function performing the JSON serialization. This interception mechanism is internally used to implement eventTemplateRootObjectKey and eventTemplateAdditionalField features. In a nutshell, one needs to create a @Plugin-annotated class extending from EventResolverInterceptor interface.

To see the interception in action, check out the EventRootObjectKeyInterceptor class which is responsible for implementing the eventTemplateRootObjectKey feature:

Event interceptor to add eventTemplateRootObjectKey, if present
import org.apache.logging.log4j.layout.template.json.resolver.EventResolverContext;
import org.apache.logging.log4j.layout.template.json.resolver.EventResolverInterceptor;
import org.apache.logging.log4j.layout.template.json.resolver.TemplateResolverInterceptor;

/**
 * Interceptor to add a root object key to the event template.
 */
@Plugin(name = "EventRootObjectKeyInterceptor", category = TemplateResolverInterceptor.CATEGORY)
public class EventRootObjectKeyInterceptor implements EventResolverInterceptor {

    private static final EventRootObjectKeyInterceptor INSTANCE =
            new EventRootObjectKeyInterceptor();

    private EventRootObjectKeyInterceptor() {}

    @PluginFactory
    public static EventRootObjectKeyInterceptor getInstance() {
        return INSTANCE;
    }

    @Override
    public Object processTemplateBeforeResolverInjection(
            final EventResolverContext context,
            final Object node) {
        String eventTemplateRootObjectKey = context.getEventTemplateRootObjectKey();
        return eventTemplateRootObjectKey != null
                ? Collections.singletonMap(eventTemplateRootObjectKey, node)
                : node;
    }

}

Here, processTemplateBeforeResolverInjection() method checks if the user has provided an eventTemplateRootObjectKey. If so, it wraps the root node with a new object; otherwise, returns the node as is. Note that node refers to the root Java object of the event template read by JsonReader.

Extending Recycler Factories

recyclerFactory input String read from the layout configuration is converted to a RecyclerFactory using the default RecyclerFactoryConverter extending from TypeConverter<RecyclerFactory>. If one wants to change this behavior, they simply need to add their own TypeConverter<RecyclerFactory> implementing Comparable<TypeConverter<?>> to prioritize their custom converter.

Custom TypeConverter for RecyclerFactory
package com.acme.logging.log4j.layout.template.json;

import org.apache.logging.log4j.core.config.plugins.Plugin;
import org.apache.logging.log4j.core.config.plugins.convert.TypeConverter;
import org.apache.logging.log4j.core.config.plugins.convert.TypeConverters;

@Plugin(name = "AcmeRecyclerFactoryConverter", category = TypeConverters.CATEGORY)
public final class AcmeRecyclerFactoryConverter
        implements TypeConverter<RecyclerFactory>, Comparable<TypeConverter<?>> {

    @Override
    public RecyclerFactory convert(final String recyclerFactorySpec) {
        return AcmeRecyclerFactory.ofSpec(recyclerFactorySpec);
    }

    @Override
    public int compareTo(final TypeConverter<?> ignored) {
        return -1;
    }

}

Here note that compareTo() always returns -1 to rank it higher compared to other matching converters.

Features

Below is a feature comparison matrix between JsonTemplateLayout and alternatives.

Table 3. Feature comparison matrix

Feature

JsonTemplateLayout

JsonLayout

GelfLayout

EcsLayout

Java version

8

8

8

6

Dependencies

None

Jackson

None

None

Schema customization?

Timestamp customization?

(Almost) garbage-free?

Custom typed Message serialization?

?[1]

Custom typed MDC value serialization?

Rendering stack traces as array?

Stack trace truncation?

JSON pretty print?

Additional string fields?

Additional JSON fields?

Custom resolvers?

F.A.Q.

Are lookups supported in templates?

Yes, lookups (e.g., ${java:version}, ${env:USER}, ${date:MM-dd-yyyy}) are supported in string literals of templates. Though note that they are not garbage-free.

Are recursive collections supported?

No. Consider a Message containing a recursive value as follows:

Object[] recursiveCollection = new Object[1];
recursiveCollection[0] = recursiveCollection;

While the exact exception might vary, you will most like get a StackOverflowError for trying to render recursiveCollection into a String. Note that this is also the default behaviour for other Java standard library methods, e.g., Arrays.toString(). Hence mind self references while logging.

Is JsonTemplateLayout garbage-free?

Yes, if the garbage-free layout behaviour toggling properties log4j2.enableDirectEncoders and log4j2.garbagefreeThreadContextMap are enabled. Take into account the following caveats:

  • The configured recycling strategy might not be garbage-free.

  • Since Throwable#getStackTrace() clones the original StackTraceElement[], access to (and hence rendering of) stack traces are not garbage-free.

  • Serialization of MapMessages and ObjectMessages are mostly garbage-free except for certain types (e.g., BigDecimal, BigInteger, Collections, except List).

  • Lookups (that is, ${…​} variables) are not garbage-free.

Don’t forget to check out the notes on garbage footprint of resolvers you employ in templates.


1. Only for ObjectMessages and if Jackson is in the classpath.