JSON Template Layout

JsonTemplateLayout is a customizable, efficient, and garbage-free JSON emitting layout. It encodes LogEvents according to the structure described by the JSON template provided. In a nutshell, it shines with its

  • Customizable JSON structure (see eventTemplate[Uri] and stackTraceElementTemplate[Uri] parameters)

  • Customizable timestamp formatting (see timestamp parameter)

Usage

Adding log4j-layout-template-json artifact to your list of dependencies is enough to enable access to JsonTemplateLayout in your Log4j configuration:

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-layout-template-json</artifactId>
    <version>2.14.0</version>
</dependency>

For instance, given the following JSON template modelling the the official Logstash JSONEventLayoutV1 (accessible via classpath:LogstashJsonEventLayoutV1.json)

{
  "mdc": {
    "$resolver": "mdc"
  },
  "exception": {
    "exception_class": {
      "$resolver": "exception",
      "field": "className"
    },
    "exception_message": {
      "$resolver": "exception",
      "field": "message",
      "stringified": true
    },
    "stacktrace": {
      "$resolver": "exception",
      "field": "stackTrace",
      "stringified": true
    }
  },
  "line_number": {
    "$resolver": "source",
    "field": "lineNumber"
  },
  "class": {
    "$resolver": "source",
    "field": "className"
  },
  "@version": 1,
  "source_host": "${hostName}",
  "message": {
    "$resolver": "message",
    "stringified": true
  },
  "thread_name": {
    "$resolver": "thread",
    "field": "name"
  },
  "@timestamp": {
    "$resolver": "timestamp"
  },
  "level": {
    "$resolver": "level",
    "field": "name"
  },
  "file": {
    "$resolver": "source",
    "field": "fileName"
  },
  "method": {
    "$resolver": "source",
    "field": "methodName"
  },
  "logger_name": {
    "$resolver": "logger",
    "field": "name"
  }
}

in combination with the below log4j2.xml configuration:

<JsonTemplateLayout eventTemplateUri="classpath:LogstashJsonEventLayoutV1.json"/>

or with the below log4j2.properties configuration:

appender.console.json.type = JsonTemplateLayout
appender.console.json.eventTemplateUri = classpath:LogstashJsonEventLayoutV1.json

JsonTemplateLayout emits JSON strings as follows:

{
  "exception": {
    "exception_class": "java.lang.RuntimeException",
    "exception_message": "test",
    "stacktrace": "java.lang.RuntimeException: test\n\tat org.apache.logging.log4j.JsonTemplateLayoutDemo.main(JsonTemplateLayoutDemo.java:11)\n"
  },
  "line_number": 12,
  "class": "org.apache.logging.log4j.JsonTemplateLayoutDemo",
  "@version": 1,
  "source_host": "varlik",
  "message": "Hello, error!",
  "thread_name": "main",
  "@timestamp": "2017-05-25T19:56:23.370+02:00",
  "level": "ERROR",
  "file": "JsonTemplateLayoutDemo.java",
  "method": "main",
  "logger_name": "org.apache.logging.log4j.JsonTemplateLayoutDemo"
}

Layout Configuration

JsonTemplateLayout is configured with the following parameters:

Table 1. JsonTemplateLayout parameters

Parameter Name

Type

Description

charset

Charset

Charset used for String encoding

locationInfoEnabled

boolean

toggles access to the LogEvent source; file name, line number, etc. (defaults to false set by log4j.layout.jsonTemplate.locationInfoEnabled property)

stackTraceEnabled

boolean

toggles access to the stack traces (defaults to true set by log4j.layout.jsonTemplate.stackTraceEnabled property)

eventTemplate

String

inline JSON template for rendering LogEvents (has priority over eventTemplateUri, defaults to null set by log4j.layout.jsonTemplate.eventTemplate property)

eventTemplateUri

String

URI pointing to the JSON template for rendering LogEvents (defaults to classpath:EcsLayout.json set by log4j.layout.jsonTemplate.eventTemplateUri property)

eventTemplateAdditionalFields

EventTemplateAdditionalField[]

additional key-value pairs appended to the root of the event template

stackTraceElementTemplate

String

inline JSON template for rendering StackTraceElements (has priority over stackTraceElementTemplateUri, defaults to null set by log4j.layout.jsonTemplate.stackTraceElementTemplate property)

stackTraceElementTemplateUri

String

JSON template for rendering StackTraceElements (defaults to classpath:StackTraceElementLayout.json set by log4j.layout.jsonTemplate.stackTraceElementTemplateUri property)

eventDelimiter

String

delimiter used for separating emitted LogEvents (defaults to System.lineSeparator() set by log4j.layout.jsonTemplate.eventDelimiter property)

nullEventDelimiterEnabled

boolean

append \0 (null) character to the end of every emitted eventDelimiter (defaults to false set by log4j.layout.jsonTemplate.nullEventDelimiterEnabled property)

maxStringLength

int

truncate string values longer than the specified limit (defaults to 16384 set by log4j.layout.jsonTemplate.maxStringLength property)

truncatedStringSuffix

String

suffix to append to strings truncated due to exceeding maxStringLength (defaults to set by log4j.layout.jsonTemplate.truncatedStringSuffix property)

recyclerFactory

RecyclerFactory

recycling strategy that can either be dummy, threadLocal, or queue (set by log4j.layout.jsonTemplate.recyclerFactory property)

Additonal event template fields

Additional event template field is a convenient short-cut to add custom fields to a template or override the fields of a template. Following configuration overrides the host field of the GelfLayout.json template and adds two new custom fields:

<JsonTemplateLayout eventTemplateUri="classpath:GelfLayout.json">
    <EventTemplateAdditionalFields>
        <EventTemplateAdditionalField key="host" value="www.apache.org"/>
        <EventTemplateAdditionalField key="_serviceName" value="auth-service"/>
        <EventTemplateAdditionalField key="_containerId" value="6ede3f0ca7d9"/>
    </EventTemplateAdditionalFields>
</JsonTemplateLayout>

One can also pass JSON literals into additional fields:

<EventTemplateAdditionalField
     key="marker"
     type="JSON"
     value='{"$resolver": "marker", "field": "name"}'/>
<EventTemplateAdditionalField
     key="aNumber"
     type="JSON"
     value="1"/>
<EventTemplateAdditionalField
     key="aList"
     type="JSON"
     value='[1,2,"string"]'/>

Recycling strategy

RecyclerFactory plays a crucial role for determining the memory footprint of the layout. Template resolvers employ it to create recyclers for objects that they can reuse. The function of each RecyclerFactory and when one should prefer one over another is explained below:

  • dummy performs no recycling, hence each recycling attempt will result in a new instance. This will obviously create a load on the garbage-collector. It is a good choice for applications with low and medium log rate.

  • threadLocal performs the best, since every instance is stored in ThreadLocals and accessed without any synchronization cost. Though this might not be a desirable option for applications running with hundreds of threads or more, e.g., a web servlet.

  • queue is the best of both worlds. It allows recycling of objects up to a certain number (capacity). When this limit is exceeded due to excessive concurrent load (e.g., capacity is 50 but there are 51 threads concurrently trying to log), it starts allocating. queue is a good strategy where threadLocal is not desirable.

    queue also accepts optional supplier (of type java.util.Queue, defaults to org.jctools.queues.MpmcArrayQueue.new if JCTools is in the classpath; otherwise java.util.concurrent.ArrayBlockingQueue.new) and capacity (of type int, defaults to max(8,2*cpuCount+1)) parameters:

    queue:supplier=org.jctools.queues.MpmcArrayQueue.new
    queue:capacity=10
    queue:supplier=java.util.concurrent.ArrayBlockingQueue.new,capacity=50

The default RecyclerFactory is threadLocal, if log4j2.enable.threadlocals=true; otherwise, queue.

Template Configuration

Templates are configured by means of the following JsonTemplateLayout parameters:

  • eventTemplate[Uri] (for serializing LogEvents)

  • stackTraceElementTemplate[Uri] (for serializing StackStraceElements)

  • eventTemplateAdditionalFields (for extending the used event template)

Event Templates

eventTemplate[Uri] describes the JSON structure JsonTemplateLayout uses to serialize LogEvents. The default configuration (accessible by log4j.layout.jsonTemplate.eventTemplate[Uri] property) is set to classpath:EcsLayout.json provided by the log4j-layout-template-json artifact:

{
  "@timestamp": {
    "$resolver": "timestamp",
    "pattern": {
      "format": "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",
      "timeZone": "UTC"
    }
  },
  "log.level": {
    "$resolver": "level",
    "field": "name"
  },
  "message": {
    "$resolver": "message",
    "stringified": true
  },
  "process.thread.name": {
    "$resolver": "thread",
    "field": "name"
  },
  "log.logger": {
    "$resolver": "logger",
    "field": "name"
  },
  "labels": {
    "$resolver": "mdc",
    "flatten": true,
    "stringified": true
  },
  "tags": {
    "$resolver": "ndc"
  },
  "error.type": {
    "$resolver": "exception",
    "field": "className"
  },
  "error.message": {
    "$resolver": "exception",
    "field": "message"
  },
  "error.stack_trace": {
    "$resolver": "exception",
    "field": "stackTrace",
    "stringified": true
  }
}

log4j-layout-template-json artifact contains the following predefined event templates:

Below is the list of supported event template resolvers:

Table 2. LogEvent template resolvers

Resolver Name

Syntax

Description

Garbage Footprint

Examples

endOfBatch

logEvent.isEndOfBatch()

none

{
  "$resolver": "endOfBatch"
}

exception

config      = field , [ stringified ]
field       = "field" -> (
                "className"  |
                "message"    |
                "stackTrace" )
stringified = "stringified" -> boolean

Resolves fields of the Throwable returned by logEvent.getThrown().

Note that this resolver is toggled by log4j.layout.jsonTemplate.stackTraceEnabled property.

Since Throwable#getStackTrace() clones the original StackTraceElement[], access to (and hence rendering of) stack traces are not garbage-free.

Resolve logEvent.getThrown().getClass().getCanonicalName():

{
  "$resolver": "exception",
  "field": "className"
}

Resolve the stack trace into a list of StackTraceElement objects:

{
  "$resolver": "exception",
  "field": "stackTrace"
}

Resolve the stack trace into a string field:

{
  "$resolver": "exception",
  "field": "stackTrace",
  "stringified": true
}

exceptionRootCause

identical to exception resolver

Resolves the fields of the innermost Throwable returned by logEvent.getThrown().

Note that this resolver is toggled by log4j.layout.jsonTemplate.stackTraceEnabled property.

identical to exception resolver

identical to exception resolver

level

config         = field , [ severity ]
field          = "field" -> ( "name" | "severity" )
severity       = severity-field
severity-field = "field" -> ( "keyword" | "code" )

resolves the fields of the logEvent.getLevel()

none

Resolve the level name:

{
  "$resolver": "level",
  "field": "name"
}

Resolve the Syslog severity keyword:

{
  "$resolver": "level",
  "field": "severity",
  "severity": {
    "field": "keyword"
  }
}

Resolve the Syslog severity code:

{
  "$resolver": "level",
  "field": "severity",
  "severity": {
    "field": "code"
  }
}

logger

config = "field" -> ( "name" | "fqcn" )

resolves logEvent.getLoggerFqcn() and logEvent.getLoggerName()

none

Resolve the logger name:

{
  "$resolver": "logger",
  "field": "name"
}

Resolve the logger’s fully qualified class name:

{
  "$resolver": "logger",
  "field": "fqcn"
}

main

config = ( index | key )
index  = "index" -> number
key    = "key" -> string

performs Main Argument Lookup for the given index or key

none

Resolve the 1st main() method argument:

{
  "$resolver": "main",
  "index": 0
}

Resolve the argument coming right after --userId:

{
  "$resolver": "main",
  "key": "--userId"
}

map

config      = key , [ stringified ]
key         = "key" -> string
stringified = "stringified" -> boolean

resolves the given key of MapMessages

stringified flag translates to String.valueOf(value), hence mind not-String-typed values.

Resolve the userRole field of the message:

{
  "$resolver": "map",
  "key": "userRole"
}

marker

config = "field" -> "name"

logEvent.getMarker().getName()

none

Resolve the marker name:

{
  "$resolver": "marker",
  "field": "name"
}

mdc

config        = singleAccess | multiAccess

singleAccess  = key , [ stringified ]
key           = "key" -> string
stringified   = "stringified" -> boolean

multi-access  = [ pattern ] , [ flatten ] , [ stringified ]
pattern       = "pattern" -> string
flatten       = "flatten" -> ( boolean | flattenConfig )
flattenConfig = [ flattenPrefix ]
flattenPrefix = "prefix" -> string

Mapped Diagnostic Context (MDC), aka. Thread Context Data, resolver.

singleAccess resolves the MDC value as is, whilst multiAccess resolves a multitude of MDC values. If flatten is provided, multiAccess merges the values with the parent, otherwise creates a new JSON object containing the values.

Enabling stringified flag converts each value to its string representation.

Regex provided in the pattern is used to match against the keys.

log4j2.garbagefreeThreadContextMap flag needs to be turned on to iterate the map without allocations.

stringified allocates a new String for values that are not of type String.

Writing certain non-primitive values (e.g., BigDecimal, Set, etc.) to JSON generates garbage, though most (e.g., int, long, String, List, boolean[], etc.) don’t.

Resolve the userRole MDC value:

{
  "$resolver": "mdc",
  "key": "userRole"
}

Resolve the string representation of the userRank MDC value:

{
  "$resolver": "mdc",
  "key": "userRank",
  "stringified": true
}

Resolve all MDC entries into an object:

{
  "$resolver": "mdc"
}

Resolve all MDC entries into an object such that values are converted to string:

{
  "$resolver": "mdc",
  "stringified": true
}

Merge all MDC entries whose keys are matching with the user(Role|Rank) regex into the parent:

{
  "$resolver": "mdc",
  "flatten": true,
  "pattern": "user(Role|Rank)"
}

After converting the corresponding entries to string, merge all MDC entries to parent such that keys are prefixed with _:

{
  "$resolver": "mdc",
  "stringified": true,
  "flatten": {
    "prefix": "_"
  }
}

message

config      = [ stringified ] , [ fallbackKey ]
pattern = "pattern" -> string
includeStackTrace = "includeStacktrae" -> boolean
stringified = "stringified" -> boolean
fallbackKey = "fallbackKey" -> string

logEvent.getMessage()

For simple string messages, the resolution is performed without allocations. For ObjectMessages and MultiformatMessages, it depends.

Resolve the message into a string:

{
  "$resolver": "message",
  "stringified": true
}

Resolve the message into a string using a pattern:

{
  "$resolver": "message",
  "pattern": ""[%t] %-5p %X{requestId, sessionId, loginId, userId, ipAddress, corpAcctNumber} %C{1.}.%M:%L - %m"",
  "stringified": true
}

Resolve the message such that if it is an ObjectMessage or a MultiformatMessage with JSON support, its type (string, list, object, etc.) will be retained:

{
  "$resolver": "message"
}

Given the above configuration, a SimpleMessage will generate a "sample log message", whereas a MapMessage will generate a {"action": "login", "sessionId": "87asd97a"}. Certain indexed log storage systems (e.g., Elasticsearch) will not allow both values to coexist due to type mismatch: one is a string while the other is an object. Here one can use a fallbackKey to work around the problem:

{
  "$resolver": "message",
  "fallbackKey": "formattedMessage"
}

Using this configuration, a SimpleMessage will generate a {"formattedMessage": "sample log message"} and a MapMessage will generate a {"action": "login", "sessionId": "87asd97a"}. Note that both emitted JSONs are of type object and have no type-conflicting fields.

messageParameter

config      = [ stringified ] , [ index ]
stringified = "stringified" -> boolean
index       = "index" -> number

logEvent.getMessage().getParameters()

stringified flag translates to String.valueOf(value), hence mind not-String-typed values. Further, logEvent.getMessage() is expected to implement ParameterVisitable interface, which is the case if log4j2.enableThreadlocals property set to true.

Resolve the message parameters into an array:

{
  "$resolver": "messageParameter"
}

Resolve the string representation of all message parameters into an array:

{
  "$resolver": "messageParameter",
  "stringified": true
}

Resolve the first message parameter:

{
  "$resolver": "messageParameter",
  "index": 0
}

Resolve the string representation of the first message parameter:

{
  "$resolver": "messageParameter",
  "index": 0,
  "stringified": true
}

ndc

config  = [ pattern ]
pattern = "pattern" -> string

Resolves the Nested Diagnostic Context (NDC), aka. Thread Context Stack, String[] returned by logEvent.getContextStack()

none

Resolve all NDC values into a list:

{
  "$resolver": "ndc"
}

Resolve all NDC values matching with the pattern regex:

{
  "$resolver": "ndc",
  "pattern": "user(Role|Rank):\\w+"
}

pattern

config            = pattern , [ stackTraceEnabled ]
pattern           = "pattern" -> string
stackTraceEnabled = "stackTraceEnabled" -> boolean

Resolver delegating to PatternLayout.

The default value of stackTraceEnabled is inherited from the parent JsonTemplateLayout.

none

Resolve the string produced by %p %c{1.} [%t] %X{userId} %X %m%ex pattern:

{
  "$resolver": "pattern",
  "pattern": "%p %c{1.} [%t] %X{userId} %X %m%ex"
}

source

config = "field" -> (
           "className"  |
           "fileName"   |
           "methodName" |
           "lineNumber" )

Resolves the fields of the StackTraceElement returned by logEvent.getSource().

Note that this resolver is toggled by log4j.layout.jsonTemplate.locationInfoEnabled property.

none

Resolve the line number:

{
  "$resolver": "source",
  "field": "lineNumber"
}

thread

config = "field" -> ( "name" | "id" | "priority" )

resolves logEvent.getThreadId(), logEvent.getThreadName(), logEvent.getThreadPriority()

none

Resolve the thread name:

{
  "$resolver": "thread",
  "field": "name"
}

timestamp

config        = [ patternConfig | epochConfig ]

patternConfig = "pattern" -> (
                  [ format ]   ,
                  [ timeZone ] ,
                  [ locale ]   )
format        = "format" -> string
timeZone      = "timeZone" -> string
locale        = "locale" -> (
                   language                                   |
                 ( language , "_" , country )                 |
                 ( language , "_" , country , "_" , variant )
               )

epochConfig   = "epoch" -> ( unit , [ rounded ] )
unit          = "unit" -> (
                   "nanos"         |
                   "millis"        |
                   "secs"          |
                   "millis.nanos"  |
                   "secs.nanos"    |
                )
rounded       = "rounded" -> boolean

resolves logEvent.getInstant() in various forms

none

Table 3. timestamp template resolver examples

Configuration

Output

{
  "$resolver": "timestamp"
}

2020-02-07T13:38:47.098+02:00

{
  "$resolver": "timestamp",
  "pattern": {
    "format": "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",
    "timeZone": "UTC",
    "locale": "en_US"
  }
}

2020-02-07T13:38:47.098Z

{
  "$resolver": "timestamp",
  "epoch": {
    "unit": "secs"
  }
}

1581082727.982123456

{
  "$resolver": "timestamp",
  "epoch": {
    "unit": "secs",
    "rounded": true
  }
}

1581082727

{
  "$resolver": "timestamp",
  "epoch": {
    "unit": "secs.nanos"
  }
}

982123456

{
  "$resolver": "timestamp",
  "epoch": {
    "unit": "millis"
  }
}

1581082727982.123456

{
  "$resolver": "timestamp",
  "epoch": {
    "unit": "millis",
    "rounded": true
  }
}

1581082727982

{
  "$resolver": "timestamp",
  "epoch": {
    "unit": "millis.nanos"
  }
}

123456

{
  "$resolver": "timestamp",
  "epoch": {
    "unit": "nanos"
  }
}

1581082727982123456

Stack Trace Element Templates

stackTraceElement[Uri] describes the JSON structure JsonTemplateLayout uses to format StackTraceElements. The default configuration (accessible by log4j.layout.jsonTemplate.stackTraceElementTemplate[Uri] property) is set to classpath:StackTraceElementLayout.json provided by the log4j-layout-template-json artifact:

{
  "class": {
    "$resolver": "stackTraceElement",
    "field": "className"
  },
  "method": {
    "$resolver": "stackTraceElement",
    "field": "methodName"
  },
  "file": {
    "$resolver": "stackTraceElement",
    "field": "fileName"
  },
  "line": {
    "$resolver": "stackTraceElement",
    "field": "lineNumber"
  }
}

The allowed template configuration syntax is as follows:

config = "field" -> (
           "className"  |
           "fileName"   |
           "methodName" |
           "lineNumber" )

All above accesses to StackTraceElement is garbage-free.

Features

Below is a feature comparison matrix between JsonTemplateLayout and alternatives.

Table 4. Feature comparison matrix

Feature

JsonTemplateLayout

JsonLayout

GelfLayout

EcsLayout

Java version

8

8

8

6

Dependencies

None

Jackson

None

None

Schema customization?

Timestamp customization?

(Almost) garbage-free?

Custom typed Message serialization?

?[1]

Custom typed MDC value serialization?

Rendering stack traces as array?

JSON pretty print?

Additional field support?

F.A.Q.

Are lookups supported in templates?

Yes, lookups (e.g., ${java:version}, ${env:USER}, ${date:MM-dd-yyyy}) are supported in string literals of templates. Though note that they are not garbage-free.

Is JsonTemplateLayout garbage-free?

Yes, if the garbage-free layout behaviour toggling properties log4j2.enableDirectEncoders and log4j2.garbagefreeThreadContextMap are enabled. Take into account the following caveats:

  • The configured recycling strategy might not be garbage-free.

  • Since Throwable#getStackTrace() clones the original StackTraceElement[], access to (and hence rendering of) stack traces are not garbage-free.

  • Serialization of MapMessages and ObjectMessages are mostly garbage-free except for certain types (e.g., BigDecimal, BigInteger, Collections with the exception of List).

  • Lookups (that is, ${…​} variables) are not garbage-free.

Don’t forget to checkout the notes on garbage footprint of resolvers you employ in templates.


1. Only for ObjectMessages and if Jackson is in the classpath.