Delegating Appenders
Log4j Core supplies multiple appenders that do not perform any work themselves, but modify the way other appenders work. The following behaviors can be modified:
-
If you want to perform all I/O from a dedicated thread, see
AsyncAppender. -
If you want to provide a backup appender in case an appender fails, see
FailoverAppender. -
If you want to modify the log event, before it is sent to the target destination, see
RewriteAppender. -
If you want to create appenders dynamically or choose a different appender for each log event, see
RoutingAppender.
Async Appender
The Async Appender stores log events in a blocking queue and forwards them to other appenders on a separate thread.
Due to the asynchronous barrier, exceptions occurring in those appenders will not be forwarded to the caller of the log statement.
The Async should be configured after the appenders it references to allow it to shut down properly.
|
The blocking queue is susceptible to lock contention, and performance may become worse when more threads are logging concurrently. Consider using lock-free asynchronous loggers instead, for optimal performance. |
|
Log4j 2 brought the following enhancements to the Log4j 1 async appender:
|
Async configuration
The Async Appender supports the following configuration options:
| Attribute | Type | Default value | Description | ||
|---|---|---|---|---|---|
Required |
|||||
|
The name of the appender. |
||||
Optional |
|||||
|
|
If If false, the event will be written to the error appender if the queue is full. The default is true. |
|||
|
1024 |
Specifies the maximum number of events that can be queued. When using a disruptor-style When the application is logging faster than the underlying appender can keep up with for a long enough time to fill up the queue, the behavior is determined by the Queue full policy. |
|||
String |
The name of the appender to invoke if none of the appenders can be called, either due to exceptions in the appenders or because the queue is full. If not specified then errors will be ignored. |
||||
boolean |
|
If set to See location information for more information. |
|||
|
|
If Logging exceptions are always also logged to Status Logger
|
|||
|
|
Timeout in milliseconds to wait before stopping the asynchronous thread. A value of |
|||
| Type | Multiplicity | Description |
|---|---|---|
zero or one |
Allows filtering log events just before they are appended to the blocking queue. See also appender filtering stage. |
|
one or more |
A list of appenders to invoke asynchronously. See appender references for more information. |
|
zero or one |
The blocking queue factory implementation to use. If not provided, See Blocking Queue Factories below. |
As an example, you can instrument a File appender to perform asynchronous I/O, by using the following appender configurations:
-
XML
-
JSON
-
YAML
-
Properties
log4j2.xml<File name="FILE"
fileName="app.log">
<JsonTemplateLayout/>
</File>
<Async name="ASYNC">
<AppenderRef ref="FILE"/>
</Async>
log4j2.json"File": {
"name": "FILE",
"fileName": "app.log"
},
"Async": {
"name": "ASYNC"
}
log4j2.yamlFile:
name: "FILE"
fileName: "app.log"
Async:
name: "ASYNC"
AppenderRef:
ref: "FILE"
log4j2.propertiesappender.0.type = File
appender.0.name = FILE
appender.0.fileName = app.log
appender.1.type = Async
appender.1.name = ASYNC
appender.1.appenderRef.type = AppenderRef
appender.1.appenderRef.ref = FILE
Queue full policy
When the queue is full the Async Appender uses an
AsyncQueueFullPolicy
to decide whether to:
-
drop the log event.
-
busy wait until the log event can be added to the queue.
-
log the event on the current thread.
The queue full policy can only be configured through configuration properties. See Async components for more details.
Blocking Queue Factories
The Async appender allows you to customize the blocking queue used by specifying a nested
BlockingQueueFactory
element.
You can specify the size of the queue using the bufferSize configuration attribute.
ArrayBlockingQueue-
This is the default implementation that produces
ArrayBlockingQueues.
DisruptorBlockingQueue-
This queue factory uses the Conversant Disruptor implementation of
BlockingQueue.Table 3. DisruptorBlockingQueueFactory configuration attributesAttribute Type Default value Description spinPolicyThe
SpinPolicyto apply, when adding elements to the queue.Additional dependencies are required to use
DisruptorBlockingQueue-
Maven
-
Gradle
<dependency> <groupId>com.conversantmedia</groupId> <artifactId>disruptor</artifactId> <version>1.2.21</version> <scope>runtime</scope> </dependency>runtimeOnly 'com.conversantmedia:disruptor:1.2.21' -
JCToolsBlockingQueue-
This queue factory uses JCTools, specifically the MPSC bounded lock-free queue.
Additional dependencies are required to use
JCToolsBlockingQueue-
Maven
-
Gradle
<dependency> <groupId>org.jctools</groupId> <artifactId>jctools-core</artifactId> <version>4.0.5</version> <scope>runtime</scope> </dependency>runtimeOnly 'org.jctools:jctools-core:4.0.5' -
LinkedTransferQueue-
This queue factory produces
LinkedTransferQueues. Note that this queue does not have a maximum capacity and ignores thebufferSizeattribute.
Failover Appender
The Failover Appender can protect your logging pipeline against I/O exceptions in other appenders.
During normal operations the Failover Appender forwards all log events to a primary appender.
However, if the primary appender fails, a set of secondary appenders will be checked until one succeeds.
Failover configuration
The Failover Appender supports the following configuration options:
| Attribute | Type | Default value | Description |
|---|---|---|---|
Required |
|||
|
The name of this appender. |
||
|
The name of the primary appender to use. |
||
Optional |
|||
|
|
It specifies how many seconds to wait after a failure of the primary appender before the primary appender can be used again. |
|
|
|
If Logging exceptions are always also logged to Status Logger |
|
| Type | Multiplicity | Description |
|---|---|---|
zero or one |
Allows filtering log events just before they are appended to the blocking queue. See also appender filtering stage. |
|
one |
A container element for a list of
|
|
The primary appender must be configured to forward exceptions to the caller, by setting the
|
The following example shows how to configure Failover to use an appender named FILE as primary and fall back to CONSOLE if an error occurs:
-
XML
-
JSON
-
YAML
-
Properties
log4j2.xml<File name="FILE"
fileName="app.log"
ignoreExceptions="false"/> (1)
<Console name="CONSOLE"/>
<Failover name="FAILOVER"
primary="FILE">
<Failovers>
<AppenderRef ref="CONSOLE"/>
</Failovers>
</Failover>
log4j2.json"File": {
"name": "FILE",
"fileName": "app.log",
"ignoreExceptions": false (1)
},
"Console": {
"name": "CONSOLE"
},
"Failover": {
"name": "FAILOVER",
"primary": "FILE",
"Failovers": {
"AppenderRef": {
"ref": "CONSOLE"
}
}
}
log4j2.yamlFile:
name: "FILE"
fileName: "app.log"
ignoreExceptions: false
Console:
name: "CONSOLE"
Failover:
name: "FAILOVER"
primary: "FILE"
Failovers:
AppenderRef:
ref: "CONSOLE"
log4j2.propertiesappender.0.type = File
appender.0.name = FILE
appender.0.fileName = app.log
appender.0.ignoreExceptions = false
appender.1.type = Console
appender.1.name = CONSOLE
appender.2.type = Failover
appender.2.name = FAILOVER
appender.2.primary = FILE
appender.2.fail.type = Failovers
appender.2.fail.0.type = AppenderRef
appender.2.fail.0.ref = CONSOLE
| 1 | The primary appender must set ignoreExceptions to false. |
Rewrite Appender
The Rewrite allows the log events to be manipulated before they are processed by another Appender.
This can be used to inject additional information into each event.
|
Although this appender can be used to mask sensitive information contained in log events, we strongly discourage such practice. Sensitive data like passwords and credit card numbers can appear in log files in many formats, and it is challenging to detect them all. A better approach to sensitive data is not to log them at all.
Third-party frameworks like
Palantir |
Rewrite Configuration
| Attribute | Type | Default value | Description |
|---|---|---|---|
Required |
|||
|
The name of this appender. |
||
Optional |
|||
|
|
If Logging exceptions are always also logged to Status Logger |
|
| Type | Multiplicity | Description |
|---|---|---|
one |
The reference to an appender that will perform the actual logging. |
|
zero or one |
Allows filtering log events just before they are appended to the blocking queue. See also appender filtering stage. |
|
one |
The rewrite policy to apply to all logged events. |
Rewrite Policies
A rewrite policy is a Log4j plugin that implements the
RewritePolicy
interface.
Rewrite policies allow to apply arbitrary modifications to log events.
Log4j Core provides three rewrite policies out-of-the-box:
MapRewritePolicy-
The
MapRewritePolicyonly modifies events that contain aMapMessage. It allows adding or updating the keys of theMapMessage.Table 8. MapRewritePolicyconfiguration attributesAttribute Type Default value Description It determines which map entries to modify:
- Add
-
All the configured map entries will be added to the
MapMessage, modifying the existing ones. - Update
-
The rewrite policy will add to the
MapMessageonly entries corresponding to existing keys.
Table 9. MapRewritePolicynested elementsTypeMultiplicity
Description
one or more
A list of map entries to add to the
MapMessage.
PropertiesRewritePolicy-
The
PropertiesRewritePolicywill add properties to the context data of the log event.Only the context data of the log event will be modified. The contents of the Thread Context will remain unchanged.
Table 10. PropertiesRewritePolicynested elementsType Multiplicity Description one or more
A list of map entries to add to the context data of the log event.
The
valueattribute of eachPropertyelement supports runtime property substitution in the global context.
- LoggerNameLevelRewritePolicy
-
You can use this policy to change the log level of loggers from third-party libraries. The
LoggerNameLevelRewritePolicywill rewrite the level of log event for a given logger name prefix.The new log levels will only be used by the filter attached to the
Rewriteappender and those downstream of the appender. Filters configured on loggers will use the previous levels. See Filters for more details on filteringTable 11. LoggerNameLevelRewritePolicyconfiguration attributesAttribute Type Default value Description StringThe rewrite policy will only be applied to loggers with this logger name and their children.
Table 12. LoggerNameLevelRewritePolicynested elementsType Multiplicity Description one or more
Provides a mapping between old level names and new level names.
Configuration example
If a library org.example over-evaluates the severity of its log events, you decrease their severity with the following configuration:
-
XML
-
JSON
-
YAML
-
Properties
log4j2.xml<Rewrite name="REWRITE">
<LoggerNameLevelRewritePolicy logger="org.example"> (1)
<KeyValuePair key="WARN" value="INFO"/>
<KeyValuePair key="INFO" value="DEBUG"/>
</LoggerNameLevelRewritePolicy>
<AppenderRef level="INFO" ref="CONSOLE"/> (2)
</Rewrite>
log4j2.json"Rewrite": {
"name": "REWRITE",
"LoggerNameLevelRewritePolicy": { (1)
"logger": "org.example",
"KeyValuePair": [
{
"key": "WARN",
"value": "INFO"
},
{
"key": "INFO",
"value": "DEBUG"
}
]
},
"AppenderRef": {
"level": "INFO", (2)
"ref": "CONSOLE"
}
}
log4j2.yamlRewrite:
name: "REWRITE"
LoggerNameLevelRewritePolicy: (1)
logger: "org.example"
KeyValuePair:
- key: "WARN"
value: "INFO"
- key: "INFO"
value: "DEBUG"
AppenderRef:
level: "INFO" (2)
ref: "CONSOLE"
log4j2.propertiesappender.1.type = Rewrite
appender.1.name = REWRITE
(1)
appender.1.policy.type = LoggerNameLevelRewritePolicy
appender.1.policy.logger = org.example
appender.1.policy.kv0.type = KeyValuePair
appender.1.policy.kv0.key = WARN
appender.1.policy.kv0.value = INFO
appender.1.policy.kv1.type = KeyValuePair
appender.1.policy.kv1.key = INFO
appender.1.policy.kv1.value = DEBUG
appender.1.appenderRef.type = AppenderRef
(2)
appender.1.appenderRef.level = INFO
appender.1.appenderRef.ref = CONSOLE
| 1 | Decreases the severity of WARN and INFO messages, so they appear with the new severity in your log viewer. |
| 2 | If additionally you don’t want to log DEBUG log events, you must apply a filter. |
Routing Appender
The Routing Appender evaluates log events and then routes them to one of its subordinate appenders.
The target appender may be:
-
an existing appender referenced by its name.
-
a new appender obtained by evaluating a configuration snippet.
The Routing Appender should be configured after any appenders it references to allow it to shut down properly.
Routing Configuration
| Attribute | Type | Default value | Description |
|---|---|---|---|
Required |
|||
|
The name of this appender. |
||
Optional |
|||
|
|
If Logging exceptions are always also logged to Status Logger |
|
| Type | Multiplicity | Description |
|---|---|---|
zero or one |
This script has two purposes:
The script has the following bindings:
See also Scripts for more details on scripting in Log4j Core. |
|
zero or one |
Allows filtering log events before routing them to a subordinate appender. See also appender filtering stage. |
|
zero or one |
The purge policy to apply to handle the lifecycle of automatically instantiated appenders. See Purge Policy for more details. |
|
zero or one |
The rewrite policy to apply to all logged events. If set, the |
|
one |
Determines the routing configuration of the appender. See |
Route selection
At the base of route selection there are two configuration elements:
Routes-
The
Routeselement is a container forRoutedefinitions. It provides two additional properties, which are used to determine the appropriate route for each log event:Table 15. Routesconfiguration attributesAttribute Type Default value Description StringIf present, this pattern is evaluated at each log event to determine the key of the route to use.
This attribute supports runtime property substitution using the current event as context.
Required, unless a nested
AbstractScriptis provided.Table 16. Routesnested elementsType Multiplicity Description zero or one
If present, this script is evaluated at each log event to determine the key of the route to use. The script has the following bindings:
staticVariables-
A
Map<String, Object>that is reused between script calls. This is the same map, which is passed to theAbstractScriptofRouting. logEvent-
The
LogEventbeing processed. configuration-
The current
Configurationobject. statusLogger-
The Status Logger to use to print diagnostic messages in the script.
See also Scripts for more details on scripting in Log4j Core.
Required, unless the
patternattribute is provided.one or more
Route-
The
Routeelement determines the appender to use if the route is selected. The appender can be:-
A previously declared appender, from the
Appenderssection of the configuration file. -
A new appender that is instantiated based on a nested appender definition, when the route becomes active. See also Purge Policy to learn more about the lifecycle of such an appender.
Table 17. Routeconfiguration attributesAttributeType
Default value
Description
StringnullA key that is compared with the evaluation of either the
patternattribute or nested script of the `Routes element.StringThe reference to an existing appender to use.
You cannot specify both this attribute and a nested `Appender definition.
Table 18. Routenested elementsTypeMultiplicity
Description
zero or one
The definition of an
Appenderto create, when this route is used for the first time.You cannot specify both this nested element and the
refconfiguration attribute.Lookups in the children of the
Routecomponent are not evaluated at configuration time. The substitution is delayed until theRouteelement is evaluated. This means that${...}expression should not be escaped as$${...}.The appender definition is evaluated in the context of the current event, instead of the global context.
See lazy property substitution for more details.
-
For each log event, the appropriate route is selected as follows:
-
First the
patternattribute orRoutesscript are evaluated to obtain a key. -
The key is compared with the
keyattribute of eachRouteelement. -
If there is a
Routefor that key, it is selected. -
Otherwise, the default
Routeis selected. The key of the defaultRouteis determined by theRoutingscript or isnull(lack ofkeyattribute) if the script is absent.
|
If the
|
Purge Policy
If your default Route element contains an appender definition, the Routing Appender can instantiate a large number of appenders, one for each value of the routing key.
These appenders might be useful only for a short period of time, but will consume system resources unless they are stopped.
The purge policy is a Log4j plugin that implements the
PurgePolicy
interface and handles the lifecycle of automatically instantiated appenders.
|
If an appender has been destroyed, it can be created again when the |
Log4j Core provides one implementation of PurgePolicy:
IdlePurgePolicy-
This policy destroys appenders if they have not been used for a certain amount of time. It supports the following configuration attributes:
Table 19. IdlePurgePolicyconfiguration attributesAttribute Type Default value Description longIt specifies the number of time units that an appender can be idle, before it is destroyed.
Required
longIt specifies the number of time units between two runs of this purge policy.
It specifies the time unit to use for the other attributes.
Configuration examples
Using appender references
You can deliver log events for different Markers into separate log files, using the following configuration:
-
XML
-
JSON
-
YAML
-
Properties
log4j2.xml<Routing name="ROUTING">
<Routes pattern="$${event:Marker}">
<Route key="AUDIT" ref="AUDIT_LOG"/> (1)
<Route key="$${event:Marker}" ref="MAIN_LOG"/> (2)
<Route ref="MARKED_LOG"/> (3)
</Routes>
</Routing>
log4j2.json"Routing": {
"name": "ROUTING",
"Routes": {
"pattern": "$${event:Marker}}",
"Route": [
{ (1)
"key": "AUDIT",
"ref": "AUDIT_LOG"
},
{ (2)
"key": "$${event:Marker}",
"ref": "MAIN_LOG"
},
{ (3)
"ref": "MARKED_LOG"
}
]
}
}
log4j2.yamlRouting:
Routes:
pattern: "$${event:Marker}"
Route:
- key: "AUDIT" (1)
ref: "AUDIT_LOG"
- key: "$${event:Marker}" (2)
ref: "MAIN_LOG"
- ref: "MARKED_LOG" (3)
log4j2.propertiesappender.3.type = Routing
appender.3.name = ROUTING
appender.3.route.type = Routes
appender.3.route.pattern = $${event:Marker}
(1)
appender.3.route.0.type = Route
appender.3.route.0.key = AUDIT
appender.3.route.0.ref = AUDIT_LOG
(2)
appender.3.route.1.type = Route
appender.3.route.1.key = $${event:Marker}
appender.3.route.1.ref = MAIN_LOG
(3)
appender.3.route.2.type = Route
appender.3.route.2.ref = MARKED_LOG
| 1 | This route is selected if the log event is marked with an AUDIT marker. |
| 2 | This route is selected if the log event has no marker.
In this case the expression ${event:Marker} evaluates to itself.
See Property evaluation for more details. |
| 3 | This is the default route.
It is selected if the log event has a marker, but it is not the AUDIT marker. |
Using appender definitions
If the number of appenders is high or unknown, you might want to use appender definitions instead of appender references. In the example below, a different log file is created for each marker.
-
XML
-
JSON
-
YAML
-
Properties
log4j2.xml<Routing name="ROUTING">
<Routes pattern="$${event:Marker}"> (1)
<Route>
<File name="${event:Marker}"
fileName="${event:Marker:-main}.log"> (2)
<JsonTemplateLayout/>
</File>
</Route>
</Routes>
<IdlePurgePolicy timeToLive="15"/> (3)
</Routing>
log4j2.json"Routing": {
"name": "ROUTING",
"Routes": {
"pattern": "$${event:Marker}}", (1)
"Route": {
"File": { (2)
"name": "${event:Marker}",
"fileName": "${event:Marker:-main}.log",
"JsonTemplateLayout": {}
}
}
},
"IdlePurgePolicy": { (3)
"timeToLive": 15
}
}
log4j2.yamlRouting:
Routes:
pattern: "$${event:Marker}" (1)
Route:
File: (2)
name: "${event:Marker}"
fileName: "${event:Marker:-main}.log"
IdlePurgePolicy: (3)
timeToLive: 15
log4j2.propertiesappender.0.type = Routing
appender.0.name = ROUTING
appender.0.route.type = Routes
(1)
appender.0.route.pattern = $${event:Marker}
appender.0.route.0.type = Route
(2)
appender.0.route.0.appender.type = File
appender.0.route.0.appender.name = ${event:Marker}
appender.0.route.0.appender.fileName = ${event:Marker:-main}.log
appender.0.route.0.appender.layout.type = JsonTemplateLayout
(3)
appender.0.purge.type = IdlePurgePolicy
appender.0.purge.timeToLive = 15
| 1 | The pattern attribute is evaluated at configuration time, so the ${event:Marker} lookup needs to be escaped. |
| 2 | The appender definition is not evaluated at configuration time, so no escaping is necessary. |
| 3 | To prevent resource leaks, consider using a Purge Policy. |
Using scripts
If the flexibility of Lookups is not enough to express your routing logic, you can also resort to scripts. In the example below, we route messages in a round-robin fashion to three different Syslog servers:
-
XML
-
JSON
-
YAML
-
Properties
log4j2.xml<Routing name="ROUTING">
<Script language="groovy"> (1)
staticVariables.servers = ['server1', 'server2', 'server3'];
staticVariables.count = 0;
</Script>
<Routes>
<Script language="groovy"> (2)
int count = staticVariables.count++;
String server = staticVariables.servers[count % 3];
return configuration.properties['server'] = server;
</Script>
<Route>
<Socket name="${server}"
protocol="TCP"
host="${server}"
port="500"> (3)
<Rfc5424Layout/>
</Socket>
</Route>
</Routes>
</Routing>
log4j2.json"Routing": {
"name": "ROUTING",
"Script": {
"language": "groovy",
(1)
"scriptText": "staticVariables.servers = ['server1', 'server2', 'server3']; staticVariables.count = 0;"
},
"Routes": {
"Script": {
"language": "groovy",
(2)
"scriptText": "int count = staticVariables.count++; String server = staticVariables.servers[count % 3]; return configuration.properties['server'] = server;"
},
"Route": {
"Socket": { (3)
"name": "${server}",
"protocol": "TCP",
"host": "${server}",
"port": "500",
"Rfc5425Layout": {}
}
}
}
}
log4j2.yamlRouting:
Script:
language: "groovy"
(1)
scriptText: |
staticVariables.servers = ['server1', 'server2', 'server3'];
staticVariables.count = 0;
Routes:
Script:
language: "groovy"
(2)
scriptText: |
int count = staticVariables.count++;
String server = staticVariables.servers[count % 3];
return configuration.properties['server'] = server;
Route:
(3)
Socket:
name: "${server}"
protocol: "TCP"
host: "${server}"
port: 500
Rfc5424Layout: {}
log4j2.propertiesappender.0.type = Routing
appender.0.name = ROUTING
appender.0.script.type = Script
appender.0.script.language = groovy
(1)
appender.0.script.scriptText = \
staticVariables.servers = ['server1', 'server2', 'server3']; \
staticVariables.count = 0;
appender.0.route.type = Routes
appender.0.route.script.type = Script
appender.0.route.script.language = groovy
(2)
appender.0.route.script.scriptText = \
int count = staticVariables.count++; \
String server = staticVariables.servers[count % 3]; \
return configuration.properties['server'] = server;
appender.0.route.0.type = Route
(3)
appender.0.route.0.appender.type = Socket
appender.0.route.0.appender.name = ${server}
appender.0.route.0.appender.protocol = TCP
appender.0.route.0.appender.host = ${server}
appender.0.route.0.appender.port = 500
appender.0.route.0.appender.layout = Rfc5424Layout
| 1 | The Routing script performs the initialization of state variables. |
| 2 | The Routes script returns the name of the server to use.
It also exports the value as server entry in
Configuration.getProperties(). |
| 3 | The exported value can be used as ${server} in the appender definition. |
ScriptAppenderSelector
The ScriptAppenderSelector plugin allows using different appender definitions based on the output of a script.
At configuration time:
-
The nested script element is evaluated to obtain the name of an appender.
-
The plugin looks for the appropriate appender definition inside the
<AppenderSet>container.
|
The functionality of |
|
Type |
Default value |
Description |
|
The name of this appender. Required |
| Type | Multiplicity | Description |
|---|---|---|
one |
The script to determine the appender name. |
|
one |
A lazy container for appender definitions. |