Structured logging in Spring Boot 3.4

Engineering | Moritz Halbritter | August 23, 2024 | ...

Logging is a long established part of troubleshooting applications and one of the three pillars of observability, next to metrics and traces. No one likes flying blind in production, and when incidents happen, developers are happy to have log files. Logs are often written out in a human-readable format.

Structured logging is a technique where the log output is written in a well-defined, often machine-readable format. This format can be fed into log management systems and enables powerful search and analytics capabilities. One of the most commonly used formats for structured logging is JSON.

With Spring Boot 3.4, structured logging is supported out of the box. It supports the Elastic Common Schema (ECS) and Logstash formats, but it's also possible to extend it with your own formats.

Let's jump straight in and see how it works!

Structured logging Hello World

Create a new project on start.spring.io. You don't need to add any dependencies, but make sure to select at least Spring Boot 3.4.0-M2.

To enable structured logging on the console, add this to your application.properties:

logging.structured.format.console=ecs

This will instruct Spring Boot to log in the Elastic Common Schema (ECS) format.

Now start the application, and you'll see that the log is formatted in JSON:

{"@timestamp":"2024-07-30T08:41:10.561295200Z","log.level":"INFO","process.pid":67455,"process.thread.name":"main","service.name":"structured-logging-demo","log.logger":"com.example.structured_logging_demo.StructuredLoggingDemoApplication","message":"Started StructuredLoggingDemoApplication in 0.329 seconds (process running for 0.486)","ecs.version":"8.11"}

Quite cool, right? Now let's dive into the more advanced topics.

Structured logging to a file

You can also enable structured logging to a file. This can be used, for example, to print human-readable logs on the console and write structured logs to a file for machine ingestion.

To enable that, add this to your application.properties and make sure to delete the logging.structured.format.console=ecs setting:

logging.structured.format.file=ecs
logging.file.name=log.json

Now start your application and you'll see that there's human-readable log on the console, and the file log.json contains machine-readable JSON content.

Add additional fields

One powerful feature of structured logging is that developers can add information to the log event in a structured way. You can, for example, add the user id to every log event and then later filter on that id to see what this particular user did.

Both Elastic Common Schema and Logstash include the content of the Mapped Diagnostic Context in the JSON. To see that in action, let's create our own log message:

@Component
class MyLogger implements CommandLineRunner {
    private static final Logger LOGGER = LoggerFactory.getLogger(MyLogger.class);

    @Override
    public void run(String... args) {
        MDC.put("userId", "1");
        LOGGER.info("Hello structured logging!");
        MDC.remove("userId");
    }
}

Before logging the log message, this code also sets the user id in the MDC. Spring Boot automatically includes the user id in the JSON:

{ ... ,"message":"Hello structured logging!","userId":"1" ... }

You can also use the fluent logging API to add additional fields without relying on the MDC:

@Component
class MyLogger implements CommandLineRunner {

    private static final Logger LOGGER = LoggerFactory.getLogger(MyLogger.class);

    @Override
    public void run(String... args) {
        LOGGER.atInfo().setMessage("Hello structured logging!").addKeyValue("userId", "1").log();
    }

}

Elastic Common Schema defines a lot of field names, and Spring Boot has built-in support for the service name, the service version, the service environment and the node name. To set values for those fields, you can use the following in your application.properties:

logging.structured.ecs.service.name=MyService
logging.structured.ecs.service.version=1
logging.structured.ecs.service.environment=Production
logging.structured.ecs.service.node-name=Primary

When looking at the JSON output, there are now fields for service.name, service.version, service.environment and service.node.name. With that, you can now filter in your logging system on the node name, service versions, etc.

Custom log formats

As said above, Spring Boot supports the Elastic Common Schema and Logstash formats out of the box. To add your own format, you have to do the following steps:

Create a custom implementation of the StructuredLogFormatter interface
Reference your custom implementation in the application.properties

First, let's create our own custom implementation:

class MyStructuredLoggingFormatter implements StructuredLogFormatter<ILoggingEvent> {

    @Override
    public String format(ILoggingEvent event) {
        return "time=" + event.getTimeStamp() + " level=" + event.getLevel() + " message=" + event.getMessage() + "\n";
    }

}

As you can see, the structured logging support is not limited to JSON, you can return any String you want. In this example, we chose to use key=value pairs.

We now need to make Spring Boot aware of our custom implementation. To do that, add this to the application.properties:

logging.structured.format.console=com.example.structured_logging_demo.MyStructuredLoggingFormatter

It's time to start the application and marvel at the log output!

time=1722330118045 level=INFO message=Hello structured logging!

Wow, look at that! What a beautiful piece of log output!

If you want to write JSON, there's a handy new JsonWriter in Spring Boot 3.4, which you can use:

class MyStructuredLoggingFormatter implements StructuredLogFormatter<ILoggingEvent> {

    private final JsonWriter<ILoggingEvent> writer = JsonWriter.<ILoggingEvent>of((members) -> {
        members.add("time", (event) -> event.getInstant());
        members.add("level", (event) -> event.getLevel());
        members.add("thread", (event) -> event.getThreadName());
        members.add("message", (event) -> event.getFormattedMessage());
        members.add("application").usingMembers((application) -> {
            application.add("name", "StructuredLoggingDemo");
            application.add("version", "1.0.0-SNAPSHOT");
        });
        members.add("node").usingMembers((node) -> {
           node.add("hostname", "node-1");
           node.add("ip", "10.0.0.7");
        });
    }).withNewLineAtEnd();

    @Override
    public String format(ILoggingEvent event) {
        return this.writer.writeToString(event);
    }

}

Of course you can also use any other JSON library (e.g. Jackson) to create the JSON, you don't have to use the JsonWriter.

The resulting log message looks something like this:

{"time":"2024-07-30T09:14:49.377308361Z","level":"INFO","thread":"main","message":"Hello structured logging!","application":{"name":"StructuredLoggingDemo","version":"1.0.0-SNAPSHOT"},"node":{"hostname":"node-1","ip":"10.0.0.7"}}

Wrapping up

We hope you like that new feature in Spring Boot 3.4! The documentation has been updated for structured logging, too.

Please let us know what you think and if you find any issues, our issue tracker is always open!

Spring blog

Get the Spring newsletter

Get ahead

Get support

Upcoming events