Structured Logging

Historically logs were meant to be read by humans. If an error occurred the log might look like: ERROR 2023-08-30 Something bad happened. A human could view that message and glean that its severity was ERROR, the error occurred on 2023-08-30, and that the error message was Something bad happened.

Now imagine you send these logs to a service, and that service puts them into a table structure so you can query them. The table structure might look like this:

It would be tricky for the service to parse the logs into anything more granular than that given their format.

What if you wanted to find all the logs with a severity of error? You could write a query against your logs like this:

SELECT *
FROM "logs"
WHERE "log" LIKE "ERROR%"

Not too bad, but what if you wanted to get the errors that occurred in the last week? That would be a tricky query to write with unstructured logging.

The idea of structured logging is quite straightforward: you log in a format that is machine readable. The log entry above could instead look like:

{
  "severity": "ERROR",
  "meessage": "Something bad happened",
  "occurred": "2023-08-30"
}

The imaginary service we are sending these logs to would now have an easier time putting the logs into a more granular table structure. This time it could look like this:

And our previous query would become:

SELECT *
FROM "logs"
WHERE "severity" = "ERROR"

This small change makes it much easier to interact with logs.

What fields you log will vary system to system, but there is a set of fields that is almost always good to have:

• The log message
• The thing that did the logging
• The current timestamp
• The severity level of the log message
• The stack trace (if an error occurred)
• The ID of the current trace

The service you send the logs to will dictate their format, but it will probably be JSON. An example of one of these formats is the one Cloud Logging uses: https://cloud.google.com/logging/docs/structured-logging.

Where you write logs to will depend on where you are sending them. My preference is services like Cloud Run that will automatically push logs up to a store from the console. This has several advantages:

• You aren't using any vendor specific libraries for logging
• The logging works the same locally as it does in the cloud
• Writing a log to the console is fast

Generally speaking you will use a logging library for structured logging. Any reasonably mature language will have options for this, and it's generally straightforward to configure them for your use case. For example say you want to write logs in TypeScript for a Cloud Run service. The requirements would be:

• You write the logs to console
• You write the logs in JSON
• You write the logs to this spec: https://cloud.google.com/logging/docs/structured-logging

First you would select a library. For this example we'll select pino. It already writes logs in JSON to the console by default. The rest of the configuration looks like:

import { pino } from "pino";

type Severity =
  | "DEFAULT"
  | "DEBUG"
  | "INFO"
  | "NOTICE"
  | "WARNING"
  | "ERROR"
  | "CRITICAL"
  | "ALERT"
  | "EMERGENCY";

function logLevelToSeverity(logLevel: string): Severity {
  switch (logLevel) {
    case "debug":
      return "DEBUG";
    case "trace":
      return "DEBUG";
    case "info":
      return "INFO";
    case "warn":
      return "WARNING";
    case "error":
      return "ERROR";
    case "fatal":
      return "CRITICAL";
    default:
      return "DEFAULT";
  }
}

export const logger = pino({
  redact: { paths: ["pid", "hostname"], remove: true },
  timestamp: false,
  messageKey: "message",
  mixin: () => ({
    timestamp: new Date().toISOString(),
  }),
  formatters: {
    level(label) {
      return { severity: logLevelToSeverity(label) };
    },
  },
});

Structured logging is in widespread use already, but if you aren't using it perhaps you will consider it. It makes your logs more valuable.