Configure Log Forwarding

Configure Couchbase Server logs to be forwarded to standard console output.

Overview

Couchbase Server produces a variety of log files. By default, these logs cannot be collected from the Couchbase Server container’s standard console output. This limits the ability to integrate Couchbase logging with 3rd-party monitoring or collection technologies deployed within a Kubernetes cluster.

However, the Autonomous Operator can optionally enable log forwarding by deploying a third party log processor in a sidecar container on each Couchbase pod, which then reads the log files and forwards them to standard console output. For this purpose, Couchbase supplies a default log processor image based on Fluent Bit.

The sections on this page describe how to enable and configure log forwarding using the Couchbase-supplied log processor image.

The Couchbase-supplied log processor container image is only supported on Kubernetes platforms in conjunction with the Couchbase Autonomous Operator.

Log forwarding requires that logs be written to a persistent volume (i.e. the Couchbase deployment’s default or logs volumes are backed by persistent storage). Fully-ephemeral clusters are not supported by this feature.

Log forwarding requires that couchbaseclusters.spec.securityContext is set as appropriate for your Kubernetes distribution, as the sidecar requires read access to the shared volume.

Enabling Log Forwarding

Log forwarding is fundamentally provided by two components:

the log processor image that is used by the Autonomous Operator to deploy the logging sidecar container onto each Couchbase Server pod, and
the log forwarding configuration, stored in a Kubernetes Secret, that gets consumed by the logging sidecar container and which controls its behavior.

Enabling log forwarding for a particular Couchbase cluster involves enabling these two components in the CouchbaseCluster resource specification.

The required configuration parameters for enabling log forwarding are described in the example below. Specified values represent the defaults for their respective fields unless otherwise noted in a callout. (The Autonomous Operator will set the default values for any fields that are not specified by the user.)

Basic CouchbaseCluster Parameters That Enable Log Forwarding

apiVersion: couchbase.com/v2
kind: CouchbaseCluster
spec:
  servers:
  - size: 3
    name: all_services
    volumeMounts:
      default: couchbase
  volumeClaimTemplates:
      - metadata:
          name: couchbase
        spec:
          storageClassName: "standard"
          resources:
            requests:
              storage: 1Gi
  logging:
    server:
      enabled: true (1)
      sidecar:
        image: "couchbase/fluent-bit:1.2.2" (2)
    audit:
      enabled: false (3)
      garbageCollection:
        sidecar:
          enabled: false

1	`couchbaseclusters.spec.logging.server.enabled`: This is technically the only field that must be changed in order to enable log forwarding. Setting this field to `true` (defaults to `false`) instructs the Autonomous Operator to deploy the `logging` sidecar container on each pod.
2	`couchbaseclusters.spec.logging.server.sidecar.image`: This field specifies the container image that the Autonomous Operator will use for deploying the `logging` sidecar container. This field defaults to `couchbase/fluent-bit:1.2.2`, which pulls the Couchbase-supplied log processor image. You may need to modify this field if your Kubernetes nodes can’t pull from the Docker public registry.
3	(Optional) `couchbaseclusters.spec.logging.audit.enabled`: This field is used to enable Couchbase audit logging, and defaults to `false`. Unlike other Couchbase logs, audit logging is not turned on by default. If you intend to forward audit logs, you must configure audit logging first.

After deploying the CouchbaseCluster resource specification, the logs will be available on the standard output of the logging container on each pod:

$ kubectl logs cb-example-0000 logging

Configuring Log Forwarding

As described in Enabling Log Forwarding, there are two primary components that provide log forwarding: the log processor image and the log forwarding configuration. The Autonomous Operator uses Couchbase-provided defaults for both of these components, which can be used as-is (refer to Using Default Log Forwarding). However, these components can be further customized to better meet the needs of your environment (refer to Using Custom Log Forwarding).

Enable memory buffer limits

The Fluent Bit input plugins have memory buffer limits that can be used to help prevent the container from being terminated for exceeding memory limits. Once an input plugin reaches its memory buffer limit, all new incoming data is instead buffered to file system.

Memory buffer limits are disabled by default and can be enabled by adding the annotation fluentbit.couchbase.com/mem.buf.limits.enabled to a pod.

apiVersion: couchbase.com/v2
kind: CouchbaseCluster
spec:
  servers:
  - name: default
    pod:
      metadata:
        annotations:
          fluentbit.couchbase.com/mem.buf.limits.enabled: "true"

See Fluent Bit Memory Buffer Limits and Fluent Bit Buffering and Storage from more information.

Using Default Log Forwarding

A Couchbase deployment is considered to be using default log forwarding when the Couchbase-supplied defaults for both the log processor image and the log forwarding configuration are being used.

CouchbaseCluster Parameters Involved with Default Log Forwarding

apiVersion: couchbase.com/v2
kind: CouchbaseCluster
spec:
  logging:
    server:
      enabled: true
      manageConfiguration: true (1)
      configurationName: “fluent-bit-config” (2)
      sidecar:
        image: couchbase/fluent-bit:1.2.2” (3)
        configurationMountPath: “/fluent-bit/config/”
        resources:
          requests:
            cpu: 100m
            memory: 500Mi

1 couchbaseclusters.spec.logging.server.manageConfiguration: When this field is set to true (the default), the Autonomous Operator ensures that the logging sidecar container always uses the default log forwarding configuration. The Autonomous Operator accomplishes this by creating a Secret that contains the default log forwarding configuration, which is then consumed by the logging sidecar container. The name of the Secret will be whatever name is specified in couchbaseclusters.spec.logging.server.configurationName. If a Secret already exists with the same name, the Autonomous Operator overwrites it, ensuring that the default log forwarding configuration is maintained.

couchbaseclusters.spec.logging.server.configurationName: This field defines the name of the Secret that contains the log forwarding configuration that will be consumed by the logging sidecar container. This field defaults to fluent-bit-config and will be the name of the Secret that gets automatically created by the Autonomous Operator when couchbaseclusters.spec.logging.server.manageConfiguration is set to true.

If running multiple clusters in the same Kubernetes namespace, make sure to use a different name for the Secret of each cluster rather than attempting to share the same Kubernetes Secret.

couchbaseclusters.spec.logging.server.sidecar.image: This field specifies the container image that the Autonomous Operator will use for deploying the logging sidecar container. This field defaults to couchbase/fluent-bit:1.2.2, which pulls the Couchbase-supplied log processor image. You may need to modify this field if your Kubernetes nodes can’t pull from the Docker public registry.

When couchbaseclusters.spec.logging.server.manageConfiguration is set to true, then the default log processor image supplied by Couchbase (or an image that is effectively equivalent) must also be used. This is because the default log forwarding configuration requires certain features that are a part of the default image.

By default, this configuration will only output to the standard console output, however, there are other output plugins included that do not match any existing streams. It is possible to enable matching of streams to output plugins by setting the fluentbit.couchbase.com/<PLUGIN>_MATCH annotation. The following is an example for Loki usage.

      pod:
        metadata:
          annotations:
            fluentbit.couchbase.com/loki.match: "*"
            fluentbit.couchbase.com/loki.host: loki.monitoring

Other supported output plugins include Elasticsearch and Splunk. See Configuring Couchbase Fluent Bit for more detailed output configuration options.

Using Custom Log Forwarding

A Couchbase deployment is considered to be using custom log forwarding when either a custom log processor image and/or a custom log forwarding configuration is being used for log forwarding.

CouchbaseCluster Parameters Involved with Custom Log Forwarding

apiVersion: couchbase.com/v2
kind: CouchbaseCluster
spec:
  logging:
    server:
      enabled: true
      manageConfiguration: false (1)
      configurationName: fluent-bit-config” (2)
      sidecar:
        image: couchbase/fluent-bit:1.2.2” (3)
        configurationMountPath: “/fluent-bit/config/” (4)
        resources:
          requests:
            cpu: 100m
            memory: 500Mi

couchbaseclusters.spec.logging.server.manageConfiguration: When this field is set to false (defaults to true), the Autonomous Operator allows the Secret specified in couchbaseclusters.spec.logging.server.configurationName to contain a non-default log forwarding configuration. This field must be set to false in order to use a custom log forwarding configuration, otherwise the Autonomous Operator will always reconcile and update the Secret with the default log forwarding configuration.

If couchbaseclusters.spec.logging.server.manageConfiguration was ever set to true when you enabled log forwarding, then the Autonomous Operator will have already created the default Secret. In this situation, if you were to then set this field to false, the logging sidecar container would continue to consume the default Secret. You could then choose to add your custom log forwarding configuration to this same Secret. However, if you were to change couchbaseclusters.spec.logging.server.configurationName to point to a different Secret, it won’t get picked up until the next time a pod starts.

couchbaseclusters.spec.logging.server.configurationName: This field defines the name of the Secret that contains the custom log forwarding configuration that will be consumed by the logging sidecar container.

When couchbaseclusters.spec.logging.server.manageConfiguration is set to false, then couchbaseclusters.spec.logging.server.configurationName must specify the name of a Secret that currently exists in the same namespace. Otherwise, new Couchbase pods will not start, since they will be waiting on a non-existent Secret.

Since the log forwarding configuration is stored in a Secret, it can be shared by two or more Couchbase clusters in the same namespace. This has the benefit of allowing you to manage a single log forwarding configuration for multiple clusters. However, when sharing a custom log forwarding configuration between clusters, it’s important to remember that setting couchbaseclusters.spec.logging.server.manageConfiguration to true on any of the clusters will overwrite the shared Secret with the default log forwarding configuration.

3 couchbaseclusters.spec.logging.server.sidecar.image: If desired, you can use this field to specify a custom log processing image. A few reasons for specifying a custom image include adding certificates for self-signed endpoints internally, or using another implementation of either the Fluent Bit image or any other log forwarder/processor.

4 couchbaseclusters.spec.logging.server.sidecar.configurationMountPath: This field defines the location to mount the Secret into the logging sidecar container. This field currently defaults to the usual Fluent Bit mount path (/fluent-bit/config/). This field only needs to be modified if a custom image is specified in couchbaseclusters.spec.logging.server.sidecar.image, and only if that image requires a non-default mount path.

Creating a Custom Log Forwarding Configuration

Since a log forwarding configuration can contain sensitive information, it is stored in a Kubernetes Secret. The Secret is used to populate the volume mounted into the sidecar so that the actual configuration files get picked up by the log processor.

When using default log forwarding, the Autonomous Operator automatically creates a Secret named fluent-bit-config that contains the default log forwarding configuration. This configuration, which can be viewed on GitHub, is used by the Couchbase-supplied log processor image in order to provide features like automatic parsing of a number of supported logs, automatic filtering for adding useful information to the logs (e.g. pod host names), as well as optional log redaction (another type of filtering).

When using custom log forwarding, the Autonomous Operator does not automatically create a log forwarding configuration Secret. Instead, the administrator is expected to create a custom Secret themselves, and specify that Secret in couchbaseclusters.spec.logging.server.configurationName.

The rest of this section covers some relatively simple ways in which you can customize the default log forwarding configuration, such as limiting the number of forwarded logs, and enabling log redaction. However, since the Couchbase-supplied log processor image is based on Fluent Bit, you can technically customize the log forwarding configuration with any kind of configuration that is supported by Fluent Bit. Some examples of these types of customizations are covered in the tutorial Configure Log Forwarding.

Only custom configurations that leverage the built-in parsing, redaction, and other standard features of the Couchbase-supplied log processor image are supported.

Creating a Secret That Contains a Custom Log Forwarding Configuration

The following is an example of a custom log forwarding configuration Secret that is based on the default configuration, but which limits the number of Couchbase logs that are processed and forwarded to just the Couchbase audit log:

Example: Custom Log Forwarding Configuration Secret

apiVersion: v1
kind: Secret
metadata:
  name: fluent-bit-config (1)
stringData:
  fluent-bit.conf: | (2)
    [SERVICE]
        flush        1
        daemon       Off
        log_level    Warning
        parsers_file /fluent-bit/etc/parsers-couchbase.conf (3)

    # @include /fluent-bit/etc/couchbase/in-*.conf (4)
    @include /fluent-bit/etc/couchbase/in-audit-log.conf (5)

    [FILTER]
        Name modify
        Match couchbase.log.*
        Add pod ${HOSTNAME}
        Add logshipper couchbase.sidecar.fluentbit

    [OUTPUT] (6)
        name  stdout
        match couchbase.log.audit (7)

1 metadata.name: The Autonomous Operator expects the default name fluent-bit-config. If you use a different name, then you will need to specify that name in couchbaseclusters.spec.logging.server.configurationName.

2 stringData.fluent-bit.conf: Log forwarding configuration files are defined here in the stringData field of the Secret (the data field is also potentially an option). The fluent-bit.conf file is specified with a field that starts with a space and a vertical bar (|), followed by the contents of the main configuration. Note that Fluent Bit is very particular about its format and schema — files must all follow the same indentation.

/fluent-bit/etc/parsers-couchbase.conf: This is the path to the built-in file that contains all of the default parser configurations in the Couchbase-supplied log processor image (you can view this file on GitHub).

You should only have to modify this if you’re using a custom log processor image that invokes a parser file from a different location.

@include /fluent-bit/etc/couchbase/in-*.conf: This is the default input configuration that processes most Couchbase logs. Since the example is meant to only process the Couchbase audit log, this line is commented out.

Couchbase logs are supported by individual input configurations, with those that are included in the default configuration being prefixed with in-. Therefore by including a wildcard in-*.conf, all default input configurations are invoked.

Some logs are not included by default. The individual input configurations for non-default logs are ones that are not prefixed with in-. The reason that a particular log is left out of the default input configuration is often because the log has negligible value compared to the load that it puts on the log processor.

About Input Configurations

Input configurations determine which Couchbase logs are processed, as well as some details about how they are processed. A typical input configuration for a Couchbase log will specify (among other things): the input plugin (usually tail), the logs that the input configuration applies to, the parsers to use on those logs, and the tags to apply to those logs.

Each Couchbase log is supported by an input configuration, with many input configurations supporting more than one log. The reason input configurations usually support more than one log is because certain logs are similar enough that they can share the same input plugin, parser(s), etc. Across all available input configurations, each Couchbase logging event can be captured.

5 @include /fluent-bit/etc/couchbase/in-audit-log.conf: This is the path to the built-in file that contains the input configuration that processes just the Couchbase audit log (you can view this file on GitHub). Since this line is included, and @include /fluent-bit/etc/couchbase/in-\*.conf is commented out, only the Couchbase audit log will be processed.

A log forwarding configuration includes one or more output configurations. The default output configuration forwards the processed logs to the standard output of the logging container.

A log forwarding configuration can include more than one output configuration. This allows for specific logs to be routed to different, or multiple, destinations. For example, all Couchbase logs could be sent to standard output, with the Couchbase audit log also being sent to AWS S3.

7 couchbase.log.audit: By default, all parsed Couchbase log events are tagged with couchbase.log.<name-of-log>. Therefore, specifying the match couchbase.log.audit ensures that only Couchbase audit log events are sent to the specified output destination. (The default output configuration normally uses the wildcard match couchbase.log.* to forward all tagged Couchbase logs to the output.)

You can copy the example Secret above and create it in your Kubernetes cluster for use with existing and future Couchbase cluster deployments. However, please review Using Custom Log Forwarding, as there are a few things that you need to be aware of when creating a custom log forwarding configuration Secret. In particular, you should make sure that any Couchbase cluster deployment that references the Secret in couchbaseclusters.spec.logging.server.configurationName also has couchbaseclusters.spec.logging.server.manageConfiguration set to false, otherwise the Autonomous Operator will overwrite the custom Secret with the default log forwarding configuration.

Updating an Existing Custom Log Forwarding Configuration

Further updates can be made to a custom log forwarding configuration Secret, even after it has been created in Kubernetes and consumed by the logging sidecar containers in Couchbase deployments. Changing the Secret will update the volume in the sidecar, and when the configuration change is detected in the couchbaseclusters.spec.logging.server.sidecar.configurationMountPath directory, the watcher process restarts Fluent Bit internally to pick it up without having to restart the entire Couchbase Server pod.

The ability to restart Fluent Bit internally is a special characteristic of the Couchbase-supplied log processor image. If you’re using a custom sidecar container image, be aware that Fluent Bit, on its own, does not currently support dynamic reload of its configuration.

The Couchbase-supplied log forwarding implementation does not currently support log buffering during restart. Therefore, log events that occur while Fluent Bit is restarting may be lost.

Configuring Log Redaction

The Couchbase-supplied log processor image provides optional support for log redaction, whereby sensitive data from Couchbase log events are redacted before the events are forwarded to standard console output or other locations. Log redaction isn’t enabled by default, and therefore must be enabled via the log forwarding configuration Secret.

This section describes how to configure the built-in LUA-based log redaction facility. However, there are simpler, more rudimentary methods of performing log redaction that may be more desirable depending on the use-case. Refer to log forwarding tutorial for examples.

The following example is the same custom log forwarding configuration Secret from the section Creating a Secret That Contains a Custom Log Forwarding Configuration, except that it is configured to additionally perform redaction of the audit log:

Example: Custom Log Forwarding Configuration That Redacts the Couchbase Audit Log

apiVersion: v1
kind: Secret
metadata:
  name: fluent-bit-config
stringData:
  fluent-bit.conf: |
    [SERVICE]
        flush        1
        daemon       Off
        log_level    Warning
        parsers_file /fluent-bit/etc/parsers-couchbase.conf

    @include /fluent-bit/etc/couchbase/in-audit-log.conf

    [FILTER]
        Name    lua
        Match   couchbase.log.audit (1)
        script  /fluent-bit/etc/redaction/redaction.lua (2)
        call    cb_sub_message (3)

    [FILTER]
        Name modify
        Match couchbase.log.*
        Add pod ${HOSTNAME}
        Add logshipper couchbase.sidecar.fluentbit

    [OUTPUT]
        name  stdout
        match couchbase.log.audit (4)

  redaction.salt: salty (5)

1 couchbase.log.audit: Log events that match this tag will be processed for redaction. In this example, only events from the audit log will be processed. To have all logs processed for redaction, you can specify the wildcard couchbase.log.*.

2 redaction.lua: This is the path to the Couchbase-supplied LUA script file in the default Couchbase-supplied log processor image. You should only have to modify this if you’re using a custom log processor image that invokes a LUA script from a different location.

3 cb_sub_message: This is the function that is called within the Couchbase-supplied LUA script file. You should only have to modify this if you’re using a custom log processor image and a custom LUA script.

couchbase.log.audit: Specifying couchbase.log.audit as the match ensures that only Couchbase audit log events are forwarded to standard console output.

As a means of reducing load on the logging container, it is recommended to only match the tags of the specific logs that you want to redact. LUA scripting has an overhead and it is recommended to use at least one extra Fluent Bit worker thread if you enable redaction.

5 redaction.salt: You can optionally define a custom salt for the hashing of tagged data. The salt is specified in the configuration Secret as a separate file/value (in the example, the salt value is salty). If a salt is not specified, the Autonomous Operator will default to using the cluster name as the salt.

After updating the log forwarding configuration with the example above, you’ll notice output similar to the following, showing the redacted strings as hashes:

[0] couchbase.log.audit: [1615320737.445039000, {"filename":"/fluent-bit/test/logs/audit.log","bucket":"default","description":"Cats are <ud>00b335216f27c1e7d35149b5bbfe19d4eb2d6af1</ud> than dogs, and <ud>888f807d45ff6ce47240c7ed4e884a6f9dc7b4fb</ud>","id":20492,"name":"select bucket","peername":"127.0.0.1:56021","real_userid":{"domain":"local","user":"@ns_server"},"sockname":"127.0.0.1:11209","timestamp":"2021-03-09T20:12:17.445039Z"}]

One thing to note about the above configuration is that the log processor is only outputting a redacted version of the audit log. There may be situations where it is desired to have the log processor output both a redacted and an unredacted version of a particular log.

The following is an example of a custom log forwarding configuration Secret that creates an additional stream of the Couchbase audit log that is then redacted and forwarded to a separate output:

Example: Custom Log Forwarding Configuration with Separate Redaction Stream

apiVersion: v1
kind: Secret
metadata:
  name: fluent-bit-config
stringData:
  fluent-bit.conf: |
    [SERVICE]
        flush        1
        daemon       Off
        log_level    Warning
        parsers_file /fluent-bit/etc/parsers-couchbase.conf

    @include /fluent-bit/etc/couchbase/in-*.conf (1)

    [FILTER]
        Name modify
        Match couchbase.log.*
        Add pod ${HOSTNAME}
        Add logshipper couchbase.sidecar.fluentbit

    [FILTER] (2)
        Name rewrite_tag
        Match couchbase.log.audit
        Rule message .* couchbase.log.audit.redact true

    [FILTER] (3)
        Name    lua
        Match   couchbase.log.*.redact
        script  /fluent-bit/etc/redaction/redaction.lua
        call    cb_sub_message

    [OUTPUT] (4)
        name  stdout
        match couchbase.log.*

    [OUTPUT] (5)
        Name                         s3
        Match                        couchbase.log.audit.redact
        bucket                       my-bucket
        region                       us-west-2
        total_file_size              250M
        s3_key_format                /$TAG[2]/$TAG[0]/%Y/%m/%d/%H/%M/%S
        s3_key_format_tag_delimiters .-

  redaction.salt: salty

1	`@include /fluent-bit/etc/couchbase/in-*.conf`: specifies the default input configuration that processes most Couchbase logs.
2	A rewrite_tag filter is defined in order to duplicate the Couchbase audit log events. `Match couchbase.log.audit`: specifies that only the events with the tag `couchbase.log.audit` are to be selected. `Rule message .* couchbase.log.audit.redact true`: specifies that a duplicate of each selected Couchbase audit log event is to be emitted with the new tag `couchbase.log.audit.redact`. The final value in the rule — `true` — specifies that the events with the old tag (`couchbase.log.audit`) are to be preserved. The final result is two streams of the same audit log events: one stream with the tag `couchbase.log.audit`, and another with the tag `couchbase.log.audit.redact`.
3	A LUA filter is defined in order to perform redaction on one of the streams of audit log events. `Match couchbase.log.*.redact`: specifies that events with the tag suffix `.redact` are to be selected and subject to redaction.
4	An output configuration is defined that forwards all unredacted Couchbase log events to the standard output of the `logging` container. `match couchbase.log.`: specifies that the log events with the wildcard tag `couchbase.log.` are to be selected and forwarded. Note that this includes the unredacted audit log events tagged `couchbase.log.audit`.
5	A second output configuration is defined that forwards redacted Couchbase audit log events to the Amazon S3. `Match couchbase.log.audit.redact`: specifies that only the audit log events with the tag `couchbase.log.audit.redact` (the ones that have been redacted) are to be selected and forwarded.

About Custom Log Processor Images

A different container image may be used in order to implement your own log processor and forwarding configuration. couchbaseclusters.spec.logging.server.sidecar.image supports specifying a different image and/or version for log forwarding support. For example, this field can be used to specify a custom image with self-signed certificates. It can also be used to specify an image made available by a public cloud provider, such a AWS for Fluent Bit.

It’s important to note that the built-in parsing, redaction, and other configurations provided by the Couchbase-supplied log processor image are lost if a stock log processor image is used.