Please use the form below to provide your feedback. Because your feedback is valuable to us, the information you submit in this form is recorded in our issue tracking system (JIRA), which is publicly available. You can track the status of your feedback using the ticket number displayed in the dialog once you submit the form.

Log Streaming

  • Capella Operational
      +
      Log Streaming provides a mechanism for real-time streaming of App Services operational logs to third-party observability platforms or self-hosted HTTP logs collectors. This is a crucial tool to gain instant insights into application behavior, enabling rapid issue detection and resolution to enhance application reliability, performance, and security.

      As a managed, distributed service, Capella App Services is implemented by multiple nodes. In a self-hosted service, like Couchbase’s Sync Gateway, you can directly connect to a node to figure figuring out the root cause and fix for issues with access control and synchronization. With a managed service, however, access to a given node’s logging data is not exposed directly.

      With the opt-in Log Streaming feature, we can request to stream the logs from each of the nodes to an industry standard log collector for full analysis.

      Log Streaming has implications for cost and sizing, and is turned off by default.

      When configuring Log Streaming for App Services, you must make sure that all streamed log data complies with your company’s security and privacy standards as well as with any regulatory standards you adhere to.

      Supported Log Collector Providers

      We support log streaming to Datadog, Sumo Logic, and to self-hosted log collectors via HTTP. See Enable Log Streaming for full details.

      You must set up your Log Collector as a prerequisite to using the Log Streaming feature.

      Couchbase is not responsible for any third-party endpoints you configure.

      Resource Considerations

      The amount of data will depend on factors such as:

      • Which logging filters and level you have requested.

      • How many App Services endpoints are running.

      • The number and size of documents in the cluster.

      • Read/write/import throughput.

      • Number of client connections per node.

      You will incur egress data charges from the App Services nodes for the logging data. It is therefore important to configure this feature carefully, to make sure that you receive useful information at a reasonable cost.

      Do not enable log streaming until you have validated what data you want, and understand the costs and resources involved in streaming it for your current and predicted data patterns.

      By default, we stream everything from Info level and below, and enable a preset set of filters as detailed in App Endpoint configuration page.

      The log level and log filters are configurable, and can dramatically affect the amount of data streamed, which will have cost implications.

      Couchbase recommends keeping the defaults until you have verified that you need the data, and understand the costs involved.

      In addition to network traffic, a node that is streaming logs will have some marginal impact on CPU and RAM usage.

      Troubleshooting

      Prerequisites

      Log Streaming is available for App Services on version 3.1.2 or later. Upgrade your App Services clusters to the newest version if you want to utilize this feature.

      You must set up your Log Collector (see Enable Log Streaming),ensure that it is reachable, and configure its location in the Capella UI.

      You must make sure that the log collector is able to keep up with the rate of logs streamed. App Services will not indefinitely buffer or maintain logs, therefore if the log collector falls behind, then you may lose logs. See Log rotation and retention.

      Restarts and Redeployments

      The logs are transient in App Services. Therefore if you redeploy (including to effect a log configuration change), then any logs that weren’t already streamed before the restart will be lost.

      When App Services comes back again, logging will also restart, with an entirely fresh set of logs.

      Turning the App Service or Cluster Off or On

      When an App Service is turned Off, any Log Streaming, if enabled for those App Services, will also be turned off.

      This also holds when the when the Capella cluster is turned Off to reduce costs, and any linked App Services are also turned off.

      When the App Services is turned back on again, Log Streaming will be started, if enabled, in that state it was in before the App Service was turned off (Paused or Running).

      Log rotation and retention

      You may want to know our log retention, to understand the maximum partition window. This may help identify if there is any danger of missing logs due to disruption.

      Due to the many factors previously discussed in Resource Considerations which affect your log streaming throughput (flow rate), we cannot give a general figure. However, once you know that rate, you can calculate a value for your use-case.

      For example, if you stream at 10 MB/hour/node, then the partition window is 100 hours (1 GB/node log retention, divided by 10 MB/hour/node).

      Calculate the log streaming throughput from the Prometheus metric fluentbit_output_proc_bytes_total.

      Role-Based Access

      Roles are enforced as follows:

      Role Permissions

      Org Owner, Project Owner

      Ability to update and view log.

      Project Manager

      Streaming configuration for App Services associated with projects that the user has access to.

      Project Viewer

      Ability to only view log streaming configuration for App Services associated with projects that the user has access to.

      Org Member

      Depends on Project level access granted to user. User with only Org Member role and no project access will not be able to access this feature.

      Cluster Data Reader, Cluster Data Reader/Writer

      Cannot view or update configuration.

      Compatibility guarantees

      The structure of the JSON logs (the key names and type of values sent) is stable.

      The contents of the message string in the JSON log field are an internal detail. The format and contents of this string are subject to change.

      Migration from Sync Gateway

      The underlying logging is based on Sync Gateway logging. This means that existing self-managed users of Sync Gateway should find the logging format familiar from console logging, and should in general be able to point App Services log streaming to any existing Datadog or Sumo Logic log collector agents.

      We expect that your log processing code on prem should continue to work with Capella App Services log streaming with no or minimal changes.

      Though the structure of the logs is stable, if your processing code relies on specific format of the log message, it may not work after migration. We don’t recommend scraping information from this field.

      Billing

      Couchbase bills data transfer costs for Log Streaming at the same rate as all other egress costs from App Services, and bundles these costs together in your account statement.

      Next Steps