cbbackupmgr cloud

Backing up directly to an external cloud provider will mean that we require permissions to access the given data store. Each provider has its own way of authentication; see the sections below on how to authenticate for your chosen cloud provider.

One of the most important concepts behind how backup to object store works is the staging directory. The staging directory is a location on disk where temporary data is stored during the execution of a sub-command. For a backup/restore this will be DCP metadata and storage indexes.

When creating an archive to store in a cloud provider you are required to provide a location for the obj-staging-dir. This is a local location where archive meta data will be stored. During a backup, files will be stored here before they are uploaded to the cloud. Note that cbbackupmgr doesn’t store any document values in the staging directory; they are streamed directly to the cloud.

Each cloud archive must have a unique staging directory i.e. they can’t be shared. cbbackupmgr will detect cases where the staging directory is being reused across archives.

Any modifications to the cloud archive (using the web-ui or cli tools) and not cbbackupmgr are not supported whilst using the same staging directory. If a cloud archive has been modified, the staging directory should be removed and recreated before using cbbackupmgr to interact with the archive again.

The staging directory is only used during operations e.g. backup/restore and can be safely deleted once an operation completes; this is because all the files will have been uploaded to the cloud.

The staging directory can become quite large during a normal backup depending on the number of documents being backed up, and the size of their keys. See the DISK REQUIREMENTS section for more information about how to provision the staging directory.

If the staging directory is empty, then cbbackupmgr will repopulate it using the files from the cloud archive. Depending on the size of the archive, the network speed, and the CPU resources available, this could take a very long time. As a rule of thumb, we recommend limiting the number of backups in a repository to around 50, and periodically creating new repositories. By doing this, you can avoid excessive staging directory repopulation times in case your staging directory is deleted.

The first step is to create a backup archive in object store. This can be done with the config command and only needs to be done once. All other commands will automatically download the archive meta data in the directory provided via the obj-staging-dir argument prior to performing any operations; this is done regardless of whether the archive exists locally because we must ensure the archive in the staging directory is up to date. Below is an example of how you would configure an archive in AWS S3.

Assuming your credentials are correct, the archive should now reside directly in the provided S3 bucket. To verify, you could use the aws-cli to list the contents off the bucket and they should be identical to that which would exist for a local backup.

Although it’s possible to have cbbackupmgr coexist in the same S3 bucket as other general purpose storage, we recommend using a bucket which cbbackupmgr has exclusive access too.

Once an archive is configured performing a backup works in a similar fashion to performing a local backup. It’s important to note that when backing up directly to S3 a certain amount of disk space will be used to stage local meta data files and storage indexes. See the THE STAGING DIRECTORY section for more information. Below is an example of doing a backup and storing directly in AWS S3.

Performing incremental backups works exactly as it would if you were performing an incremental locally; simply rerun the command above and an incremental backup would be created.

When choosing the amount of threads to use it’s important to consider that when backing up to the cloud, cbbackupmgr buffers data in memory before uploading it. This means that choosing an extremely large amount of threads when using a poor internet connection could lead to a scenario where your machine runs out of memory.

To learn more about backup options see cbbackupmgr-backup.

Once you have created a backup, restoring it works in a similar way to restoring a local backup. It’s worth noting that restoring a backup to a cluster that’s hosted outside of AWS is likely to be significantly more expensive than performing a backup (depending on the size of your dataset). See COSTING for more information.

Below is an example of restoring a backup that is store in AWS S3.

One of the main reasons for merging incremental backups is to save disk space. In AWS S3 space is cheap and bandwidth (to the broader internet) is expensive. This means that there isn’t a financially viable reason for merging cloud backups. For this reason merging incremental backups stored in the cloud is not supported.

Restoring will continue to support applying incremental backups in chronological order in the same fashion that it’s would when merging e.g. you will end up with the same data in your Couchbase cluster.

Several tools have been made available for use with archives stored directly in the cloud, currently these are:

It’s important that only one instance of cbbackupmgr has access to the archive at a time; this is enforced using a lockfile meaning most of the time you shouldn’t need to worry about this. However, there are some situations where cbbackupmgr may fail to ensure exclusive access to the archive:

In cases where cbbackupmgr fails to lock an archive a few simple steps can be taken:

It’s extremely important that you only manually remove the lockfile if you a certain that there isn’t another instance of cbbackupmgr using the archive. Having two instances of cbbackupmgr running against the same archive could cause data loss through overlapping key prefixes.

Below is an example of an archive which contains a lockfile from a system that crashed where the lockfile was never cleaned up.

If we attempt to use cbbackupmgr to create a backup, we should see a message similar to the one below:

In this case, cbbackupmgr will not remove the lock automatically since it cannot safely determine whether the other process is active or not. We can use the information about which machine the other instance of cbbackupmgr is running on to check whether it is active. If this machine has crashed and that instance of cbbackupmgr is no longer using the archive, we can manually remove the lockfile.

If we attempt to perform the backup once again we will see that it continues successfully; in the case that the other machine failed during a backup you may be asked to purge the previous backup using the --purge flag before you can create a new backup:

It should be possible to use cbbackupmgr with S3 compatible object stores, however, there are some things that need to be taken into consideration. First and foremost is the features that cbbackupmgr leverages. Below is a list of S3 API features that cbbackupmgr uses but not all compatible object stores support:

It’s important that you check whether these features are implemented on your S3 compatible object store because without them cbbackupmgr will not work as expected.

AWS also has a slightly newer virtual addressing style the documentation for which can be found at https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html. Not all S3 compatible object stores support this style of addressing. The errors that are returned by the SDK (and therefore cbbackupmgr) in these cases are not always clear. Before raising a support ticket about cbbackupmgr not working with an S3 compatible object store you should first try using the --s3-force-path-style argument. This will force cbbackupmgr to use the old path style addressing. From our testing with S3 compatible object stores it’s very common for this flag to be required.

As with AWS S3 support, Azure/GCP should work compatible object storage solutions. You also shouldn’t need to provide any specific flags when using compatible storage solutions.

It should however, be noted that Azure/GCP will only work when the underlying compatible object storage solution implements all the required features and it should be noted that this behavior is not exhaustively validated by Couchbase.

When running cbbackupmgr in an AWS instance, it may use the EC2 instance metadata to get credentials. This is disabled by default, however, may be enabled by either supplying the --obj-auth-by-instance-metadata flag or setting the CB_AWS_ENABLE_EC2_METADATA environment variable to true.

For example, if we wanted to use cbbackupmgr with the EC2 instance metadata we would: . Create a role with a policy which allows S3 data manipulation (e.g. S3 Full Admin) . Attach that role to the instance .. Run export CB_AWS_ENABLE_EC2_METADATA=true to enable fetching EC2 instance metadata .. Add the --obj-auth-by-instance-metadata flag to your cbbackupmgr command. . Run cbbackupmgr as described elsewhere in this tutorial

When running in an Azure VM, cbbackupmgr will attempt to fetch a service principle token from the environment which will be used to authenticate against blob storage. In this case, the correct RBAC permissions should be provided to allow cbbackupmgr access to the chosen storage account. As apposed to AWS, this behavior is enabled by default.

When running in Google Compute, cbbackupmgr will using the instance metadata server to fetch credentials which may be used to authenticate against Google Storage. As with AWS/Azure, the correct RBAC permissions should be provided to allow cbbackupmgr access to the given bucket. As apposed to AWS, this behavior is enabled by default.

The following is a list of the actions required by cbbackupmgr when interacting with a remote archive in AWS.

The following are the required RBAC roles required by cbbackupmgr when interacting with a remote archive in Azure.

The following are the required RBAC roles required by cbbackupmgr when interacting with a remote archive in GCP.