Configure Server Certificates

Couchbase Server Enterprise Edition supports X.509 certificates, for the encryption of communications between the server and networked clients.

Configure Server Certificates

This section demonstrates how server certificates can be configured for Couchbase Server. Note that the procedures are provided only as limited examples, giving guidance as to the basic steps typically involved in certificate creation. Modification of the procedures will likely be required, for the preparation of certificates for different platforms and cluster-configurations.

Two procedures are provided, each of which configures X.509 certificates on Ubuntu 16 for a one-node Couchbase Server-cluster. Before attempting to follow either procedure, see the conceptual and architectural information provided in Certificates.

The first procedure, Cluster Protection with Root and Node Certificates, is the simpler: it shows how to create a root certificate that is a trusted, self-signed authority; and how to use this to sign individual, per node certificates.

The second procedure, Cluster Protection with Root, Intermediate, and Node Certificates, demonstrates how a created root certificate and its private key are used to sign one or more intermediate certificates; which are then in turn used with their own private keys to sign individual, per node certificates: such use of intermediate certificates and their private keys increases security — in that it minimizes use of the private key associated with the root certificate, when the signing of many node-certificates or client-certificates is required.

Although these procedures demonstrate certificate management and deployment on single-node clusters, the steps they contain can also be used on multi-node clusters; with some of the steps being necessarily repeated, across the different nodes. See Protection of Multi-Node Clusters, below, for details.

Note also that once a cluster has been protected with administrator-defined certificates in accordance with these procedures, any new nodes subsequently to be added to the cluster must be individually protected with conformant certificates, before addition can take place. See Adding New Cluster-Nodes, below, for details.

Using an Externally Provided Root Certificate

The examples below show how to create a root certificate, and how to use that certificate’s private key to sign (and thereby confer limited authority to) other certificates. However, production deployments of Couchbase Server will frequently deploy an externally provided root certificate for the cluster, this having been provided by a recognized certificate authority. Additionally, an intermediate (sometimes referred to as a subordinate) certificate, defined by the system administrator for the production environment, is likely to have been signed by the external authority, using its root certificate’s private key: this allows the intermediate certificate and its private key to be used by the system administrator in creating additional certificates for deployment, each of these certificates thereby acquiring authority indirectly from the root.

Therefore, to use the procedures provided below in the context of a root certificate having been provided by, and an intermediate certificate signed by, an external authority, substitute the externally provided root certificate for the generated ca.pem; use the externally signed intermediate in place of the generated intermediate; and use the private key of the externally signed intermediate to sign node and client certificates as appropriate.

Node-to-Node Encryption and Certificate Management

Couchbase Server supports Node-to-Node Encryption, whereby network traffic between the individual nodes of a cluster is encrypted.

Node-to-node encryption, which is managed by means of the Couchbase CLI, must be disabled before management of either root or intermediate certificates is performed. To determine the status of node-to-node encryption, use the node-to-node-encryption command with the --get flag. If the command output shows that node-to-node encryption is enabled, disable it; using the same command, with the --disable flag. Once management of node and intermediate certificates has been concluded, use the same command with the --enable flag, to re-enable node-to-node encryption.

Cluster Protection with Root and Node Certificates

The following procedure shows how to create a root certificate that is the trusted, self-signed authority used to sign individual, per node certificates. Note that corresponding, subsequent procedures that create certificates for client-authentication are provided in Client Access: Root Certificate Authorization and Java Client Access: Root Certificate Authorization.

Proceed as follows:

On the server to be certificate-protected, create working directories:
```
mkdir servercertfiles
cd servercertfiles
mkdir -p {public,private,requests}
```
The public directory will be used to store certificates, which contain public keys. The private directory will contain private keys. The requests directory will store certificate signing requests, which are files generated from private keys: when a signing request is granted the signature of an appropriate authority, a signed certificate is produced.
Create a private key for the cluster.

A private key can be used to decrypt data previously encrypted by the corresponding public key. It can also be used to sign a message that is then sent by the client to the server; allowing the client’s identity to be verified by the server, using the client’s public key. In the key-creation sequence, the private key is created first. Then, the public key is created, being derived from the private key.

Enter the following:
```
openssl genrsa -out ca.key 2048
```
The output of this command, ca.key, is the private key for the cluster.
Create the certificate (that is, the file that will contain the public key) for the cluster. The certificate is intended to be self-signed, meaning that it will not be vouched for by any other authority. This means that it can be created directly, based on the existing private key ca.key, without assistance from a third party.

Enter the following:
```
openssl req -new -x509 -days 3650 -sha256 -key ca.key -out ca.pem \
-subj "/CN=Couchbase Root CA"
```
The x509 flag indicates that in this case, an x509 structure, rather than a request is to be generated. (By contrast, a request will need to be generated whenever the signature of a third-party authority is required: this is demonstrated below.) The days flag specifies the number of days for which the certificate should be active. The hashing algorithm to be used for digital-signature creation is specified as sha256. The private key file on which the certificate is to be based is specified as ca.key, and the output-certificate is named as ca.pem. The certificate’s issuer is specified to have the CN (Common Name) of Couchbase Root CA: as this name indicates, the certificate will be the root certificate for the Couchbase Server-cluster.

The output of the command is the certificate ca.pem; which contains the public key corresponding to the cluster’s private key, ca.key.

Optionally, the public key within the certificate can be displayed as follows:
```
openssl x509 -in ./ca.pem -noout -pubkey
```
The output has approximately the following appearance:
```
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3HMfiSjCwakfMbA20HUd
V372JbQG9UGjf9V3xyMa90IHFD8cFjPYao7SZOpe0nkm2UmZRgQbTwWxC4CZqrYZ
pyrWLn9rjDFkzzbRjMRcZv2D0s0KkPrNYxfHj3cL/j5bpB4/hquvb4RglMkyyJo9
mVx19lF4mtEsBqPGZBGArbzeArn4c1e6I4mqIfb9Vne/7vhIzLLSXoT5FmifWyGQ
4B9BSIrE9Ildwhez699MGfj+N+0xg2wTOIUVNvS1c5gF/uDS6t9Aswb60W+hjtF4
d1ZBKBIVkmPGX0XOgGtdndXza4sjVkh3bB/ipWo9zUJYwFCWkofbqGeSnSz9n9o6
fwIDAQAB
-----END PUBLIC KEY-----
```
Note that by substituting other flags for -pubkey, other characteristics of the certificate can be displayed. -issuer displays the certificate’s issuer, and -subject its subject (in both cases, subject= /CN=Couchbase Root CA). The -version, -serial, -subject-hash, and more can be displayed.

The entire certificate can be displayed as text, by means of the following command:
```
openssl x509 -text -noout -in ./ca.pem
```
The initial part of the output, which is extensive, is as follows:
```
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 18276610881715621025 (0xfda390c366b2cca1)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Couchbase Root CA
        Validity
            Not Before: Sep  2 08:32:31 2019 GMT
            Not After : Aug 30 08:32:31 2029 GMT
        Subject: CN=Couchbase Root CA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:d7:a6:ba:5d:e2:e2:fd:6e:1b:33:9a:4b:bf:77:
                    6f:28:c3:37:60:33:da:09:b2:0b:73:1f:f9:65:2a:
                                  .
                                  .
```
The displayed text thus provides information including the Version, the Serial Number, and the Signature Algorithm of the certificate. The certificate’s Issuer, Subject, and period of Validity are also shown. The Algorithm and Modulus (and, further below, the Exponent) of the public key are shown.

For detailed information on keys and key-generation, see RSA (cryptosystem).
Create a private key for the individual node.

In addition to the root certificate and private key for the entire cluster, which are ca.pem and ca.key, a node certificate and private key must also be created. The node certificate, along with its corresponding node-private key, will reside on its own, corresponding node. When deployed, each node certificate must be named chain.pem, and each node private key pkey.key. Consequently, if the node certificates and private keys for multiple nodes are being prepared on a single system, the files should be given individual, distinctive names on creation; and then each deployed on its appropriate node as either chain.pem or pkey.key. This renaming procedure is indeed followed here for demonstration purposes, even though only a one-node cluster is involved.

Create the node private key as follows:
```
openssl genrsa -out private/couchbase.default.svc.key 2048
```
The output file is couchbase.default.svc.key, which is the private key for the node.
Create a certificate signing request for the node certificate. This step allows the materials required for certificate-creation to be passed to a third-party, who will digitally sign the certificate as part of its creation-process, and thereby confirm its validity. (In this demonstration, however, no actual third-party is involved: the certificate will be signed by means of the root private key, which is owned by the current user.)

Enter the following command:
```
openssl req -new -key private/couchbase.default.svc.key \
-out requests/couchbase.default.svc.csr -subj "/CN=Couchbase Server"
```
The key specified as the input for the request is couchbase.default.svc.key, which was created in the last step. The output request-file is specified as couchbase.default.svc.csr. Note that this can be inspected as text, by entering the following command:
```
openssl req -text -noout -verify -in ./requests/couchbase.default.svc.csr
```
The initial part of the displayed output, which is extensive, is as follows:
```
verify OK
Certificate Request:
    Data:
        Version: 0 (0x0)
        Subject: CN=Couchbase Server
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:be:26:e5:06:c6:8e:43:bb:9d:bc:84:20:34:8e:
                    db:2f:d1:8b:b4:ff:c2:66:c0:61:70:8d:c3:8c:df:
                                      .
                                      .
```
The Version and Subject of the request are listed, along with information on the public key that is to be included in the certificate.
Define certificate extensions for the node.

Certificate extensions specify constraints on how a certificate is to be used. Extensions are submitted to the signing authority, along with the certificate signing request.

For example, the certificate’s public key can be specified, by means of the keyUsage extension, to support digital signatures, but not to support key encipherment — or, the opposite can be specified; or, support of both digital signatures and key encipherment can be specified. Meanwhile, the subjectAltName extension can be used to specify the DNS name and IP address of the server on which the certificate resides; so that if the certificate is deployed in any other context, it becomes invalid.

For detailed information on certificate extensions, see the Standard Extensions section of the Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL Profile).

Certificate extensions can be defined in a file, whose pathname is then provided as a parameter to the openssl command used to create the certificate. Thus, such server-certificate extensions as are intended to be generic across all cluster-nodes might be written as follows:
```
cat > server.ext <<EOF
basicConstraints=CA:FALSE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
extendedKeyUsage=serverAuth
keyUsage = digitalSignature,keyEncipherment
EOF
```
The value of extendedKeyUsage is specified as serverAuth, indicating that the certificate is to be used for server authentication. The values of keyUsage are digitalSignature, specifying that the certificate’s public key can be used in the verifying of information-origin; and keyEncipherment, specifying that the public key can be used in the encrypting of symmetric keys (through the exchange and use of which symmetrically encrypted communications between server and client can occur).
Create a customized certificate extensions file, which adds per node constraints to the generic constraints already specified.
```
cp ./server.ext ./server.ext.tmp

echo "subjectAltName = IP:10.143.192.102" \
>> ./server.ext.tmp
```
This customized extensions file is to be used to authenticate a single node, whose IP address is 10.143.192.102. Note that if the DNS naming-convention is used by the cluster, the node’s DNS name might be specified instead: for example, DNS:node2.cb.com. If the node is not identified appropriately in the certificate, authentication fails.

The creation of the customized extensions file should occur once for each node, with each customized extensions file containing only those extensions that apply to the current node.
Create the node certificate, applying the certificate and digital signature of the appropriate authority, and the customized extensions file for the node, to the materials in the signing request.

Enter the following:
```
openssl x509 -CA ca.pem -CAkey ca.key -CAcreateserial -days 365 -req \
-in requests/couchbase.default.svc.csr \
-out public/couchbase.default.svc.pem \
-extfile server.ext.tmp
```
The file generated by this command, couchbase.default.svc.pem, is the node certificate. The root certificate and private key, ca.pem and ca.key, are specified as input values to the certificate-creation command. This ensures that the new certificate’s chain of trust includes the root certificate, ca.pem, and is digitally signed by ca.key; allowing that signature to be verified by means of the public key.

The following confirmatory output is displayed:
```
Signature ok
subject=/CN=Couchbase Server
Getting CA Private Key
```
Note that if a node certificate were actually submitted to an external authority for signing, then the authority’s own pem and key would be specified as inputs, rather than ca.pem and ca.key: and in such a case, the authority’s pem would need to become the root certificate for the cluster.
Rename the node certificate and node private key.

For deployment on the node, the node certificate must be renamed chain.pem; and the node private key renamed pkey.key. Proceed as follows:
```
cd ./public
mv couchbase.default.svc.pem chain.pem
cd ../private
mv couchbase.default.svc.key pkey.key
```
Deploy the node certificate and node private key.

These are deployed by being moved to the inbox directory of the server, and made executable. The inbox directory must be created by the administrator. Proceed as follows:
```
cd ..
sudo mkdir /opt/couchbase/var/lib/couchbase/inbox/
sudo cp ./public/chain.pem /opt/couchbase/var/lib/couchbase/inbox/chain.pem
sudo cp ./private/pkey.key /opt/couchbase/var/lib/couchbase/inbox/pkey.key
```
Upload the root certificate for the cluster. Use the following REST command:
```
curl -X POST --data-binary "@./ca.pem" \
http://Administrator:password@10.143.192.102:8091/controller/uploadClusterCA
```
The root certificate is now activated for the entire cluster, and ready for use. This can be verified by means of Couchbase Web Console: access the Security screen, by means of the Security tab in the left-hand navigation bar. Then, left-click on the Root Certificate tab, located on the upper, horizontal navigation bar.

The screen appears as follows:

As this indicates, the signed certificate has now been substituted for the default certificate (an example of whose appearance is provided in Root Certificate).
Reload the node certificate from disk, for the current node:
```
curl -X POST \
http://Administrator:password@10.143.192.102:8091/node/controller/reloadCertificate
```
The node certificate is now activated for the current node. Note that when, as is typical, the cluster contains more than one node, this step must be performed on each node of the cluster, with each individual IP address thereby specified in turn.

For more information using the REST API to manage certificates, see Certificate Management API.

Configuring Client Access

Once the cluster has been protected by the deployment of root and node certificates described above, a client certificate can be signed by the root certificate, to allow a client to access the cluster. Client-certificate preparation varies, depending on the type of client to be supported. For steps to prepare a certificate supportive of Couchbase Server, see Client Access: Root-Certificate Authorization. For steps to prepare a certificate supportive of a Java client, see Java Client Access: Root-Certificate Authorization.

Note that access by means of a client certificate must be specifically enabled, on the cluster that is to be accessed: see Enable Client-Certificate Handling.

Cluster Protection with Root, Intermediate, and Node Certificates

Optionally, a root certificate can be used to sign an intermediate certificate, which is then itself used to sign node certificates. This increases security, since it minimizes use of the private key associated with the root certificate, when many node or client-certificates are to be signed.

The steps and descriptions below assume that the previous procedure, Cluster Protection with Root and Node Certificates, has already been successfully completed; and that familiarity with the basic certificate-related concepts explained there has been attained.

Note that corresponding, subsequent procedures that create certificates for client-authentication are provided in Client Access: Intermediate Certificate Authorization and Java Client Access: Intermediate Certificate Authorization

Proceed as follows:

On the server to be certificate-protected, create working directories:
```
mkdir servercertfiles2
cd servercertfiles2
mkdir -p {root,servers,clients}/{issued,reqs,private}
```
The directories root, servers, and clients will contain the issued certificates, requests, and private keys generated for the root, the individual nodes, and clients wishing to access the nodes. Each directory therefore contains issued, reqs, and private subdirectories.

Note that this directory infrastructure will also be used in the subsequent process, Client Access: Intermediate Certificate Authorization; where the contents of the clients directory will be created.
Change directory to root. Then, create a configuration file for the root certificate that is to be created.
```
cd root

cat > config <<EOF
[req]
distinguished_name = cn_only
x509_extensions = ca_ext
[ cn_only ]
commonName = Common Name (eg: your user, host, or server name)
commonName_max = 64
commonName_default = CA
[ca_ext]
basicConstraints = CA:TRUE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer:always
keyUsage = cRLSign, keyCertSign
EOF
```
The config file has three sections. The first, req, specifies values to be passed to the req command, which is used to create and process certificate requests: use man req to obtain information on the values passed. The second section, cn_only, provides specifications for the Common Name to be used in the certificate, including the maximum number of characters and the default name. The third section, ca_ext, provides basic extensions that limit the capability of the certificate. These include a value of TRUE for CA, indicating that the certificate will be able to provide signing authority for other certificates. Additionally, the values for keyUsage are provided as cRLSign, indicating that the certificate’s public key will be usable to verify signatures on Certificate Revocation Lists; and keyCertSign, indicating that the certificate’s public key will be usable to verify signatures on other certificates.
Create the root certificate, specifying the created config file.
```
openssl req -config config -new -x509 -days 3650 -sha256 -newkey rsa:2048 \
-keyout ca.key -out ca.pem -subj '/C=UA/O=MyCompany/CN=RootCA'
```
This specifies that both the root certificate for the cluster and its private key be created. The key is additionally specified to be encrypted. In consequence, during execution, the following prompt is displayed:
```
Generating a 2048 bit RSA private key
....+++
...................+++
writing new private key to 'ca.key'
Enter PEM pass phrase:
```
This requires that a pass phrase be entered, for inclusion of the key in command-line procedures, such as those used for certificate generation. The phrase will be stored in the certificate, and prompted for whenever administrative access is attempted. Enter an appropriate phrase: a second prompt then appears, requesting confirmation of the phrase. Enter the phrase again, and the operation completes.

The output file, ca.pem is the root certificate for the cluster, and is saved in the root folder. (Note that in the steps that follow, other certificates named ca.pem are created in additional folders: these should not be confused with the certificate of the same name in root.)
Create an extensions file that will limit the capabilities of the intermediate certificate that is to be created.

Enter the following:
```
cat > ca.ext <<EOF
basicConstraints = CA:TRUE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer:always
keyUsage = cRLSign, keyCertSign
EOF
```
Here, CA is set to TRUE, meaning that the intermediate certificate will be able to act as an authority for other certificates (specifically, for the individual, per node certificates used by the cluster). The specified keyUsage includes the value keyCertSign, meaning that the intermediate certificate’s public key will be used to verify signatures that appear on other certificates.
Create a private key and corresponding certificate signing request for the intermediate certificate.
```
openssl req -new -sha256 -newkey rsa:2048 -keyout ../servers/ca.key \
-out reqs/server-signing.csr \
-subj '/C=UA/O=MyCompany/OU=Servers/CN=ServerSigningCA'
```
Again, the key is specified to be encrypted. Therefore, prompts appear, asking for a pass phrase for the certificate. Enter an appropriate phrase in response to the prompts.

The output from the request consists of the encrypted private key ../servers/ca.key and the signing-request req/server-signing.csr.
Create the intermediate certificate, specifying the root certificate ca.pem and its key ca.key, to establish the root certificate’s authority.
```
openssl x509 -CA ca.pem -CAkey ca.key -CAcreateserial \
-CAserial serial.srl -days 3650 -req -in reqs/server-signing.csr \
-out issued/server-signing.pem -extfile ca.ext
```
Since this specifies that the encrypted key ca.key be used to sign the intermediate certificate, the user is prompted for the appropriate pass phrase. Enter the phrase against the prompt.

The extension file ca.ext is thus applied to the certificate, so as to limit the certificate’s capabilities. The certificate is generated and saved in the reqs folder as server-signing.pem.
Save the intermediate certificate as the authority for the node certificates that are to be created.
```
cp issued/server-signing.pem ../servers/ca.pem
```
Within the ../servers directory, create an extension file containing the information that will be generic across all the individual nodes of the cluster.
```
cd ../servers

cat > server.ext <<EOF
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
extendedKeyUsage = serverAuth
keyUsage = digitalSignature,keyEncipherment
EOF
```
The extendedKeyUsage value serverAuth indicates that the certificate will be used for server authentication. The keyUsage value digitalSignature specifies that the certificate’s public key can be used in the verifying of information-origin; while keyEncipherment allows the public key to be used in the encrypting of symmetric keys.
Generate the private key to be used for the individual cluster-node.
```
openssl genrsa -out private/couchbase.node.svc.key 2048
```
The private key couchbase.node.svc.key is thus saved in the private folder, as the private key for the node.

Generate the certificate signing request for the node certificate.

openssl req -new -key private/couchbase.node.svc.key \
-out reqs/couchbase.node.svc.csr \
-subj "/C=UA/O=MyCompany/OU=Servers/CN=couchbase.node.svc"

The signing-request file couchbase.node.svc.csr is thus saved in the reqs folder.

Add node-specific information for each node, in turn. Although the current example features a single-node cluster, this step would be repeated for each node in the cluster, if the cluster contained multiple nodes: in each case, the node-specific information (here, the node’s IP address) being different.
```
cp server.ext temp.ext

echo 'subjectAltName = IP:10.143.192.102' >> temp.ext
```
This creates temp.ext as an extension file that will be used for one node only. The file specifies the IP address specific to the node.
Create the node certificate for an individual node, specifying the unique extension file for the node, and specifying the intermediate certificate and key as the signing authority.
```
openssl x509 -CA ca.pem -CAkey ca.key -CAcreateserial \
-CAserial serial.srl -days 365 -req -in reqs/couchbase.node.svc.csr \
-out issued/couchbase.node.svc.pem -extfile temp.ext
```
Since this specifies that the certificate should be signed by the encrypted intermediate key, ca.key, a prompt appears, requesting the appropriate pass phrase. Enter the phrase against the prompt.

The node-certificate file couchbase.node.svc.pem is hereby saved in the issued folder. The certificate bears the constraints specified in temp.ext, and is granted the authority of the intermediate certificate and key, which are ca.pem and ca.key respectively.
Check that the node certificate is valid. The following use of the openssl command verifies the relationship between the root certificate, the intermediate certificate, and the node certificate.
```
openssl verify -trusted ../root/ca.pem -untrusted ca.pem \
issued/couchbase.node.svc.pem
```
If the certificate is valid, the following output is displayed:
```
issued/couchbase.node.svc.pem: OK
```
Prepare to deploy the certificate and private key for the node. First, concatenate the node certificate and the intermediate certificate, to establish the chain of authority. Then, rename the private key for the node.
```
cat issued/couchbase.node.svc.pem ca.pem > chain.pem

cp private/couchbase.node.svc.key pkey.key
```

Move the node certificate and node private key into the inbox for the current node.

sudo mkdir /opt/couchbase/var/lib/couchbase/inbox/  # if needed

sudo cp ./chain.pem /opt/couchbase/var/lib/couchbase/inbox/chain.pem
sudo cp ./pkey.key /opt/couchbase/var/lib/couchbase/inbox/pkey.key

Upload the root certificate, thereby activating it for the entire cluster. Then, reload the node certificate from disk, for the current node.
```
cd ../root

curl -X POST --data-binary "@./ca.pem" \
http://Administrator:password@10.143.192.102:8091/controller/uploadClusterCA

curl -X POST http://Administrator:password@10.143.192.102:8091/node/controller/reloadCertificate
```
Note that when, as is typical, the cluster contains more than one node, the /node/controller/reloadCertificate command must be executed on each node, specifying the IP address of the node on which execution is occurring.

This concludes the certificate-deployment process. The root certificate can be examined by means of Couchbase Web Console, as shown in Step 11 of the previous example on this page.

For more information using the REST API to manage certificates, see Certificate Management API.

Configuring Client Access

Once the cluster has been protected by the deployment of root, intermediate, and node certificates described above, a client certificate can be signed by a client-intermediate certificate that itself inherits the authority of the root: this allows the client certificate to access the cluster. Client-certificate preparation varies, depending on the type of client to be supported. For steps to prepare a certificate supportive of Couchbase Server, see Client Access: Intermediate-Certificate Authorization. For steps to prepare a certificate supportive of a Java client, see Java Client Access: Intermediate-Certificate Authorization.

Note that access by means of a client certificate must be specifically enabled, on the cluster that is to be accessed: see Enable Client-Certificate Handling.

Protection of Multi-Node Clusters

The certificate-management procedures described above, for single-node clusters, can be used for multi-node clusters, if the following modifications are made:

A separate chain.pem must be prepared for each node. Each chain.pem should be generated from a new, unique private key (pkey.key); must be an appropriate concatenation of the node certificate with whatever intermediate certificates have formed its chain; and must have its own node’s IP address specified as a subjectAltName.
If created on the same system as all other keys and certificates, the chain.pem and pkey.key for each node must be independently transferred onto the node they are intended to protect. An inbox must be created on that node, and the chain.pem and pkey.key files then moved there.
The node certificate must be reloaded individually for each node in the cluster, after the chain.pem and pkey.key file have been moved into the node’s inbox. Each reload command must therefore specify the node’s own IP address.

Since this is a gradual process, prior to its conclusion, the certificates for the cluster will go through periods of mismatch. Error messages are duly provided:

When the root certificate is uploaded and reloaded for any one of the nodes in the cluster, it becomes the root certificate for all the nodes in the cluster. However, for each node whose corresponding chain.pem and pkey.key files have not yet been copied to its inbox, the following error message appears on the Root Certificate panel of the Couchbase Web Console Security screen: Warning: ns_1:<ip-address>: Certificate is not signed with cluster CA.

Respond by copying the chain.pem and pkey.key for the node into its inbox, then perform a reload of the node certificate, specifying the IP address of the node. This coordinates and activates the new certificates for the node, and removes the error message.
If a new cluster certificate is uploaded and reloaded for a node before the node’s chain.pem and pkey.pem have been copied into the node’s inbox, the successful reload gives the following error message: "Unable to read certificate chain file /opt/couchbase/var/lib/couchbase/inbox/chain.pem. The file does not exist".

Respond by copying the chain.pem and pkey.pem files to the inbox, and then reloading the node certificate, specifying the node’s IP address. This coordinates and activates the new certificates for the node.

Adding and Joining New Nodes

The default certificates provided with Couchbase Server allow nodes to be either added or joined to a cluster, without any explicit certificate-management required. This is because the Cluster Manager automatically handles the creation and deployment of the default cluster and node certificates.

However, when new cluster and other certificates have been (as in the examples above) explicitly prepared by the system administrator and deployed on an existing cluster, no new node can become a member of that cluster until conformant certificates have been explicitly deployed on the node by the administrator. Consequently, a node can be added or joined to a cluster only after the appropriate certificates have been deployed on it. If certificate-preparation has not been completed, the following error message appears in the Add Server Node dialog of Couchbase Web Console: Attenton: Prepare join failed. failed to establish TLS connection to <ip-address>:8091: "unknown ca".

Therefore, to perform node-addition, prepare a chain.pem and pkey.key for the node; transfer these files to the running node, create an inbox, and move the files into the inbox; then upload the root certificate for the cluster to which the node is to be added; the reload the node certificate for the node that is to be added. When this sequence has been completed, the new node can be added to the cluster. (Note that although other aspects of a node’s prior provisioning will be erased by the addition, the new certificates will remain in place.)

Re-Adding a Previously Removed Node

When a node is removed from a cluster, its configuration is deleted. If the removed node is subsequently re-added to the cluster, it is added as a new node, with a new definition of its configuration. Consequently, the appropriate root certificate and chain certificate for the node must again be loaded.

For more information on node removal, see Removal.

Regenerating the Default Certificates

The default certificates provided automatically by Couchbase Server — including the root certificate and each of the node certificates for the cluster — can be regenerated at any time, by means of the REST API. This means that the administrator-uploaded certificates are removed, and default certificates re-substituted. For information, see Regenerate All Certificates.

Further Information

For information on certificate-management by means of the REST API, see ssl-manage and Certificate Management API.

For step-by-step instructions on creating client certificates, see Configure Client Certificates.

For an example of using the certificates and keys created on the current page and on Configure Client Certificates to secure an XDCR replication, see Specify Root and Client Certificates, and Client Private Key.