Data Operations

how-to

The Key Value (KV) service, sometimes called the "data service", is often the best way to get or change a document when you know its ID. Here we cover CRUD operations and locking strategies.

Before You Start

You should know how to connect to a Couchbase cluster.

You should know about documents and collections, and how to get a Couchbase Collection object.

You should know how to call a Kotlin suspending function.

CRUD Operations

The KV service has CRUD operations for working with whole documents. This table shows the Couchbase KV method for each CRUD operation.

CRUD operation	Couchbase KV method
Create	Collection.insert
Read	Collection.get
Update	Collection.replace
Delete	Collection.remove
Create or Update	Collection.upsert

CRUD operation

Couchbase KV method

Create

Read

Update

Delete

Create or Update

Insert (Create)

The insert method creates a new document in a collection.

This method has two required parameters:

id: String - The new document’s ID.
content: Any? - The new document’s value.

If the collection already has a document with the same ID, the insert method throws DocumentExistsException.

For example, let’s pretend we’re writing a program that helps a storyteller remember details about characters in a story.

To start, let’s insert a document that represents a character in a story. The document ID is the character’s name. The document content is some information about the character.

Creating a new document

try {
    collection.insert(
        id = "alice",
        content = mapOf("favoriteColor" to "blue"), (1)
    )
} catch (t: DocumentExistsException) {
    println("Insert failed because the document already exists.")
}

1	The content doesn’t have to be a `Map`. To learn more, please read Working with JSON.

Get (Read)

The get method reads a document from a collection.

It has one required parameter:

id: String - The ID of the document to get.

If the collection does not have a document with this ID, the get method throws DocumentNotFoundException.

Reading a document

try {
    val result: GetResult = collection.get(id = "alice")
    val content = result.contentAs<Map<String, Any?>>()
    println("The character's favorite color is ${content["favoriteColor"]}")
} catch (t: DocumentNotFoundException) {
    println("Get failed because the document does not exist.")
}

Replace (Update)

The replace method updates the value of an existing document.

This method has two required parameters:

id: String - The ID of the document to replace.
content: Any? - The document’s new value.

If the collection does not have a document with this ID, the replace method throws DocumentNotFoundException.

Updating an existing document

try {
    collection.replace(
        id = "alice",
        content = mapOf("favoriteColor" to "red"),
    )
} catch (t: DocumentNotFoundException) {
    println("Replace failed because there was no document to replace.")
}

When you replace a document, it’s usually good to use optimistic locking. Otherwise, changes might get lost if two people change the same document at the same time.

Remove (Delete)

The remove method deletes a document from a collection.

This method has one required parameter:

id: String - The ID of the document to remove.

If the collection does not have a document with this ID, the remove method throws DocumentNotFoundException.

Deleting a document

try {
    collection.remove(id = "alice")
} catch (t: DocumentNotFoundException) {
    println("Remove failed because there was no document to remove.")
}

Upsert (Create or Update)

The word "upsert" is a portmanteau word that means "update or insert."

If the document already exists, the upsert method updates (replaces) it. If the document does not exist, the upsert method inserts it.

This method has two required parameters:

id: String - The ID of the document to create or update.
content: Any? - The document’s new value.

Creating or updating a document

collection.upsert(
    id = "alice",
    content = mapOf("favoriteColor" to "blue"),
)

You can run this example many times. It should succeed each time, because the upsert method does not care if the document already exists.

Bulk Operations

The Couchbase Kotlin SDK does not have bulk operations. However, coroutines make it easy to do many things at the same time.

Here is an example bulkGet function you can add to your project. This function can get many documents at the same time.

Extension function Collection.bulkGet

/**
 * Gets many documents at the same time.
 *
 * @param ids The IDs of the documents to get.
 * @param maxConcurrency Limits how many operations happen
 * at the same time.
 * @return A map where the key is a document ID, and the value
 * is a [kotlin.Result] indicating success or failure.
 */
suspend fun com.couchbase.client.kotlin.Collection.bulkGet(
    ids: Iterable<String>,
    maxConcurrency: Int = 128,
): Map<String, Result<GetResult>> {
    val result = ConcurrentHashMap<String, Result<GetResult>>()
    val semaphore = kotlinx.coroutines.sync.Semaphore(maxConcurrency)

    coroutineScope { (1)
        ids.forEach { id ->
            launch { (2)
                semaphore.withPermit { (3)
                    result[id] = runCatching { get(id) }
                }
            }
        }
    }
    return result
}

1	Starting a new coroutine scope ensures the `bulkGet` method does not return until all coroutines launched inside the scope finish.
2	Start a new coroutine for each `get` operation, so all the gets happen at the same time.
3	The semaphore prevents sending too many requests at once. You can change the function to take the semaphore as a parameter, and pass the same semaphore whenever calling the method. If you do that, the concurrency limit will be shared by all bulk requests.

After adding the Collection.bulkGet extension function to your project, call it like this:

Calling the Collection.bulkGet extension function

val ids = listOf("airline_10", "airline_10123", "airline_10226")

collection.bulkGet(ids).forEach { (id, result) ->
    println("$id = $result")
}

You can copy the bulkGet extension function and change it to do other operations, like upsert.

Locking

What is an "atomic" operation?

An atomic operation succeeds completely or fails completely. When Couchbase Server works on an atomic operation, you never see the result of incomplete work. A failed atomic operation never changes a document.

If two or more atomic operations use the same document, Couchbase Server works on only one of the operations at a time.

A KV operation is atomic. However, a sequence of KV operations is not atomic.

You can use a locking strategy to make a sequence of KV operations on the same document succeed or fail together. This makes the sequence of operations behave like a single atomic operation.

The locking strategy can be optimistic or pessimistic.

Optimistic Locking

When you use optimistic locking, you assume nobody else will change a document while you work with it. If somebody else does change the document, start again. Keep trying until you succeed or decide to give up.

How do you tell if the document changed? Every Couchbase document has a Compare-And-Swap (CAS) value. The CAS value is a number that changes every time the document changes.

Most KV operations that change documents have a cas parameter. If you set this parameter, the operation fails with CasMismatchException if the document’s current CAS value does not match the cas parameter value.

Optimistic locking can make get and replace behave like an atomic unit:

Read a document using the get method. Remember the document’s CAS value.
Use the old document content to make new content. For example, add or remove a field, or change a field value.
Replace the document content using the replace method. Pass the new content and the CAS value from step 1. If replace throws CasMismatchException, start again at step 1.

If you pass a CAS value to replace, the operation succeeds only if nobody changed the document after you got the CAS value.

This example shows how to safely change a document, without losing changes made by somebody else at the same time:

while (true) { (1)
    val result: GetResult = collection.get(id = "alice")

    val oldContent = result.contentAs<Map<String, Any?>>()
    val newContent = oldContent + ("favoriteFood" to "hamburger")

    try {
        collection.replace(
            id = "alice",
            content = newContent,
            cas = result.cas
        )
        return

    } catch (t: CasMismatchException) {
        // Someone else changed the document after we read it!
        // Start again.
    }
}

1	This example keeps trying until the coroutine is cancelled. Another choice would be to set a time limit, or limit the number of tries.

You don’t need to write all of that code every time you want to use optimistic locking. Instead, you can define your own extension function like this:

suspend inline fun <reified T> Collection.mutate(
    id: String,
    expiry: Expiry = Expiry.none(),
    preserveExpiry: Boolean = false,
    transcoder: Transcoder? = null,
    durability: Durability = Durability.none(),
    common: CommonOptions = CommonOptions.Default,
    transform: (GetResult) -> T,
): MutationResult {
    while (true) {
        val old = get(
            id = id,
            withExpiry = preserveExpiry,
            common = common,
        )

        val newContent = transform(old)
        val newExpiry = if (preserveExpiry) old.expiry else expiry

        try {
            return replace(
                id = id,
                content = newContent,
                common = common,
                transcoder = transcoder,
                durability = durability,
                expiry = newExpiry,
                cas = old.cas
            )
        } catch (_: CasMismatchException) {
            // Someone else modified the document. Start again.
        }
    }
}

Now the optimistic locking example from before looks like this:

collection.mutate("alice") { old: GetResult ->
    val oldContent = old.contentAs<Map<String, Any?>>()
    return@mutate oldContent + ("favoriteFood" to "hamburger")
}

Pessimistic Locking

Pessimistic locking stops anyone except you from changing a document.

When a document is locked, only people who know the CAS value from getAndLock can modify the document.

The lock is released when you change the document using the correct CAS, or when you call the unlock method.

Changing a document safely, using pessimistic locking

val result: GetResult = collection.getAndLock(
    id = "alice",
    lockTime = 15.seconds, (1)
)

val oldContent = result.contentAs<Map<String, Any?>>()
val newContent = oldContent + ("favoriteFood" to "hamburger")

collection.replace( (2)
    id = "alice",
    content = newContent,
    cas = result.cas,
)

1	The lock is automatically released (unlocked) after this duration. The lock time can be as short as 1 second, or as long as 30 seconds.
2	`replace` automatically releases the lock. Alternatively, you can release the lock by calling `unlock`.

Pessimistic locking is expensive. It’s usually better to use optimistic locking.

Selecting Fields

The project() feature allows you to select a couple of fields — specify a path or paths within the JSON document, and this list is fetched rather than the whole document.

	project(Iterable<String> paths)
	project(String path, String... morePaths)

This feature is implemented by internally using our subdocument API, which you can access directly — for more sophisticated selection of portions of a document — subdocument-operations.adoc.

KV Range Scan

A range scan gives you documents from a collection, even if you don’t know the document IDs. This feature requires Couchbase Server 7.6 or newer.

KV range scan is suitable for use cases that require relatively low concurrency and tolerate relatively high latency. If your application does many scans at once, or requires low latency results, we recommend using SQL++ (with a primary index on the collection) instead of KV range scan.

Range scan

Here’s an example of a KV range scan that gets all documents in a collection:

KV Range Scan for all documents in a collection

val results: Flow<GetResult> = collection.scanDocuments(
    type = ScanType.range() (1)
)
results.collect { println(it) }

1 The ScanType.range() method has two optional parameters: from and to. If you omit them like in this example, you’ll get all documents in the collection. These parameters are for advanced use cases; you probably won’t need to specify them. Instead, it’s more common to use the "prefix" scan type shown in the next example.

Prefix scan

KV range scan can also give you all documents whose IDs start with the same prefix.

Imagine you have a collection where documents are named like this: <username>::<uuid>. In other words, the document ID starts with the name of the user associated with the document, followed by a delimiter, and then a UUID. If you use this document naming scheme, you can use a prefix range scan to get all documents associated with a user.

For example, to get all documents associated with user "alice", you would write:

KV Range Scan for all documents in a collection whose IDs start with alice::

val results: Flow<GetResult> = collection.scanDocuments(
    type = ScanType.prefix("alice::") (1)
)
results.collect { println(it) }

1	Note the scan type is prefix.

Sample scan

If you want to get random documents from a collection, use a sample scan.

KV Range Scan for 100 random documents

val results: Flow<GetResult> = collection.scanDocuments(
    type = ScanType.sample(limit = 100)
)
results.collect { println(it) }

Get IDs instead of full document

If you only want the document IDs, call scanIds() instead of scanDocuments(), like this:

KV Range Scan for all document IDs in a collection

val ids: Flow<String> = collection.scanIds( (1)
    type = ScanType.range()
)
ids.collect { println(it) }

1	Note the call to `scanIds()` instead of `scanDocuments()`.

The scanIds() methods also works with the other scan types described above.

Summary

The Couchbase Key Value (KV) service is the fastest way to work with single documents when you know the document ID.

The insert, get, replace, remove, and upsert methods of the Collection object do the standard CRUD operations on full documents.

When changing a document, use a locking strategy if the new content depends on the old content. Optimistic locking usually performs better than pessimistic locking.

You can do a KV range scan to get documents even if you don’t know their IDs. However, it’s faster to use SQL++ for this.