Sub-Document Operations with the C SDK

how-to

Sub-document operations can be used to efficiently access parts of documents. Sub-document operations may be quicker and more network-efficient than full-document operations such as upsert, replace and get because they only transmit the accessed sections of the document over the network. Sub-document operations are also atomic, allowing safe modifications to documents with built-in concurrency control.

Sub-documents

Starting with Couchbase Server 4.5 you can atomically and efficiently update and retrieve parts of a document. These parts are called sub-documents.

While full-document retrievals retrieve the entire document and full document updates require sending the entire document, sub-document retrievals only retrieve relevant parts of a document and sub-document updates only require sending the updated portions of a document. You should use sub-document operations when you are modifying only portions of a document, and full-document operations when the contents of a document is to change significantly.

The sub-document operations described on this page are for Key-Value requests only: they are not related to sub-document SQL++ (formerly N1QL) queries. (Sub-document SQL++ queries are explained in the section Querying with SQL++.)

Considering the document:

customer123.json

{
  "name": "Douglas Reynholm",
  "email": "douglas@reynholmindustries.com",
  "addresses": {
    "billing": {
      "line1": "123 Any Street",
      "line2": "Anytown",
      "country": "United Kingdom"
    },
    "delivery": {
      "line1": "123 Any Street",
      "line2": "Anytown",
      "country": "United Kingdom"
    }
  },
  "purchases": {
    "complete": [
      339, 976, 442, 666
    ],
    "abandoned": [
      157, 42, 999
    ]
  }
}

The paths name, addresses.billing.country and purchases.complete[0] are all valid paths.

Retrieving

The lookup-in operations query the document for certain path(s); these path(s) are then returned. You have a choice of actually retrieving the document path using the subdoc-get sub-document operation, or simply querying the existence of the path using the subdoc-exists sub-document operation. The latter saves even more bandwidth by not retrieving the contents of the path if it is not needed.

Check existence of Subdocument path

        check(lcb_subdocspecs_exists(specs, 2, 0, paths[2].c_str(), paths[2].size()),
                "create SUBDOC-EXISTS operation");

Retrieve Subdocument value

        check(lcb_subdocspecs_get(specs, 1, 0, paths[1].c_str(), paths[1].size()),
                "create SUBDOC-GET operation");

See the code sample for use in context.

Choosing an API

libcouchbase is an asynchronous library which means that operations results are passed to callbacks you define rather than being returned to functions. Callbacks are passed a cookie parameter which is a user defined pointer (i.e your own pointer, which can be NULL) to associate a specific command with a specific callback invocation.

        lcb_cmdstore_create(&scmd, LCB_STORE_UPSERT);
        lcb_cmdstore_key(scmd, key.data(), key.size());
        lcb_cmdstore_value(scmd, value.data(), value.size());

        err = lcb_store(instance, nullptr, scmd);
        lcb_cmdstore_destroy(scmd);
        if (err != LCB_SUCCESS) {
            die("Couldn't schedule storage operation", err);
        }
        lcb_wait(instance, LCB_WAIT_DEFAULT);

For simple synchronous use, you will need to call lcb_wait() after each set of scheduled operations. During lcb_wait the library will block for I/O, and invoke your callbacks as the results of the operations arrive.

            // This snippet lives inside the callback, so it is not necessary to call lcb_wait here
            lcb_CMDSTORE *cmd;
            lcb_cmdstore_create(&cmd, LCB_STORE_INSERT);
            lcb_cmdstore_key(cmd, key, nkey);
            lcb_cmdstore_value(cmd, value, nvalue);

            lcb_STATUS err = lcb_store(instance, nullptr, cmd);
            lcb_cmdstore_destroy(cmd);
            if (err != LCB_SUCCESS) {
                die("Couldn't schedule storage operation", err);
            }

See the code sample for use in context.

Mutating

Mutation operations modify one or more paths in the document. The simplest of these operations is subdoc-upsert, which, similar to the fulldoc-level upsert, will either modify the value of an existing path or create it if it does not exist.

        lcb_CMDSTORE *cmd = nullptr;
        check(lcb_cmdstore_create(&cmd, LCB_STORE_UPSERT), "create UPSERT command");
        check(lcb_cmdstore_key(cmd, key.c_str(), key.size()), "assign ID for UPSERT command");
        check(lcb_cmdstore_value(cmd, value.c_str(), value.size()),
                "assign value for UPSERT command");
        check(lcb_store(instance, nullptr, cmd), "schedule UPSERT command");
        check(lcb_cmdstore_destroy(cmd), "destroy UPSERT command");
        lcb_wait(instance, LCB_WAIT_DEFAULT);

See the code sample for use in context.

mutateIn is an atomic operation. If any single ops fails, then the entire document is left unchanged.

Array insertion

New elements can also be inserted into an array. While append will place a new item at the end of an array and prepend will place it at the beginning, insert allows an element to be inserted at a specific position. The position is indicated by the last path component, which should be an array index. For example, to insert "42" as the last element in the array [1,2,3,4], the code would look like:

std::string value_to_add{ "42" };
check(lcb_subdocspecs_array_add_last(specs, 0, 0, paths[0].c_str(), paths[0].size(), value_to_add.c_str(), value_to_add.size()),"create ARRAY_ADD_LAST operation");

Note that the array must already exist and that the index must be valid (i.e. it must not point to an element which is out of bounds).

Executing multiple operations

Multiple sub-document operations can be executed at once on the same document, allowing you to retrieve or modify several sub-documents at once. When multiple operations are submitted within the context of a single lookup-in or mutate-in command, the server will execute all the operations with the same version of the document.

Unlike batched operations which is simply a way of sending multiple individual operations efficiently on the network, multiple subdoc operations are formed into a single command packet, which is then executed atomically on the server. You can submit up to 16 operations at a time.

When submitting multiple mutation operations within a single mutate-in command, those operations are considered to be part of a single transaction: if any of the mutation operations fail, the server will logically roll-back any other mutation operations performed within the mutate-in, even if those commands would have been successful had another command not failed.

When submitting multiple retrieval operations within a single lookup-in command, the status of each command does not affect any other command. This means that it is possible for some retrieval operations to succeed and others to fail. While their statuses are independent of each other, you should note that operations submitted within a single lookup-in are all executed against the same version of the document.

Creating parents

Sub-document mutation operations such as subdoc-upsert or subdoc-insert will fail if the immediate parent is not present in the document. Consider:

{
    "level_0": {
        "level_1": {
            "level_2": {
                "level_3": {
                    "some_field": "some_value"
                }
            }
        }
    }
}

Looking at the some_field field (which is really level_0.level_1.level_2.level_3.some_field), its immediate parent is level_3. If we were to attempt to insert another field, level_0.level_1.level_2.level_3.another_field, it would succeed because the immediate parent is present. However if we were to attempt to subdoc-insert to level_1.level_2.foo.bar it would fail, because level_1.level_2.foo (which would be the immediate parent) does not exist. Attempting to perform such an operation would result in a Path Not Found error.

By default the automatic creation of parents is disabled, as a simple typo in application code can result in a rather confusing document structure.

Error handling

Subdoc operations have their own set of errors. When programming with subdoc, be prepared for any of the full-document errors (such as Document Not Found) as well as special sub-document errors which are received when certain constraints are not satisfied. Some of the errors include:

Path does not exist: When retrieving a path, this means the path does not exist in the document. When inserting or upserting a path, this means the immediate parent does not exist.
Path already exists: In the context of an insert, it means the given path already exists. In the context of array-add-unique, it means the given value already exists.
Path mismatch: This means the path may exist in the document, but that there is a type conflict between the path in the document and the path in the command. Consider the document:
```
{ "tags": ["reno", "nevada", "west", "sierra"] }
```
The path tags.sierra is a mismatch, since tags is actually an array, while the path assumes it is a JSON object (dictionary).
Document not JSON: This means you are attempting to modify a binary document using sub-document operations.
Invalid path: This means the path is invalid for the command. Certain commands such as subdoc-array-insert expect array elements as their final component, while others such as subdoc-upsert and subdoc-insert expect dictionary (object) keys.

If a Sub-Document command fails a top-level error is reported (Multi Command Failure), rather than an individual error code (e.g. Path Not Found). When receiving a top-level error code, you should traverse the results of the command to see which individual code failed.

Path syntax

Path syntax largely follows SQL++ conventions: A path is divided into components, with each component referencing a specific level in a document hierarchy. Components are separated by dots (.) in the case where the element left of the dot is a dictionary, or by brackets ([n]) where the element left of the bracket is an array and n is the index within the array.

As a special extension, you can indicate the last element of an array by using an index of -1, for example to get the last element of the array in the document

{"some":{"array":[1,2,3,4,5,6,7,8,9,0]}}

Use some.array[-1] as the path, which will return the element 0.

Each path component must conform as a JSON string, as if it were surrounded by quotes, and any character in the path which may invalidate it as a JSON string must be escaped by a backslash (\). In other words, the path component must match exactly the path inside the document itself. For example:

{"literal\"quote": {"array": []}}

must be referenced as literal\"quote.array.

If the path also has special path characters (i.e. a dot or brackets) it may be escaped using SQL++ escapes. Considering the document

{"literal[]bracket": {"literal.dot": true}}

A path such as `literal[]bracket`.`literal.dot`. You can use double-backticks (``) to reference a literal backtick.

If you need to combine both JSON and path-syntax literals you can do so by escaping the component from any JSON string characters (e.g. a quote or backslash) and then encapsulating it in backticks (`path`).

Currently, paths cannot exceed 1024 characters, and cannot be more than 32 levels deep.

Extended Attributes

Extended Attributes (also known as XATTRs), built upon the Sub-Document API, allow developers to define application-specific metadata that will only be visible to those applications that request it or attempt to modify it. This might be, for example, meta-data specific to a programming framework that should be hidden by default from other frameworks or libraries, or possibly from other versions of the same framework. They are not intended for use in general applications, and data stored there cannot be accessed easily by some Couchbase services, such as Search.