Function: Document Controlled Expiry

Enterprise Edition

Goal: Purge a document automatically based on self-contained start and duration fields.

This function docControlledSelfExpiry demonstrates self-expiry of a document for example a user trial.
Requires Eventing Storage (or metadata collection) and a "source" collection.
Needs a Binding of type "bucket alias" (as documented in the Scriptlet).
When documents are created, they will have no expiration value. This function processes the initial mutation to calculate and set the proper TTL.
In Couchbase, when using a simple integer expiry value (as opposed to a proper date or time object), the expiration can be specified in two ways:
- As an offset from the current time. If the absolute value of the expiry is less than 30 days (60 * 60 * 24 * 30 seconds), it is considered an offset.
- As an absolute Unix time stamp. If the value is greater than 30 days (60 * 60 * 24 * 30 seconds), it is considered an absolute time stamp.
- As described in Expiration, if a "Bucket Max Time-To-Live" is set (specified in seconds), it is an enforced hard upper limit. As such, any subsequent document mutation (by SQL++, Eventing, or any Couchbase SDK) will result in the document having its expiration adjusted and set to the bucket’s maximum TTL if the operation has:
  - No TTL.
  - A TTL of zero.
  - A TTL greater than the bucket TTL.
Will operate on any document with type == "trial_customers".
Will ignore any doc with a non-zero TTL (prevents infinite recursion)
Uses the N1QL(…) function to update the source bucket instead of an inline SQL++ statement because inline SQL++ is prohibited from updating the source bucket of an Eventing handler to prevent infinite recursion scenarios.
The recursion from the N1QL(…) statement is ignored via the if (meta.expiration !== 0) { … } filter.
This is different than setting a TTL on a bucket which will typically update (or extend) the TTL of a document on each mutation.

You must use the function N1QL(…) with great caution when updating the source bucket of your Eventing handler as you can easily create infinite recursion which may crash your server.

docControlledSelfExpiry
Input Data/Mutation
Output Data/Mutation

Two variants of this function are available: a 6.6 version (this Function) that relies on SQL++, and a 6.6.1 version that directly sets the expiration. Using N1QL(…) is much slower than using couchbase.replace(bucket_binding, meta, doc) in the advancedDocControlledSelfExpiry variant.

docControlledSelfExpiry (indirect TTL via SQL++)

// To run configure the settings for this Function, docControlledSelfExpiry, as follows:
//
// Version 7.1+
//   "Function Scope"
//     *.* (or try bulk.data if non-privileged)
// Version 7.0+
//   "Listen to Location"
//     bulk.data.source
//   "Eventing Storage"
//     rr100.eventing.metadata
//   Binding(s)
//    1. "binding type", "alias name...", "bucket.scope.collection", "Access"
//       "bucket alias", "src_col",       "bulk.data.source",        "read and write"
//
// Version 6.X
//   "Source Bucket"
//     source
//   "MetaData Bucket"
//     metadata
//   Binding(s)
//    1. "binding type", "alias name...", "bucket",     "Access"
//       "bucket alias", "src_col",       "source", "read and write"

function OnUpdate(doc, meta) {
    // Filter items that don't have been updated, this also stops
    // any recursion when we update meta.expiration via N1QL(...)
    if (meta.expiration !== 0) {
        log(meta.id, "IGNORE expiration "+meta.expiration+" !== 0 or "+
            new Date(meta.expiration).toString());
        return;
    }

    // Optional filter to a specic field like 'type'
    if (doc.type !== 'trial_customers') return;

    // Our expiry is based on a JavaScript date parsable field, it must exist
    if (!doc.trialStartDate || !doc.trialDurationDays) return;

    // Convert the doc's field timeStamp and convert to unix epoch time (in ms.).
    var docTimeStampMs = Date.parse(doc.trialStartDate);

    var keepDocForMs = doc.trialDurationDays * 1000 * 60 * 60 * 24 ;
    var nowMs = Date.now();  // get current unix time (in ms.).

    // Archive if we have kept it for too long no need to set the expiration
    if( nowMs >= (docTimeStampMs + keepDocForMs) ) {

        // Delete the document form the source bucket via the map alias
        delete src_col[meta.id];

        log(meta.id, "DELETE from src_col to dst_bkt alias as our expiration " +
            new Date(docTimeStampMs + keepDocForMs).toString()) + " is already past";
    } else {
        var key = meta.id;
        //set the meta.expiration=ttlMs
        var	ttlMs = docTimeStampMs + keepDocForMs;

        // Use SQL++ to write back a non-zero TTL to the document hear we actually
        // have to use the real bucket name "source" instead of the alias src_col
        // as we are using SQL++. This will cause recursion but it will be ignored
        // since we ignore all non-zero TTLs

        if (ttlMs !== 0) {
            log(meta.id, "UPDATE expiration "+meta.expiration+" === 0 set to "+
                ttlMs+" or " + new Date(ttlMs).toString());

            // Ensure non-zero, just be safe just in case somehow 1) doc.timeStamp
            // evals to 0, and 2) keepDocForMs is set to 0
            var stmt = "UPDATE `source` USE KEYS \""+key+
                       "\" SET meta().expiration = " + Math.floor(ttlMs/1000);
            N1QL(stmt);

            // Future in 6.6.1+ we can avoid SQL++ via Eventing's new Advanced Bucket Ops
            // couchbase.replace(src_col,{"id":meta.id,"expiry_date":new Date(ttlMs)},doc);
        }
    }
}

We want to create a test set of four (4) documents, use the Query Editor to insert the the data items (you do not need an index).

Note, if the today is past 08-25-2021 (MM-DD-YYYY) just change the trialStartDate for the last two records to at least 90 days from now.

  INSERT INTO `bulk`.`data`.`source` (KEY,VALUE)
  VALUES ( "trial_customers::0", {
    "type": "trial_customers",
    "id": 0,
    "trialStartDate": "08-25-2019",
    "trialDurationDays": 30,
    "note": "this is old will get immeadiately deleted"
  } ),
  VALUES ( "trial_customers::1",
  {
    "type": "trial_customers",
    "id": 1,
    "trialStartDate": "01-27-2020",
    "trialDurationDays": 30,
    "note": "this is old will get immeadiately deleted"
  } ),
  VALUES ( "trial_customers::2",
  {
    "type": "trial_customers",
    "id": 2,
    "trialStartDate": "08-25-2021",
    "trialDurationDays": 30,
    "note": "this will get an exiration set"
  } ),
  VALUES ( "trial_customers::3",
  {
    "type": "trial_customers",
    "id": 3,
    "trialStartDate": "08-26-2021",
    "trialDurationDays": 60,
    "note": "this will get an exiration set"
  } );

NEW/OUTPUT: KEY trial_customers::2

{
  "id": 2,
  "note": "this will get an exiration set",
  "trialDurationDays": 30,
  "trialStartDate": "08-25-2021",
  "type": "trial_customers"
}

NEW/OUTPUT: KEY trial_customers::3

{
  "id": 3,
  "note": "this will get an exiration set",
  "trialDurationDays": 60,
  "trialStartDate": "08-26-2021",
  "type": "trial_customers"
}

We end up with two (2) of the four documents (obviously you may need to adjust the N1QL INSERT in a few months as all the document would be immediately deleted).

* "trial_customers::0" was deleted
* "trial_customers::1" was deleted
* "trial_customers::2" has an meta.expiration set for 1632466800 (or 2021-09-24 07:00:00 UTC) in it's metadata
* "trial_customers::3" has an meta.expiration set for 1635145200 (or 2021-10-25 07:00:00 UTC) in it's metadata