Overview

reference

With the SELECT statement, you can query and manipulate JSON data. You can select, join, project, nest, unnest, group, and sort in a single SELECT statement.

The SELECT statement takes a set of JSON documents from keyspaces as its input, manipulates it and returns a set of JSON documents in the result array. Since the schema for JSON documents is flexible, JSON documents in the result set have flexible schema as well.

A simple query in N1QL consists of three parts:

SELECT: specifies the projection, which is the part of the document that is to be returned.
FROM: specifies the keyspaces to work with.
WHERE: specifies the query criteria (filters or predicates) that the results must satisfy.

To query on a keyspace, you must either specify the document keys or use an index on the keyspace.

Prerequisites

The user executing the SELECT statement must have the Query Select privileges granted on all keyspaces referred in the query. For more details about user roles, see Authorization.

For example,

To execute the following statement, the user does not need any special privileges.

SELECT 1

To execute the following statement, the user must have the Query Select privilege on `travel-sample`.inventory.airline.

SELECT * FROM `travel-sample`.inventory.airline;

To execute the following statement, the user must have the Query Select privilege on `travel-sample`.inventory.route and `travel-sample`.inventory.airline.

SELECT * FROM `travel-sample`.inventory.route
JOIN `travel-sample`.inventory.airline
ON KEYS route.airlineid
WHERE route.airlineid IN ["airline_330", "airline_225"]

To execute the following statement, the user must have the Query Select privilege on `travel-sample`.inventory.airport and `travel-sample`.inventory.landmark.

SELECT * FROM `travel-sample`.inventory.airport
WHERE city IN (SELECT RAW city FROM `travel-sample`.inventory.landmark);

To execute the following statement, the user must have the Query Select privilege on `travel-sample`.inventory.hotel and `travel-sample`.inventory.landmark.

SELECT * FROM `travel-sample`.inventory.hotel WHERE city = "Gillingham"
UNION
SELECT * FROM `travel-sample`.inventory.landmark WHERE city = "Gillingham";

The following example uses an index to query the keyspace for airports that are in the America/Anchorage timezone and at an altitude of 2100ft or higher, and returns an array with the airport name and city name for each airport that satisfies the conditions.

Query

SELECT t.airportname, t.city
FROM   `travel-sample`.inventory.airport t
WHERE  tz = "America/Anchorage"
       AND geo.alt >= 2100;

Results

[
  {
        "airportname": "Anaktuvuk Pass Airport",
        "city": "Anaktuvuk Pass",
  }
]

The next example queries the keyspace using the document key "airport_3469".

Query

SELECT * FROM `travel-sample`.inventory.airport USE KEYS "airport_3469";

Results

[
  {
    "airport": {
      "airportname": "San Francisco Intl",
      "city": "San Francisco",
      "country": "United States",
      "faa": "SFO",
      "geo": {
        "alt": 13,
        "lat": 37.618972,
        "lon": -122.374889
      },
      "icao": "KSFO",
      "id": 3469,
      "type": "airport",
      "tz": "America/Los_Angeles"
    }
  }
]

With projections, you retrieve just the fields that you need and not the entire document. This is especially useful when querying for a large dataset as it results in shorter processing times and better performance.

The SELECT statement provides a variety of data processing capabilities such as filtering, querying across relationships using JOINs or subqueries, deep traversal of nested documents, aggregation, combining result sets using operators, grouping, sorting, and more. Follow the links for examples that demonstrate each capability.

SELECT Statement Processing

The SELECT statement queries a keyspace and returns a JSON array that contains zero or more objects.

The following diagram shows the query execution workflow at a high level and illustrates the interaction with the query, index, and data services.

1. Client submits the query over the REST API; 2. Query Service parses, analyzes, and creates plan; 3. Query Service sends request and index filters to Index Service; 4. Index Service returns qualified doc keys to Query Service; 5. Query Service sends fetch request with doc keys to Data Service; 6. Data Service returns documents to the Query Service; 7. Query Service evaluates the documents; 8. Query Service returns result to client

Figure 1. Query Execution Workflow

The SELECT statement is executed as a sequence of steps. Each step in the process produces result objects that are then used as inputs in the next step until all steps in the process are complete. While the workflow diagram shows all the possible phases a query goes through before returning a result, the clauses and predicates in a query decide the phases and the number of times that the query goes through. For example, sort phase can be skipped when there is no ORDER BY clause in the query; scan-fetch-join phase will execute multiple times for correlated subqueries.

The following diagram shows the possible elements and operations during query execution.

Parse, Plan, Scan, Fetch, Join, Filter, Pre-Aggregate, Aggregate, Sort, Offset, Limit, Project

Figure 2. Query Execution Phases

Some phases are done serially while others are done in parallel, as specified by their parent operator.

The below table summarizes all the Query Phases that might be used in an Execution Plan:

Query Phase	Description
Parse	Analyzes the query and available access path options for each keyspace in the query to create a query plan and execution infrastructure.
Plan	Selects the access path, determines the Join order, determines the type of Joins, and then creates the infrastructure needed to execute the plan.
Scan	Scans the data from the Index Service.
Fetch	Fetches the data from the Data Service.
Join	Joins the data from the Data Service.
Filter	Filters the result objects by specifying conditions in the WHERE clause.
Pre-Aggregate	Internal set of tools to prepare the Aggregate phase.
Aggregate	Performs aggregating functions and window functions.
Sort	Orders and sorts items in the resultset in the order specified by the ORDER BY clause
Offset	Skips the first n items in the result object as specified by the OFFSET clause.
Limit	Limits the number of results returned using the LIMIT clause.
Project	Receives only the fields needed for final displaying to the user.

Query Phase

Description

Parse

Analyzes the query and available access path options for each keyspace in the query to create a query plan and execution infrastructure.

Plan

Selects the access path, determines the Join order, determines the type of Joins, and then creates the infrastructure needed to execute the plan.

Scan

Scans the data from the Index Service.

Fetch

Fetches the data from the Data Service.

Join

Joins the data from the Data Service.

Filter

Filters the result objects by specifying conditions in the WHERE clause.

Pre-Aggregate

Internal set of tools to prepare the Aggregate phase.

Aggregate

Performs aggregating functions and window functions.

Sort

Orders and sorts items in the resultset in the order specified by the ORDER BY clause

Offset

Skips the first n items in the result object as specified by the OFFSET clause.

Limit

Limits the number of results returned using the LIMIT clause.

Project

Receives only the fields needed for final displaying to the user.

The possible elements and operations in a query include:

Specifying the keyspace that is queried.
Specifying the document keys or using indexes to access the documents.
Fetching the data from the data service.
Filtering the result objects by specifying conditions in the WHERE clause.
Removing duplicate result objects from the resultset by using the DISTINCT clause.
Grouping and aggregating the result objects.
Ordering (sorting) items in the resultset in the order specified by the ORDER BY expression list.
Skipping the first n items in the result object as specified by the OFFSET clause.
Limiting the number of results returned using the LIMIT clause.

Data Processing Capabilities

Filtering

You can filter the query results using the WHERE clause. Consider the following example which queries for all airports in the America/Anchorage timezone that are at an altitude of 2000ft or more. The WHERE clause specifies the conditions that must be satisfied by the documents to be included in the resultset, and the resultset is returned as an array of airports that satisfy the condition.

The keys in the result object are ordered alphabetically at each level.

Query

SELECT *
FROM   `travel-sample`.inventory.airport
WHERE  tz = "America/Anchorage"
       AND geo.alt >= 2000;

Result

[
  {
    "airport": {
      "airportname": "Arctic Village Airport",
      "city": "Arctic Village",
      "country": "United States",
      "faa": "ARC",
      "geo": {
        "alt": 2092,
        "lat": 68.1147,
        "lon": -145.579
      },
      "icao": "PARC",
      "id": 6729,
      "type": "airport",
      "tz": "America/Anchorage"
    }
  },
  {
    "airport": {
      "airportname": "Anaktuvuk Pass Airport",
      "city": "Anaktuvuk Pass",
      "country": "United States",
      "faa": "AKP",
      "geo": {
        "alt": 2103,
        "lat": 68.1336,
        "lon": -151.743
      },
      "icao": "PAKP",
      "id": 6712,
      "type": "airport",
      "tz": "America/Anchorage"
    }
  }
]

Querying Across Relationships

You can use the SELECT statement to query across relationships using the JOIN clause or subqueries.

JOIN Clause

Before we delve into examples, let’s take a look at the data model of the inventory scope in the travel-sample bucket, which is used in the following examples. For more details about the data model, see Travel App Data Model.

Figure 3. Data model of inventory scope

The first example uses a JOIN clause to find the distinct airline details which have routes that start from SFO. This example JOINS the document from the route keyspace with documents from the airline keyspace using the KEY "airlineid".

Documents from the route keyspace are on the left side of the JOIN, and documents from the airline keyspace are on the right side of the JOIN.
The documents from the route keyspace (on the left) contain the foreign key "airlineid" of documents from the airline keyspace (on the right).

Query

SELECT DISTINCT airline.name, airline.callsign,
  route.destinationairport, route.stops, route.airline
FROM `travel-sample`.inventory.route
  JOIN `travel-sample`.inventory.airline
  ON KEYS route.airlineid
WHERE route.sourceairport = "SFO"
LIMIT 2;

Results

[
  {
    "airline": "B6",
    "callsign": "JETBLUE",
    "destinationairport": "AUS",
    "name": "JetBlue Airways",
    "stops": 0
  },
  {
    "airline": "B6",
    "callsign": "JETBLUE",
    "destinationairport": "BOS",
    "name": "JetBlue Airways",
    "stops": 0
  }
]

Let’s consider another example which finds the number of distinct airports where AA has routes. In this example:

Documents from the airline keyspace are on the left side of the JOIN, and documents from the route keyspace are on the right side.
The WHERE clause predicate airline.iata = "AA" is on the left side keyspace airline.

This example illustrates a special kind of JOIN where the documents on the right side of JOIN contain the foreign key reference to the documents on the left side. Such JOINs are referred to as index JOIN. For details, see Index JOIN Clause.

Index JOIN requires a special inverse index route_airlineid on the JOIN key route.airlineid. Create this index using the following command:

CREATE INDEX route_airlineid ON `travel-sample`.inventory.route(airlineid);

Now we can execute the following query.

Query

SELECT Count(DISTINCT route.sourceairport) AS distinctairports1
FROM `travel-sample`.inventory.airline
  JOIN `travel-sample`.inventory.route
  ON KEY route.airlineid FOR airline
WHERE airline.iata = "AA";

Results

[
   {
      "distinctairports1": 429
   }
]

Subqueries

A subquery is an expression that is evaluated by executing an inner SELECT query. Subqueries can be used in most places where you can use an expression such as projections, FROM clauses, and WHERE clauses.

A subquery is executed once for every input document to the outer statement and it returns an array every time it is evaluated. See Subqueries for more details.

Query

SELECT *
FROM   (SELECT t.airportname
        FROM   (SELECT *
                FROM   `travel-sample`.inventory.airport t
                WHERE  country = "United States"
                LIMIT  1) AS s1) AS s2;

Results

[
  {
    "s2": {
      "airportname": "Barter Island Lrrs"
    }
  }
]

Deep Traversal for Nested Documents

When querying a keyspace with nested documents, SELECT provides an easy way to traverse deep nested documents using the dot notation and NEST and UNNEST clauses.

Dot Notation

The following query looks for the schedule, and accesses the flight id for the destination airport "ALG". Since a given flight has multiple schedules, the attribute schedule is an array containing all schedules for the specified flight. You can access the individual array elements using the array indexes. For brevity, we’re limiting the number of results in the query to 1.

Query

SELECT t.schedule[0].flight AS flightid
FROM `travel-sample`.inventory.route t
WHERE destinationairport="ALG"
LIMIT 1;

Results

[
  {
    "flightid": "AH631"
  }
]

NEST and UNNEST

Note that an array is created with the matching nested documents. In this example:

The airline field in the result is an array of the airline documents that are matched with the key route.airlineid.
Hence, the projection is accessed as airline[0] to pick the first element of the array.

Query

SELECT DISTINCT route.sourceairport,
                route.airlineid,
                airline[0].callsign
FROM `travel-sample`.inventory.route
NEST `travel-sample`.inventory.airline
  ON KEYS route.airlineid
WHERE route.airline = "AA"
LIMIT 4;

Results

[
  {
    "airlineid": "airline_24",
    "callsign": "AMERICAN",
    "sourceairport": "ABE"
  },
  {
    "airlineid": "airline_24",
    "callsign": "AMERICAN",
    "sourceairport": "ABI"
  },
  {
    "airlineid": "airline_24",
    "callsign": "AMERICAN",
    "sourceairport": "ABQ"
  },
  {
    "airlineid": "airline_24",
    "callsign": "AMERICAN",
    "sourceairport": "ABZ"
  }
]

The following example uses the UNNEST clause to retrieve the author names from the reviews object.

Query

SELECT r.author
FROM `travel-sample`.inventory.hotel t
UNNEST t.reviews r
LIMIT 4;

Results

[
  {
    "author": "Ozella Sipes"
  },
  {
    "author": "Barton Marks"
  },
  {
    "author": "Blaise O'Connell IV"
  },
  {
    "author": "Nedra Cronin"
  }
]

Aggregation

As part of a single SELECT statement, you can also perform aggregation using the SUM, COUNT, AVG, MIN, MAX, or ARRAY_AVG functions.

The following example counts the total number of flights to SFO:

Query

SELECT count(schedule[*]) AS totalflights
FROM `travel-sample`.inventory.route t
WHERE destinationairport="SFO";

Results

[
  {
    "totalflights": 250
  }
]

Combining Resultsets Using Operators

You can combine the result sets using the UNION or INTERSECT operators.

Consider the following example which looks for the first schedule for flights to "SFO" and "BOS":

Query

(SELECT t.schedule[0]
 FROM `travel-sample`.inventory.route t
 WHERE destinationairport = "SFO"
 LIMIT 1)
UNION ALL
(SELECT t.schedule[0]
 FROM `travel-sample`.inventory.route t
 WHERE destinationairport = "BOS"
 LIMIT 1);

Results

[
  {
    "$1": {
      "day": 0,
      "flight": "AI339",
      "utc": "23:05:00"
    }
  },
  {
    "$1": {
      "day": 0,
      "flight": "AM982",
      "utc": "09:11:00"
    }
  }
]

Grouping, Sorting, and Limiting Results

You can perform further processing on the data in your result set before the final projection is generated. You can group data using the GROUP BY clause, sort data using the ORDER BY clause, and you can limit the number of results included in the result set using the LIMIT clause.

The following example looks for the number of airports at an altitude of 5000ft or higher and groups the results by country and timezone. It then sorts the results by country names and timezones (ascending order by default).

Query

SELECT COUNT(*) AS count,
       t.country AS country,
       t.tz AS timezone
FROM `travel-sample`.inventory.airport t
WHERE geo.alt >= 5000
GROUP BY t.country, t.tz
ORDER BY t.country, t.tz;

Results

[
  {
    "count": 2,
    "country": "France",
    "timezone": "Europe/Paris"
  },
  {
    "count": 57,
    "country": "United States",
    "timezone": "America/Denver"
  },
  {
    "count": 7,
    "country": "United States",
    "timezone": "America/Los_Angeles"
  },
  {
    "count": 4,
    "country": "United States",
    "timezone": "America/Phoenix"
  },
  {
    "count": 1,
    "country": "United States",
    "timezone": "Pacific/Honolulu"
  }
]

Index Selection

The optimizer attempts to select an appropriate secondary index for a query based on the filters in the WHERE clause. If it cannot select a secondary query, the query service falls back on the primary index for the keyspace.

Secondary indexes do not index a document if the leading index key is MISSING in the document. This means that when a query selects a field which is MISSING in some documents, the optimizer will not be able to choose a secondary index which uses that field as a leading key.

To enable the optimizer to choose the required index, you must use a WHERE clause of some kind to filter out any documents in which the required field is MISSING. The minimum filter you can use to do this is IS NOT MISSING. This is usually only necessary in queries which do not otherwise have a WHERE clause; for example, some GROUP BY and aggregate queries.

A query is said to be sargable if the optimizer is able to select an index to speed up the execution of the query.

Example 1. Field with MISSING values — cannot choose the secondary index

This example uses an index idx_image_direct_url that is defined by following statement.

Index

CREATE INDEX `idx_image_direct_url`
ON `travel-sample`.inventory.landmark(`image_direct_url`);

Query

EXPLAIN SELECT image_direct_url FROM `travel-sample`.inventory.landmark; (1)

Result

[
  {
    "plan": {
      "#operator": "Sequence",
      "~children": [
        {
          "#operator": "PrimaryScan3",
          "bucket": "travel-sample",
          "index": "def_inventory_landmark_primary", (2)
          "index_projection": {
            "primary_key": true
          },
          "keyspace": "landmark",
          "namespace": "default",
          "scope": "inventory",
          "using": "gsi"
        },
// ...
      ]
    },
    "text": "SELECT image_direct_url FROM `travel-sample`.inventory.landmark;"
  }
]

1	The query selects the `image_direct_url` field, which is MISSING for many documents in the `landmark` keyspace.
2	Therefore the optimizer falls back on the `def_inventory_landmark_primary` index.

Example 2. Filter out MISSING values — correct secondary index is chosen

EXPLAIN SELECT image_direct_url FROM `travel-sample`.inventory.landmark
WHERE image_direct_url IS NOT MISSING; (1)

[
  {
    "plan": {
      "#operator": "Sequence",
      "~children": [
        {
          "#operator": "IndexScan3",
          "bucket": "travel-sample",
          "covers": [
            "cover ((`landmark`.`image_direct_url`))",
            "cover ((meta(`landmark`).`id`))"
          ],
          "filter": "(cover ((`landmark`.`image_direct_url`)) is not missing)",
          "index": "idx_image_direct_url", (2)
          "index_id": "ed5be54231ea184",
          "index_projection": {
            "entry_keys": [
              0
            ]
          },
/// ...
        }
      ]
    },
    "text": "SELECT image_direct_url FROM `travel-sample`.inventory.landmark\nWHERE image_direct_url IS NOT MISSING;"
  }
]

1	The WHERE clause filters out documents where `image_direct_url` is not MISSING.
2	The optimizer correctly chooses the `idx_image_direct_url` index.

For further examples, refer to Group By and Aggregate Performance.