From b3792659aa6318a2e2a3f9af929f903b60d9a8b6 Mon Sep 17 00:00:00 2001 From: Phil Varner Date: Fri, 3 Dec 2021 13:59:28 -0500 Subject: [PATCH] Updates to Filter Extension for CQL2 (#225) * numerous updates to CQL2 definitions * few more basic-cql2 replacements * update cql2 prefixes * add between and like * fix quotes in cql text examples * add filter-lang to examples * remove Filter Ext Case-sensitive Comparison class --- CHANGELOG.md | 20 + extensions.md | 23 +- fragments/filter/README.md | 661 +++++++++++++++++++++------------- fragments/filter/openapi.yaml | 18 +- 4 files changed, 455 insertions(+), 267 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 664530c7..b38847ac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed - Limit parameter semantics now match OAFeat. Previously, the behavior was not precisely defined. +- Filter Extension updates to align with changes to OAFeat CQL2 spec + - Updated all "CQL" usages to "CQL2" + - Most conformance class URIs are now prefixed with `http://www.opengis.net/spec/cql2/` instead + of `http://www.opengis.net/spec/ogcapi-features-3/` + - Conformance classes `http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/basic-cql`, + `http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/cql-text`, and + `http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/cql-json` have had `cql` replaced + with `cql2` (in addition to the prefix change) to + become `http://www.opengis.net/spec/cql2/1.0/conf/basic-cql2`, + `http://www.opengis.net/spec/cql2/1.0/conf/cql2-text`, and + `http://www.opengis.net/spec/cql2/1.0/conf/cql2-json` + - Significant changes to CQL2 JSON format, now using `op` and `args` structure + - Spatial operator `INTERSECTS` is now `S_INTERSECTS` + - Temporal operator `ANYINTERACTS` is now `T_INTERSECTS` + - Updated Example 3 (now Example 5) to make it clear that property to property comparisons require the + Property-Property Comparisons conformance class + - The CQL2 Case-insensitive Comparison + (`http://www.opengis.net/spec/cql2/1.0/conf/case-insensitive-comparison`) conformance class + that adds UPPER/LOWER terms or function CASEI for case-insensitive comparison has not been added + to this spec yet, since the definition in CQL2 is in flux. - `service-desc` endpoint may return any service description format, typically a machine-consumable one (previous restricted required to be OpenAPI 3.0 JSON) - `service-doc` endpoint may return any service description format, typically a diff --git a/extensions.md b/extensions.md index 59765d4e..01c04474 100644 --- a/extensions.md +++ b/extensions.md @@ -70,18 +70,19 @@ the service supports. This are listed at the top of each extension description, - - [Filter](item-search/README.md#filter) - - - + - - - - - - - - - - - - - - - - - - - - - - + - + - + - + - + - + - + - + - + - + - + - [Context](item-search/README.md#context) - - diff --git a/fragments/filter/README.md b/fragments/filter/README.md index 8c725447..9da6bc86 100644 --- a/fragments/filter/README.md +++ b/fragments/filter/README.md @@ -4,17 +4,18 @@ - **Conformance Classes:** - Filter: - Item Search Filter: - - CQL Text: - - CQL JSON: - - Basic CQL: - - Advanced Comparison Operators: - - Basic Spatial Operators: - - Spatial Operators: - - Temporal Operators: - - Custom Functions: - - Arithmetic Expressions: - - Array Operators: - - Property-Property Comparisons: + - CQL2 Text: + - CQL2 JSON: + - Basic CQL2: + - Advanced Comparison Operators: + - Basic Spatial Operators: + - Spatial Operators: + - Temporal Operators: + - Custom Functions: + - Arithmetic Expressions: + - Array Operators: + - Property-Property Comparisons: + - **Extension [Maturity Classification](../../extensions.md#extension-maturity):** Pilot - **Dependents:** - [Item Search](../../item-search) @@ -30,58 +31,68 @@ - [Interaction with Endpoints](#interaction-with-endpoints) - [Examples](#examples) - [Example 1](#example-1) - - [Example 1: GET with cql-text](#example-1-get-with-cql-text) - - [Example 1: POST with cql-json](#example-1-post-with-cql-json) + - [Example 1: GET with cql2-text](#example-1-get-with-cql2-text) + - [Example 1: POST with cql2-json](#example-1-post-with-cql2-json) - [Example 2](#example-2) - - [Example 2: GET with cql-text](#example-2-get-with-cql-text) - - [Example 2: POST with cql-json](#example-2-post-with-cql-json) - - [Example 3](#example-3) - - [Example 3: GET with cql-text](#example-3-get-with-cql-text) - - [Example 3: POST with cql-json](#example-3-post-with-cql-json) - - [Example 4](#example-4) - - [Example 4: AND cql-text (GET)](#example-4-and-cql-text-get) - - [Example 4: AND cql-json (POST)](#example-4-and-cql-json-post) - - [Example 5](#example-5) - - [Example 5: OR cql-text (GET)](#example-5-or-cql-text-get) - - [Example 5: OR cql-json (POST)](#example-5-or-cql-json-post) - - [Example 6: Temporal](#example-6-temporal) - - [Example 6: ANYINTERACTS cql-text (GET)](#example-6-anyinteracts-cql-text-get) - - [Example 6: ANYINTERACTS cql-json (POST)](#example-6-anyinteracts-cql-json-post) - - [Example 7: Spatial](#example-7-spatial) - - [Example 7: INTERSECTS cql-text (GET)](#example-7-intersects-cql-text-get) - - [Example 7: INTERSECTS cql-json (POST)](#example-7-intersects-cql-json-post) - - [Example 8: Spatial Disjunction](#example-8-spatial-disjunction) - - [Example 8: INTERSECTS cql-text (GET)](#example-8-intersects-cql-text-get) - - [Example 8: INTERSECTS cql-json (POST)](#example-8-intersects-cql-json-post) + - [Example 2: GET with cql2-text](#example-2-get-with-cql2-text) + - [Example 2: POST with cql2-json](#example-2-post-with-cql2-json) + - [Example 3: Conjunction with AND](#example-3-conjunction-with-and) + - [Example 3: AND cql2-text (GET)](#example-3-and-cql2-text-get) + - [Example 3: AND cql2-json (POST)](#example-3-and-cql2-json-post) + - [Example 4: Disjunction with OR](#example-4-disjunction-with-or) + - [Example 4: OR cql2-text (GET)](#example-4-or-cql2-text-get) + - [Example 4: OR cql2-json (POST)](#example-4-or-cql2-json-post) + - [Example 5: Property-Property Comparisons](#example-5-property-property-comparisons) + - [Example 5: GET with cql2-text](#example-5-get-with-cql2-text) + - [Example 5: POST with cql2-json](#example-5-post-with-cql2-json) + - [Example 6: Temporal Intersection](#example-6-temporal-intersection) + - [Example 6: T_INTERSECTS cql2-text (GET)](#example-6-t_intersects-cql2-text-get) + - [Example 6: T_INTERSECTS cql2-json (POST)](#example-6-t_intersects-cql2-json-post) + - [Example 7: Spatial Intersection](#example-7-spatial-intersection) + - [Example 7: S_INTERSECTS cql2-text (GET)](#example-7-s_intersects-cql2-text-get) + - [Example 7: S_INTERSECTS cql2-json (POST)](#example-7-s_intersects-cql2-json-post) + - [Example 8: Spatial Intersection Disjunction](#example-8-spatial-intersection-disjunction) + - [Example 8: S_INTERSECTS cql2-text (GET)](#example-8-s_intersects-cql2-text-get) + - [Example 8: S_INTERSECTS cql2-json (POST)](#example-8-s_intersects-cql2-json-post) - [Example 9: Using IS NULL](#example-9-using-is-null) - - [Example 9: cql-text (GET)](#example-9-cql-text-get) - - [Example 9: cql-json (POST)](#example-9-cql-json-post) + - [Example 9: cql2-text (GET)](#example-9-cql2-text-get) + - [Example 9: cql2-json (POST)](#example-9-cql2-json-post) + - [Example 10: Using BETWEEN](#example-10-using-between) + - [Example 10: cql2-text (GET)](#example-10-cql2-text-get) + - [Example 10: cql2-json (POST)](#example-10-cql2-json-post) + - [Example 11: Using LIKE](#example-11-using-like) + - [Example 11: cql2-text (GET)](#example-11-cql2-text-get) + - [Example 11: cql2-json (POST)](#example-11-cql2-json-post) ## Overview The Filter extension provides an expressive mechanism for searching based on Item attributes. This extension references behavior defined in the -[OGC API - Features - Part 3: Filtering and the Common Query Language (CQL2)](https://github.com/opengeospatial/ogcapi-features/tree/master/extensions/cql) -specification. As of August 2021, this specification is in draft status. Several behaviors have changed since the +[OGC API - Features - Part 3: Filtering and the Common Query Language (CQL2)](https://github.com/opengeospatial/ogcapi-features/tree/master/extensions/cql) and [Common Query Language (CQL2) +](https://github.com/opengeospatial/ogcapi-features/blob/master/cql2/README.md) +specifications. As of November 2021, these specifications are still in draft status, but rapidly converging on +finalized behavior. Several behaviors have changed since the last published [draft](https://portal.ogc.org/files/96288), so this spec references the latest revision in the -[spec's GitHub repo](https://github.com/opengeospatial/ogcapi-features/tree/master/extensions/cql). There are no -major anticipated changes, so implementers are encouraged to move ahead with implementation, and to simply be -aware that minor changes may need to be made in the future. - -OAFeat Part 3 CQL2 formally defines the syntax of "CQL2" as both a text format (cql-text) as an ABNF grammar -(largely similar to the BNF grammar in the General Model for CQL) and a JSON format (cql-json) as a JSON Schema and -OpenAPI schema, and provides a precise natural -language description of the declarative semantics. The CQL Text format has long-standing use within -geospatial software (e.g., GeoServer), is expected not to change before final. -OGC CQL Text has been previously described in [OGC Filter Encoding](https://www.ogc.org/standards/filter) and +[OAFeat Part 3 spec's GitHub repo](https://github.com/opengeospatial/ogcapi-features/tree/master/extensions/cql) +and [Common Query Language (CQL2)](https://github.com/opengeospatial/ogcapi-features/blob/master/cql2/README.md)). +Implementers should proceed with implementation, but must be aware that minor changes may be made +before these specs are final. + +OAFeat Part 3 CQL2 formally defines the syntax of "CQL2" as both a text format (cql2-text) as an ABNF grammar +(largely similar to the BNF grammar in the General Model for CQL) and a JSON format (cql2-json) as a JSON Schema and +OpenAPI schema. Additionally, it defines a natural +language description of the declarative semantics, which were never well-defined for the original CQL language. +The CQL2 Text format has had long-standing use within +geospatial software (e.g., GeoServer), is not expected to change before final. +OGC CQL2 Text has been previously described in [OGC Filter Encoding](https://www.ogc.org/standards/filter) and [OGC Catalogue Services 3.0 - General Model](http://docs.opengeospatial.org/is/12-168r6/12-168r6.html#62) -(including a BNF grammar in Annex B). The CQL JSON format is newly-defined, but also not -expected to change before final. +(including a BNF grammar in Annex B). The CQL2 JSON format is newly-defined and has changed significantly during +the draft process, but is believed to be stable now. It should be noted that the "CQL" referred to here is "CQL2" defined in OGC API - Features - Part 3. This is a related, but -different language to the "classic" OGC CQL defined in the General Model. CQL is also **not** referencing or related two -other "CQL" languages, +different language to the "classic" OGC CQL defined in the General Model. Relatedly, CQL is **not** +referencing or related two other "CQL" languages, the [SRU (Search/Retrieve via URL) Contextual Query Language](https://www.loc.gov/standards/sru/cql/index.html) (formerly known as Common Query Language) or the [Cassandra Query Language](https://cassandra.apache.org/doc/latest/cql/) used by the Cassandra database. @@ -109,77 +120,81 @@ by providing an expressive query language to construct more complex filter predi those provided by SQL. This extension also supports the Queryables mechanism that allows discovery of what Item fields can be used in predicates. -CQL enables more expressive queries than supported by STAC API Item Search. These include: +CQL2 enables more expressive queries than supported by STAC API Item Search. These include: - Use of Item Property values in predicates (e.g., `item.properties.eo:cloud_cover`), using comparison operators - Items whose `datetime` values are in the month of August of the years 2017-2021, using OR and datetime comparisons -- Items whose `geometry` values intersect any one of several Polygons, using OR and INTERSECTS +- Items whose `geometry` values intersect any one of several Polygons, using `OR` and `S_INTERSECTS` - Items whose `geometry` values intersect one Polygon, but do not intersect another one, using AND, NOT, and - INTERSECTS + S_INTERSECTS ## Conformance Classes -OAFeat Part 3 CQL defines several conformance classes that allow implementers to create compositions of +OAFeat Part 3 CQL2 defines several conformance classes that allow implementers to create compositions of functionality that support whatever expressiveness they need. This allows implementers to incrementally support CQL syntax, without needing to implement a huge spec all at once. Some implementers choose not to incur the cost of implementing functionality they do not need or may not be able to implement functionality that is not supported by their underlying datastore, e.g., Elasticsearch does not support the spatial predicates required by the -Spatial Operators conformance class, only the `intersects` operator in the Basic Spatial Operators class. +Spatial Operators conformance class, only the `S_INTERSECTS` operator in the Basic Spatial Operators class. The precise decomposition of the OAFeat conformance classes is still a work in progress, but is being finalized rapidly (see [ogcapi-features/issues/579](https://github.com/opengeospatial/ogcapi-features/issues/579)). The STAC API Filter Extension reuses the definitions and conformance classes in OAFeat CQL, adding only the Item Search Filter conformance class (`https://api.stacspec.org/v1.0.0-beta.5/item-search#filter:item-search-filter`) to bind -the CQL filter behavior to the Item Search resource. +the CQL2 Filter behavior to the Item Search resource. The implementation **must** support these conformance classes: - Filter (`http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/filter`) defines the Queryables mechanism and parameters `filter-lang`, `filter-crs`, and `filter`. -- Basic CQL (`http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/basic-cql`) defines the basic operations allowed in +- Basic CQL2 (`http://www.opengis.net/spec/cql2/1.0/conf/basic-cql2`) defines the basic operations allowed in the query language used for the `filter` parameter defined by Filter. This includes logical operators (`AND`, `OR`, `NOT`), comparison operators (`=`, `<>`, `<`, `<=`, `>`, `>=`), and `IS NULL`. The comparison operators are allowed against string, numeric, boolean, and datetime types. - Item Search Filter (`https://api.stacspec.org/v1.0.0-beta.5/item-search#filter:item-search-filter`) binds the Filter and - Basic CQL conformance classes to apply to the Item Search endpoint (`/search`). This class is the correlate of the OAFeat CQL Features - Filter class that binds Filter and Basic CQL to the Features resource (`/collections/{cid}/items`). + Basic CQL2 conformance classes to apply to the Item Search endpoint (`/search`). This class is the correlate of the OAFeat CQL2 Features + Filter class that binds Filter and Basic CQL2 to the Features resource (`/collections/{cid}/items`). -The implementation **must** support at least one of the "CQL Text" or "CQL JSON" conformance classes that define -the CQL format used in the filter parameter: +The implementation **must** support at least one of the "CQL2 Text" or "CQL2 JSON" conformance classes that +define the CQL2 format used in the filter parameter: -- CQL Text (`http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/cql-text`) defines that the CQL Text format is supported by Item Search -- CQL JSON (`http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/cql-json`) defines that the CQL JSON format is supported by Item Search +- CQL2 Text (`http://www.opengis.net/spec/cql2/1.0/conf/cql2-text`) defines that the CQL2 Text format is supported by Item Search +- CQL2 JSON (`http://www.opengis.net/spec/cql2/1.0/conf/cql2-json`) defines that the CQL2 JSON format is supported by Item Search If both are advertised as being supported, it is only required that both be supported for GET query parameters, and that -only that CQL JSON be supported for POST JSON requests. It is recommended that clients use CQL Text in GET requests and -CQL JSON in POST requests. +only that CQL2 JSON be supported for POST JSON requests. It is recommended that clients use CQL2 Text in GET requests and +CQL2 JSON in POST requests. For additional capabilities, the following classes can be implemented: - Advanced Comparison Operators - (`http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/advanced-comparison-operators`) defines the `LIKE`, - `BETWEEN`, and `IN` operators. Note: this conformance class does **not** require implementing the - `lower` and `upper` functions as defined in the latest OAFeat CQL spec, as these will soon be - removed from the corresponding OAFeat CQL conformance class. -- Basic Spatial Operators (`http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/basic-spatial-operators`) defines the `INTERSECTS` predicate. + (`http://www.opengis.net/spec/cql2/1.0/conf/advanced-comparison-operators`) defines the `LIKE`, + `BETWEEN`, and `IN` operators. **Note**: this conformance class no longer requires implementing the + `lower` and `upper` functions. +- Basic Spatial Operators (`http://www.opengis.net/spec/cql2/1.0/conf/basic-spatial-operators`) defines the intersects operator (`S_INTERSECTS`). - Spatial Operators - (`http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/spatial-operators`) defines the - same operators as OAF Part 3 CQL Advanced Spatial Operators. + (`http://www.opengis.net/spec/cql2/1.0/conf/spatial-operators`) defines a set of operators that + are part of the Dimensionally Extended Nine-intersection Model (DE-9IM) relation operators + (`S_CONTAINS`, `S_CROSSES`, `S_DISJOINT`, `S_EQUALS`, `S_INTERSECTS`, `S_OVERLAPS`, `S_TOUCHES`, and `S_WITHIN`) - Temporal Operators - (`http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/temporal-operators`) defines the - same operators as OAF Part 3 CQL Enhanced Temporal Operators. -- Custom Functions (`http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/functions`) defines the - same operators as OAF Part 3 CQL Custom Functions. -- Arithmetic Expressions: (`http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/arithmetic`) defines - the same operators as OAF Part 3 CQL Arithmetic Expressions. -- Array Operators: (`http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/array-operators`) - defines the same operators as OAF Part 3 CQL Array Operators. -- Property-Property Comparisons: (`http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/property-property`) allows the - use of queryables (e.g., properties) in both positions of a clause, not just in the first position. This allows - predicates like `property1 == property2` be expressed, whereas the Basic CQL conformance class only requires - comparisons against literal values. + (`http://www.opengis.net/spec/cql2/1.0/conf/temporal-operators`) defines several temporal + operators that provide more expressivity with datetime types than the relative comparison + operators + in the Basic CQL2 class. +- Custom Functions (`http://www.opengis.net/spec/cql2/1.0/conf/functions`) defines support + for function definition and usage. +- Arithmetic Expressions: (`http://www.opengis.net/spec/cql2/1.0/conf/arithmetic`) defines + support for arithmetic expressions. +- Array Operators: (`http://www.opengis.net/spec/cql2/1.0/conf/array-operators`) + defines array operators (`A_EQUALS`, `A_CONTAINS`, `A_CONTAINED_BY`, and `A_OVERLAPS`). +- Property-Property Comparisons: (`http://www.opengis.net/spec/cql2/1.0/conf/property-property`) + allows the use of queryables (e.g., properties) in both positions of a clause, not just in the + first position. This allows predicates like `property1 == property2` be expressed, whereas the + Basic CQL2 conformance class only requires comparisons against right-hand-side literals. + + Additionally, if an API implements the OGC API Features endpoint, it is **recommended** that the OAFeat Part 3 Filter, -Features Filter, and Basic CQL conformance classes be implemented, which allow use of CQL filters against the +Features Filter, and Basic CQL2 conformance classes be implemented, which allow use of CQL2 filters against the OAFeat Part 1 Features endpoint (`/collections/{collectionId}/items`). Note that POST with a JSON body to the Features resource is not supported, as POST is used by the [Transaction Extension](../../ogcapi-features/extensions/transaction/README.md) for creating items. @@ -187,35 +202,32 @@ to the Features resource is not supported, as POST is used by the ## Getting Started with Implementation It recommended that implementers start with fully implementing only a subset of functionality. A good place to start is -implementing only the Basic CQL conformance class of logical and comparison operators, defining a static Queryables +implementing only the Basic CQL2 conformance class of logical and comparison operators, defining a static Queryables schema with no queryables advertised and the `additionalProperties` field set to `true`, and -only implementing CQL Text. Following from that can be support for -INTERSECTS, defining a static Queryables schema with only the basic Item properties, and -implementing CQL JSON. From there, other comparison operators can be implemented and a more +only implementing CQL2 Text. Following from that can be support for +S_INTERSECTS, defining a static Queryables schema with only the basic Item properties, and +implementing CQL2 JSON. From there, other comparison operators can be implemented and a more dynamic Queryables schema. -Formal definitions and grammars for CQL can be found here: +Formal definitions and grammars for CQL2 can be found here: -- The [OAFeat (CQL) spec](https://portal.ogc.org/files/96288) includes an ABNF for cql-text and both JSON Schema and - OpenAPI specifications for cql-json. The standalone files are: +- The [OAFeat (CQL) spec](https://portal.ogc.org/files/96288) includes an ABNF for cql2-text and both JSON Schema and + OpenAPI specifications for cql2-json. The standalone files are: - [cql.bnf](https://github.com/opengeospatial/ogcapi-features/blob/master/extensions/cql/standard/schema/cql.bnf) - [cql.json](https://github.com/opengeospatial/ogcapi-features/blob/master/extensions/cql/standard/schema/cql.json) - [cql.yml](https://github.com/opengeospatial/ogcapi-features/blob/master/extensions/cql/standard/schema/cql.yml) -- A JSON Schema for only the parts of the CQL JSON encoding required by this extension is [here](cql.json) -- A OpenAPI specification for only the parts of the CQL JSON encoding required by this extension is [here](cql.yml) -- xtraplatform-spatial has a CQL [ANTLR 4 grammer](https://github.com/interactive-instruments/xtraplatform-spatial/tree/master/xtraplatform-cql/src/main/antlr/de/ii/xtraplatform/cql/infra) +- A JSON Schema for only the parts of the CQL2 JSON encoding required by this extension is [here](cql.json) +- A OpenAPI specification for only the parts of the CQL2 JSON encoding required by this extension is [here](cql.yml) +- xtraplatform-spatial has a CQL2 [ANTLR 4 grammer](https://github.com/interactive-instruments/xtraplatform-spatial/tree/master/xtraplatform-cql/src/main/antlr/de/ii/xtraplatform/cql/infra) -These projects have or are developing CQL support: - -Also ECQL is the old CQL standard defined in some old OGC CSW spec, not a superset of the current CQL Text -standard from OGC API Features. (Which will be implemented soon). +These projects have or are developing CQL or CQL2 support: - [pygeofilter](https://github.com/geopython/pygeofilter) has support for the older ECQL standard - (similar to CQL Text) and will soon have support for OGC API Part 3 CQL + (similar to CQL2 Text) and will soon have support for OGC API Part 3 CQL2 - [GeoPython PyCQL](https://github.com/geopython/pycql/tree/master/pycql) (discontinued), and the [Bitner fork](https://github.com/bitner/pycql) to be used in stac-fastapi - [Franklin](https://github.com/azavea/franklin) is working on it in [this PR](https://github.com/azavea/franklin/pull/750). -- [Geotools](https://github.com/geotools/geotools) has support for [CQL text](https://github.com/geotools/geotools/tree/main/modules/library/cql/src/main/java/org/geotools/filter/text/cql2) +- [Geotools](https://github.com/geotools/geotools) has support for [CQL2 text](https://github.com/geotools/geotools/tree/main/modules/library/cql/src/main/java/org/geotools/filter/text/cql2) Note that the [xbib CQL library (JVM)](https://github.com/xbib/cql) is the OASIS Contextual Query Language, not OGC CQL, and should not be used to implement this extension, as they are significantly different query languages. @@ -283,10 +295,10 @@ collection, e.g., `/collections/collection1/queryables`. The queryables endpoint returns a JSON Schema describing the names and types of terms that may be used in filter expressions. This response is defined in JSON Schema because it is a well-specified typed schema, but it is not used for validating a JSON -document derived from it. This schema defines the terms that may be used in a CQL filter. +document derived from it. This schema defines the terms that may be used in a CQL2 filter. These queryable terms are mapped by the service to filter Items. For example, the service may define a queryable with the -name "eo:cloud_cover" that can be used in a CQL expression like `eo:cloud_cover <= 10`, with the semantics that only Items where the +name "eo:cloud_cover" that can be used in a CQL2 expression like `eo:cloud_cover <= 10`, with the semantics that only Items where the `properties.eo:cloud_cover` value was <= 10 would match the filter. The server would then translate this into an appropriate query for the data within its datastore. @@ -295,16 +307,16 @@ may have a set of enumerated values dynamically determined by an aggregation at interactive client to dynamically expose to the user the fields that are available for filtering, and provide a precise group of options for the values of these terms. -Queryables can also be used to advertise "synthesized" property values. The only requirement in CQL is that the property have a type +Queryables can also be used to advertise "synthesized" property values. The only requirement in CQL2 is that the property have a type and evaluate to literal value of that type or NULL. For example, a filter like "Items must have an Asset with an eo:band with the common_name of 'nir'" can be expressed. A Queryable `assets_bands` could be defined to have a type of array of string and have the semantics that it contains all of `common_name` values across all assets and bands for an Item. This could then be -filtered with the CQL expression `'nir' in assets_bands`. Implementations would then expand this expression into the +filtered with the CQL2 expression `'nir' in assets_bands`. Implementations would then expand this expression into the appropriate query against its datastore. (TBD if this will actually work or not. This is also related to the upcoming restriction on property/literal comparisons) An implementation may also choose not to advertise any queryables, and provide the user with out-of-band information or -simply let them try querying against fields. While this is not allowed according to the OGC CQL Queryable spec, it is allowed +simply let them try querying against fields. While this is not allowed according to the OGC CQL2 Queryable spec, it is allowed in STAC API by the Filter Extension. In this case, the queryables endpoint (`/queryables`) would return this document: ```json @@ -324,15 +336,15 @@ in STAC API by the Filter Extension. In this case, the queryables endpoint (`/qu This extension adds three GET query parameters or POST JSON fields to an Item Search request: -- filter-lang:`cql-text` or `cql-json`. If undefined, defaults to `cql-text` for a GET request and `cql-json` for a POST request. +- filter-lang:`cql2-text` or `cql2-json`. If undefined, defaults to `cql2-text` for a GET request and `cql2-json` for a POST request. - filter-crs: recommended to not be passed, but server must only accept `http://www.opengis.net/def/crs/OGC/1.3/CRS84` as a valid value, may reject any others -- filter: CQL filter expression +- filter: CQL2 filter expression API implementations advertise which `filter-lang` values are supported via conformance classes in the Landing Page. At least one must be implemented, but it is recommended to implement both. If both are advertised as conformance classes, the -server should process either for a GET request, but may only process cql-json for a POST request. If POST of cql-text is not -supported, the server must return a 400 error if `filter-lang=cql-text`. +server should process either for a GET request, but may only process cql2-json for a POST request. If POST of cql2-text is not +supported, the server must return a 400 error if `filter-lang=cql2-text`. ## Interaction with Endpoints @@ -355,11 +367,11 @@ at least these values: "http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/filter", "http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/features-filter", - "http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/basic-cql", - "http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/cql-text", - "http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/cql-json", - "http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/basic-spatial-operators", - "http://www.opengis.net/spec/ogcapi-features-3/1.0/conf/advanced-comparison-operators" + "http://www.opengis.net/spec/cql2/1.0/conf/basic-cql2", + "http://www.opengis.net/spec/cql2/1.0/conf/cql2-text", + "http://www.opengis.net/spec/cql2/1.0/conf/cql2-json", + "http://www.opengis.net/spec/cql2/1.0/conf/basic-spatial-operators", + "http://www.opengis.net/spec/cql2/1.0/conf/advanced-comparison-operators" ], "links": [ @@ -437,7 +449,7 @@ but also binds an arbitrarily-named Queryable to a specific STAC field. For exam `eo:cloud_cover` field is referring to the field of the same name in the EO Extension not because they happen to have the same name, but rather because the `$ref` indicates it. The field could just as well be named "cloud_cover", "CloudCover", or "cc", and we would still know it filtered on the EO extension `eo:cloud_cover` field. For example, if the queryable was named -"CloudCover", a CQL expression using that queryable would look like `CloudCover <= 10`. +"CloudCover", a CQL2 expression using that queryable would look like `CloudCover <= 10`. While these do seem quite complex to write and understand, keep in mind that query construction will likely be done with a more ergonomic SDK, and query parsing will be done with the help of a ABNF grammar and OpenAPI schema. @@ -458,26 +470,33 @@ in any of these examples. This example uses the queryables definition in (Interaction with Endpoints)(#interaction-with-endpoints). -#### Example 1: GET with cql-text +#### Example 1: GET with cql2-text -Note that `filter-lang` defaults to `cql-text` in this case. The parameter `filter-crs` defaults +Note that `filter-lang` defaults to `cql2-text` in this case. The parameter `filter-crs` defaults to `http://www.opengis.net/def/crs/OGC/1.3/CRS84` for a STAC API. ```http filter=id='LC08_L1TP_060247_20180905_20180912_01_T1_L1TP' AND collection='landsat8_l1tp' ``` -#### Example 1: POST with cql-json +#### Example 1: POST with cql2-json -Note that `filter-lang` defaults to `cql-json` in this case. The parameter `filter-crs` defaults +Note that `filter-lang` defaults to `cql2-json` in this case. The parameter `filter-crs` defaults to `http://www.opengis.net/def/crs/OGC/1.3/CRS84` for a STAC API. ```json { "filter": { - "and": [ - { "eq": [ { "property": "id" }, "LC08_L1TP_060247_20180905_20180912_01_T1_L1TP" ] }, - { "eq": [ { "property": "collection" }, "landsat8_l1tp" ] } + "op" : "and", + "args": [ + { + "op": "=", + "args": [ { "property": "id" }, "LC08_L1TP_060247_20180905_20180912_01_T1_L1TP" ] + }, + { + "op": "=", + "args" : [ { "property": "collection" }, "landsat8_l1tp" ] + } ] } } @@ -490,29 +509,40 @@ This example uses the queryables definition in [Interaction with Endpoints](#int Note that filtering on the `collection` field is relevant in Item Search, since the queries are cross-collection, whereas OGC API Features filters only operate against a single collection already. -#### Example 2: GET with cql-text +#### Example 2: GET with cql2-text ```http filter=collection = 'landsat8_l1tp' AND gsd <= 30 AND eo:cloud_cover <= 10 - AND datetime >= "2021-04-08T04:39:23Z" - AND datetime <= "2021-05-07T12:27:57Z" - AND INTERSECTS(geometry, POLYGON((43.5845 -79.5442, 43.6079 -79.4893, 43.5677 -79.4632, 43.6129 -79.3925, 43.6223 -79.3238, 43.6576 -79.3163, 43.7945 -79.1178, 43.8144 -79.1542, 43.8555 -79.1714, 43.7509 -79.6390, 43.5845 -79.5442)) + AND datetime >= '2021-04-08T04:39:23Z' + AND datetime <= '2021-05-07T12:27:57Z' + AND S_INTERSECTS(geometry, POLYGON((43.5845 -79.5442, 43.6079 -79.4893, 43.5677 -79.4632, 43.6129 -79.3925, 43.6223 -79.3238, 43.6576 -79.3163, 43.7945 -79.1178, 43.8144 -79.1542, 43.8555 -79.1714, 43.7509 -79.6390, 43.5845 -79.5442)) ``` -#### Example 2: POST with cql-json +#### Example 2: POST with cql2-json ```json { - "filter-lang": "cql-json", + "filter-lang": "cql2-json", "filter": { - "and": [ - { "eq": [ { "property": "collection" }, "landsat8_l1tp" ] }, - { "lte": [ { "property": "eo:cloud_cover" }, "10" ] }, - { "gte": [ { "property": "datetime" }, "2021-04-08T04:39:23Z" ] }, + "op": "and", + "args": [ + { + "op": "=", + "args": [ { "property": "collection" }, "landsat8_l1tp" ] + }, + { + "op": "<=", + "args": [ { "property": "eo:cloud_cover" }, "10" ] + }, + { + "op": ">=", + "args": [ { "property": "datetime" }, "2021-04-08T04:39:23Z" ] + }, { - "intersects": [ + "op": "s_intersects", + "args": [ { "property": "geometry" }, @@ -541,55 +571,7 @@ filter=collection = 'landsat8_l1tp' } ``` -### Example 3 - -Queryable properties can be used on either side of an operator. This is a generic example, as there are are few STAC properties -that are comparable in a meaningful way. This example uses a contrived example of two proprietary properties, `prop1` and `prop2` that are of the -same type. - -This queryables JSON Schema is used in these examples: - -```json -{ - "$schema" : "https://json-schema.org/draft/2019-09/schema", - "$id" : "https://example.org/queryables", - "type" : "object", - "title" : "Queryables for Example STAC API", - "description" : "Queryable names for the example STAC API Item Search filter.", - "properties" : { - "prop1" : { - "description" : "Property 1", - "type": "integer" - }, - "prop2" : { - "description" : "Property 2", - "type": "integer" - } - } -} -``` - -#### Example 3: GET with cql-text - -```http -filter=prop1 = prop2 -``` - -#### Example 3: POST with cql-json - -```json -{ - "filter-lang": "cql-json", - "filter": { - "eq": [ - { "property": "prop1" }, - { "property": "prop2" } - ] - } -} -``` - -### Example 4 +### Example 3: Conjunction with AND We'll be imagining these as queries against [EarthSearch Sentinel 2 COG](https://stacindex.org/catalogs/earth-search#/Cnz1sryATwWudkxyZekxWx6356v9RmvvCcLLw79uHWJUDvt2?t=items) data. @@ -681,21 +663,28 @@ recording path intersection only a corner of a grid square. This examples shows Show me all imagery that has low cloud cover (less than 10), and high data coverage (50), as I'd like a cloud free image that is not just a tiny sliver of data. -#### Example 4: AND cql-text (GET) +#### Example 3: AND cql2-text (GET) ```http filter=sentinel:data_coverage > 50 AND eo:cloud_cover < 10 ``` -#### Example 4: AND cql-json (POST) +#### Example 3: AND cql2-json (POST) ```json { - "filter-lang": "cql-json", + "filter-lang": "cql2-json", "filter": { - "and": [ - { "gt": [ { "property": "sentinel:data_coverage" }, 50 ] }, - { "lt": [ { "property": "eo:cloud_cover" }, 10 ] } + "op": "and", + "args": [ + { + "op": ">", + "args": [ { "property": "sentinel:data_coverage" }, 50 ] + }, + { + "op": "<", + "args": [ { "property": "eo:cloud_cover" }, 10 ] + } ] } } @@ -704,50 +693,108 @@ filter=sentinel:data_coverage > 50 AND eo:cloud_cover < 10 An 'or' is also supported, matching if either condition is true. Though it's not a sensible query you could get images that have full data coverage or low cloud cover. -### Example 5 +### Example 4: Disjunction with OR This uses the same queryables as Example 4. -#### Example 5: OR cql-text (GET) +#### Example 4: OR cql2-text (GET) ```http filter=sentinel:data_coverage > 50 OR eo:cloud_cover < 10 ``` -#### Example 5: OR cql-json (POST) +#### Example 4: OR cql2-json (POST) ```json { - "filter-lang": "cql-json", + "filter-lang": "cql2-json", + "filter": { + "op": "or", + "args": [ + { + "op": ">", + "args": [ { "property": "sentinel:data_coverage" }, 50 ] + }, + { + "op": "<", + "args": [ { "property": "eo:cloud_cover" }, 10 ] + } + ] + } +} +``` + +### Example 5: Property-Property Comparisons + +The Property-Property Comparisons conformance class permits queryable properties to be used on either side +of an operator. This is a generic example, as there are few STAC properties +that are comparable in a meaningful way. This example uses a contrived example of two proprietary properties, +`prop1` and `prop2` that are of the same type. + +This queryables JSON Schema is used in these examples: + +```json +{ + "$schema" : "https://json-schema.org/draft/2019-09/schema", + "$id" : "https://example.org/queryables", + "type" : "object", + "title" : "Queryables for Example STAC API", + "description" : "Queryable names for the example STAC API Item Search filter.", + "properties" : { + "prop1" : { + "description" : "Property 1", + "type": "integer" + }, + "prop2" : { + "description" : "Property 2", + "type": "integer" + } + } +} +``` + +#### Example 5: GET with cql2-text + +```http +filter=prop1 = prop2 +``` + +#### Example 5: POST with cql2-json + +```json +{ + "filter-lang": "cql2-json", "filter": { - "or": [ - { "gt": [ { "property": "sentinel:data_coverage" }, 50 ] }, - { "lt": [ { "property": "eo:cloud_cover" }, 10 ] } + "op": "eq", + "args": [ + { "property": "prop1" }, + { "property": "prop2" } ] } } ``` -### Example 6: Temporal +### Example 6: Temporal Intersection This uses the same queryables as Example 4. The only temporal operator required is `ANYINTERACTS`. This is effectively that the datetime or interval operands have any overlap between them. -#### Example 6: ANYINTERACTS cql-text (GET) +#### Example 6: T_INTERSECTS cql2-text (GET) ```http -filter=datetime ANYINTERACTS 2020-11-11T00:00:00Z/2020-11-12T00:00:00Z +filter=datetime T_INTERSECTS 2020-11-11T00:00:00Z/2020-11-12T00:00:00Z ``` -#### Example 6: ANYINTERACTS cql-json (POST) +#### Example 6: T_INTERSECTS cql2-json (POST) ```json { - "filter-lang": "cql-json", + "filter-lang": "cql2-json", "filter": { - "anyinteracts": [ + "op": "t_intersects", + "args": [ { "property": "datetime" }, [ "2020-11-11T00:00:00Z", "2020-11-12T00:00:00Z"] ] @@ -755,26 +802,27 @@ filter=datetime ANYINTERACTS 2020-11-11T00:00:00Z/2020-11-12T00:00:00Z } ``` -### Example 7: Spatial +### Example 7: Spatial Intersection The only spatial operator that must be implemented for Basic Spatial Operators -is `INTERSECTS`. This has the same semantics as the one provided -in the Item Search `intersects` parameter. The `cql-text` format uses WKT geometries and the `cql-json` +is `S_INTERSECTS`. This has the same semantics as provided +by the Item Search `intersects` parameter. The `cql2-text` format uses WKT geometries and the `cql2-json` format uses GeoJSON geometries. -#### Example 7: INTERSECTS cql-text (GET) +#### Example 7: S_INTERSECTS cql2-text (GET) ```http -filter=INTERSECTS(geometry,POLYGON((-77.0824 38.7886,-77.0189 38.7886,-77.0189 38.8351,-77.0824 38.8351,-77.0824 38.7886))) +filter=S_INTERSECTS(geometry,POLYGON((-77.0824 38.7886,-77.0189 38.7886,-77.0189 38.8351,-77.0824 38.8351,-77.0824 38.7886))) ``` -#### Example 7: INTERSECTS cql-json (POST) +#### Example 7: S_INTERSECTS cql2-json (POST) ```json { - "filter-lang": "cql-json", + "filter-lang": "cql2-json", "filter": { - "intersects": [ + "op": "s_intersects", + "args": [ { "property": "geometry" } , { "type": "Polygon", @@ -789,47 +837,55 @@ filter=INTERSECTS(geometry,POLYGON((-77.0824 38.7886,-77.0189 38.7886,-77.0189 3 } ``` -### Example 8: Spatial Disjunction +### Example 8: Spatial Intersection Disjunction One limitation of the `intersects` parameter is that only a single geometry may be provided. While most GeoJSON geometries can be combined to form a composite (e.g., multiple Polygons can be combined to form a -MultiPolygon), this is much easier to do in the query by combining `INTERSECTS` predicates with the `OR` +MultiPolygon), this is much easier to do in the query by combining `S_INTERSECTS` predicates with the `OR` logical operator. -#### Example 8: INTERSECTS cql-text (GET) +#### Example 8: S_INTERSECTS cql2-text (GET) ```http -filter=INTERSECTS(geometry,POLYGON((-77.0824 38.7886,-77.0189 38.7886,-77.0189 38.8351,-77.0824 38.8351,-77.0824 38.7886))) OR INTERSECTS(geometry,POLYGON((-79.0935 38.7886,-79.0290 38.7886,-79.0290 38.8351,-79.0935 38.8351,-79.0935 38.7886))) +filter=S_INTERSECTS(geometry,POLYGON((-77.0824 38.7886,-77.0189 38.7886,-77.0189 38.8351,-77.0824 38.8351,-77.0824 38.7886))) OR S_INTERSECTS(geometry,POLYGON((-79.0935 38.7886,-79.0290 38.7886,-79.0290 38.8351,-79.0935 38.8351,-79.0935 38.7886))) ``` -#### Example 8: INTERSECTS cql-json (POST) +#### Example 8: S_INTERSECTS cql2-json (POST) ```json { + "filter-lang": "cql2-json", "filter": { - "or" : [ - "intersects": [ - { "property": "geometry" } , - { - "type": "Polygon", - "coordinates": [[ - [-77.0824, 38.7886], [-77.0189, 38.7886], - [-77.0189, 38.8351], [-77.0824, 38.8351], - [-77.0824, 38.7886] - ]] - } - ], - "intersects": [ - { "property": "geometry" } , - { - "type": "Polygon", - "coordinates": [[ - [-79.0935, 38.7886], [-79.0290, 38.7886], - [-79.0290, 38.8351], [-79.0935, 38.8351], - [-79.0935, 38.7886] - ]] - } - ] + "op": "or" , + "args": [ + { + "op": "s_intersects", + "args": [ + { "property": "geometry" } , + { + "type": "Polygon", + "coordinates": [[ + [-77.0824, 38.7886], [-77.0189, 38.7886], + [-77.0189, 38.8351], [-77.0824, 38.8351], + [-77.0824, 38.7886] + ]] + } + ] + }, + { + "op": "s_intersects", + "args": [ + { "property": "geometry" } , + { + "type": "Polygon", + "coordinates": [[ + [-79.0935, 38.7886], [-79.0290, 38.7886], + [-79.0290, 38.8351], [-79.0935, 38.8351], + [-79.0935, 38.7886] + ]] + } + ] + } ] } } @@ -844,27 +900,138 @@ different sets of properties. For example, a collection of Sentinel 2 data may h data. However, we many also want to also include in our result items that do not have a value defined for either of those properties. -#### Example 9: cql-text (GET) +#### Example 9: cql2-text (GET) ```http filter=sentinel:data_coverage > 50 OR landsat:coverage_percent < 10 OR (sentinel:data_coverage IS NULL AND landsat:coverage_percent IS NULL) ``` -#### Example 9: cql-json (POST) +#### Example 9: cql2-json (POST) ```json { + "filter-lang": "cql2-json", "filter": { - "or": [ - { "gte": [ { "property": "sentinel:data_coverage" }, 50 ] }, - { "gte": [ { "property": "landsat:coverage_percent" }, 50 ] }, + "op": "or", + "args": [ + { + "op": ">=", + "args": [ { "property": "sentinel:data_coverage" }, 50 ] + }, + { + "op": ">=", + "args": [ { "property": "landsat:coverage_percent" }, 50 ] + }, { - "and": [ - { "isNull": { "property": "sentinel:data_coverage" } }, - { "isNull": { "property": "landsat:coverage_percent" } } + "op": "and", + "args": [ + { + "op": "isNull", + "args": { "property": "sentinel:data_coverage" } + }, + { + "op": "isNull", + "args": { "property": "landsat:coverage_percent" } + } ] } ] } } ``` + +### Example 10: Using BETWEEN + +The BETWEEN operator allows for checking if a numeric value is within a specified inclusive range. + +#### Example 10: cql2-text (GET) + +```http +filter=eo:cloud_cover BETWEEN 0 AND 50 +``` + +#### Example 10: cql2-json (POST) + +```json +{ + "filter-lang": "cql2-json", + "filter": { + "op": "between", + "args": [ + { "property": "eo:cloud_cover" }, + [ 0, 50 ] + ] + } +} +``` + +### Example 11: Using LIKE + +The LIKE operator allows for pattern-based string matching. + +#### Example 11: cql2-text (GET) + +```http +filter=mission LIKE 'sentinel%' +``` + +#### Example 11: cql2-json (POST) + +```json +{ + "filter-lang": "cql2-json", + "filter": { + "op": "like", + "args": [ + { "property": "mission" }, + "sentinel%" + ] + } +} +``` + + diff --git a/fragments/filter/openapi.yaml b/fragments/filter/openapi.yaml index 48a2088a..7d2ff30d 100644 --- a/fragments/filter/openapi.yaml +++ b/fragments/filter/openapi.yaml @@ -31,10 +31,10 @@ paths: /queryables: get: - summary: Get the JSON Schema defining the list of variable terms that can be used in CQL expressions. + summary: Get the JSON Schema defining the list of variable terms that can be used in CQL2 expressions. operationId: getQueryables description: |- - This endpoint returns a list of variable terms that can be used in CQL expressions. The + This endpoint returns a list of variable terms that can be used in CQL2 expressions. The precise definition of this can be found in the OGC API - Features - Part 3: Filtering and the Common Query Language (CQL) specification. tags: @@ -70,10 +70,10 @@ paths: A link with rel=queryables for queryables to only apply to this collection. /collections/{collectionId}/queryables: get: - summary: Get the JSON Schema defining the list of variable terms that can be used in CQL expressions. + summary: Get the JSON Schema defining the list of variable terms that can be used in CQL2 expressions. operationId: getQueryablesForCollection description: |- - This endpoint returns a list of variable terms that can be used in CQL expressions. The + This endpoint returns a list of variable terms that can be used in CQL2 expressions. The precise definition of this can be found in the OGC API - Features - Part 3: Filtering and the Common Query Language (CQL) specification. parameters: @@ -99,7 +99,7 @@ components: description: |- **Extension:** Filter - A CQL filter expression for filtering items. + A CQL2 filter expression for filtering items. required: true schema: oneOf: @@ -112,7 +112,7 @@ components: description: |- **Extension:** Filter - The CQL filter encoding that the 'filter' value uses. Must be one of 'cql-text' or 'cql-json'. + The CQL2 filter encoding that the 'filter' value uses. Must be one of 'cql-text' or 'cql-json'. required: false schema: $ref: '#/components/schemas/filter-lang' @@ -145,13 +145,13 @@ components: $ref: '#/components/schemas/filter-crs' filter-cql-text: description: | - A CQL filter expression in the 'cql-text' encoding. + A CQL2 filter expression in the 'cql-text' encoding. type: string filter-cql-json: $ref: './cql.yml#/components/schemas/booleanValueExpression' filter-lang: description: | - The CQL filter encoding that the 'filter' value uses. + The CQL2 filter encoding that the 'filter' value uses. type: string enum: - 'cql-text' @@ -164,7 +164,7 @@ components: format: uri responses: Queryables: - description: A JSON Schema defining the Queryables allowed in CQL expressions + description: A JSON Schema defining the Queryables allowed in CQL2 expressions content: application/schema+json: schema: