The purpose of Geostore is to store geospatial datasets. This document should provide the technical know-how to create and maintain such datasets.
The keywords "must", "must not", "required", "shall", "shall not", "should", "should not", "recommended", "may", and "optional" in this document are to be interpreted as described in RFC 2119.
You'll need the following information to start using Geostore:
- Geostore AWS account ID (
GEOSTORE_AWS_ACCOUNT_ID
) - Geostore user role name (
GEOSTORE_USER_ROLE_NAME
)
Replace the GEOSTORE_…
strings in the following documentation with the actual values given by the
Geostore instance maintainer.
Geostore needs to be able to read all the files in the dataset in the source bucket. This is
achieved using role assumption: you need to create a role which GEOSTORE_AWS_ACCOUNT_ID
is allowed
to assume (see the policy below) and which has read access to the dataset files in your bucket. You
then pass the ARN of that role into the
dataset version endpoint.
Technical note: Using role assumption means that it's easy for you to verify that the role you have created has access to the right things, without having to ask the Geostore team to verify it for you. Only the role assumption itself needs to be checked by the Geostore team.
Template trust policy on your role:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Resource": "arn:aws:iam::<GEOSTORE_AWS_ACCOUNT_ID>:role/<GEOSTORE_USER_ROLE_NAME>"
}
]
}
Template dataset source S3 bucket policy:
{
"Id": "<ID>",
"Version": "2012-10-17",
"Statement": [
{
"Sid": "<STATEMENT_ID>",
"Action": ["s3:GetObject", "s3:GetObjectAcl", "s3:GetObjectTagging"],
"Effect": "Allow",
"Resource": "arn:aws:s3:::<YOUR_BUCKET>/<YOUR_DATASET>/*",
"Principal": {
"AWS": "arn:aws:iam::<YOUR_ACCOUNT_ID>:role/<YOUR_ROLE_NAME>"
}
}
]
}
A dataset consists of a set of files in Amazon Web Services Simple Storage Service (AWS S3). The dataset consists of geospatial metadata files in SpatioTemporal Asset Catalogs (STAC) format and data files, which are called "assets" in STAC.
Geostore performs many checks on datasets. If any of the checks fail the dataset will not be imported, so it's important to know what they are. The following list is a reference of all the checks which are currently in place.
- Every metadata file must follow the STAC Collection Specification.
- Every metadata URL (in the
links
property) must be an S3 URL of the forms3://BUCKET_NAME/KEY
, for example,s3://my-bucket/some-path/foo.tif
. - Every asset (in the
assets
property) must have: - Every metadata and asset file must be in the same S3 bucket.
- Every metadata and asset URL must be readable by Geostore.
- A dataset may refer to the same asset more than once. All references to the same asset must have the same multihash. That is, having a SHA-1 and a SHA-256 checksum for the same file will be considered invalid, even if both checksums are valid. This is to enable a simpler checksum validation.
Geostore allows read/write access for users authorized by SAML identity provider
(GEOSTORE_SAML_IDENTITY_PROVIDER_ARN
) configured during deployment time (see
README).
Example of AWS service account authentication and authorization in to Geostore users role via Azure:
-
Log in to Geostore AWS account
aws-azure-login --profile=li-geostore-prod
-
Set AWS profile for subsequent commands
export AWS_PROFILE=li-geostore-prod
Example of assuming the Geostore roles from a Toitū Te Whenua LINZ AWS account:
-
Choose any Toitū Te Whenua LINZ AWS account to assume the role from
export SOURCE_PROFILE=<YOUR_AWS_SOURCE_PROFILE> # e.g. li-small-apps-nonprod
-
Log in to any Toitū Te Whenua LINZ AWS account:
aws-azure-login --profile="$SOURCE_PROFILE"
-
Assume either Geostore role:
-
API user:
aws sts assume-role --role-arn="$(aws iam get-role --role-name=api-users | jq --raw-output .Role.Arn)" --role-session-name="$USER"
-
S3 user:
aws sts assume-role --role-arn="$(aws iam get-role --role-name=s3-users | jq --raw-output .Role.Arn)" --role-session-name="$USER"
-
You can communicate with a Geostore instance using either the low-level application programming
interface (API) or the high-level command-line interface (CLI). The following documentation explains
how to work with both. For an example of working with Python, see the
should_successfully_run_dataset_version_creation_process_with_multiple_assets
function in
tests/test_processing_stack.py
.
The API endpoints include:
- Dataset space, to create, get, update or delete individual datasets, and to list all datasets.
- Dataset versions, to create new versions of datasets. The S3 files which constitute the dataset are all linked to a specific dataset version.
- Import status, to get information about the status of dataset version import, including errors and issues.
These are implemented as AWS Lambda functions, which means they can be run ("invoked") either via
the AWS web interface (links above) or via any tool using the AWS API, such as the
official AWS CLI. AWS CLI API requests have the form
aws lambda invoke --function-name=LAMBDA-FUNCTION-NAME --payload=JSON /dev/stdout
.
The CLI is a convenience function built on top of the API for easier use. Rather than using JSON as the primary input and output format it takes only the minimal input necessary and prints only the minimal new information. In cases of more complex output like the dataset version import status the JSON response is output verbatim, since it can contain an arbitrary amount of relevant information.
To install the Geostore CLI, run pip3 install geostore
.
The general synopsis is geostore [GLOBAL_PARAMETER…] NOUN VERB [ACTION_PARAMETER…]
. NOUN
is
the type of thing the command is operating on, for example version
when dealing with dataset
versions. VERB
is the action it is telling the system to take, for example list
to show a
listing of the relevant objects, or create
to create such an object. Verbs may have parameters in
the form of key/value pairs. --KEY=VALUE
means the parameter is mandatory, and [--KEY=VALUE]
means the parameter is optional.
Run geostore --help
to show the overall synopsis. geostore NOUN --help
and
geostore NOUN VERB --help
show subcommand synopsis.
Synopsis: geostore dataset VERB [PARAMETER…]
Synopsis: geostore dataset create --title=TITLE --description=DESCRIPTION
This creates a new dataset space and prints the new dataset ID on standard output when successful.
Note: it is important to choose an accurate and stable title. Changing the title is complex, time-consuming, risky and lossy. If you need to change the title, choose between changing the dataset title by creating a copy of the latest dataset version and changing the dataset title by renaming and moving the files below.
CLI example:
$ geostore dataset create --title=Auckland_2020 --description='Aerial imagery from April 2020'
01F9ZFRK12V0WFXJ94S0DHCP65
API example:
# AWS CLI version 1
$ aws lambda invoke --function-name=datasets --payload='{"http_method": "POST", "body": {"title": "Auckland_2020", "description": "Aerial imagery from April 2020"}}' /dev/stdout
# AWS CLI version 2
$ aws lambda invoke --function-name=datasets --invocation-type RequestResponse --cli-binary-format raw-in-base64-out --payload='{"http_method": "POST", "body": {"title": "Auckland_2020", "description": "Aerial imagery from April 2020"}}' /dev/stdout
# Sample response:
{"status_code": 201, "body": {"created_at": "2021-05-26T21:17:47.758448+0000", "pk": "DATASET#01F6N8MSVEY2Y6EPFZ5XR0KFW1", "title": "Auckland_2020", "updated_at": "2021-05-26T21:17:47.758538+0000", "id": "01F6N8MSVEY2Y6EPFZ5XR0KFW1"}}
Synopsis: geostore dataset list [--id=ID]
Prints a listing of datasets, optionally filtered by the dataset ID.
Examples:
-
List all datasets using the CLI:
$ geostore dataset list Auckland_2020 Wellington_2020
-
List all datasets using the API:
# AWS CLI version 1 $ aws lambda invoke --function-name=datasets --payload='{"http_method": "GET", "body": {}}' /dev/stdout # AWS CLI version 2 $ aws lambda invoke --function-name=datasets --invocation-type RequestResponse --cli-binary-format raw-in-base64-out --payload='{"http_method": "GET", "body": {}}' /dev/stdout # Sample response: {"status_code": 200, "body": [{"created_at": "2021-02-01T13:38:40.776333+0000", "id": "cb8a197e649211eb955843c1de66417d", "title": "Auckland_2020", "updated_at": "2021-02-01T13:39:36.556583+0000"}]}
-
Filter to a single dataset using the CLI:
$ geostore dataset list --id=01F9ZFRK12V0WFXJ94S0DHCP65 Auckland_2020
-
Filter to a single dataset using the API:
# AWS CLI version 1 $ aws lambda invoke --function-name=datasets --payload='{"http_method": "GET", "body": {"id": "cb8a197e649211eb955843c1de66417d"}}' /dev/stdout # AWS CLI version 2 $ aws lambda invoke --function-name=datasets --invocation-type RequestResponse --cli-binary-format raw-in-base64-out --payload='{"http_method": "GET", "body": {"id": "cb8a197e649211eb955843c1de66417d"}}' /dev/stdout # Sample response: {"status_code": 200, "body": {"created_at": "2021-02-01T13:38:40.776333+0000", "id": "cb8a197e649211eb955843c1de66417d", "title": "Auckland_2020", "updated_at": "2021-02-01T13:39:36.556583+0000"}}
Synopsis: geostore dataset delete --id=ID
ID
is the dataset ID.
Deletes a dataset. Will only work if there are no versions in the dataset. This command does not print anything when successful.
CLI example:
$ geostore dataset delete --id=Auckland_2020
API example:
# AWS CLI version 1
$ aws lambda invoke --function-name=datasets --payload='{"http_method": "DELETE", "body": {"id": "cb8a197e649211eb955843c1de66417d"}}' /dev/stdout
# AWS CLI version 2
$ aws lambda invoke --function-name=datasets --invocation-type RequestResponse --cli-binary-format raw-in-base64-out --payload='{"http_method": "DELETE", "body": {"id": "cb8a197e649211eb955843c1de66417d"}}' /dev/stdout
# Sample response
{"status_code": 204, "body": {}}
Synopsis: geostore version VERB [PARAMETER…]
Synopsis: geostore version create --dataset-id=ID --metadata-url=URL --s3-role-arn=ROLE_ARN
Creates a new dataset version. It returns immediately, while the import process continues in AWS. It prints the new dataset version ID and the ID of the import process on standard output in case of success.
CLI example:
$ geostore version create --dataset-id=01FKPEP0SQG4W2QF8KSQB6EJCD --metadata-url=s3://my-staging/Auckland_2020/catalog.json --s3-role-arn=arn:aws:iam::702361495692:role/s3-readers
2021-11-08T01-13-37-203Z_CJD6XKVJKS29ZXPA arn:aws:states:ap-southeast-2:702361495692:execution:processingdatasetversioncreation55809360-7likTQJZBsBG:2021-11-08T01-13-37-203Z_CJD6XKVJKS29ZXPA
API example:
# AWS CLI version 1
$ aws lambda invoke --function-name=dataset-versions --payload='{"http_method": "POST","body": {"id": "cb8a197e649211eb955843c1de66417d","metadata_url": "s3://example-s3-url","s3_role_arn": "arn:aws:iam::1234567890:role/example-role"}}' /dev/stdout
# AWS CLI version 2
$ aws lambda invoke --function-name=datasets-versions --invocation-type RequestResponse --cli-binary-format raw-in-base64-out --payload='{"http_method": "POST","body": {"id": "cb8a197e649211eb955843c1de66417d","metadata_url": "s3://example-s3-url","s3_role_arn": "arn:aws:iam::1234567890:role/example-role"}}' /dev/stdout
# Sample response:
{"status_code": 201, "body": {"dataset_version": "example_dataset_version_id", "execution_arn": "arn:aws:batch:ap-southeast-2:xxxx:job/example-arn"}}
Synopsis: geostore version status --execution-arn=EXECUTION_ARN
EXECUTION_ARN
is the import process ID printed by geostore version create
.
This prints the current status of the dataset version import process started by
geostore version create
.
CLI example:
$ geostore version status --execution-arn=arn:aws:states:ap-southeast-2:702361495692:execution:processingdatasetversioncreation55809360-7likTQJZBsBG:2021-11-08T01-13-37-203Z_CJD6XKVJKS29ZXPA
{"step_function": {"status": "Succeeded"}, "validation": {"status": "Passed", "errors": []}, "metadata_upload": {"status": "Complete", "errors": {"failed_tasks": 0, "failure_reasons": []}}, "asset_upload": {"status": "Complete", "errors": {"failed_tasks": 0, "failure_reasons": []}}}
API example:
# AWS CLI version 1
$ aws lambda invoke --function-name=import-status --payload='{"http_method": "GET", "body": {"execution_arn": "arn:aws:batch:ap-southeast-2:xxxx:job/example-arn"}}' /dev/stdout
# AWS CLI version 2
$ aws lambda invoke --function-name=import-status --invocation-type RequestResponse --cli-binary-format raw-in-base64-out --payload='{"http_method": "GET", "body": {"execution_arn": "arn:aws:batch:ap-southeast-2:xxxx:job/example-arn"}}' /dev/stdout
# Sample response:
{"step_function": {"status": "Succeeded"}, "validation": {"status": "Passed", "errors": []}, "metadata_upload": {"status": "Complete", "errors": {"failed_tasks": 0, "failure_reasons": []}}, "asset_upload": {"status": "Complete", "errors": {"failed_tasks": 0, "failure_reasons": []}}}
The ARN of our SNS Topic is
arn:aws:sns:ap-southeast-2:<GEOSTORE_AWS_ACCOUNT_ID>:geostore-import-status
which you may choose
to subscribe to.
You may also choose to apply a subscription filter policy, which will filter notifications for a specific dataset or specific statuses. Included in the example is all the valid statuses.
The Geostore will store a dataset in a top level directory with the same name as the dataset title specified when importing it. You can filter SNS topics for a dataset by providing the title:
{
"dataset_title": ["Taranaki_2020"],
"status": ["RUNNING", "SUCCEEDED", "FAILED", "TIMED_OUT", "ABORTED"]
}
Below is an example payload received when subscribed to our SNS Topic. The 'Message' field will contain a JSON string with specific details regarding the Step Function Execution.
{
"Type": "Notification",
"MessageId": "xxxx-xxxx-xxxx",
"TopicArn": "arn:aws:sns:ap-southeast-2:<GEOSTORE_AWS_ACCOUNT_ID>:geostore-import-status",
"Message": "{\"version\": \"0\", […]}",
"Timestamp": "2021-07-07T01:49:33.471Z",
"SignatureVersion": "1",
"Signature": "xxxxx",
"SigningCertURL": "https://example.com",
"UnsubscribeURL": "https://example.com",
"MessageAttributes": {
"dataset_title": {
"Type": "String",
"Value": "01F9ZFRK12V0WFXJ94S0DHCP65"
},
"status": {
"Type": "String",
"Value": "SUCCEEDED"
}
}
}
Below is an example of the AWS EventBridge 'Step Functions Execution Status Change' payload. This will be formatted as a string and associated with the 'Message' attribute in the above SNS payload. See AWS EventBridge Payload Examples
{
"version": "0",
"id": "3afc69d1-1291-6372-ae76-9a9c11ee35c4",
"detail-type": "Step Functions Execution Status Change",
"source": "aws.states",
"account": "<GEOSTORE_AWS_ACCOUNT_ID>",
"time": "2021-07-07T01:49:29Z",
"region": "ap-southeast-2",
"resources": ["arn:aws:states:example"],
"detail": {
"executionArn": "arn:aws:states:example",
"stateMachineArn": "arn:aws:states:example",
"name": "Aerial_Imagery_xxxx",
"status": "SUCCEEDED",
"startDate": 1625622391067,
"stopDate": 1625622569782,
"input": "{\"dataset_id\": \"01F9ZA9ZZZDM815S20EHXEAT40\", \"dataset_title\": \"test_1625622377\", \"version_id\": \"2021-07-07T01-46-30-787Z_9NJEAD3VXRCH5W05\", \"metadata_url\": \"s3://example/catalog.json\", \"s3_role_arn\": \"arn:aws:iam::715898075157:role/example\"}",
"inputDetails": {
"included": true
},
"output": "{\"dataset_id\":\"01F9ZA9ZZZDM815S20EHXEAT40\",\"dataset_title\":\"test_1625622377\",\"version_id\":\"2021-07-07T01-46-30-787Z_9NJEAD3VXRCH5W05\",\"metadata_url\":\"s3://example/catalog.json\",\"s3_role_arn\":\"arn:aws:iam::715898075157:role/example\",\"content\":{\"first_item\":\"0\",\"iteration_size\":1,\"next_item\":-1,\"assets_table_name\":\"example\",\"results_table_name\":\"example\"},\"validation\":{\"success\":true},\"import_dataset\":{\"asset_job_id\":\"e4ad8b0d-4358-4c42-bb0d-3577c96f7039\",\"metadata_job_id\":\"84a7b4fc-7d00-403c-a5fb-91257f406afb\"},\"upload_status\":{\"validation\":{\"status\":\"Passed\",\"errors\":[]},\"asset_upload\":{\"status\":\"Complete\",\"errors\":[]},\"metadata_upload\":{\"status\":\"Complete\",\"errors\":[]}},\"update_root_catalog\":{\"new_version_s3_location\":\"s3://linz-geostore/example/catalog.json\"}}",
"outputDetails": {
"included": true
}
}
}
Note: the output field will only be populated above when the Step Function has succeeded.
This is the simplest way to change a dataset title, but there will be no explicit connection between the new and old datasets. Anyone wishing to go back beyond the rename of a dataset needs to be aware of this rename and has to either know or find the original dataset title.
- Create a new dataset
- Create a new dataset version for the dataset created above,
using a
metadata_url
pointing to the latest version of the original dataset. - Wait for the import to finish.
- Optional: if the original dataset can be removed at this point (or sometime in the future), please let the Geostore product team know, and we'll arrange it.
This process is more cumbersome and time-consuming than the above, and results links to the old dataset being broken rather than slowly phased out, but can be followed if necessary.
- Send a request to the product team asking for a rename, specifying the current and new title of
the dataset. The product team then takes care of the rest of the process:
- Schedule necessary downtime and notify users.
- Turn off external access to the whole or part of the system to avoid any conflicts.
- Run a manual rename of all the files in the relevant dataset.
- Use the
${ENV}-datasets
endpoint to rename the dataset in the database. - Re-enable external access and notify users.
- Notify requester about the name change completion.
- The root catalog has the ID
root_catalog
. - The dataset catalogs have IDs consisting of the dataset title, a
hyphen, and a Universally Unique Lexicographically Sortable Identifier (ULID), for example
Wellington_2020
. - IDs within the dataset versions are unchanged.