Learn how to use <geosys/> platform capabilities in your own business workflow! Build your processor and learn how to run them on your platform.
Who we are
Report Bug
·
Request Feature
Table of Contents
The aim of this project is to help our customers valuing <geosys/> platform capabilities to build their own analytic of interest.
This directory exposes an example of code that will enable you to create an Analytics Datacube of clear images and store the result on a cloud storage provider (Azure Blob Storage or AWS S3).
This directory allows you to run this example both through a notebook and as a local application on your machine.
To be able to run this example, you will need to have the following tools to be installed
-
Install Git
Please install Git on your computer. You can download and install it by visiting the [official Git website] (https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) and following the provided instructions
-
Install Conda
Please install Conda on your computer. You can download and install it by following the instructions provided on the official Conda website
-
Install Docker Desktop
Please install Docker Desktop on your computer. You can download and install it by following the instructions provided on the official Docker Desktop website
-
Install Jupyter Notebook
Please install jupyter Notebook on your computer. You can install it by following the instructions provided on the official Jupyter website
Make sure you have valid credentials. If you need to get trial access, please register here.
This package has been tested on Python 3.12.0
To set up the project, follow these steps:
-
Clone the project repository:
git clone http://github.com/earthdaily/analytics-datacube-processor -
Change the directory:
cd analytics-datacube-processor
To use the project with Jupyter Notebook, follow these steps:
-
Create a Conda environment:
To create a Conda environment, ensure first you have installed Conda on your computer. You can download and install it by following the instructions provided on the official Conda website.
conda create -y --name demo -
Activate the Conda environment:
conda activate demo -
Install the project dependencies. You can do this by running the following command in your terminal:
conda install -y pip pip install -r requirements.txt pip install ipykernel -
Set up the Jupyter Notebook kernel for the project:
python -m ipykernel install --user --name demo --display-name demo -
Open jupyter and then the example notebook (analytics_datacube_v2.ipynb) by clicking on it.
-
Select the "Kernel" menu and choose "Change Kernel". Then, select "demo" from the list of available kernels.
-
Run the notebook cells to execute the code example.
To set up and run the project using Docker, follow these steps:
-
Create an environment file at root directory. You must specify following values to make the tool run:
# geosys identity server information IDENTITY_SERVER_URL = https://identity.geosys-na.com/v2.1/connect/token # optional (to check token validity) # CIPHER_CERTIFICATE_PUBLIC_KEY = # optional (to use credentials from .env file) # API_CLIENT_ID = # API_CLIENT_SECRET = # API_USERNAME = # API_PASSWORD = # AWS credentials AWS_ACCESS_KEY_ID = AWS_SECRET_ACCESS_KEY = # optional #AWS_BUCKET_NAME = # Azure credentials AZURE_ACCOUNT_NAME = AZURE_BLOB_CONTAINER_NAME = AZURE_SAS_CREDENTIAL = # Example input file path to run the processor in local INPUT_JSON_PATH=data/processor_input_example.json -
Build the Docker image locally:
docker build -t template . -
Run the Docker container:
- Processor Mode
docker run -d --name template_container templateSome options can be provided to the processor:
--input_path: Path to the input data file
--bearer_token: Geosys Api bearer token value
--aws_s3_bucket_name: AWS S3 Bucket name
--cloud_storage_provider: Cloud storage provider to store the zarr file (AWS/AZURE)
--entity_id: Provide an entity_id value added to the zarr output file
--metrics: Display bandwitdh & time metrics in results (bool)
For example:
docker run -d --name template_container template --aws_s3_bucket_name byoa-demo
- Processor with API Mode
```
docker run -e RUN_MODE_ENV=API -d --name template_container -p 8081:80 template
```
-
Access the API by opening a web browser and navigating to the following URL:
http://127.0.0.1:8081/docsThis URL will open the Swagger UI documentation, click on the "Try it out" button for the POST endpoint.
- Select first a cloud storage provider to store the zarr file produced as output (AWS or Azure Blob Storage)
- You can specify a value for the AWS S3 bucket where the file will be stored (default value can be set in env file: AWS_BUCKET_NAME).
- Select then one or several indicator values to build the datacube in zarr format.
- As example, you can then enter the following request body (polygon can be wkt or geojson)
Body Example for analytics_datacube_processor endpoint: (WKT) ```json { "polygon": "POLYGON((-90.41 41.6663, -90.41 41.6545, -90.3775 41.6541, -90.3778 41.6660, -90.41 41.6663))", "startDate": "2023-06-01", "endDate": "2023-07-01" } ```
(GeoJson)
{
"polygon": "{\"type\": \"Polygon\",\"coordinates\": [[[-90.41, 41.6663],[-90.41, 41.6545],[-90.3775, 41.6541],[-90.3778, 41.666],[-90.41, 41.6663]]]}",
"startDate": "2023-06-01",
"endDate": "2023-07-01"
}-
Closing the Docker container:
To delete the container when it is not needed anymore run :
docker stop demo
├── README.md <- The top-level README for developers using this project.
├── notebooks <- Jupyter notebooks. Naming convention is a number (for ordering),
│ the creator's initials, and a short `-` delimited description, e.g.
│ `1.0-jqp-initial-data-exploration`.
│
├── requirements.txt <- The requirements file for reproducing the analysis environment, e.g.
│ generated with `pip freeze > requirements.txt`
├── environment.yml <- The conda requirements file for reproducing the analysis environment, e.g.
│ generated with `conda env export > environment.yml`, or manually
│
├── pyproject.toml <- Makes project pip installable (pip install -e .) so src can be imported
├── MANIFEST.in <- Used to include/exclude files for package genration.
├───src <- Source code for use in tis project.
│ ├───main.py
│ ├───api
│ │ ├── files
│ │ │ └── favicon.svg
│ │ ├── __init__.py
│ │ ├── constants.py
│ │ └── api.py
│ ├───data
│ │ └── processor_input_example.json
│ ├───data
│ │ ├── __init__.py
│ │ ├── input_schema.py
│ │ └── output_schema.py
│ ├───utils
│ │ ├── __init__.py
│ │ └── file_utils.py
│ └───analytics_datacube_processor
│ ├── __init__.py
│ ├── processor.py
│ └── utils.py
└── manifests
├── analytics-datacube.yml
└── analytics-datacube-svc.yml
The following links will provide access to more information:
If this project has been useful, that it helped you or your business to save precious time, don't hesitate to give it a star.
Distributed under the MIT License.
For any additonal information, please email us.
© 2023 Geosys Holdings ULC, an Antarctica Capital portfolio company | All Rights Reserved.