Merge pull request #16 from stac-utils/feature/aster

Add subpackage for processing ASTER L1T data

.github/workflows/continuous-integration.yml

@@ -3,7 +3,6 @@ name: CI
 on:
   push:
     branches:
-      - develop
       - master
   pull_request:
 
@@ -22,20 +21,13 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
 
-      - name: Cache dependencies
-        uses: actions/cache@v2
-        with:
-          path: ~/.cache/pip
-          key: pip-${{ hashFiles('requirements-dev.txt') }}
-          restore-keys: pip-
-
       - name: Execute linters and test suites
-        run: ./scripts/cibuild
+        run: ./docker/cibuild
 
 
       - name: Upload All coverage to Codecov
         uses: codecov/codecov-action@v1
         with:
           token: ${{ secrets.CODECOV_TOKEN }}
           file: ./coverage.xml
-          fail_ci_if_error: false 
+          fail_ci_if_error: false

.readthedocs.yml

@@ -21,7 +21,11 @@ python:
       path: stactools_core/
     - method: pip
       path: stactools_cli/
+    - method: pip
+      path: stactools_aster/
     - method: pip
       path: stactools_landsat/
     - method: pip
       path: stactools_planet/
+    - method: pip
+      path: stactools_browse/

README.md

@@ -58,52 +58,71 @@ to install
 ## Running
 
 ```
-> stactools --help
+> stac --help
 ```
 
 ## Documentation
 
 See the [documentation page](https://stactools.readthedocs.io/en/latest/) for the latest docs.
 
+## Sub-packages
+
+`stactools` is comprised of subpackages that provide library and CLI functionality. Below is a list of available subpackages.
+
+| subpackage                    | description                                                                     |
+| ------------------------------| ------------------------------------------------------------------------------- |
+| `stactools_core`              | Contains core library functionality that is used across the other projects      |
+| `stactools_cli`               | Contains the command line interface (cli) for running the `stactools` command   |
+| `stactools_aster`             | Methods and commands for working with ASTER data                 |
+| `stactools_planet`            | Methods and commands for working with planet data                |
+| `stactools_landsat`           | Methods and commands for working with landsat data (TODO)        |
+| `stactools_browse`            | Contains a command for launching stac-browser against a local STAC |
+
+Subpackages are symlinked to the `stactools` directory in this repo to allow them to be importable for python running at the top level directory of the repository clone.
+
 ## Developing
 
+### Using docker
+
+Some subpackages require environments with more complex environments than can be set up just through pip. For example, the `stactools.aster` package uses rasterio functionality that required a GDAL enabled with the HDF4 format. Because of this, it's recommended to utilize the docker environment provided by this repository.
+
+Once the container is built, you can run the `scripts/` scripts inside a docker console by running:
+
+```
+> docker/console
+```
+
 ### Using virtualenv
 
-It's recommended to use [virtualenv](https://virtualenv.pypa.io/en/latest/index.html) to keep isolate the python environment used to develop stactools. See virtualenv documentation for more detailed information, but as a shortcut here's some quick steps:
+If not using docker, itt's recommended to use [virtualenv](https://virtualenv.pypa.io/en/latest/index.html) to keep isolate the python environment used to develop stactools. See virtualenv documentation for more detailed information, but as a shortcut here's some quick steps:
 
 - Make sure [virtualenv](https://virtualenv.pypa.io/en/latest/installation.html) is installed
 - Run `virtualenv venv`
 - Activate the virtualenv with `source venv/bin/active`
 
-### Installing development requirements
+#### Installing development requirements
 
 To install all the requirements for subpackages and the development requirements, use:
 
 ```
 > scripts/update
 ```
 
+Note that some packages might fail requirement installs because of required environment setup that cannot be controlled with pip-install. For instance, you may need to install rasterio with a GDAL that has non-standard formats enabled. You can avoid errors by installing the proper environment, by installing the failing requirements like rasterio manually, or by using the docker environment.
+
 ### Running the CLI against development code
 
-You can run the CLI through the source code by running
+You can run the CLI through docker by running
 
 ```
-> scripts/stac --help
+> docker/stac --help
 ```
 
-## Sub-packages
-
-`stactools` is comprised of subpackages that provide library and CLI functionality. Below is a list of available subpackages.
+or in the local environment with
 
-| subpackage                    | description                                                                     |
-| ------------------------------| ------------------------------------------------------------------------------- |
-| `stactools_core`              | Contains core library functionality that is used across the other projects      |
-| `stactools_cli`               | Contains the command line interface (cli) for running the `stactools` command   |
-| `stactools_planet`            | Contains methods and commands for working with planet data                      |
-| `stactools_landsat`           | Contains methods and commands for working with landsat data (TODO)              |
-| `stactools_browse`            | Contains a command for launching stac-browser against a local STAC              |
-
-Subpackages are symlinked to the `stactools` directory in this repo to allow them to be importable for python running at the top level directory of the repository clone.
+```
+> scripts/stac --help
+```
 
 ### Unit Tests
 
@@ -115,6 +134,12 @@ Unit tests are in the `tests` folder. To run unit tests, use `unittest`:
 
 To run linters, code formatters, and test suites all together, use `test`:
 
+```
+> ./docker/test
+```
+
+or
+
 ```
 > ./scripts/test
 ```
@@ -141,16 +166,36 @@ To check for spelling mistakes in modified files:
 > git diff --name-only | xargs codespell -I .codespellignore -f
 ```
 
-You can also run the `./scripts/test` script to check for linting, spelling, and run unit tests.
+You can also run the `./docker/test` or `./scripts/test` script to check for linting, spelling, and run unit tests.
 
 ### Documentation
 
-To build and develop the documentation locally, make sure sphinx is available (which is installed with `requirements-dev.txt`), and use the Makefile in the docs folder:
+To build and serve the docs, all of the requirements must be installed with `scripts/update`. Make sure [Pandocs](https://pandoc.org/installing.html) is installed. Also make sure sphinx is available, which should be installed with `requirements-dev.txt`. You can also run the following in the docker container using
+
+```
+> docker/console
+```
+
+To build the docs, you can use `make html`, and to build the docs and start a server that watches for changes, use `make livhtml`:
 
 ```
 > cd docs
 > make html
 > make livehtml
 ```
 
+If using `make livehtml`, once the server starts, navigate to [http://localhost:8000](http://localhost:8000/) to see the docs.
+
 Use 'make' without arguments to see a list of available commands.
+
+### Adding a new sub-package
+
+stactools is happy to take contributions of new subpackages for working with specific data types! Below is a list of steps to add a new subpackage:
+
+- Add the new subpackage as `stactools_{pkg}`, where `{pkg}` is a short name for the dataset the subpackage works with (e.g. "landsat")
+- Create a `setup.py`, `requirements.txt`, and `README.md` in the subpackage directory.
+- The code should exist in the `stactools/{pkg}/` directory in that package subdirectory. Note that the `stactools` does not have an __init__.py (look at the other subpackages for examples).
+- Add the subpackage to the appropriate variables in `scripts/env`
+- Add the subpackage to the appropriate tables in the README.
+- Add documentation for the subpackage.
+- Add subpackage to .readthedocs.yml install

docker/Dockerfile

@@ -1,8 +1,38 @@
-FROM python:3.8.6
+FROM ubuntu:18.04
 
-RUN apt-get update && apt-get install --no-install-recommends -y \
-    unzip && \
-    rm -rf /var/lib/apt/lists/*
+# Setup timezone info
+ENV TZ=UTC
+
+ENV LC_ALL=C.UTF-8
+ENV LANG=C.UTF-8
+
+RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
+
+RUN apt-get update && apt-get install -y software-properties-common
+
+RUN add-apt-repository ppa:ubuntugis/ppa && \
+    apt-get update && \
+    apt-get install -y wget=1.* git=1:2.* python-protobuf=3.* python3-tk=3.* \
+                       build-essential python3-dev python3-numpy \
+                       jq=1.5* unzip curl \
+                       libsqlite3-dev=3.22.* zlib1g-dev=1:1.2.* \
+                       libhdf4-dev && \
+    apt-get autoremove && apt-get autoclean && apt-get clean
+
+ENV CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
+
+# Install Python 3.8
+RUN wget -q -O ~/miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-4.7.12.1-Linux-x86_64.sh && \
+     chmod +x ~/miniconda.sh && \
+     ~/miniconda.sh -b -p /opt/conda && \
+     rm ~/miniconda.sh
+ENV PATH /opt/conda/bin:$PATH
+ENV LD_LIBRARY_PATH /opt/conda/lib/:$LD_LIBRARY_PATH
+RUN conda install -y python=3.8
+RUN python -m pip install --upgrade pip
+
+# Install GDAL
+RUN conda install -y -c conda-forge gdal=3.2.0
 
 # Core
 COPY stactools_core/requirements.txt /tmp/core/requirements.txt
@@ -12,6 +42,10 @@ RUN pip install -r /tmp/core/requirements.txt
 COPY stactools_cli/requirements.txt /tmp/cli/requirements.txt
 RUN pip install -r /tmp/cli/requirements.txt
 
+# Aster
+COPY stactools_aster/requirements.txt /tmp/aster/requirements.txt
+RUN pip install -r /tmp/aster/requirements.txt
+
 # Landsat
 COPY stactools_landsat/requirements.txt /tmp/landsat/requirements.txt
 RUN pip install -r /tmp/landsat/requirements.txt
@@ -29,3 +63,4 @@ ENV PYTHONPATH=/opt/src:$PYTHONPATH
 
 COPY stactools /opt/src/stactools
 
+ENTRYPOINT [ "python", "-m", "stactools.cli" ]

docker/Dockerfile-dev

@@ -0,0 +1,7 @@
+FROM stactools:latest
+
+# Install Pandoc
+RUN apt-get install -y pandoc
+
+COPY requirements-dev.txt /tmp/requirements-dev.txt
+RUN pip install -r /tmp/requirements-dev.txt

docker/build

@@ -17,6 +17,7 @@ Build stactools containers. Must be run from the repository root
 function build_containers() {
     echo "Building containers"
     docker build -t stactools -f docker/Dockerfile .
+    docker build -t stactools-dev -f docker/Dockerfile-dev .
 }
 
 if [ "${BASH_SOURCE[0]}" = "${0}" ]; then

docker/cibuild

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+set -e
+
+if [[ -n "${STACTOOLS_DEBUG}" ]]; then
+    set -x
+fi
+
+function usage() {
+    echo -n \
+        "Usage: $(basename "$0")
+Runs CI in the dev docker container.
+"
+}
+
+if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
+    # Build containers
+    ./docker/build
+
+    docker run --rm \
+        -v `pwd`:/opt/src \
+        --entrypoint /opt/src/scripts/cibuild \
+        stactools-dev:latest
+fi

docker/console

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+set -e
+
+if [[ -n "${STACTOOLS_DEBUG}" ]]; then
+    set -x
+fi
+
+function usage() {
+    echo -n \
+        "Usage: $(basename "$0")
+Run a console in a docker container with all prerequisites installed.
+"
+}
+
+if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
+    docker run --rm -it \
+        -v `pwd`:/opt/src \
+        -p 8000:8000 \
+        --entrypoint /bin/bash \
+        stactools-dev:latest
+fi

docker/docs-server

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+set -e
+
+if [[ -n "${STACTOOLS_DEBUG}" ]]; then
+    set -x
+fi
+
+function usage() {
+    echo -n \
+        "Usage: $(basename "$0")
+Build and serve documentation from a docker container with all prerequisites installed.
+"
+}
+
+if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
+    docker run --rm -it \
+        -p 8000:8000 \
+        -v `pwd`:/opt/src \
+        -w /opt/src/docs \
+        --entrypoint /bin/bash \
+        stactools-dev:latest \
+        -c "make livehtml"
+fi

docker/format

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+set -e
+
+if [[ -n "${STACTOOLS_DEBUG}" ]]; then
+    set -x
+fi
+
+function usage() {
+    echo -n \
+        "Usage: $(basename "$0")
+Run code formatters in a docker container with all prerequisites installed.
+"
+}
+
+if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
+    docker run --rm -it \
+        -v `pwd`:/opt/src \
+        --entrypoint /opt/src/scripts/format \
+        stactools-dev:latest
+fi

docker/stac

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+set -e
+
+if [[ -n "${STACTOOLS_DEBUG}" ]]; then
+    set -x
+fi
+
+function usage() {
+    echo -n \
+        "Usage: $(basename "$0")
+Run the stactools CLI in a docker container. Mounts the source
+directory inside the container, so any edits to the repository
+will be reflected in the execution.
+"
+}
+
+if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
+    docker run --rm -it \
+        -v `pwd`:/opt/src \
+        stactools:latest "${@}"
+fi