cads-broker

CADS broker service

Quick Start

>>> import cads_broker

Workflow for developers/contributors

For best experience create a new conda environment (e.g. DEVELOP) with Python 3.11:

conda create -n DEVELOP -c conda-forge python=3.11
conda activate DEVELOP

Before pushing to GitHub, run the following commands:

1. Update conda environment: make conda-env-update
2. Install this package: pip install -e .
3. Sync with the latest template (optional): make template-update
4. Run quality assurance checks: make qa
5. Run tests: make unit-tests
6. Run the static type checker: make type-check
7. Build the documentation (see Sphinx tutorial): make docs-build

Instructions for database updating

The package cads-broker comes with its 'broker' database. In case of database structure upgrade, developers must follow these steps:

1. Update the new database structure modifying /cads_broker/database.py, using SQLAlchemy ORM technologies
2. Execute from the cads-broker work folder:

   alembic revision -m "message about the db modification"
   

3. The last command will create a new python file inside /alembic/versions. Fill the upgrade function with the operations that must be executed to migrate the database from the old structure to the new one. Keep in mind both DDL (structure modification) and DML (data modification) instructions. For reference, use https://alembic.sqlalchemy.org/en/latest/ops.html#ops. Similarly, do the same with the downgrade function.
4. Commit and push the modifications and the new file.

For details about the alembic migration tool, see the Alembic tutorial.

Quality of Service rules examples

# User limits
# user "Limit for anonymous"       (user == "anonymous") : numberOfWorkers
# user "Default per-user limit"   (user ~ ".*")        : 8

# limits for finished requests in the last hours
# user "Limit for test user 1: 10 finished requests in the last 24 hours" (user == "00000000-0000-4000-a000-000000000000")   : 10 - userRequestCount(hour(24))
# user "Limit for users: 10 finished requests in the last 24 hours" (user ~ ".*")   : 10 - userRequestCount(hour(24))

# Limits
# limit "Limit for dummy-dataset"    (dataset == "test-dummy-adaptor")      : numberOfWorkers
# limit "Limit for cads_adaptors:DummyAdaptor" (adaptor == "cads_adaptors:DummyAdaptor")   : numberOfWorkers - 6

# Permissions
# permission "anonymous cannot access dummy-dataset"  (dataset == "test-dummy-adaptor"): user != 'anonymous'

# Priorities
priority "Priority for test user 1"    (user == "00000000-0000-4000-a000-000000000000")  :  hour(1)
priority "Priority for test user 2"    (user == "00000000-0000-3000-abcd-000000000001")  :  -hour(1)

# Functions examples

# Request contains any of the specified variable
# priority "Priority for temperature and humidity" (request_contains_any("variable", ["temperature", "relative_humidity"])): -hour(1)

# Request contains all the specified months
# limit "Limit for retrieve with all months" (request_contains_all("month", ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12"])): 2

# The adaptor is tagged with "block"
# permission "The adaptor is blocked." (tagged("block")): false

License

Copyright 2022, European Union.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.