diff --git a/.github/.cSpellWords.txt b/.github/.cSpellWords.txt index af0d4cf..3159b02 100644 --- a/.github/.cSpellWords.txt +++ b/.github/.cSpellWords.txt @@ -151,6 +151,8 @@ MQTT MQTT's mqttexample MVEI +mytestthing +myTestThing Mzrdfkvi nents NIOS @@ -196,6 +198,7 @@ SECP sntp Sntp SNTP +someTestThing speex Speex SPEEX @@ -208,6 +211,7 @@ SSSZ suppr SYSWDOG TALGORITHMS +testThing TGENERAL THEIGHT tinycbor diff --git a/docs/components/aws_iot/aws_tool.md b/docs/components/aws_iot/aws_tool.md index 39687c2..eca538e 100644 --- a/docs/components/aws_iot/aws_tool.md +++ b/docs/components/aws_iot/aws_tool.md @@ -39,7 +39,7 @@ export AWS_SECRET_ACCESS_KEY= export AWS_SESSION_TOKEN= export AWS_REGION=eu-west-1 ``` -The above keys and tokens can be found on the AWS online portal. They are also called AWS API keys. They need to be reset occasionally as they expire. See [Troubleshooting](#troubleshooting) for more information. +The above keys and tokens can be found on the [AWS access portal](https://docs.aws.amazon.com/signin/latest/userguide/sign-in-urls-defined.html#access-portal-url). They are also called AWS API keys. They need to be reset occasionally as they expire. See [Troubleshooting](#troubleshooting) for more information. In order to do an OTA update, you should have built an application that uses AWS, e.g. Keyword Detection. This is because the script assumes that `build/update-signature.txt` file exists when doing an OTA update. Alternatively, you can provide the directory holding signatures and credentials as a command line argument (see `--build_dir`). Script-generated credentials will be written here. @@ -56,15 +56,20 @@ See `--help` for each command. For example: python tools/scripts/createIoTThings.py --help python tools/scripts/createIoTThings.py create-thing-and-policy --help ``` -For examples of how to use each command, see [Creating AWS IoT Firmware update](#creating-aws-iot-firmware-update-job-using-the-automated-script). +For examples of how to use each command, see [Creating AWS IoT Firmware update](#creating-aws-iot-firmware-update-jobs-using-the-automated-script). ### Create an AWS IOT setup manually See the `docs/applications/aws_iot/setting_up_aws_connectivity.md` file for instructions on setting up IoT manually. -## Creating AWS IoT firmware update job using the automated script +#### Limitations -Please refer to [AWS IOT basic concepts](#aws-iot-basic-concepts) and [Using the commands](#using-the-commands) for the general instructions and limitation about using the script. +These limitations apply for [creating AWS IoT firmware update jobs using the automated script](#creating-aws-iot-firmware-update-jobs-using-the-automated-script). +If you are re-using a role when making an OTA update, you must be able to assume the role. Most of the time, this means the credentials (e.g. `AWS_ACCESS_KEY_ID`) you are using must be associated with the account that created the role. The script will throw an error if this is not the case. + +## Creating AWS IoT firmware update jobs using the automated script + +Please refer to [AWS IOT basic concepts](#aws-iot-basic-concepts) and [Using the commands](#using-the-commands) for the general instructions and limitations about using the script. Performing an OTA update will require you to: * Create a role with the required permissions @@ -77,7 +82,7 @@ Create a thing, an IOT policy, and attach the two together with: ```sh python tools/scripts/createIoTThings.py create-thing-and-policy --thing_name --policy_name --target_application ``` -Where `` is one of `keyword_detection`, `object_detection`, `speech_recognition`. +Where `` is one of `keyword-detection`, `object-detection`, `speech-recognition`. You must specify the `--target_application` argument if creating a Thing. The script will update your `aws_clientcredential.h` config file automatically. If you have already modified the entries `clientcredentialMQTT_BROKER_ENDPOINT` or `clientcredentialIOT_THING_NAME` in the file, the script will warn you and ask before overwriting. > Note: You may also create each things and policies individually, but you'll have to make sure to pass the certificate created by the first command to the second. Certificates will be printed upon creation during the first command. Use `--use_existing_certificate_arn ` on the second command. @@ -93,7 +98,7 @@ You may now rebuild keyword with those certificates: ``` Next, we'll create the bucket, upload the binary there, create a role capable of running an OTA update, and create the update. All of those with the following command: ```sh -python tools/scripts/createIoTThings.py create-bucket-role-update --thing_name --bucket_name --iam_role_name - --update_name --ota_binary keyword-detection-update_signed.bin --permissions_boundary arn:aws:iam::: +python tools/scripts/createIoTThings.py create-bucket-role-update --thing_name --bucket_name --iam_role_name - --update_name --ota_binary keyword-detection-update_signed.bin --permissions_boundary arn:aws:iam::: ``` The above assumes you have `keyword-detection-update_signed.bin` saved in the directory specified by the `--ota_binary_build_dir` argument. This argument defaults to the project's root directory. If you want to run an OTA update for another example application, use another signed binary name. @@ -121,6 +126,124 @@ python tools/scripts/createIoTThings.py delete-thing -p --thing_name Note: Your role is what allow you to interact with the OTA update. It is important you don't delete it before successfully deleting the OTA update or you will lose the permission to delete the update. If such a thing happen, you'll have to recreate the role and manually delete the update. +## Creating AWS IoT firmware update job (simplified) + +The `create-update-simplified` command that (1) creates a Thing and Policy, (2) runs build, (3) creates a bucket, role, and update. +This command also re-uses AWS entities where possible, validating entities being re-used. + + +To use this command: +1. Fill the following fields in the `.json` config file: + * `thing_name` with the name of your AWS Thing. + * `permissions_boundary` with the name of your Role's permission boundary - if applicable. + For more information, see `python tools/scripts/createIoTThings.py create-iam-role-only --help` or [AWS IOT basic concepts](#aws-iot-basic-concepts). + Creating a role without a permissions boundary is not supported by this script. + * `role_prefix` with the prefix for your role. This prefix will be pre-pended to your role name with a hyphen by default. For example, with the prefix `Proj` and role name `role`, the completed role name will become `Proj-role`. + * `target_application` with one of `keyword-detection`, `object-detection`, or `speech-recognition`. Alternatively, you can override this with a command-line argument (see `--help`). +2. Set up following [prerequisites](#prerequisites). +3. Run the command below. + +```sh +python tools/scripts/createIoTThings.py create-update-simplified +``` +You may optionally specify a different `.json` config file. See `--help`. + +That's it. You can now run `run.sh` and wait for OTA update to complete. + +If you have changed an example application and want to re-create a new OTA update, simply repeat step 3. +To create a new OTA update but keep the old one, modify the `update_name` field in the `.json` file. + + +### Overview of the .json config file + +```json +{ + # There are only 3 required fields. These need to be filled in step (1) of using this command. + "thing_name":"", + "permissions_boundary":"arn:aws:iam:::policy/", + "role_prefix":"", + "target_application":"", + + # Everything past this point need not be filled in + + # USED TO REPLACE e.g. '${thing_name}' with the field's value + "format_vars":"thing_name;role_prefix;target_application;credentials_path", + + "policy_name":"", + "bucket_name":"${thing_name}{lower}-bucket", + "iam_role_name":"", + "update_name":"", + "existing_certificate_arn":"CREATE_NEW_CERTIFICATE", + + # Where to find certificates and binaries + "credentials_path":"certificates", + "build_dir":"build", + "ota_binary":"${target_application}-update_signed.bin", + + # Build.sh script inputs are below + "skip_build":"", + "build_script_path":"", + "private_key_path":"", + "certificate_path":"", + "target_platform":"", + "inference":"", + "audio":"", + "toolchain":"", + "clean_build":"", + + # Default values are below. + "policy_name_DEFAULT":"${thing_name}_policy", + "bucket_name_DEFAULT":"${thing_name}_bucket", + "iam_role_name_DEFAULT":"${role_prefix}-${thing_name}_role", + "update_name_DEFAULT":"${thing_name}_update", + "existing_certificate_arn_DEFAULT":"", + "credentials_path_DEFAULT":"", + "build_dir_DEFAULT":"", + "ota_binary_DEFAULT":"", + "skip_build_DEFAULT":"false", + "build_script_path_DEFAULT":"./tools/scripts/build.sh", + "private_key_path_DEFAULT":"${credentials_path}/thing_private_key_${thing_name}.pem.key", + "certificate_path_DEFAULT":"${credentials_path}/thing_certificate_${thing_name}.pem.crt", + "target_platform_DEFAULT":"corstone315", + "inference_DEFAULT":"SOFTWARE", + "audio_DEFAULT":"ROM", + "toolchain_DEFAULT":"GNU", + "clean_build_DEFAULT":"auto" +} +``` +Note the \# comments are not valid JSON syntax and are purely included for this documentation. + +Most settings in the config file are identical to those passed by the command-line to either `createIoTThings.py` or `build.sh`. +If a setting is left empty (`""`), then the script will look at the definition in `{FIELD_NAME}_DEFAULT`. E.g. if `policy_name:""`, then the script will use `"${thing_name}_policy"` as the policy name. + +Some of the less obvious settings include: +- `format_vars` lists settings and variables that are used in the definitions of other settings. For more information, see [how the formatter works](#how-the-formatter-works-for-the-config-file). +- `skip_build`: if `true`, will cause `build.sh` to not run. This takes precedence over `clean_build`. +- `clean_build`: if `auto`, will run the `build.sh` script for a clean build (with the `-c` flag) only when necessary. I.e. if `aws_clientcredential` is updated by the script. Otherwise, the script runs `build.sh` not from clean. If `true`, always run `build.sh` for a clean build. +- `existing_certificate_arn` should be set to either a valid ARN for a certificate, or if you want the script to generate certificates for you, should be set to `CREATE_NEW_CERTIFICATE`. +- `target_application` can be specified in the `.json` file, and if not otherwise specified on the command line this value will be taken as a default by the script. + +Changing the `_DEFAULT` setting values is not recommended. Try to change the user settings instead of the default settings. +Other commands in the Python file (such as `create-policy-only`) will not adhere to changes made to this settings file. + +Another useful tip is that the script will not empty any buckets. It is the responsibility of the user to periodically empty a bucket being re-used, if the user wants to avoid storing too much data in said bucket. + +#### How the formatter works for the config file + +The script defines some default values in terms of user-specified variables. For example, `policy_name_DEFAULT` is specified as `${thing_name}_policy`. To do this, some formatting is supported. +For these examples, assume that `"thing_name":"myTestThing"`. +Because `thing_name` is in `format_vars`, the following formatting happens (in order of priority): + +1. `${thing_name}{lower}` maps to the value of `mytestthing`. +2. `${thing_name}{replace my with some}` maps to `someTestThing`. I.e. the value of `testThing` where occurrences of `my` are replaced by `some`. This is case-sensitive. +3. `${thing_name}` maps to `myTestThing`. + +These are the only 3 types of formatting supported. Even `${thing_name}{lower}{replace A with B}` and similar chaining is not supported. Additionally, trying to use `}` or `{` in setting definitions may break the formatter. + +If you want to add a setting, for example `update_name` to the definitions you can use in formatting, then append `;update_name` to the end of `format_vars`. This makes it possible to use `${update_name}` in the definitions of other settings. + +The `target_application` setting is special because it is not defined in the `json` file but can still be mentioned in definitions. + ## Troubleshooting ##### 1. My AWS credentials are rejected, despite being accepted earlier. diff --git a/release_changes/202408300915.change b/release_changes/202408300915.change new file mode 100644 index 0000000..c19b086 --- /dev/null +++ b/release_changes/202408300915.change @@ -0,0 +1 @@ +tools: Add command to createIoTThings.py for creating OTA updates. diff --git a/tools/scripts/createIoTThings.py b/tools/scripts/createIoTThings.py index 72b937f..1bffb25 100644 --- a/tools/scripts/createIoTThings.py +++ b/tools/scripts/createIoTThings.py @@ -18,6 +18,7 @@ from enum import Enum import os +import pathlib import time import traceback as tb import re @@ -34,7 +35,13 @@ import click import logging -DEFAULT_LOG_LEVEL = "warning" +# used to build the application +import subprocess + +# used to display progress for building application +import threading + +DEFAULT_LOG_LEVEL = "info" # Default path of OTA binary directory is build/ DEFAULT_BUILD_DIR = "build" DEFAULT_CREDENTIALS_PATH = "certificates" @@ -45,7 +52,7 @@ # try to build one of the FRI applications (e.g. keyword-detection) DEFAULT_OTA_UPDATE_SIGNATURE_FILENAME = "update-signature.txt" -CREATE_NEW_CERTIFICATE = "" +CREATE_NEW_CERTIFICATE = "CREATE_NEW_CERTIFICATE" OTA_JOB_NAME_PREFIX = "AFR_OTA-" @@ -63,6 +70,7 @@ s3 = boto3.client("s3", AWS_REGION) iam = boto3.client("iam", AWS_REGION) sts_client = boto3.client("sts", AWS_REGION) +org_client = boto3.client("organizations") validCredentials = False try: @@ -83,7 +91,15 @@ def cli(): def read_whole_file(path, mode="r"): - with open(path, mode) as fp: + """ + For example, to read 'createIoTThings.py', call: + >>> read_whole_file("tools/scripts/createIoThings.py") + + Parameters: + path (str): the path to the file to open, relative to the ROOT of this directory. + """ + absolutePath = pathlib.Path(path).resolve() + with open(absolutePath, mode) as fp: return fp.read() @@ -97,7 +113,7 @@ class ApplicationType(Enum): def app_type_from_string(s): for app in AWS_APPLICATIONS: - if s == app.value: + if s == app.value.replace("_", "-"): return app return ApplicationType.UNDEFINED @@ -164,6 +180,9 @@ def __init__( self.updateHasBeenCreated = False self.policyHasBeenCreated = False self.thingHasBeenCreated = False + # Used for the create_update_simplified. + # If true, build.sh runs with `-c` flag. + self.rebuildRequired = False # JSONS self.POLICY_DOCUMENT = { "Version": "2012-10-17", @@ -222,13 +241,13 @@ def __init__( } ], } - signaturePath = os.path.join( + self.signaturePath = os.path.join( self.BUILD_DIR, ota_update_signature_filename, ) signature = "unavailable" try: - signature = read_whole_file(signaturePath) + signature = read_whole_file(self.signaturePath) except FileNotFoundError: pass # this check is done at the CLI stage. self.OTA_UPDATE_FILES = [ @@ -256,6 +275,28 @@ def __init__( } ] + def reload_signature(self): + """ + Parameters: + self (Flags): the Flags object to reload the signature of. + + This function reloads the update signature from the file + at self.signaturePath. + + Returns: Nothing + """ + signature = "unavailable" + try: + signature = read_whole_file(self.signaturePath) + except FileNotFoundError: + pass + self.OTA_UPDATE_FILES[0]["codeSigning"]["customCodeSigning"]["signature"][ + "inlineDocument" + ] = bytearray( + signature.strip(), + "utf-8", + ) + def set_log_level(loglevel): logging.basicConfig(level=loglevel.upper()) @@ -428,83 +469,53 @@ def replace_between(start, end, replaceWith, target): return re.sub(start + "(.*?)" + end, start + replaceWith + end, target, re.DOTALL) -def _write_credentials(flags: Flags, credentials_path): +def aws_clientcredential_needs_to_be_updated(flags): """ - This function writes certificates ('credentials') for any Things - created to credential files in the designated directory. - Will create the 'credentials_path' directory if it does not exist. - Credentials are found from the 'flags' parameter. - - This function assumes that the application type is not undefined. + This function identifies whether 'aws_clientcredential.h' + contains the Thing specified in 'flags'. + Additionally, whether 'aws_clientcredential.h' contains + the AWS endpoint. Parameters: flags (Flags): contains metadata needed for AWS and OTA updates (e.g. credentials). - credentials_path (string): directory to write new files to. """ fileDir = os.path.dirname(os.path.realpath("__file__")) - # Create folder storing credentials if it does not exist. - try: - logging.info( - "Creating/opening directory '" - + credentials_path - + "' to store credentials generated for Thing : '" - + flags.THING_NAME - + "'" - ) - os.makedirs(os.path.join(fileDir, credentials_path), exist_ok=True) - except Exception: - err_msg = ( - "Failed to write credentials, could not make path: '" - + credentials_path - + "'. This may be due to an invalid path being provided. " - ) - logging.error(err_msg) - # Errors can occur here, e.g. due to invalid path names. - # This script will not sanitise path inputs, so checking them is - # the responsibility of the user. - """ - Assuming the name of the new Thing is . - The below section of code saves: - 1. the certificate of the new Thing to - 'thing_certificate_.pem.crt" - 2. the private key of the new Thing to - 'thing_private_key_.pem.key" - 3. the public key of the new Thing to - 'thing_public_key_.pem.key" - """ - # This script does not sanitise THING_NAME for file name conventions. - certificateFile = os.path.join( - fileDir, credentials_path, "thing_certificate_" + flags.THING_NAME + ".pem.crt" - ) - privateKeyFile = os.path.join( - fileDir, credentials_path, "thing_private_key_" + flags.THING_NAME + ".pem.key" + credentialFileTemplate = os.path.join( + fileDir, + "applications/", + flags.targetApplication.value, + "configs/aws_configs/", + "aws_clientcredential.h", ) - publicKeyFile = os.path.join( - fileDir, credentials_path, "thing_public_key_" + flags.THING_NAME + ".pem.key" + contents = "" + try: + with open(credentialFileTemplate, "r") as file: + contents = file.read() + except FileNotFoundError: + logging.warning("The file '" + credentialFileTemplate + "' was not found") + return True + return not ( + flags.THING_NAME in contents + and flags.endPointAddress["endpointAddress"] in contents ) - with open(certificateFile, "w") as file: - file.write(key_to_pem_formatter(flags.certificate)) - logging.info("Saved Thing certificate to: " + certificateFile) - with open(privateKeyFile, "w") as file: - file.write(key_to_pem_formatter(flags.privateKey)) - logging.info("Saved Thing private key to: " + privateKeyFile) - with open(publicKeyFile, "w") as file: - file.write(key_to_pem_formatter(flags.publicKey)) - logging.info("Saved Thing public key to: " + publicKeyFile) + + +def _write_aws_clientcredential_h(flags): """ The below writes to `aws_clientcredential.h`. This can be specified to overwrite `aws_clientcredential.h` in an example application directory. + + Parameters: + flags (Flags): contains metadata needed for AWS and OTA updates + (e.g. credentials). """ + fileDir = os.path.dirname(os.path.realpath("__file__")) template_has_correct_format = ( lambda contents: "clientcredentialMQTT_BROKER_ENDPOINT " in contents and "clientcredentialIOT_THING_NAME " in contents ) - template_has_been_edited = ( - lambda contents: "dummy.endpointid.amazonaws.com" not in contents - or "dummy_thingname" not in contents - ) # Try to find a template for 'aws_clientcredential.h' template = "" # We assume flags.targetApplication is not ApplicationType.UNDEFINED @@ -534,7 +545,22 @@ def _write_credentials(flags: Flags, credentials_path): ) logging.warning(err_msg) return False - if template_has_been_edited(template): + # if template has been edited, ask before overwriting. + DEFAULT_ENDPOINT = ( + "#define clientcredentialMQTT_BROKER_ENDPOINT " + + 'dummy.endpointid.amazonaws.com"' + ) + DEFAULT_THING_NAME = ( + "#define clientcredentialIOT_THING_NAME " '"dummy_thingname"' + ) + # Remove spaces for robustness. + template_without_spaces = template.replace(" ", "") + if ( + DEFAULT_ENDPOINT.replace(" ", "") in template_without_spaces + and DEFAULT_THING_NAME.replace(" ", "") in template_without_spaces + ): + logging.debug("aws_clientcredential.h has not been edited. Overwriting.") + else: warn_msg = ( "Your aws_clientcredential.h file at " + credentialFileTemplate @@ -580,9 +606,82 @@ def _write_credentials(flags: Flags, credentials_path): with open(credentialFile, "w") as file: file.write(template) logging.info("Wrote AWS client credentials to: " + credentialFile) + flags.rebuildRequired = True return True +def _write_credentials(flags: Flags, credentials_path): + """ + This function writes certificates ('credentials') for any Things + created to credential files in the designated directory. + Will create the 'credentials_path' directory if it does not exist. + Credentials are found from the 'flags' parameter. + + This function assumes that the application type is not undefined. + + Parameters: + flags (Flags): contains metadata needed for AWS and OTA updates + (e.g. credentials). + credentials_path (string): directory to write new files to. + """ + fileDir = os.path.dirname(os.path.realpath("__file__")) + # Create folder storing credentials if it does not exist. + try: + logging.info( + "Creating/opening directory '" + + credentials_path + + "' to store credentials generated for Thing : '" + + flags.THING_NAME + + "'" + ) + os.makedirs(os.path.join(fileDir, credentials_path), exist_ok=True) + except Exception: + err_msg = ( + "Failed to write credentials, could not make path: '" + + credentials_path + + "'. This may be due to an invalid path being provided. " + ) + logging.error(err_msg) + # Errors can occur here, e.g. due to invalid path names. + # This script will not sanitise path inputs, so checking them is + # the responsibility of the user. + """ + Assuming the name of the new Thing is . + The below section of code saves: + 1. the certificate of the new Thing to + 'thing_certificate_.pem.crt" + 2. the private key of the new Thing to + 'thing_private_key_.pem.key" + 3. the public key of the new Thing to + 'thing_public_key_.pem.key" + """ + # This script does not sanitise THING_NAME for file name conventions. + certificateFile = os.path.join( + fileDir, credentials_path, "thing_certificate_" + flags.THING_NAME + ".pem.crt" + ) + privateKeyFile = os.path.join( + fileDir, credentials_path, "thing_private_key_" + flags.THING_NAME + ".pem.key" + ) + publicKeyFile = os.path.join( + fileDir, credentials_path, "thing_public_key_" + flags.THING_NAME + ".pem.key" + ) + with open(certificateFile, "w") as file: + file.write(key_to_pem_formatter(flags.certificate)) + logging.info("Saved Thing certificate to: " + certificateFile) + with open(privateKeyFile, "w") as file: + file.write(key_to_pem_formatter(flags.privateKey)) + logging.info("Saved Thing private key to: " + privateKeyFile) + with open(publicKeyFile, "w") as file: + file.write(key_to_pem_formatter(flags.publicKey)) + logging.info("Saved Thing public key to: " + publicKeyFile) + """ + The below writes to `aws_clientcredential.h`. + This can be specified to overwrite `aws_clientcredential.h` + in an example application directory. + """ + return _write_aws_clientcredential_h(flags) + + def _parse_credentials(flags: Flags): """ Modifies the keys and certificates in 'flags' to ensure correct formatting. @@ -654,7 +753,7 @@ def _create_credentials(flags: Flags, credentials_path): def _get_credential_arn(flags: Flags, existing_certificate_arn, credentials_path): """ Will create credentials (and save into 'credentials_path') if needed. - Otherwise, if use_existing_credentials_arn is not the empty string (""), + Otherwise, if existing_certificate_arn is not the empty string (""), will use existing credentials. Parameters: @@ -764,7 +863,8 @@ def _create_thing(flags: Flags, Name, certificate_arn): help="Updates the target application's aws_clientcredential.h automatically. \ Accepted values: " + reduce( - lambda y, z: y + ", " + z, map(lambda x: "'" + x.value + "'", AWS_APPLICATIONS) + lambda y, z: y + ", " + z, + map(lambda x: "'" + x.value.replace("_", "-") + "'", AWS_APPLICATIONS), ) + ".", ) @@ -1036,7 +1136,7 @@ def _create_iam_role(flags: Flags, Name, permissions_boundary=None): logging.error("The role name is undefined.") return False if permissions_boundary is None: - logging.warn( + logging.warning( "About to create role " + flags.OTA_ROLE_NAME + " without a permissions boundary." @@ -1251,6 +1351,8 @@ def _wait_for_job_deleted(job_name, timeout=20): step = 2 for i in range(0, timeout, step): if not _does_job_exist(job_name): + if i > 0: + logging.info("Job deletion complete!") return True res = iot.describe_job(jobId=job_name) job_status = res["job"]["status"] @@ -1665,6 +1767,548 @@ def create_bucket_role_update( ctx.exit(0) +def _tryGetSetting( + setting: str, + settings: dict[str, str], + ctx=None, + errorOnFailure=True, +): + """ + This function tries to find '{setting}' or '{setting}_DEFAULT' in + 'settings', where '{setting}' is the string value of setting. + If not found, the function will either error or return None, + depending on the flag 'errorOnFailure' and whether 'ctx' is set. + + Parameters: + setting (str): the field to find in 'settings'. + settings (dict[str, str]): a mapping from settings to their values. + ctx: the click CLI context. Used to send exit codes. + errorOnFailure: if True and a setting is not found, will cause the + program to exit. + Otherwise, the programm will return None if a setting is not + found. + + Returns: + str: the setting's value if the setting or default is non-empty and + found in 'settings'. + None: if 'errorOnFailure' is True, and 'ctx' is defined, and the setting is + not found, and a default value for the setting is not found. + """ + default_setting = setting + "_DEFAULT" + if setting in settings and settings[setting] != "": + return settings[setting] + elif default_setting in settings and settings[default_setting] != "": + return settings[default_setting] + elif errorOnFailure and ctx is not None: + logging.error( + "Cannot find setting " + + setting + + " in settings file given, and this setting has" + + " no default value. " + ) + ctx.exit(1) + else: + return None + + +def _formatVars( + settings: dict[str, str], + ctx, +): + # Replace all occurrences of "${thing_name}" with the value settings["thing_name"] + if "format_vars" in settings: + variablesToReplace = settings["format_vars"].split(";") + for setting in settings: + if setting != "format_vars": + for var in variablesToReplace: + variableValue = _tryGetSetting( + var, settings=settings, ctx=ctx, errorOnFailure=True + ) + settings[setting] = settings[setting].replace( + "${" + var + "}{lower}", variableValue.lower() + ) + while "${" + var + "}{replace{" in settings[setting]: + # settings[setting] = "${target_application}{replace{_ with -}}" + # .split("${" + var + "}{replace{")[1] = "_ with -}}" + # .split("}")[0] = _ with - + # args = ["_", "-"] + args = ( + settings[setting] + .split("${" + var + "}{replace{")[1] + .split("}")[0] + .split(" with ") + ) + variableValueFormatted = variableValue.replace(args[0], args[1]) + toReplace = ( + "${" + + var + + "}{replace{" + + args[0] + + " with " + + args[1] + + "}}" + ) + logging.debug( + "Replacing " + toReplace + " with " + variableValueFormatted + ) + settings[setting] = settings[setting].replace( + toReplace, variableValueFormatted + ) + settings[setting] = settings[setting].replace( + "${" + var + "}", variableValue + ) + return settings + + +def _try_delete(target, del_func, **kwargs): + has_deleted = False + try: + has_deleted = del_func(target, kwargs) + except Exception as ex: + print_exception(ex) + if has_deleted: + logging.info("Deleted: " + target) + else: + logging.error("Failed to delete: " + target) + + +def _counter(stop_event: threading.Event): + while not stop_event.is_set(): + print(".", end="", flush=True) + time.sleep(5) + + +# Defines Command-line interface for creating update using config file +@cli.command(cls=StdCommand) +@click.option( + "--target_application", + required=False, + help="Updates the target application's aws_clientcredential.h automatically. \ + Accepted values: " + + reduce( + lambda y, z: y + ", " + z, + map(lambda x: "'" + x.value.replace("_", "-") + "'", AWS_APPLICATIONS), + ) + + ". Providing this argument will take priority over the value specified" + + " in the .json config file.", +) +@click.option( + "--config_file_path", + help="Path to the .json file defining arguments for creating an OTA" + + "update. Relative to the root of this Project.", + default="tools/scripts/createIoTThings_settings.json", +) +@click.pass_context +def create_update_simplified( + ctx, + target_application, + config_file_path, + log_level, +): + """ + This command parses and formats 'config_file_path', and uses the arguments + from there to create an OTA update from scratch. + This command attempts to re-use AWS entities (e.g. Things, Policies, ...) + where possible. + This command also runs the build script for the target application + provided (unless otherwise specified in the config file). + See `aws_tool.md` for more information on usage and the config file. + + Parameters: + ctx: the click CLI context. Used to send exit codes. + target_application: one of 'keyword-detection', 'speech-recognition', + or 'object-detection'. + config_file_path: the path to the config file containing settings, + such as the Thing name. + This file must be a .json file. + log_level: unused parameter needed for compatibility with StdCommand. + """ + settings = {} + target = ApplicationType.app_type_from_string(target_application) + # Read .json file, pass parameters to flags. + try: + contents = read_whole_file(config_file_path) + settings = json.loads(contents) + if target == ApplicationType.UNDEFINED: + if target_application is not None: + logging.warning( + "Invalid target application '" + + str(target_application) + + "' provided by command line. Re-trying with .json setting." + ) + target_cfg = _tryGetSetting( + "target_application", settings=settings, ctx=ctx, errorOnFailure=False + ) + if target_cfg is None: + logging.error( + f"Target application not provided as a command line argument," + f" nor in '{config_file_path}'." + ) + ctx.exit(1) + target = ApplicationType.app_type_from_string(target_cfg) + if target == ApplicationType.UNDEFINED: + logging.error( + f"Invalid application type provided by '{config_file_path}'." + " Value is: " + + str(target_cfg) + + ". See --help for valid alternatives." + ) + ctx.exit(1) + settings["target_application"] = target.value.replace("_", "-") + settings = _formatVars(settings, ctx) + logging.debug("Settings .json file parsed to: " + str(settings)) + except FileNotFoundError: + logging.error("Config file not found at " + config_file_path) + ctx.exit(1) + except json.JSONDecodeError: + logging.error("Failed to parse .json file: " + config_file_path) + ctx.exit(1) + + # Check the required settings exist. + thing_name = _tryGetSetting( + "thing_name", settings=settings, ctx=ctx, errorOnFailure=True + ) + policy_name = _tryGetSetting( + "policy_name", settings=settings, ctx=ctx, errorOnFailure=True + ) + bucket_name = _tryGetSetting( + "bucket_name", settings=settings, ctx=ctx, errorOnFailure=True + ) + iam_role_name = _tryGetSetting( + "iam_role_name", settings=settings, ctx=ctx, errorOnFailure=True + ) + build_dir = _tryGetSetting( + "build_dir", settings=settings, ctx=ctx, errorOnFailure=True + ) + ota_binary = _tryGetSetting( + "ota_binary", settings=settings, ctx=ctx, errorOnFailure=True + ) + existing_certificate_arn = _tryGetSetting( + "existing_certificate_arn", settings=settings, ctx=ctx, errorOnFailure=True + ) + credentials_path = _tryGetSetting( + "credentials_path", settings=settings, ctx=ctx, errorOnFailure=True + ) + permissions_boundary = _tryGetSetting( + "permissions_boundary", settings=settings, ctx=ctx, errorOnFailure=True + ) + update_name = _tryGetSetting( + "update_name", settings=settings, ctx=ctx, errorOnFailure=True + ) + # Build script settings + build_script_path = _tryGetSetting( + "build_script_path", settings=settings, ctx=ctx, errorOnFailure=True + ) + private_key_path = _tryGetSetting( + "private_key_path", settings=settings, ctx=ctx, errorOnFailure=True + ) + certificate_path = _tryGetSetting( + "certificate_path", settings=settings, ctx=ctx, errorOnFailure=True + ) + target_platform = _tryGetSetting( + "target_platform", settings=settings, ctx=ctx, errorOnFailure=True + ) + inference = _tryGetSetting( + "inference", settings=settings, ctx=ctx, errorOnFailure=True + ) + audio = _tryGetSetting("audio", settings=settings, ctx=ctx, errorOnFailure=True) + toolchain = _tryGetSetting( + "toolchain", settings=settings, ctx=ctx, errorOnFailure=True + ) + clean_build = _tryGetSetting( + "clean_build", settings=settings, ctx=ctx, errorOnFailure=False + ) + skip_build = _tryGetSetting( + "skip_build", settings=settings, ctx=ctx, errorOnFailure=True + ) + + ctx.flags = Flags( + bucket_name=bucket_name, + role_name=iam_role_name, + target_application=target, + build_dir=build_dir, + ota_binary=ota_binary, + ota_update_signature_filename=DEFAULT_OTA_UPDATE_SIGNATURE_FILENAME, + ) + ctx.flags.THING_NAME = thing_name + ctx.flags.endPointAddress = iot.describe_endpoint(endpointType="iot:Data-ATS") + + signaturePath = os.path.join( + ctx.flags.BUILD_DIR, DEFAULT_OTA_UPDATE_SIGNATURE_FILENAME + ) + + # All parameters are now validated. + certificate_arn = "" + + if not _does_thing_exist(thing_name): + certificate_arn = _get_credential_arn( + ctx.flags, existing_certificate_arn, credentials_path + ) + if not _create_thing(ctx.flags, thing_name, certificate_arn): + logging.error("Thing '" + thing_name + "' creation failed.") + cleanup_aws_resources(ctx.flags) + ctx.exit(1) + else: + logging.info("Re-using existing Thing: " + thing_name) + + logging.warning( + "When using an existing Thing, it is the responsibility of the User" + + " to ensure certificates are where specified. " + + "\n Expecting private key path: '" + + private_key_path + + "'\n Expecting certificate path: '" + + certificate_path + + "'" + ) + + if not _does_policy_exist(policy_name): + if not _create_policy(ctx.flags, policy_name, certificate_arn): + logging.error("Policy '" + policy_name + "' creation failed.") + cleanup_aws_resources(ctx.flags) + ctx.exit(1) + else: + logging.info("Re-using existing Policy: " + policy_name) + + # Check the certificates match. + thing_certificates = iot.list_thing_principals(thingName=thing_name)["principals"] + policy_certificates = iot.list_targets_for_policy(policyName=policy_name)["targets"] + logging.debug("Thing certificates: " + str(thing_certificates)) + logging.debug("Policy certificates: " + str(policy_certificates)) + matching = list(set(thing_certificates).intersection(policy_certificates)) + logging.debug("Shared certificates: " + str(matching)) + if len(matching) != 0: + logging.info("Thing and Policy have a shared Certificate. ") + certificate_arn = matching[0] + else: + logging.info( + "the Thing and Policy provided by the .json config " + "do not have the same certificate." + "This is probably because at least one of the Thing " + "or Policy existed before running this command." + ) + if input("Delete the Thing and Policy? (Y/N): ").lower() == "y": + # Delete the Thing and Policy. + _try_delete( + thing_name, + _delete_thing, + ) + _try_delete(policy_name, _delete_policy) + else: + logging.warning( + "Using a Thing and Policy with different certificates may cause errors." + ) + cleanup_aws_resources(ctx.flags) + ctx.exit(1) + + if aws_clientcredential_needs_to_be_updated(ctx.flags): + if _write_aws_clientcredential_h(ctx.flags): + logging.info( + "'aws_clientcredential.h' was updated in the Application. " + "Build.sh will need to run from clean. " + ) + else: + logging.warn( + "'aws_clientcredential.h' failed to updated in " "the Application." + ) + else: + logging.info( + "'aws_clientcredential.h' is already up-to-date in the " "Application." + ) + + # Re-build the application. + if skip_build != "true": + cmnd = [ + build_script_path, + target.value.replace("_", "-"), + "--certificate_path", + certificate_path, + "--private_key_path", + private_key_path, + "--target", + target_platform, + "--inference", + inference, + "--audio", + audio, + "--toolchain", + toolchain, + ] + if clean_build == "true" or ( + clean_build == "auto" and ctx.flags.rebuildRequired + ): + cmnd.append("-c") + logging.info("Executing build command: \n" + " ".join(cmnd)) + logging.info("Grab a coffee - this may take a minute or so... ") + # Call and wait to finish + stop_event = threading.Event() + progressThread = threading.Thread(target=_counter, args=(stop_event,)) + progressThread.start() + result = subprocess.run(cmnd, capture_output=True, text=True) + stop_event.set() + progressThread.join() + if result.returncode != 0: + logging.error( + "Build script failed with error code: " + str(result.returncode) + ) + logging.error("Errors produced by build script: \n" + result.stderr) + cleanup_aws_resources(ctx.flags) + ctx.exit(1) + else: + logging.info("Build successful.") + # Signature stored in ctx.flags is now invalid - reload it. + ctx.flags.reload_signature() + else: + logging.info( + "Skipping re-build of application. To change this, see " + f"'skip_build' in {config_file_path}." + ) + + try: + open(signaturePath, "r") + except FileNotFoundError: + logging.error( + "The OTA update signature was not found at '" + + signaturePath + + "'. It is not possible to upload OTA updates." + ) + cleanup_aws_resources(ctx.flags) + ctx.exit(1) + + # create or re-use bucket. + if not _does_bucket_exist(bucket_name): + # create new bucket. + if not _create_aws_bucket(ctx.flags, bucket_name): + logging.error("Bucket '" + bucket_name + "' creation failed.") + cleanup_aws_resources(ctx.flags) + ctx.exit(1) + else: + logging.info( + "A bucket already exists with the name '" + bucket_name + "'. Re-using." + ) + if _does_role_exist(iam_role_name=iam_role_name): + can_access_bucket = False + try: + iam_role_arn = iam.get_role(RoleName=iam_role_name)["Role"]["Arn"] + # assume user-specified IAM role + response = None + try: + response = sts_client.assume_role( + RoleArn=iam_role_arn, RoleSessionName="test_bucket_access" + ) + except botocore.exceptions.ClientError: + # Try to modify permissions of role so it is possible to assume. + logging.error( + "You do not have permission to assume the role you " + f"have provided ({iam_role_name}). You need to add " + "the following under the role's " + "'Trusted Entities/Statements' to use the role " + "with this script." + ) + role = iam.get_role(RoleName=iam_role_name)["Role"] + AWS_ACCOUNT_ID = sts_client.get_caller_identity()["Account"] + newStatement = { + "Effect": "Allow", + "Principal": { + "Service": [ + "s3.amazonaws.com", + "iot.amazonaws.com", + "iam.amazonaws.com", + ], + "AWS": str(AWS_ACCOUNT_ID), + }, + "Action": "sts:AssumeRole", + } + print(json.dumps(newStatement, indent=4)) + logging.error("For example, the full document will look like: ") + role["AssumeRolePolicyDocument"]["Statement"].append(newStatement) + print(json.dumps(role["AssumeRolePolicyDocument"], indent=4)) + cleanup_aws_resources(ctx.flags) + ctx.exit(1) + credentials = response["Credentials"] + # create a new boto3 session so we do not modify + # the default session used by variable 's3'. + test_session = boto3.session.Session() + test_s3_client = test_session.client( + "s3", + region_name=AWS_REGION, + aws_access_key_id=credentials["AccessKeyId"], + aws_secret_access_key=credentials["SecretAccessKey"], + aws_session_token=credentials["SessionToken"], + ) + # try to put object + testKey = "testRolePermissionsObject" + test_s3_client.put_object( + Bucket=bucket_name, Key=testKey, Body=bytes("dummy", "utf-8") + ) + # try to get object version + test_s3_client.list_object_versions(Bucket=bucket_name) + # try to get object + test_s3_client.get_object(Bucket=bucket_name, Key=testKey) + # We can put, get, and get object versioning with this role. + # So it has all the permissions needed for an OTA update. + can_access_bucket = True + except botocore.exceptions.ClientError as e: + logging.error( + f"Failed to access bucket {bucket_name} " + f"using {iam_role_name} with error:" + ) + logging.error(repr(e)) + can_access_bucket = False + + # Cleanup object used to test that role has access to bucket. + try: + s3.delete_object(BucketName=bucket_name, key=testKey) + except Exception: + pass # object doesn't exist. + + if can_access_bucket: + logging.info( + "Verified the role '" + + iam_role_name + + "' provides access to bucket '" + + bucket_name + + "'" + ) + else: + logging.error( + "The role (" + iam_role_name + ") found does not have " + "access to the bucket (" + bucket_name + ")." + ) + cleanup_aws_resources(ctx.flags) + ctx.exit(1) + + if not _does_role_exist(iam_role_name): + # create role + if not _create_iam_role(ctx.flags, iam_role_name, permissions_boundary): + logging.error("Role '" + iam_role_name + "' creation failed.") + cleanup_aws_resources(ctx.flags) + ctx.exit(1) + else: + logging.info(f"Attempting to re-use existing role: '{iam_role_name}'") + + ota_job_name = OTA_JOB_NAME_PREFIX + update_name + if _does_job_exist(job_name=ota_job_name): + logging.error("There is already an update job with name '" + ota_job_name + "'") + if input("Delete update job? (Y/N) ").lower() == "y": + if not ( + _delete_ota_update(update_name, True) + and _wait_for_job_deleted(ota_job_name) + ): + logging.warning("Failed to delete update! Try doing this manually. ") + cleanup_aws_resources(ctx.flags) + ctx.exit(1) + + # create update. + if not _does_job_exist(job_name=ota_job_name): + if not _create_aws_update(ctx.flags, update_name): + logging.error("Update '" + update_name + "' creation failed.") + cleanup_aws_resources(ctx.flags) + ctx.exit(1) + + logging.info("All done!") + ctx.exit(0) + + def print_exception(ex): x = tb.extract_stack()[1] logging.error(f"{x.filename}:{x.lineno}:{x.name}: {ex}") @@ -2157,7 +2801,7 @@ def _delete_job(job_name, force_delete=False): # Define command-line interface for Job deletion command. @cli.command(cls=StdCommand) @click.option( - "--job_name", prompt="Enter role name", help="Name of the job to be deleted." + "--job_name", prompt="Enter job name", help="Name of the job to be deleted." ) @click.option( "-f", @@ -2185,36 +2829,6 @@ def _delete_ota_update(ota_update_name, force_delete=False): False otherwise. """ ota_job_name = OTA_JOB_NAME_PREFIX + ota_update_name - if not _does_job_exist(ota_job_name): - return True - job_status = None - try: - job_status = iot.describe_job(jobId=ota_job_name)["job"]["status"] - except botocore.exceptions.ClientError as ex: - if ex.response["Error"]["Code"] == "ResourceNotFoundException": - return False - else: - raise ex - # Handle edge cases. - if job_status == "IN_PROGRESS": - if not force_delete: - # we do have a job but don't force delete: - # we won't be able to delete the ota yet - logging.warning( - "The ota update is currently in progress. \ - Try re-running with the -f option to delete it anyway." - ) - return False - # If a deletion is ongoing, give it time (until maximum timeout) to complete. - elif job_status == "DELETION_IN_PROGRESS" and not _wait_for_job_deleted( - ota_job_name - ): - logging.error( - "Job deletion has been started but is taking an abnormal amount of time." - ) - # Cannot delete the OTA update without the job being deleted first. - return False - # tries to delete the ota update try: if force_delete: @@ -2337,6 +2951,26 @@ def delete_ota_update(ctx, ota_update_name, log_level, force_delete): ctx.exit(1) +def _empty_bucket(bucket_name): + """ + Parameters: + bucket_name (string): name of the S3 Bucket to be emptied. + + This function does not check if the bucket exists, and does + no error handling. + + Returns: + bool: True if emptying succeeds. + """ + versioning = s3.get_bucket_versioning(Bucket=bucket_name) + bucket = boto3.resource("s3").Bucket(bucket_name) + if versioning.get("Status") == "Enabled": + logging.debug("Removing bucket versioning") + bucket.object_versions.delete() + bucket.objects.delete() + logging.info("Bucket " + bucket_name + " emptied.") + + def _delete_bucket(bucket_name, force_delete=False): """ Parameters: @@ -2351,12 +2985,7 @@ def _delete_bucket(bucket_name, force_delete=False): """ try: if force_delete: - versioning = s3.get_bucket_versioning(Bucket=bucket_name) - bucket = boto3.resource("s3").Bucket(bucket_name) - if versioning.get("Status") == "Enabled": - logging.debug("Removing bucket versioning") - bucket.object_versions.delete() - bucket.objects.delete() + _empty_bucket(bucket_name) s3.delete_bucket(Bucket=bucket_name) except botocore.exceptions.ClientError as ex: if ex.response["Error"]["Code"] == "NoSuchBucket": @@ -2490,6 +3119,73 @@ def delete_certificate(ctx, certificate_id, log_level, force_delete): ctx.exit(1) +# Defines Command-line interface for deleting the Thing, +# Policy, Bucket, Role, and Update +# specified by createIoTThings_settings.json. +@cli.command(cls=StdCommand) +@click.option( + "--config_file_path", + help=".json file defining arguments for creating an OTA update.", + default="createIoTThings_settings.json", +) +@click.pass_context +def cleanup_simplified( + ctx, + config_file_path, + log_level, +): + settings = {} + # Read .json file, pass parameters to flags. + try: + contents = read_whole_file(config_file_path) + settings = json.loads(contents) + settings["format_vars"] = settings["format_vars"].replace( + "target_application;", "" + ) + settings = _formatVars(settings, ctx) + logging.debug("Settings .json file parsed to: " + str(settings)) + except FileNotFoundError: + logging.error("Config file not found at " + config_file_path) + ctx.exit(1) + except json.JSONDecodeError: + logging.error("Failed to parse .json file: " + config_file_path) + ctx.exit(1) + + # Check the required settings exist. + thing_name = _tryGetSetting( + "thing_name", settings=settings, ctx=ctx, errorOnFailure=True + ) + policy_name = _tryGetSetting( + "policy_name", settings=settings, ctx=ctx, errorOnFailure=True + ) + bucket_name = _tryGetSetting( + "bucket_name", settings=settings, ctx=ctx, errorOnFailure=True + ) + iam_role_name = _tryGetSetting( + "iam_role_name", settings=settings, ctx=ctx, errorOnFailure=True + ) + update_name = _tryGetSetting( + "update_name", settings=settings, ctx=ctx, errorOnFailure=True + ) + ota_job_name = OTA_JOB_NAME_PREFIX + update_name + if _does_job_exist(ota_job_name): + _delete_ota_update(ota_update_name=update_name, force_delete=True) + if _wait_for_job_deleted(ota_job_name): + logging.info("Deleted OTA update " + update_name + " successfully.") + else: + logging.warning("Failed to delete OTA update job.") + if _does_bucket_exist(bucket_name): + _try_delete(bucket_name, _delete_bucket, force_delete=True) + if _does_role_exist(iam_role_name): + _try_delete(iam_role_name, _delete_iam_role, force_delete=True) + if _does_policy_exist(policy_name): + _try_delete(policy_name, _delete_policy, prune=True) + if _does_thing_exist(thing_name): + _try_delete(thing_name, _delete_thing) + logging.info("All done!") + ctx.exit(0) + + def cleanup_aws_resources( flags: Flags, ): diff --git a/tools/scripts/createIoTThings_settings.json b/tools/scripts/createIoTThings_settings.json new file mode 100644 index 0000000..80737a9 --- /dev/null +++ b/tools/scripts/createIoTThings_settings.json @@ -0,0 +1,47 @@ +{ + "thing_name":"", + "permissions_boundary":"arn:aws:iam:::policy/", + "role_prefix":"", + "target_application":"", + + "format_vars":"thing_name;role_prefix;target_application;credentials_path", + + "policy_name":"", + "bucket_name":"${thing_name}{lower}-bucket", + "iam_role_name":"", + "update_name":"", + "existing_certificate_arn":"CREATE_NEW_CERTIFICATE", + + + "credentials_path":"certificates", + "build_dir":"build", + "ota_binary":"${target_application}-update_signed.bin", + + "skip_build":"", + "build_script_path":"", + "private_key_path":"", + "certificate_path":"", + "target_platform":"", + "inference":"", + "audio":"", + "toolchain":"", + "clean_build":"", + + "policy_name_DEFAULT":"${thing_name}_policy", + "bucket_name_DEFAULT":"${thing_name}_bucket", + "iam_role_name_DEFAULT":"${role_prefix}-${thing_name}_role", + "update_name_DEFAULT":"${thing_name}_update", + "existing_certificate_arn_DEFAULT":"", + "credentials_path_DEFAULT":"", + "build_dir_DEFAULT":"", + "ota_binary_DEFAULT":"", + "skip_build_DEFAULT":"false", + "build_script_path_DEFAULT":"./tools/scripts/build.sh", + "private_key_path_DEFAULT":"${credentials_path}/thing_private_key_${thing_name}.pem.key", + "certificate_path_DEFAULT":"${credentials_path}/thing_certificate_${thing_name}.pem.crt", + "target_platform_DEFAULT":"corstone315", + "inference_DEFAULT":"SOFTWARE", + "audio_DEFAULT":"ROM", + "toolchain_DEFAULT":"GNU", + "clean_build":"auto" +}