Weldr/Cloud API: simplify GCP upload options

[thozza]

In both, the Weldr and Cloud APIs, users had to previously provide values, which could have been generated automatically by osbuild-composer, or were in reality static and could be configured on the worker.

Specifically:

• Bucket - bucket name usually does not change for Cloud API calls, since it is specific to the GCP credentials provisioned on the worker and as a result, to the respective GCP project. Therefore it makes more sense to allow setting the GCP bucket in the worker configuration when composer is running in a SaaS deployment.
• Object - object name used for the image when uploading it to the GCP Storage had to be previously provided by the user of the Weldr API. For Cloud API, the value has been generated by the implementation. In reality, the object name is not very important, as long as it is unique. The reason is that the object gets deleted after the image importing finishes.

Notable changes:

• Allow setting of the GCP Bucket in the worker configuration.
• Make the Bucket optional for GCP upload options in the Cloud API. Adjust the Cloud API test case to set the Bucket in the worker configuration, instead of passing it in the compose request.
• Extend Weldr API to generate a unique GCP Object name if it was not provided in the job.
• Make the GCP Object name optional for Weldr API. Adjust the Weldr API test case to not provide the Object name when passing the GCP upload options.
• Flip the logic in the worker when uploading image to AWS EC2/S3 WRT the Bucket name preference. The Bucket value set in the worker configuration is now used only if the Bucket was not provided in the job.
• Extend the packer template ansible role to configure the GCP Bucket in worker configuration.

This pull request includes:

• adequate testing for the new functionality or fixed issue
• adequate documentation informing people about the change such as
  • submit a PR for the guides repository if this PR changed any behavior described there: https://www.osbuild.org/guides/

[achilleas-k]

LGTM. One tiny question.

[ondrejbudai]

I have concerns about generating the GCP object names on multiple levels, I think that we should make it more consistent and do it in the API.

Also, I would really love to make the worker more dumb and predictive.

[croissanne]

Should we update

osbuild-composer/templates/packer/ansible/roles/common/files/worker-initialization-scripts/get_gcp_creds.sh

Lines 1 to 21 in d9cfbea

	#!/bin/bash
	set -eo pipefail
	source /tmp/cloud_init_vars
	echo "Deploy GCP credentials."
	if [[ -z "$GCP_SERVICE_ACCOUNT_IMAGE_BUILDER_ARN" ]]; then
	echo "GCP_SERVICE_ACCOUNT_IMAGE_BUILDER_ARN not defined, skipping."
	exit 0
	fi
	# Deploy the GCP Service Account credentials file.
	/usr/local/bin/aws secretsmanager get-secret-value \
	--endpoint-url "${SECRETS_MANAGER_ENDPOINT_URL}" \
	--secret-id "${GCP_SERVICE_ACCOUNT_IMAGE_BUILDER_ARN}" | jq -r ".SecretString" > /etc/osbuild-worker/gcp_credentials.json
	sudo tee -a /etc/osbuild-worker/osbuild-worker.toml > /dev/null << EOF
	[gcp]
	credentials = "/etc/osbuild-worker/gcp_credentials.json"
	EOF

in this PR?

[thozza]

Should we update
...
in this PR?

Looking at it, probably yes. Everything should keep working even without this change as the option to pass the bucket via the API call is still present. Nevertheless changing the script will be needed anyway in order to stop passing the bucket over the API, so I may as well change it now.

Thanks for pointing this out!

[thozza]

I addressed all review comments and added a change for the packer template as suggested by Sanne. Please have a look.

[ondrejbudai]

Oh no, it now needs to be rebased. :/ Otherwise, this is amazing!

[thozza]

Rebased on top of main... 🦾

[ondrejbudai]

Thanks a lot, a great cleanup!

[thozza]

Rebase on main without doing any other changes.