Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Module: Amazon MQ #432

Merged
merged 8 commits into from
Mar 20, 2023

Conversation

fotto
Copy link
Contributor

@fotto fotto commented Feb 21, 2021

SUMMARY

replicates ansible-collections/amazon.aws#266 as suggested

This PR contains some basic support for Amazon MQ. It covers

  • Managing Amazon MQ brokers
  • Managing Amazon MQ configurations
  • Managing Amazon MQ (local) users
ISSUE TYPE
  • New Module Pull Request
COMPONENT NAME

Amazon MQ

ADDITIONAL INFORMATION

We use those modules to manage our MQ Brokers. The context of usage is

  • brokers are created by different means (e.g. terraform) - but the module is capable of doing that as well
  • we've developed a custom role around those modules
  • to manage MQ user credentials we use another local extension of amazon.aws collection (will come as separate PR) that interfaces with AWS SecretsManager
  • that role uses the reboot broker feature (part of this PR) to implement a custom handler that reboots a broker whenever a configuration run sees any changes in configuration and/or users

The functionality of the added modules is illustrated in the added test suite (tests/integration/targets/mq) some of them still require a running MQ broker to be usable.

Missing functionality/limitations:

  • requires a recent version of boto3 library (older versions don't support all Amazon MQ features used here)
  • API results are returned "as is", i.e. there's no conversion from camel case yaml to snake yaml

mq_broker.py

  • no support for LDAP connection (external user management)
  • update configuration only supported through mq_broker_config.py
  • known to work only with EngineType=ACTIVEMQ (proper support for RABBITMQ still missing)

mq_broker_config.py

  • needs proper XML comparison between current and desired configuration (current one is too simplistic)
  • only tested with ActiveMQ configurations

@ansibullbot
Copy link

@ansibullbot ansibullbot added community_review integration tests/integration module module needs_triage new_contributor Help guide this first time contributor new_module New module new_plugin New plugin plugins plugin (any type) tests tests needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR and removed community_review labels Feb 21, 2021
@fotto
Copy link
Contributor Author

fotto commented Feb 22, 2021

I'll work on the code quality (sanity checks)
I'll rebase the PR after that.
You can ignore it until then

@tremble tremble changed the title Mq [WIP] New Module: Amazon MQ Feb 22, 2021
@ansibullbot ansibullbot added the WIP Work in progress label Feb 22, 2021
@tremble
Copy link
Contributor

tremble commented Feb 22, 2021

Couple of quick comments:

  • It would be better if mq_broker could create/delete a broker. This would then support creating the broker as part of the integration tests so they could be run in our CI environment.
  • "info" states/actions are generally discouraged. It would be better to have a dedicated mq_broker_info module.

@fotto
Copy link
Contributor Author

fotto commented Feb 22, 2021

Couple of quick comments:

* It would be better if mq_broker could create/delete a broker.  This would then support creating the broker as part of the integration tests so they could be run in our CI environment.

* "info" states/actions are generally discouraged.  It would be better to have a dedicated mq_broker_info module.

Thank you for your hints. I'll take them into account

@fotto
Copy link
Contributor Author

fotto commented Feb 28, 2021

current state:

  • logic/structure refactored as suggested
  • documentation added/fixed
  • code and doc passes sanity checks

not covered, yet:

  • the modules have some functional limitations (see PR description)
  • the current test harness (tests/integration/targets/mq) needs to be reworked to run as automatic test (is there a good example to learn from?)

How to proceed from here?

@tremble

@fotto fotto changed the title [WIP] New Module: Amazon MQ New Module: Amazon MQ Feb 28, 2021
@ansibullbot ansibullbot removed the WIP Work in progress label Feb 28, 2021
@tremble
Copy link
Contributor

tremble commented Feb 28, 2021

@jillr wrote an initial guide:
https://www.ansible.com/blog/getting-started-with-aws-ansible-module-development

tests/integration/targets/iam_saml_federation/ includes a relatively simple example (but doesn't support check_mode)
https://github.com/ansible-collections/amazon.aws/blob/main/tests/integration/targets/ec2_vpc_subnet/tasks/main.yml has some slightly more complex examples

Biggest issue is likely to be IAM policies.
mattclay/aws-terminator#125 is the pieces for iam_saml_federation which doesn't return creation time (DBTerminator)
mattclay/aws-terminator#119 is the pieces for ec2_launch_template which does return a creation time (Terminator)

@fotto fotto changed the title New Module: Amazon MQ [WIP] New Module: Amazon MQ Mar 13, 2021
@ansibullbot ansibullbot added the WIP Work in progress label Mar 13, 2021
@ansibullbot
Copy link

@fotto this PR contains the following merge commits:

Please rebase your branch to remove these commits.

click here for bot help

@softwarefactory-project-zuul
Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/1904839c2b2840e2928b7fcb118743bd

ansible-galaxy-importer FAILURE in 3m 55s
✔️ build-ansible-collection SUCCESS in 12m 19s
✔️ ansible-test-sanity-docker-devel SUCCESS in 8m 40s (non-voting)
✔️ ansible-test-sanity-docker-milestone SUCCESS in 12m 21s (non-voting)
✔️ ansible-test-sanity-docker-stable-2.12 SUCCESS in 12m 41s
✔️ ansible-test-sanity-docker-stable-2.13 SUCCESS in 12m 40s
✔️ ansible-test-sanity-docker-stable-2.14 SUCCESS in 12m 14s
✔️ ansible-test-units-amazon-aws-python36 SUCCESS in 7m 34s
✔️ ansible-test-units-amazon-aws-python38 SUCCESS in 7m 34s
✔️ ansible-test-units-amazon-aws-python39 SUCCESS in 7m 42s
✔️ ansible-test-units-amazon-aws-python310 SUCCESS in 9m 06s
✔️ ansible-test-changelog SUCCESS in 4m 13s
✔️ ansible-test-splitter SUCCESS in 4m 51s
✔️ integration-community.aws-1 SUCCESS in 8m 21s
Skipped 21 jobs

@tremble
Copy link
Contributor

tremble commented Mar 19, 2023

recheck

@softwarefactory-project-zuul
Copy link
Contributor

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/e9c1dc58cd4e410593d5dcc670d72690

✔️ ansible-galaxy-importer SUCCESS in 3m 31s
✔️ build-ansible-collection SUCCESS in 12m 49s
✔️ ansible-test-sanity-docker-devel SUCCESS in 12m 21s (non-voting)
✔️ ansible-test-sanity-docker-milestone SUCCESS in 11m 51s (non-voting)
✔️ ansible-test-sanity-docker-stable-2.12 SUCCESS in 10m 38s
✔️ ansible-test-sanity-docker-stable-2.13 SUCCESS in 11m 36s
✔️ ansible-test-sanity-docker-stable-2.14 SUCCESS in 10m 00s
✔️ ansible-test-units-amazon-aws-python36 SUCCESS in 6m 19s
✔️ ansible-test-units-amazon-aws-python38 SUCCESS in 7m 39s
✔️ ansible-test-units-amazon-aws-python39 SUCCESS in 7m 20s
✔️ ansible-test-units-amazon-aws-python310 SUCCESS in 5m 46s
✔️ ansible-test-changelog SUCCESS in 4m 15s
✔️ ansible-test-splitter SUCCESS in 4m 41s
✔️ integration-community.aws-1 SUCCESS in 5m 30s
Skipped 21 jobs

plugins/modules/mq_broker.py Outdated Show resolved Hide resolved
plugins/modules/mq_broker.py Outdated Show resolved Hide resolved
@tremble tremble added the mergeit Merge the PR (SoftwareFactory) label Mar 20, 2023
@softwarefactory-project-zuul
Copy link
Contributor

Build failed (gate pipeline). For information on how to proceed, see
http://docs.openstack.org/infra/manual/developers.html#automated-testing

https://ansible.softwarefactory-project.io/zuul/buildset/a4eb61cecfd447f98cb93f25824bdd3c

ansible-galaxy-importer FAILURE in 4m 04s
✔️ build-ansible-collection SUCCESS in 12m 37s
✔️ ansible-test-sanity-docker-devel SUCCESS in 10m 44s (non-voting)
✔️ ansible-test-sanity-docker-milestone SUCCESS in 10m 14s (non-voting)
✔️ ansible-test-sanity-docker-stable-2.12 SUCCESS in 11m 20s
✔️ ansible-test-sanity-docker-stable-2.13 SUCCESS in 10m 10s
✔️ ansible-test-sanity-docker-stable-2.14 SUCCESS in 10m 15s
✔️ ansible-test-units-amazon-aws-python36 SUCCESS in 6m 08s
✔️ ansible-test-units-amazon-aws-python38 SUCCESS in 6m 50s
✔️ ansible-test-units-amazon-aws-python39 SUCCESS in 5m 56s
✔️ ansible-test-units-amazon-aws-python310 SUCCESS in 6m 10s
✔️ ansible-test-changelog SUCCESS in 4m 27s
✔️ ansible-test-splitter SUCCESS in 4m 48s
✔️ integration-community.aws-1 SUCCESS in 5m 47s
Skipped 21 jobs

@tremble
Copy link
Contributor

tremble commented Mar 20, 2023

regate

@softwarefactory-project-zuul
Copy link
Contributor

Build failed (gate pipeline). For information on how to proceed, see
http://docs.openstack.org/infra/manual/developers.html#automated-testing

https://ansible.softwarefactory-project.io/zuul/buildset/2ae5723893d24c2a9358cc517f948711

ansible-galaxy-importer FAILURE in 3m 53s
✔️ build-ansible-collection SUCCESS in 12m 28s
✔️ ansible-test-sanity-docker-devel SUCCESS in 10m 07s (non-voting)
✔️ ansible-test-sanity-docker-milestone SUCCESS in 10m 17s (non-voting)
✔️ ansible-test-sanity-docker-stable-2.12 SUCCESS in 11m 03s
✔️ ansible-test-sanity-docker-stable-2.13 SUCCESS in 11m 09s
✔️ ansible-test-sanity-docker-stable-2.14 SUCCESS in 9m 57s
✔️ ansible-test-units-amazon-aws-python36 SUCCESS in 6m 15s
✔️ ansible-test-units-amazon-aws-python38 SUCCESS in 5m 57s
✔️ ansible-test-units-amazon-aws-python39 SUCCESS in 5m 56s
✔️ ansible-test-units-amazon-aws-python310 SUCCESS in 6m 07s
✔️ ansible-test-changelog SUCCESS in 4m 28s
✔️ ansible-test-splitter SUCCESS in 4m 57s
✔️ integration-community.aws-1 SUCCESS in 5m 43s
Skipped 21 jobs

@tremble
Copy link
Contributor

tremble commented Mar 20, 2023

regate

@softwarefactory-project-zuul
Copy link
Contributor

Build failed (gate pipeline). For information on how to proceed, see
http://docs.openstack.org/infra/manual/developers.html#automated-testing

https://ansible.softwarefactory-project.io/zuul/buildset/07edd3b1049a4ba086c398b91aede4de

ansible-galaxy-importer FAILURE in 4m 39s
✔️ build-ansible-collection SUCCESS in 12m 56s
✔️ ansible-test-sanity-docker-devel SUCCESS in 11m 11s (non-voting)
✔️ ansible-test-sanity-docker-milestone SUCCESS in 10m 15s (non-voting)
✔️ ansible-test-sanity-docker-stable-2.12 SUCCESS in 19m 44s
✔️ ansible-test-sanity-docker-stable-2.13 SUCCESS in 10m 58s
✔️ ansible-test-sanity-docker-stable-2.14 SUCCESS in 9m 59s
✔️ ansible-test-units-amazon-aws-python36 SUCCESS in 5m 32s
✔️ ansible-test-units-amazon-aws-python38 SUCCESS in 7m 13s
✔️ ansible-test-units-amazon-aws-python39 SUCCESS in 5m 36s
✔️ ansible-test-units-amazon-aws-python310 SUCCESS in 5m 33s
✔️ ansible-test-changelog SUCCESS in 4m 25s
✔️ ansible-test-splitter SUCCESS in 5m 52s
✔️ integration-community.aws-1 SUCCESS in 5m 23s
Skipped 21 jobs

@tremble
Copy link
Contributor

tremble commented Mar 20, 2023

regate

@softwarefactory-project-zuul
Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/c255e50eab574339aee75c811ba309c1

ansible-galaxy-importer FAILURE in 4m 05s
✔️ build-ansible-collection SUCCESS in 12m 47s
✔️ ansible-test-sanity-docker-devel SUCCESS in 9m 01s (non-voting)
✔️ ansible-test-sanity-docker-milestone SUCCESS in 10m 22s (non-voting)
✔️ ansible-test-sanity-docker-stable-2.12 SUCCESS in 8m 55s
✔️ ansible-test-sanity-docker-stable-2.13 SUCCESS in 10m 16s
✔️ ansible-test-sanity-docker-stable-2.14 SUCCESS in 9m 38s
✔️ ansible-test-units-amazon-aws-python36 SUCCESS in 6m 11s
✔️ ansible-test-units-amazon-aws-python38 SUCCESS in 5m 34s
✔️ ansible-test-units-amazon-aws-python39 SUCCESS in 6m 18s
✔️ ansible-test-units-amazon-aws-python310 SUCCESS in 5m 53s
✔️ ansible-test-changelog SUCCESS in 4m 17s
✔️ ansible-test-splitter SUCCESS in 4m 59s
✔️ integration-community.aws-1 SUCCESS in 6m 09s
Skipped 21 jobs

@tremble
Copy link
Contributor

tremble commented Mar 20, 2023

regate

@softwarefactory-project-zuul
Copy link
Contributor

Build succeeded (gate pipeline).
https://ansible.softwarefactory-project.io/zuul/buildset/6422f974e8d54943bde93060f3f5fda8

✔️ ansible-galaxy-importer SUCCESS in 4m 33s
✔️ build-ansible-collection SUCCESS in 12m 52s
✔️ ansible-test-sanity-docker-devel SUCCESS in 11m 20s (non-voting)
✔️ ansible-test-sanity-docker-milestone SUCCESS in 12m 13s (non-voting)
✔️ ansible-test-sanity-docker-stable-2.12 SUCCESS in 11m 23s
✔️ ansible-test-sanity-docker-stable-2.13 SUCCESS in 10m 55s
✔️ ansible-test-sanity-docker-stable-2.14 SUCCESS in 11m 31s
✔️ ansible-test-units-amazon-aws-python36 SUCCESS in 6m 07s
✔️ ansible-test-units-amazon-aws-python38 SUCCESS in 8m 23s
✔️ ansible-test-units-amazon-aws-python39 SUCCESS in 6m 07s
✔️ ansible-test-units-amazon-aws-python310 SUCCESS in 9m 10s
✔️ ansible-test-changelog SUCCESS in 4m 25s
✔️ ansible-test-splitter SUCCESS in 4m 42s
✔️ integration-community.aws-1 SUCCESS in 6m 10s
Skipped 21 jobs

@softwarefactory-project-zuul softwarefactory-project-zuul bot merged commit 43f2bf1 into ansible-collections:main Mar 20, 2023
boutetnico pushed a commit to boutetnico/community.aws that referenced this pull request Apr 3, 2023
New Module: Amazon MQ

SUMMARY
replicates ansible-collections/amazon.aws#266 as suggested
This PR contains some basic support for Amazon MQ. It covers

Managing Amazon MQ brokers
Managing Amazon MQ configurations
Managing Amazon MQ (local) users

ISSUE TYPE

New Module Pull Request

COMPONENT NAME
Amazon MQ
ADDITIONAL INFORMATION
We use those modules to manage our MQ Brokers. The context of usage is

brokers are created by different means (e.g. terraform) - but the module is capable of doing that as well
we've developed a custom role around those modules
to manage MQ user credentials we use another local extension of amazon.aws collection (will come as separate PR) that interfaces with AWS SecretsManager
that role uses the reboot broker feature (part of this PR) to implement a custom handler that reboots a broker whenever a configuration run sees any changes in configuration and/or users

The functionality of the added modules is illustrated in the added test suite (tests/integration/targets/mq) some of them still require a running MQ broker to be usable.
Missing functionality/limitations:

requires a recent version of boto3 library  (older versions don't support all Amazon MQ features used here)
API results are returned "as is", i.e. there's no conversion from camel case yaml to snake yaml

mq_broker.py

no support for LDAP connection (external user management)
update configuration only supported through mq_broker_config.py
known to work only with EngineType=ACTIVEMQ (proper support for RABBITMQ still missing)

mq_broker_config.py

needs proper XML comparison between current and desired configuration (current one is too simplistic)
only tested with ActiveMQ configurations

Reviewed-by: Mark Chappell
Reviewed-by: Alina Buzachis
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
community_review integration tests/integration mergeit Merge the PR (SoftwareFactory) module module new_contributor Help guide this first time contributor new_module New module new_plugin New plugin plugins plugin (any type) tests tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants