In-browser assisted full cloud stack deployments #2816

stefannica · 2024-06-28T17:27:06Z

Describe changes

Implements a new automated experience of deploying cloud infrastructure and registering full cloud ZenML stacks two-in-one with minimal user input. This experience is meant to reduce the friction and technical difficulties usually encountered when a user has to create a cloud ZenML stack from scratch.

The zenml stack deploy CLI command has been repurposed to support this new experience as showcased here:

$ zenml stack deploy -p aws --set

AWS ZenML Cloud Stack Deployment                                                                                                                                                                                   
================================

Provision and register a basic AWS ZenML stack authenticated and connected to all the necessary cloud infrastructure resources required to run pipelines in AWS.                                                   

**Instructions**                                                                                                                                                                                                       

You will be redirected to the AWS console in your browser where you'll be asked to log into your AWS account and create a CloudFormation ZenML stack. The stack parameters will be pre-filled with the necessary   
information to connect ZenML to your AWS account, so you should only need to review and confirm the stack.                                                                                                         

After the CloudFormation stack is deployed, you can return to the CLI to view details about the associated ZenML stack automatically registered with ZenML.                                                        

NOTE: The CloudFormation stack will create the following new resources in your AWS account. Please ensure you have the necessary permissions and are aware of any potential costs:                                 

 • An S3 bucket registered as a ZenML artifact store.                                                                                                                                                              
 • An ECR repository registered as a ZenML container registry.                                                                                                                                                     
 • Sagemaker registered as a ZenML orchestrator.                                                                                                                                                                   
 • An IAM user and IAM role with the minimum necessary permissions to access the above resources.                                                                                                                  
 • An AWS access key used to give access to ZenML to connect to the above resources through a ZenML service connector.                                                                                             

The CloudFormation stack will automatically create an AWS secret key and will share it with ZenML to give it permissions to access the resources created by the stack. You can revoke these permissions at any time
by deleting the CloudFormation stack.                                                                                                                                                                              


Proceed to continue with the deployment. You will be automatically redirected to AWS in your browser. [y/n]: y
If your browser did not open automatically, please open the following URL into your browser to deploy the stack to AWS: AWS CloudFormation Console.                                                                

**Stack successfully registered! 🚀**                                                                                                                                                                                  

Stack stack:                                                                                                                                                                                                       

 • sagemaker orchestrator: aws-cf-eu-central-1-wpyh                                                                                                                                                                
 • s3 artifact_store: aws-cf-eu-central-1-iaiy                                                                                                                                                                     
 • aws container_registry: aws-cf-eu-central-1-smep                                                                                                                                                                
 • Service Connector: aws-cf-eu-central-1-pabl                                                                                                                                                                     

**Follow-up**                                                                                                                                                                                                          

The ZenML stack has been successfully deployed and registered. You can delete the CloudFormation at any time to revoke ZenML's access to your AWS account and to clean up the resources created by the stack by    
using the AWS CloudFormation console.

AWS is currently the only supported provider with Google and Azure to follow shortly.

Side changes

Make the `zenml` repository name configurable in the container registry stack component

Currently, ZenML uses the .../zenml docker repository by default for all pipeline images it builds/pushes. This can be overridden, but it's a docker settings that must be configured in code.
At the same time, we want these full stack deployment templates to create unique cloud resources, which includes container registries. Therefore, we need the ability to configure the container registry stack components to use a "default repository name" that is different than zenml. This PR also adds that capability by modeling a new (optional) default_repository configuration attribute to all container registry stack components.

Use the region from the AWS service connector in the S3 artifact store

The S3 artifact store doesn't explicitly configure the AWS region in the s3fs client. This can lead to errors in environments such as Sagemaker, because the region might default to something else. This PR now uses the AWS region in the service connector to explicitly configure the s3fs client.

Deprecations

The zenml stack deploy CLI command has been renamed to zenml stack deploy-mlstack and marked as deprecated. Users should use the new zenml stack deploy CLI command to benefit from the new experience or use the mlstacks project stacks separately.

Pre-requisites

Please ensure you have done the following:

I have read the CONTRIBUTING.md document.
If my change requires a change to docs, I have updated the documentation accordingly.
I have added tests to cover my changes.
I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
If my changes require changes to the dashboard, these changes are communicated/requested.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Other (add details above)

coderabbitai · 2024-06-28T17:27:11Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

stefannica · 2024-06-28T20:01:18Z

@schustmi I might need you to take a quick look at the side-change documented in the PR description. It affects the way we build docker container images to run pipelines.

bcdurak

I left a few comments, mostly minor things.

infra/README.md

src/zenml/stack_deployments/stack_deployment.py

src/zenml/stack_deployments/aws_stack_deployment.py

src/zenml/client.py

infra/README.md

schustmi

Docker changes LGTM

schustmi · 2024-07-03T13:15:58Z

src/zenml/analytics/enums.py

@@ -85,6 +85,9 @@ class AnalyticsEvent(str, Enum):
    DEPLOY_STACK_COMPONENT = "Stack component deployed"
    DESTROY_STACK_COMPONENT = "Stack component destroyed"

+    # Cloud stack deployment
+    DEPLOY_STACK_CLOUD = "Stack deployed in cloud"


Where else would a stack be deployed? 😅

Isn't there a better wording for this to explain what it really does?

@schustmi there already is a DEPLOY_STACK event related to stacks deployed through mlstacks. I don't want to reuse that event. @bcdurak may be able to give me a better alternative.

Maybe something with One Click in there so we know this is what happened? Also, this is currently only being tracked in the CLI, which means we'll miss all users that run this from the dashboard (which I assume will be the majority of them?).

At some point it was mentioned that you're planning to add some special metadata to the components/stacks created using this feature, which we could then track in our Stack/Componenent/Connector registered events. Is this still planned?

Okay, I named the event DEPLOY_FULL_STACK. On the server side, there is the event REGISTERED_STACK that we already had, but it will have new metadata fields:

wizard = True if the full stack was registered via the CLI wizard

deployment = cloud-formation if the full stack was deployed from scratch in AWS

provider = aws if either happened

src/zenml/cli/stack.py

src/zenml/zen_server/routers/stack_deployment_endpoints.py

schustmi · 2024-07-03T13:33:43Z

src/zenml/stack_deployments/aws_stack_deployment.py

+        """Return the ZenML stack that was deployed and registered.
+
+        This method is called to retrieve a ZenML stack matching the deployment
+        provider and stack name.


The stack name is not an input here. This I guess just gives the latest, I'm not sure how this would be used if there are multiple such stacks?

Check the comment below. I can't rely on the stack name to be the same because the user can change it before they deploy the infrastructure in AWS. Therefore I'm using another approach: I'm searching for the stack most likely to match the one deployed by the user in AWS.

Yeah, just the docstring is wrong here. I guess this method is only used for polling until the stack gets deployed right?

IMO, this could again be improved (similar to my analytics comment) by some metadata on the stack. The lambda function registering the stack could add a specific label to the stack, which could then be passed to this endpoint when waiting for the stack.

I added labels to the full stack model. These are set differently depending on where and how the endpoint is called and used as metadata in the REGISTERED_STACK event. They are also set on the stack components and service connectors being created (not yet possible to do so directly on the stack). These labels are also used to "detect" the stack here in fewer steps.

schustmi · 2024-07-03T13:37:48Z

src/zenml/zen_server/routers/stack_deployment_endpoints.py

+    stack_name: str,
+    location: Optional[str] = None,
+    auth_context: AuthContext = Security(authorize),
+) -> Tuple[str, str]:


Should we maybe already check here that the user has permissions to create service connectors, stacks and components? Otherwise I assume the lambda function downstream will fail, which leaves users with deployed infra that is not tracked in ZenML?

Good point, but not something I can check in the client. I would need a new endpoint just for this purpose and I would need to check all permissions, not just the service connectors. Perhaps I can use the endpoint that @bcdurak already set up. Maybe add a "verify" flag to it.

Can;t we have those checks on this exact URL endpoint? Or what is the purpose for creating this URL other than to open it and create stacks?

github-actions · 2024-07-03T15:51:07Z

LLM Finetuning template updates in examples/llm_finetuning have been pushed.

github-actions · 2024-07-03T16:45:55Z

LLM Finetuning template updates in examples/llm_finetuning have been pushed.

Add cloudshell instructions Update Update GCP templates WIP new zenml deploy CLI command Change AWS template URL Implemented zenml deploy CLI command Complete zenml stack deploy CLI command Adapt stack registration to new endpoint structure Add default repository configuration attr to all container registries Update template with proper sagemaker permissions Fix template permissions Final IAM permission fixes Fix AWS service connector permissions Remove unneeded permissions Fix container registry default repository setting support Fix the S3 artifact store to use the AWS region in the service connector Fix linter Improve zenml stack deploy CLI Update cloud formation template Fix cloud formation template Move cloud stack deployment logic behind the REST API Re-add local store check Use HTTPS for the server URL Fix stack link in zenml stack deploy CLI Fix docstring errors Fixed more docstrings Apply suggestions from code review Co-authored-by: Barış Can Durak <36421093+bcdurak@users.noreply.github.com> Co-authored-by: Michael Schuster <schustmi@users.noreply.github.com> Add labels to full stacks and use them in event metadata and to detect deployed stacks Update deployment scripts Add templates publish github action Fix bugs Rename stack labels to start with zenml: Fix service connector labels to be strings Fix linter errors Fix component labels in CLI and add stack labels to CLI Fix deployed stack detection to account for new labels Move AWS CF templates to their own subdir

stefannica requested review from avishniakov and bcdurak June 28, 2024 17:27

stefannica requested a review from schustmi June 28, 2024 20:00

stefannica force-pushed the feature/prd-482-one-click-stacks branch 3 times, most recently from 5c3b989 to 1419d71 Compare July 2, 2024 19:40

bcdurak requested changes Jul 3, 2024

View reviewed changes

stefannica force-pushed the feature/prd-482-one-click-stacks branch from 50ebeb7 to 68404fc Compare July 3, 2024 13:01

stefannica added the run-slow-ci label Jul 3, 2024

schustmi reviewed Jul 3, 2024

View reviewed changes

stefannica force-pushed the feature/prd-482-one-click-stacks branch from 76a42cd to 5c44727 Compare July 3, 2024 16:43

stefannica force-pushed the feature/prd-482-one-click-stacks branch from 59df59e to a768870 Compare July 3, 2024 16:48

stefannica requested review from schustmi and bcdurak July 3, 2024 16:48

stefannica force-pushed the feature/prd-482-one-click-stacks branch 3 times, most recently from f22eb4b to 6804752 Compare July 5, 2024 10:46

bcdurak approved these changes Jul 5, 2024

View reviewed changes

Base automatically changed from feature/PRD-477-stack-wizard-cli to develop July 5, 2024 13:03

stefannica force-pushed the feature/prd-482-one-click-stacks branch from 713d44d to 62f63f1 Compare July 5, 2024 13:50

stefannica merged commit b736d25 into develop Jul 5, 2024
55 of 73 checks passed

stefannica deleted the feature/prd-482-one-click-stacks branch July 5, 2024 15:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

In-browser assisted full cloud stack deployments #2816

In-browser assisted full cloud stack deployments #2816

stefannica commented Jun 28, 2024 •

edited

Loading

coderabbitai bot commented Jun 28, 2024 •

edited

Loading

Review skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

Documentation and Community

stefannica commented Jun 28, 2024

bcdurak left a comment

schustmi left a comment

schustmi Jul 3, 2024

stefannica Jul 3, 2024

schustmi Jul 3, 2024

stefannica Jul 3, 2024

schustmi Jul 3, 2024

stefannica Jul 3, 2024

schustmi Jul 3, 2024

stefannica Jul 3, 2024

schustmi Jul 3, 2024

stefannica Jul 3, 2024

schustmi Jul 3, 2024

stefannica Jul 3, 2024

github-actions bot commented Jul 3, 2024

github-actions bot commented Jul 3, 2024

In-browser assisted full cloud stack deployments #2816

In-browser assisted full cloud stack deployments #2816

Conversation

stefannica commented Jun 28, 2024 • edited Loading

Describe changes

Side changes

Make the zenml repository name configurable in the container registry stack component

Use the region from the AWS service connector in the S3 artifact store

Deprecations

Pre-requisites

Types of changes

coderabbitai bot commented Jun 28, 2024 • edited Loading

Review skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

Documentation and Community

stefannica commented Jun 28, 2024

bcdurak left a comment

Choose a reason for hiding this comment

schustmi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jul 3, 2024

github-actions bot commented Jul 3, 2024

stefannica commented Jun 28, 2024 •

edited

Loading

Make the `zenml` repository name configurable in the container registry stack component

coderabbitai bot commented Jun 28, 2024 •

edited

Loading

CodeRabbit Configration File (`.coderabbit.yaml`)