Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(lambda-python-alpha): use function architecture (#18696) #28449

Merged
merged 2 commits into from
Dec 21, 2023

Conversation

lucacucchetti
Copy link
Contributor

With this change, architecture when bundling is inferred from the target architecture of the Lambda function.

Closes #18696.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@aws-cdk-automation aws-cdk-automation requested a review from a team December 21, 2023 11:18
@github-actions github-actions bot added beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK bug This issue is a bug. effort/small Small work item – less than a day of effort p1 labels Dec 21, 2023
Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

With this change, architecture when bundling is inferred from the target architecture of the Lambda function.

Closes aws#18696.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@lucacucchetti lucacucchetti force-pushed the lcucche/python-lambda-architecture branch from 80f8838 to 9520125 Compare December 21, 2023 11:36
@lucacucchetti
Copy link
Contributor Author

lucacucchetti commented Dec 21, 2023

Hello, I tried adding an integration test but perhaps predictably bundling failed with exec format error:

 > [2/2] RUN     python -m venv /usr/app/venv &&     mkdir /tmp/pip-cache &&     chmod -R 777 /tmp/pip-cache &&     pip install --upgrade pip &&     mkdir /tmp/poetry-cache &&     chmod -R 777 /tmp/poetry-cache &&     pip install pipenv==2022.4.8 poetry==1.5.1 &&     rm -rf /tmp/pip-cache/* /tmp/poetry-cache/*:
1.204 exec /bin/sh: exec format error
------
Dockerfile:20

...

Command: docker build -t cdk-e0fb70af5dd7d3ee8ee52bbcc32ec617017cebac8121a5ccbc396830b855b75f --platform "linux/arm64" --build-arg "IMAGE=public.ecr.aws/sam/build-python3.11" "/workspace/aws-cdk/packages/@aws-cdk/aws-lambda-python-alpha/lib"
    at dockerExec (/workspace/aws-cdk/packages/aws-cdk-lib/core/lib/private/asset-staging.js:187:15)

I tested this in a GitPod environment (from the contribution guide) which I thought would closely resemble the automated PR Linter:

uname -m
x86_64

Please let me know if I should still pursue making an integ test for this situation.

ExemptionRequest Clarification Request

@aws-cdk-automation aws-cdk-automation added pr/reviewer-clarification-requested The contributor has requested clarification on feedback, a failing build, or a failing PR Linter run pr/needs-maintainer-review This PR needs a review from a Core Team Member labels Dec 21, 2023
Copy link
Contributor

@kaizencc kaizencc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am going to have this supersede #28443 because it includes the unit test asked for there. Thanks both @lucacucchetti and @mapk-amazon, I assume the important thing is to get this in.

@lucacucchetti
Copy link
Contributor Author

Sorry about that, since I mentioned this on this issue a while back and saw no activity assumed no one was working on it.

@kaizencc do we need to get the pr-linter/exempt-integ-test label in for the Linter to pass?

@sumupitchayan sumupitchayan added pr-linter/exempt-integ-test The PR linter will not require integ test changes and removed pr/reviewer-clarification-requested The contributor has requested clarification on feedback, a failing build, or a failing PR Linter run labels Dec 21, 2023
@aws-cdk-automation aws-cdk-automation dismissed their stale review December 21, 2023 21:30

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

Copy link
Contributor

mergify bot commented Dec 21, 2023

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: f74728c
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit c724d27 into aws:main Dec 21, 2023
9 checks passed
Copy link
Contributor

mergify bot commented Dec 21, 2023

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

paulhcsun pushed a commit to paulhcsun/aws-cdk that referenced this pull request Jan 5, 2024
…28449)

With this change, architecture when bundling is inferred from the target architecture of the Lambda function.

Closes aws#18696.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@Zordrak
Copy link

Zordrak commented Feb 15, 2024

I would have loved some clear documentation here. This is a major pipeline breaking change for pipelines that are developing, building and testing lambdas in x86_64 environments, but deploying the lambdas to ARM.

I am now trying to find a way to reproduce the original behaviour, but I am struggling for a viable solution.

@lucacucchetti
Copy link
Contributor Author

I sympathise, in the original issue I commented something to that effect:

I think the contract should change, architecture should be an attribute of BundlingOptions (since it is possible in some situations to want to target one architecture while building in another), regardless, the current behaviour is not what the docs describe and so I would call it a bug.

The documented behaviour does say "platform based on the target architecture of the Lambda function".

For your case, can you do the following?

super(scope, id, {
      runtime: Runtime.PYTHON_3_11,
      bundling: {
          architecture: Architecture. X86_64,
      } as any,
      architecture: Architecture.ARM_64,
      ...

If you check the diff, bundling overrides the default of the lambda architecture.

@jbschooley
Copy link

jbschooley commented Feb 17, 2024

This change broke my pipeline as well. I'm using this for layers:

bundling: {
    image: cdk.DockerImage.fromRegistry('public.ecr.aws/sam/build-python3.11:latest-arm64'),
}

to force it to use the arm64 image to build.

If I take that out and add architecture: Architecture.X86_64 it then builds with the x86 image, installing the x86 builds of dependencies (which then break my functions).

If I set architecture: Architecture.ARM64 then it passes in --platform "linux/arm64" but tries to build with the x86 image and fails.

Both architecture: Architecture.ARM64 and setting the arm64 image, it uses the image but does not pass in --platform "linux/arm64" and fails with

WARNING: The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested
exec /usr/bin/bash: exec format error

adding platform: 'linux/arm64' to bundling does nothing.

Basically, my arm64 build seems to be broken and I have no way to update my layers without moving them, and all the functions they are used in, to x86.

@lucacucchetti
Copy link
Contributor Author

Just to confirm, you are talking about a problem while building layers and not lambdas?

This change did not update any layer related code or behaviour.

@jbschooley
Copy link

Correct. I don't use this to build functions, I keep all my dependencies in layers. If this change didn't update layer related code then I'm not sure what happened because my layer building pipeline is broken in the same way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK bug This issue is a bug. effort/small Small work item – less than a day of effort p1 pr-linter/exempt-integ-test The PR linter will not require integ test changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

(lambda-python): arm64 architecture is not respected
6 participants