[aws-lambda-python] Allow the use of CodeArtifact

[mrpackethead]

Allow the use of CodeArtifact, so that private modules can be loaded from a private repository

Use Case

Many librarys are private, which get used to build lambdas

Proposed Solution

pass parameters for the code-artifact repo to the container.

This is a 🚀 Feature Request

[PierreKiwi]

Thanks @mrpackethead (I suggest you change from code-artifact to CodeArtifact ;)).

The issue #9942 seems to be related as well.

To reiterate what I have shared on Slack, I have solved the problem with Function by first configuring pip using the login command and then I mounted a volume to the Docker container to make my pip.confg available during the build.

my_lambda = aws_lambda.Function(
    self,
    "MyLambdaFunction",
    code=aws_lambda.Code.from_asset(
        "path/to/lambda",
        asset_hash_type=core.AssetHashType.SOURCE,
        bundling=core.BundlingOptions(
            image=aws_lambda.Runtime.PYTHON_3_8.bundling_docker_image,
            command=[
                "bash",
                "-c",
                " && ".join(
                    [
                        "pip install -r requirements.txt -t /asset-output",
                        "cp -au . /asset-output",
                    ]
                ),
            ],
            user="root:root",
            volumes=[
                core.DockerVolume(
                    container_path="/root/.config/pip/pip.conf",
                    host_path=f"{Path.home()}/.config/pip/pip.conf",
                ),
            ],
        ),
    ),
)

[mrpackethead]

maybe this just needs a way to provide a pip config ( it could sit next to the requirements.txt ) to the bundling container.

[eladb]

@adamelmore thoughts?

[adamdottv]

@adamelmore thoughts?

@eladb Makes sense to me! I'll wait until #9582 lands as it makes sweeping changes to PythonFunction. Then I should have some bandwidth to take this on.

[lucasam]

👍 Just add one more customer interested on this feature

[eikeon]

@adamelmore thoughts?

@eladb Makes sense to me! I'll wait until #9582 lands as it makes sweeping changes to PythonFunction. Then I should have some bandwidth to take this on.

I believe #9582 has landed. Looking forward to this functionality. TY, all for the work on the CDK

[mrpackethead]

adding to that, I've changed jobs, but the requirment has'tn chagned... @PierreKiwi 's work ( above ) is helpful but it woudl be nice to do this more tiderly.

[mrpackethead]

Related to this ticket is

#11296

I posted a workaround detailing how to use your own ECR

[sumpfork]

To note, we hacked around the lack of direct support a little differently: we rewrite the requirements file from a template at deployment time and insert the correct codeartifact pypi index via --index-url <codeartifact_url> at the top before using it during lambda packaging in docker.

[setu4993]

I started off with #15306 to really address this, but only later realized it wouldn't be enough to add support for buildArgs to the construct, and that it'll also need an updated build image.

One potential alternative is to allow specifying a custom build image specifically for the Python Lambdas, and pass in CodeArtifact auth using build args. That requires being able to set args in the build image. This type of a setup would also allow using the various packaging tools like poetry. However, it does mean the build image would be duplicated in the local project, separate from the one here (but derived from it).

More concretely, what I'm suggesting is:

a) A build Dockerfile like:

# The correct AWS SAM build image based on the runtime of the function will be
# passed as build arg. The default allows to do `docker build .` when testing.
ARG IMAGE=public.ecr.aws/sam/build-python3.7
FROM $IMAGE

# <-- additional args begin
ARG POETRY_HTTP_BASIC_CODEARTIFACT_USERNAME
ARG POETRY_HTTP_BASIC_CODEARTIFACT_PASSWORD
# additional args end -->

# Ensure rsync is installed
RUN yum -q list installed rsync &>/dev/null || yum install -y rsync

# Upgrade pip (required by cryptography v3.4 and above, which is a dependency of poetry)
RUN pip install --upgrade pip

# Install pipenv and poetry so we can create a requirements.txt if we detect pipfile or poetry.lock respectively
RUN pip install pipenv poetry

# Install the dependencies in a cacheable layer
WORKDIR /var/dependencies
COPY Pipfile* pyproject* poetry* requirements.tx[t] ./
RUN [ -f 'Pipfile' ] && pipenv lock -r >requirements.txt; \
    [ -f 'poetry.lock' ] && poetry export --with-credentials --format requirements.txt --output requirements.txt; \
    [ -f 'requirements.txt' ] && pip install -r requirements.txt -t .;

CMD [ "python" ]

b) Then, the auth could be passed in from local env vars or by running the subprocess command inline into the PythonFunction as:

new PythonFunction(this, 'MyFunction', {
  entry: '/path/to/my/function', // required
  index: 'my_index.py', // optional, defaults to 'index.py'
  handler: 'my_exported_func', // optional, defaults to 'handler'
  buildDockerfile: "/path/to/local/build/Dockerfile",
  buildArgs: {
        POETRY_HTTP_BASIC_CODEARTIFACT_USERNAME: process.env.POETRY_HTTP_BASIC_CODEARTIFACT_USERNAME!,
        POETRY_HTTP_BASIC_CODEARTIFACT_PASSWORD: process.env.POETRY_HTTP_BASIC_CODEARTIFACT_PASSWORD!
  }
});

This would also make it more inline with the NodeJsFunction, which allows passing in a build image that is different from the default one.

This also allows flexibility in how args are passed in and interpreted without tying it to a single packaging solution given there's various possible with Python.

I'm happy to take a stab at implementing this to build on what I started in #15306.

[setu4993]

I created a draft PR here to start an implementation: #15324

I simplified my changes suggested above to reuse cdk.DockerImage instead. So, the function call would be similar to:

new PythonFunction(this, 'MyFunction', {
  entry: '/path/to/my/function', // required
  index: 'my_index.py', // optional, defaults to 'index.py'
  handler: 'my_exported_func', // optional, defaults to 'handler'
  buildImage: cdk.DockerImage.fromBuild("/path/to/file", {
    file: "Dockerfile.build",
    buildArgs: {
        POETRY_HTTP_BASIC_CODEARTIFACT_USERNAME: process.env.POETRY_HTTP_BASIC_CODEARTIFACT_USERNAME!,
        POETRY_HTTP_BASIC_CODEARTIFACT_PASSWORD: process.env.POETRY_HTTP_BASIC_CODEARTIFACT_PASSWORD!
    }
  }
});

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.