/lib64/libc.so.6: version `GLIBC_2.18' not found

[Dgame]

I'm building the lambda on Ubuntu with the basic example you've provided in the README.
It builds without any errors but if I upload and test it on aws in crashes with:

{
  "errorType": "Runtime.ExitError",
  "errorMessage": "RequestId: c24d34ab-f4a9-11e8-a9b7-d5cbfb363674 Error: Runtime exited with error: exit status 1"
}

The log output is:

START RequestId: c24d34ab-f4a9-11e8-a9b7-d5cbfb363674 Version: $LATEST
/var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /var/task/bootstrap)
END RequestId: c24d34ab-f4a9-11e8-a9b7-d5cbfb363674
REPORT RequestId: c24d34ab-f4a9-11e8-a9b7-d5cbfb363674	Duration: 61.35 ms	Billed Duration: 100 ms 	Memory Size: 128 MB	Max Memory Used: 12 MB	
RequestId: c24d34ab-f4a9-11e8-a9b7-d5cbfb363674 Error: Runtime exited with error: exit status 1
Runtime.ExitError

[davidbarsky]

Oh man, I'm not sure, and I don't have an Ubuntu instance handy. Can you try building this using musl-libc instead of GNU libc? This would entail adding the x86_64-unknown-linux-musl target using rustup.

[sapessi]

@Dgame I would probably use a docker image to build in local. AWS Lambda uses Amazon Linux 2017.03.1. We are working to add Rust support to SAM build.

[drogus]

I get the same error when compiling on Arch Linux. And it seems that using amazonlinux doesn't help, building with a docker image yields the same result. Here's a Dockerfile that I used: https://gist.github.com/drogus/e16260baa0210a30922a3808a9eed8ff

After building it I just ran the command from the README:

cargo build -p lambda_runtime --example basic --release 

[drogus]

That said, building with musl works correctly (on Arch I installed the musl package and added a x86_64-unknown-linux-musl target)

[sapessi]

Mh, I was able to cargo build successfully and run the basic example with this Dockerfile:

FROM amazonlinux:2017.03.1.20170812
RUN curl https://sh.rustup.rs -sSf | /bin/sh -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"
RUN rustup install stable
RUN yum install -y gcc gcc-c++ make openssl openssl-devel
RUN mkdir /code
WORKDIR /code

I will try to replicate with Nightly next, not sure what's causing your issue yet.

[Dgame]

@Dgame I would probably use a docker image to build in local. AWS Lambda uses Amazon Linux 2017.03.1. We are working to add Rust support to SAM build.

Well, we are already doing that, but there was the slight hope that building it locally would be faster.

Oh man, I'm not sure, and I don't have an Ubuntu instance handy. Can you try building this using musl-libc instead of GNU libc? This would entail adding the x86_64-unknown-linux-musl target using rustup.

I've tried but I get this:

rustup install stable-musl
info: syncing channel updates for 'stable-x86_64-unknown-linux-musl'
info: latest update on 2018-11-08, rust version 1.30.1 (1433507eb 2018-11-07)
error: target not found: 'x86_64-unknown-linux-musl'

[Dgame]

I've tried

cargo build --release --target=x86_64-unknown-linux-musl

but then I get those strange OpenSSL errors which I don't get if I drop --target=x86_64-unknown-linux-musl:

Compiling openssl-sys v0.9.39
error: failed to run custom build command for openssl-sys v0.9.39
process didn't exit successfully: /mnt/lambda/target/release/build/openssl-sys-a16ee2b07ae0e6c5/build-script-main (exit code: 101)
--- stdout
cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_MUSL_OPENSSL_LIB_DIR
cargo:rerun-if-env-changed=OPENSSL_LIB_DIR
cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_MUSL_OPENSSL_INCLUDE_DIR
cargo:rerun-if-env-changed=OPENSSL_INCLUDE_DIR
cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_MUSL_OPENSSL_DIR
cargo:rerun-if-env-changed=OPENSSL_DIR
run pkg_config fail: "Cross compilation detected. Use PKG_CONFIG_ALLOW_CROSS=1 to override"

--- stderr
thread 'main' panicked at '

Could not find directory of OpenSSL installation, and this -sys crate cannot
proceed without this knowledge. If OpenSSL is installed and this crate had
trouble finding it, you can set the OPENSSL_DIR environment variable for the
compilation process.

Make sure you also have the development packages of openssl installed.
For example, libssl-dev on Ubuntu or openssl-devel on Fedora.

If you're in a situation where you think the directory should be found
automatically, please open a bug at https://github.com/sfackler/rust-openssl
and include information about your system as well as this message.

$HOST = x86_64-unknown-linux-gnu
$TARGET = x86_64-unknown-linux-musl
openssl-sys = 0.9.39

', /home/dgame/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-sys-0.9.39/build/main.rs:269:9
note: Run with RUST_BACKTRACE=1 for a backtrace.

warning: build failed, waiting for other jobs to finish...
error: build failed

Any ideas?

[sapessi]

@Dgame Are you compiling on a Mac or in a container? In a Mac, you get these errors because the version of OpenSSL and source you need are not available or the compiler/linker cannot find them. There's a pretty good explanation on StackOverflow here. I would recommend using a docker container for building with the Dockerfile from my previous answer.

[drogus]

@Dgame I'm on ARCH so I'm not sure if it helps you in any way, but I had to specify openssl paths and allow cross compilation:

OPENSSL_LIB_DIR=/usr/lib/openssl-1.0
OPENSSL_INCLUDE_DIR=/usr/include/openssl-1.0
PKG_CONFIG_ALLOW_CROSS=1

[Dgame]

@sapessi
I'm using Ubuntu 18.04 and only get those pesky ssl errors if I append --target=x86_64-unknown-linux-musl. Without it, it compiles fine but will crash with the error message you can see in my initial post.

@drogus
Thanks, I'll try that next. :)

[sapessi]

On thing I should have added @Dgame, is that if you use the Amazon Linux Docker container to build then you don't have to specify a --target option in the build command.

[dbcfd]

Until some of the command line tools finish, you may want to use the docker and cargo_config from https://github.com/srijs/rust-aws-lambda/tree/master/docker

More information on setting up environment here: https://github.com/srijs/rust-aws-lambda/blob/master/docs/getting-started.md#Deploy

[marcomagdy]

@Dgame you could also try using the C++ runtime. It deals with this issue seamlessly ;) you can build on Ubuntu, ArchLinux, Alpine, etc.

[Dgame]

@marcomagdy No, thanks. ;) I've already written the code in rust. It's just about the build time. With Docker it takes ~15 min, without <4min.

[cedws]

This is quite painful. I've spent a good few hours trying to get all of the required musl tools and musl Rust target to compile, and now I'm having issues getting openssl-sys to compile on musl. I'd rather not build inside Docker if possible.
@sapessi You mentioned SAM build - will that resolve these difficulties? Thanks.

[softprops]

You don't actually have to wait for SAM.

If you are a serverless framework user you may want to check out the rust plugin It provides a more seemless build experience as it uses the lambdaci project's "provided" build env ( a faithful replication of the same env you will be deploying into ).

If you aren't a SAM user you can also just use the a replicated build env, the lambdaci project provides this for custom runtimes. Here's a published extension of that with the rust toolchain pre-installed https://github.com/softprops/lambda-rust#-usage

[sapessi]

@c-edw SAM build will automatically use docker to build the app. If you'd rather not use docker at all, your alternatives are:

1. Use an amazon linux EC2 instance (with the same AMI as Lambda)
2. Use homebrew to install the required openssl libs - few good pointers in this issue

[saethlin]

From my experience, --target=x86_64-unknown-linux-musl is only half the battle because openssl is a bit of a bugger to configure and link. To handle that, you can add this to your Cargo.toml dependencies:

openssl = { version = "0.10", features = ["vendored"] }

The feature will download and compile openssl in your project's target/ directory, with settings appropriate to the target you're building for. The only possible complication is that you may need to also use the openssl-probe crate and call openssl_probe::init_ssl_cert_env_vars(); immediately in your main. Maybe this should be suggested in this project's README?

[sapessi]

Using the vendored SSL is the easiest way to build this. I'm working with the SAM CLI folks to have a sam build command that will work for rust functions and build the project in a container automatically.

[darrell-roberts]

I had all the same issues. I was building on a mac. I went the docker build route after running into OpenSSL linking issues.

I had to use the lambci/lambda:build-provided image instead. The image mentioned here resulted in SSL errors when I ran the lambda function in AWS.

/var/task/bootstrap: /lib64/libcrypto.so.10: version `OPENSSL_1.0.2' not found (required by /var/task/bootstrap)

Here is my Dockerfile

FROM lambci/lambda:build-provided
RUN curl https://sh.rustup.rs -sSf | /bin/sh -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"
RUN rustup install stable
# RUN yum install -y gcc gcc-c++ make openssl openssl-devel
RUN mkdir /code
WORKDIR /code

Then I ran

docker build . -t lambda_builder

docker run --rm -v ${PWD}:/code -v ${HOME}/.cargo/registry:/root/.cargo/registry -v ${HOME}/.cargo/git:/root/.cargo/git lambda_builder cargo build --release
zip -j lambda.zip target/release/bootstrap

[davidbarsky]

Since most of my day-to-day work is on a Mac, I've used Cross (which recently merged support for macOS!) to build the Lambda function.

$ cargo install --force --git https://github.com/rust-embedded/cross # this pulls from master, only needed once
$ cross build --release --target x86_64-unknown-linux-musl

Note: Cross has a Docker dependency. I've been using Docker Desktop for Mac pretty happily.

[demerzel3]

Since most of my day-to-day work is on a Mac, I've used Cross (which recently merged support for macOS!) to build the Lambda function.

Saved my day! Cross should be mentioned in the README, worked flawlessly.

[konstantinvlasenko]

Hi everyone,

Having the same issue.

• My build environment is AWS EC2 instance (Amazon Linux 2 AMI 2.0.20190313 x86_64 HVM ebs).

• Lambda fails (after successful deployment) with the same error /var/task/bootstrap: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /var/task/bootstrap)

[davidbarsky]

I think the takeaway of this issue is that this project has to built against musl, as the glibc on the Lambda sandbox is too old for Rust.

[konstantinvlasenko]

@davidbarsky sorry for my ignorance. What your statement "has to built against musl" means? Can I achieve that by using Ubuntu, Debian or Amazon Linux by installing the special packages? Or/And I need to change some configuration of a system or a project?

Thank you.

[davidbarsky]

@konstantinvlasenko Sorry, that's my bad! Nothing to apologize for: I was curt in my previous message. You'll need to build against the x86_64-unknown-linux-musl target, which statically links against the musl-libc. You can install musl-libc on Ubuntu/Debian/Amazon Linux; some of the instructions on https://wiki.debian.org/musl#Installation cover that. You'll also need to add the x86_64-unknown-linux-musl target via Rustup: https://doc.rust-lang.org/edition-guide/rust-2018/platform-and-target-support/musl-support-for-fully-static-binaries.html.

All that being said, I am primarily a macOS user (with the occasional Amazon Linux usage), so I can't really tell you how to install Musl on a Debian-based system. Personally, I build everything via a Docker image.

[davidbarsky]

(I also apologize if I over-explained anything to you!)

[konstantinvlasenko]

Works! Works! Works!
@davidbarsky Thank you very much!

So in my case after following all the steps from here https://aws.amazon.com/blogs/opensource/rust-runtime-for-aws-lambda/ I was missing this one command in my Ubuntu instance:

apt-get install musl

[chumaumenze]

@davidbarsky I am currently facing this issue. I initially tried building with musl but the problem is that some crates rely heavily on C++ libraries. Static linking to the libraries definitely require some extra work (time) as described in rust-lang/rust#36710 (comment).

Is there another way to resolve this?

[davidbarsky]

@chumaumenze That makes sense. Trying to build crates with C/C++ dependencies isn't trivial. Unfortunately, there isn't anything that you can do at the moment. To properly resolve this issue, the runtime environment needs to be updated to use Amazon Linux 2, which has a significantly more recent glibc. I'm unable to share a date for when that work will be completed, but I'll let you know when that update has launched.

[rimutaka]

@softprops , would it be better to replace

Since Lambda uses Amazon Linux, you'll need to target your executable for an x86_64-linux platform.

with

Since Lambda uses Amazon Linux, you'll need to target your executable for x86_64-unknown-linux-musl platform.

in the ReadMe (https://github.com/awslabs/aws-lambda-rust-runtime#aws-cli)?

I managed to get my Lambda going after targeting -musl.

[softprops]

That doc is still true but when expanded, you've got options.

For local builds musl seems reliable with the tradeoff I'm warming up to of making sure your os or the one your building on has all of the musl prerequisites manually installed. This varies by os.

This is the approach I'm trying to get sam team to accept aws/aws-lambda-builders#174 (available someday maybe)

And one I'm experimenting with in the serverless framework https://github.com/softprops/serverless-rust/blob/master/README.md#-experimental-local-builds (available today)

The alternative is to build on a container that matches the lambda runtime. The tradeoff here is that assumes docker. Docker is most likely already present on many dev envs but comes with some limitations of what is preinstalled in that docker image https://github.com/softprops/lambda-rust. (available today)

[brainstorm]

I'd actually close this issue and direct folks to provided.al2, all the MUSL tooling is not required anymore... unless there's an edge usecase I'm not aware of nowadays?

I'm looking into building a docker container for rust and lambda that can work well with SAM CLI (and Apple Silicon trying to cross-compile ring) in aws/aws-sam-cli#3132 (comment).... so that local builds and sam local start-api works reliably for Rust.

[coltonweaver]

Hey, I've added references to provided.al2 in the README now. Closing this issue.