Slow cold-start validation when used in AWS Lambda

[paulf69487623]

Hello,

I've been experimenting with using @simplewebauthn/server in AWS Lambda as part of an AWS Cognito custom challenge setup to add WebAuthn to a normal Cognito user pool. I've gotten it working without too much trouble, due to good documentation and example code. This is a really nice project!

The one issue I've found is that after a lambda cold-start, the verifyAuthenticationResponse method can be slow. I'm timing this method call exclusively. With Node 12.x, it could take over 5 seconds the first time. AWS now recommends using Node 18.x. The results are better, but still slower than I'd expect at about 1-1.2 seconds for the first execution by a particular runtime. In the case of a warm-start, the method takes between 80-100ms.

This seems to be one of the most actively maintained WebAuthn projects, and the combination of the client and server library is fantastic. The 1s time isn't a show stopper, but I'd love to figure out a way to make it faster.

[MasterKale]

I don't have a good answer for you unfortunately. I've suspected that there's a performance issue somewhere, but I've never sat down to try and figure out how to profile execution of the library to see where the bottlenecks are. The best I've been able to do thus far is to create a TODO for myself in a couple of places that I think might be bottlenecks, but I haven't measured anything to know for sure.

For example, there's a bit of certificate chain signature verification in validateCertificatePath() that I've observed is very slow, here:

https://github.com/MasterKale/SimpleWebAuthn/blob/master/packages/server/src/helpers/validateCertificatePath.ts#L120

I think this might be related to why the "apple" attestation verification test can take up to seven seconds to complete. But again, I haven't yet spent time trying to figure out how to measure anything beyond basic use of console.time() so I don't have much data to back up my hunches.

I'm wondering, as a consumer of this library, if there's anything you can do to help with this investigation...

[MasterKale]

So I just achieved tonight a 60% reduction in the amount of time unit tests took to run by trying to tackle that TODO I linked above. My plan now is to clean this up and include it in the upcoming v7.0.0 release. Perhaps this will help with both the cold and warm start times...stay tuned!

[paulf69487623]

That sounds fantastic...looking forward to trying out the new release!

[paulf69487623]

I just wanted to report back on some test results. Your changes definitely improved the situation. Where before I was seeing about 1000ms for the verifyAuthenticationResponse call on a cold start, now it's running around 70-120ms.

[MasterKale]

Wow, that's a huge improvement! I was trying to figure out a benchmark I could use to compare before and after and hopefully have something as drastic as what you were seeing. This blows my "60% faster unit tests" anecdote out of the water 😂

What is your ideal cold boot time for your project? It's been a while since I've worked with lambdas, but I do remember every millisecond counting when it came to billing, especially if you wanted something to scale without blowing up the bank.

[paulf69487623]

Its also possible that it was made worse by a bad combination of options. When I upgraded to HEAD to start timing the different parts of the verify function, it was throwing errors and I had to modify the options to get it working. Its possible I was sending it down a bad code path before.

I don't have any specific performance or cost requirements yet. A little background might make things more clear.

I'm at the super-early stages of experimenting with a potential service that needs strong authentication, and WebAuthn fits the bill perfectly. So far I'm using a fully serverless stack on AWS, with Cognito for authentication and user management, Lambda for API backends, and DynamoDB for backend storage. All of the components are deployed using CDK for infrastructure as code. It all came out of the need for a solution for a personal pain point, but I think there is a market for it. I'm not ready to talk about it in more detail yet. Almost everything is under the free tier thresholds so far.

While exploring WebAuthn, I went looking for anything that was more or less plug-and-play for Cognito. There are other projects out there, but they don't integrate into Cognito or fit in with the serverless approach. I also looked at some of the commercial services, but I'd rather keep everything in one place. There are a number of articles about implementing WebAuthn using the Cognito custom challenge-response workflow, which really helped me get going, but nothing was really designed to be deployable.

That got me thinking about building it as an Open Source CDK construct that could be added onto an existing project already using Cognito. The idea being, pass your Cognito UserPool, your RP parameters, and a few other billing options into the library and it would create the Lambda callbacks for authentication, the credential storage, and the API for managing credentials automatically. I've also been working on an example React component that lets you view, register, and delete credentials.