AWS Lambda — Container Image .NET Benchmark

There were great announcements at AWS re:invent 2020. One of them is the support of container images for AWS Lambda. Besides that, .NET 5 is only available with container images. Standard Lambda functions don’t support .NET 5 yet.

There were new announcements about AWS Lambda at AWS re:invent 2020. You can now use Lambda functions with 10 GB of memory and 6 vCPU cores. AWS also changed the pricing of Lambda. Before, duration was rounded up to the nearest 100ms; the duration is now rounding up to the nearest millisecond with no minimum execution time.

You can deploy Lambda functions as container images of up to 10 GB. AWS also provides base images for all supported Lambda runtimes (Python, Node.js, Java, .NET, Go, Ruby), or you can create your own runtime.

In Kloia, we are supporting businesses to modernize their legacy .NET applications. So I found myself wondering if there is any performance difference between standard Lambda functions and Lambda functions in a container image. Standard Lambda functions support .NET Core 2.1 and .NET Core 3.1. I will also run .NET 5 with Lambda container images.

Architecture & Benchmarking Process

I created the same small function for every framework. Then I deployed API Gateway and integrated it with functions to make these functions reachable via an HTTP request. I created Cloud9 (t2.micro — Ubuntu 18.04) in the same region with functions to make HTTP requests. I used wrk for HTTP benchmarking. Configuration of wrk was set to 2 threads and 10 connections. This configuration lets you initiate ten functions running simultaneously.

AWS CloudWatch and wrk results let us collect and plot the metrics for our benchmark. Tests are made with 256 MB, 512 MB, 1024 MB, 3072 MB, 6144 MB (in wrk tests, 6144 MB wasn’t used in graphs) memory settings.

**In this article (C) stands for ‘Container Image’ and (F) stands for standard Lambda function.

HTTP Benchmark Results

These tests give us information about requests per second and average latency for these requests. You can find the raw data here.

As we see on the graphs, the results are so close to each other. In most of the tests, .NET 5 (C) is %9 better on average according to .NET Core 3.1 (F), and .NET Core 3.1 (F) is slightly worse. Latency got almost its best performance with 3072 MB of memory. I didn’t test higher memories because the latency graph was turning into a straight line. But we can consider this as a limit of the processing power of Lambda because our test functions did not use parallel programming. Lambda functions support 6 cores with 10240 MB of memory so that parallel programming can change performance dramatically.

Size

The size of functions could be important in the Lambda environment because every time a new Lambda function is invoked, this size moves to Lambda servers to run. Here is a table of sizes of container images and functions. These examples contain exactly the same code.

Cold Start

When the Lambda function is triggered for the first time, it takes some time to initialize the function. It changes depending on the runtime and type of function. It is called the “cold-start.”

Cold-start is a concern for most developers. I tested this on .NET functions with different memory sizes.

Cold-start times have huge differences between standard Lambda functions and container images. Standard lambda functions’ cold-time latency is gathered around 200ms, and it can go as low as 147,09 ms with .NET Core 3.1 on 6144 MB memory. Container image cold-time latency is gathered around 1000ms, and it can go as low as 2611,46 ms with .NET Core 2.1 on 1024 MB memory. Standard Lambda function is 5 times faster than container images on cold-start, but they are approaching nearly the same performance level after initiation.

Conclusion

I have performed this benchmark to understand the performance difference between standard Lambda functions and container image functions. C# is one of the best performing Lambda, but one of the worst in cold-time starts. We saw that .NET container images performed as well as standard Lambda functions after cold-time, and .NET 5 had no difference in that performance.

AWS Lambda is one of the great ways for .NET modernization. It supports the latest frameworks like .NET Core and .NET 5, and its performance is quite good.

The example that I use for benchmarking is not a real-world application, but it demonstrates Lambda functions’ performance. I hope this benchmarking helps you meet your performance requirements with .NET runtimes.