Optimizing cold starts

Recently I set out to improve cold starts for our AWS Lambda functions. On average, they took around 4 seconds to initialize, with some exceeding 10 seconds. We have around a hundred container-based Python Lambda functions behind our API gateway, and none of them perform computationally intensive work. Their runtime performance is almost always limited by database operations, which made these cold start times particularly surprising.

The first thing I looked at was improving docker image size. Our lambdas were on average around 2GB, and lambdas that needed some rust based packages were 4-5GB. Seeing image sizes this large, I was convinced reducing them would noticeably improve cold starts. I pruned down the images to 500MB of Python runtime + dependencies. In total around 750MB for every image. Our images were huge because they needed additional tooling to build the dependencies, but they were left in the final image. Lesson learned to use multi-stage builds from the start.

However, to my surprise, there was no significant improvement to the cold starts. It seems AWS really did a good job to prevent cold starts scaling linearly with docker image sizes.

The next thing I tried was increasing the memory of the lambda. Lambda cpu scales with memory and I increased memory from 512MB to 1024MB. At this point I was sceptical about the results. There were no significant improvements again.

The biggest improvement came from lazily importing heavy modules. During a cold start, Python executes all module-level imports before your handler is invoked. Every imported dependency adds to the initialization time.

Lazy imports don't come for free. They trade slightly higher latency on the first use of an import for lower Lambda initialization time.

In practice, that meant - instead of importing heavy dependencies at module level, we imported them only when used.

What it looks like in practice

# Before

# shared/ethereum.py
from web3 import Web3

class EthereumClient:
    ...

# handler.py
def handler(event, context):
    client = EthereumClient()
    ...

# After

# shared/ethereum.py
class EthereumClient:
    def __init__(self):
        from web3 import Web3
        ...

# handler.py
def handler(event, context):
    client = EthereumClient()
    ...

How to measure import time

First of all, build your docker image. Once the image is built:

1. docker run --rm -it --entrypoint /bin/bash DOCKER_IMAGE_NAME
2. python -X importtime -c "HANDLER_PATH" 2>&1 | sort -t'|' -k2 -rn | head -30

That will print out import times for each module sorted by time.

Once you are happy with the import times, you can run your lambdas and check init duration on CloudWatch:

SOURCE logGroups(namePrefix: [], class: "STANDARD") START=-604800s END=0s |
fields @timestamp, @message, @initDuration, @duration, @logStream
  | filter @type = "REPORT"
  | filter ispresent(@initDuration)
  | sort @timestamp desc
  | limit 10000
  | filterIndex @aws.region in ["eu-west-1"]

Results

Optimizing imports reduced average cold starts by roughly 2.5 seconds. The reason imports hit us this hard on average was the following:

1. Each of our APIs uses a single dedicated docker image for that api
2. Each endpoint uses the dedicated docker image with a different run command
3. If one module introduces a top level heavy import, all lambdas that import the module pay the initialization price
4. It also means that our lambda functions don't share warm environments, as they are different lambda functions

While we saw some significant improvements, there is still room for improvement. Further tuning of when modules are imported could improve our init phase further. For some of the APIs, we also added provisioned concurrency, which keeps a defined set of functions warm. In one of the next blogs I will write about profiling and monitoring lambda performance.

Takeaway:

• Image size: 2 GB -> 750 MB -> negligible improvement
• Memory: 512 MB -> 1024 MB -> negligible improvement
• Import optimization -> ~2.5 s reduction in average cold start