Creating a Lambda layer for Python functions
Why use Lambda layers?
Lambda layers make using Lambda so much easier.
You could put all your code and its dependencies in a single deployment archive. However, chances are you have a lot of libraries -- you may not think so, but they add up -- and redeploying your code with all its libraries every time you make a code change is slow and wasteful.
Plus Lambda itself has a couple of limitations. First, the maximum size of any archive is 50 MB zipped, 250 MB unzipped. It is surprisingly easy to hit those limits.
Second, if you want to make any edits in the AWS console, the limit is much lower, around 3 MB. This means if you want to make a quick test of your code, and it's bigger than 3 MB, you have to redeploy the whole thing. Every time.
A simple solution to these problems is to put all your common libraries in a layer or two, and keep your function lean. You probably don't update your dependencies more than once a month, but you update your code constantly.
Layers still have limitations though. A function can have a maximum of five layers. And the total unzipped size of the function plus all its layers still cannot exceed 250 MB.
Creating a Lambda layer
All packages must be in a .zip file with one directory called python that contains your installed packages.
Building in an AWS Linux container
In addition, it's a good idea to create your Lambda layer in an AWS Linux container. This is to avoid issues with cross-compiled dynamically linked libraries -- basically, those files ending in .so.
Why? For some packages, when you pip install them, it builds an executable for the OS you are currently on. If you build them on a Mac, for example, that executable will be compiled for Darwin and probably won't run on Amazon Linux.
So you can skip to the next section unless:
-
the packages you will be installing into your layer create executables, and
-
you are building on an OS or architecture that doesn't match the AWS instances your Lambda functions will be running on.
Still reading? Ok. We need to build in an environment that matches AWS.
Building in an environment that matches AWS
Fortunately, AWS has created public container images you can use for this. They are available from the AWS public GitHub in the AWS Lambda Base Container Images repo. Note they are based on x86_64 architecture, or in Dockerspeak, linux/amd64.
As the instructions in the README indicate, you will need to install git and docker, which you probably already have, but you will also need git-lfs, which you probably don't.
For git-lfs, the project page gives basic information, and from there you can go to the git-lfs repo for detailed installation instructions.
Then you are ready to pull down the container image you want. For example:
git clone
git checkout python3.9
If you get the error git-lfs: command not found then you forgot to install git-lfs, or git can't find it. (I probably didn't need to point this out.)
You should now have a Dockerfile in your directory. The last line of the Dockerfile will be an ENTRYPOINT command, such as ENTRYPOINT ["/lambda-entrypoint.sh"]. This launches the runtime interface client for Lambda emulation, which we don't need right now. We just want a shell prompt. So edit the Dockerfile and remove or comment out that line.
Now we can build our container:
docker build -t aws-python3.9:local -f Dockerfile.python3.9 .
That creates a Docker image called aws-python3.9:local which we will use below.
Again, this whole container thing is an optional step, for the reasons described above. But it doesn't add much complexity and can save downstream headaches, so you may find it easier just to do it every time.
The recipe for building the right environment for Lambda
1. Create a directory
First, create a directory to hold everything. Replace my_layername with the name of your layer package:
mkdir -p my-layername/python
2. Create a file with the packages to put in your Lambda layer
Next, create a requirements.txt file with all of the packages you want to put in your layer. Leave it in your my-layername directory. For example, your file could look like this:
aws-lambda-powertools>=1.22.0
boto3>=1.17.53
3. Run Docker container (optional)
Then run your Docker container (optional, as described above). Replace my_layername with the name of your layer:
docker run --name lambdalayer --rm --env HTTP_PROXY --env HTTPS_PROXY --env NO_PROXY --mount type=bind,source="$(pwd)"/my-layername,target=/var/task/lambdalayer -it aws-python3.9:local bash
4. Put everything into the Python directory
Now install everything into the python directory:
cd lambdalayer
cd python
pip3 install --target=. -r ../requirements.txt
5. Remove unnecessary files from the layer package to reduce its size (optional)
Optional, but recommended -- remove unneeded files to reduce the size of your layer package. Note the last exit is only if you are running this inside a container:
rm -rf *.dist-info
find . -name "tests" -type d | xargs -I{} rm -rf {}
find . -name "docs" -type d | xargs -I{} rm -rf {}
find . -name "__pycache__" -type d | xargs -I{} rm -rf {}
rm -rf boto*
exit
Yes, you can remove Boto! As of a couple of years ago, it's always available in Lambda. This saves a lot of space.
Back on host computer:
cd my-layername
zip -r my-layername.zip python
6. Uploading the layer
You can either use the AWS console to upload the layer, or the CLI. Uploading a layer using the console is easy. Just go to the Lambda service, choose Layers, click on Create Layer, then follow the prompts.
I prefer using the CLI. You can use the following command, replacing my-layername with your own layer name. The Python runtimes are optional:
aws lambda publish-layer-version --layer-name my-layername --zip-file fileb://my-layername.zip --compatible-runtimes "python3.7" "python3.8" "python3.9"
You should get a response like the following:
{
"Content": {
"Location": "https://something.s3.us-east-1.amazonaws.com/snapshots/something/my-layername",
"CodeSize": 27940
},
"LayerArn": "arn:aws:lambda:us-east-1::layer:my-layername",
"LayerVersionArn": "arn:aws:lambda:us-east-1::layer:my-layername:5",
"Description": "",
"CreatedDate": "2022-01-11T15:34:30.164+0000",
"Version": 5,
"CompatibleRuntimes": [
"python3.7",
"python3.8",
"python3.9"
]
}
Note the layer version (it's 5 in this example). Every time you upload a new copy of your layer, that number will increase. Also note the layer ARN. You will need to give those to your Lambda function (below).
That's it. Your layer is ready for use. Now you can import the packages in your layer into your function. Or functions – you can create a few layers and share them among all your functions.
Well, that requires one more step.
Updating your Lambda function
You need to tell your Lambda function about your layer. In the console, go to the Lambda service, click on Functions and find your function. Then scroll to the bottom to the Layers area and add your layer.
The layer version is important. Make sure you are on the latest version of your layers. (Or you can choose an older version, of course, if you want to test with that.)
A note on layer location in Lambda
As you probably already know, Lambda functions run in the /var/task directory.
However, Lambda unpacks all layers into the /opt directory. So if you have a file in a layer that you need to access, such as a certificate, look in /opt/python.
Also, since all layers are dumped into the same /opt/python directory, multiple layers can and will overwrite each other. This isn't usually a problem but if it is, you can specify the Merge Order in the Lambda function console.
Conclusion
In conclusion, creating a Lambda layer for Python functions can significantly simplify the process of deploying code and managing dependencies, while also overcoming the limitations of Lambda itself. By creating a layer or two containing common libraries, developers can keep their functions lean and easily update their dependencies as needed. When creating a Lambda layer, it is important to ensure that all packages are included in a .zip file with one directory called "python" that contains the installed packages.
Additionally, building the layer in an AWS Linux container can help avoid issues with cross-compiled dynamically linked libraries. By following the steps outlined in this article, developers can create the right environment for Lambda and streamline their development process.