How to create and deploy AWS Lambda Layer for Python functions

In this article, I will describe how to create layers with Python libraries or custom codes and share them between all your lambdas.

Once you need to use a python library for AWS Lambda you have two options:

1. Deploy your code alongside with necessary libraries — will result in large files to upload (ZIP files), you cannot edit your code in AWS Code editor, and you cannot reuse for different Lambda functions
2. Create layers and reference them in your lambda function

Background

AWS Lambda provides only the standard Python libraries. For using additional libraries, you need to deploy your code together with the required libraries or deploy your code separately from the libraries. For that, you need create a Layer for the libraries.

The intent of this article is to provide a practical guide on how to create and deploy Lambda Layers for Python. I will create a sample layer using the marshmallow library (for JSON serialization), deploy it and test it.

1. Install the Library Locally and create a Zip file

There are two options to create your python libraries:

Option #1 — Create it from your development environment if you prefer to take all the installed libraries:

Your libraries are usually installed under the venv directory (e.g.,`<your project home directory>/venv/lib/python3.8/site-packages`

rename the directory site-packages to python and create the zip file (don’t forget to rename back the directory name to site-packages once the zip file created

cd venv/lib/python3.8
mv site-packages python
zip marshmallow.zip -r ./python
mv python site-packages

Option #2 — Install the specific libraries in a local directory

mkdir python
cd python
pip3 install -t . marshmallow
cd ..
zip marshmallow.zip -r ./python

Note: It’s important that the library will be under the python directory as the lambda layer expects this specific structure. See more details about the layers’ configuration in this link

2. Create your Layer on AWS

Navigate to Lambda -> Layers and Create layer button to get the screen below.

The zip file can be uploaded from your environment or can be copied to s3.

If the zip file was created in EC2, you can upload it to s3. Verify sure you have the IAM role to access s3.

aws s3 cp marshmallow.zip s3://<your s3 path>

Make sure you enter the supported Python versions in the Compatible runtimes box otherwise you won’t be the select the specific layer for your function.

3. Add a Layer to your Lambda function

Navigate to your functions. Under your function name, click on Layers, which will get you to the Layers section

Click on the button Add a layer and select custom layers. In the dropdown list, you should see your layers.

4. Test your function

Edit your function, and add the import statement to your function code as below:

Deploy the function and test it.

Common Issues

• If you cannot find your layer in the dropdown layers list while defining it for your function, make sure your compatible runtime was selected during the upload time and that your function is defined with the same runtime (e.g., python3.8)
• If the the zip file wasn’t packed with the correct structure, AWS won’t find your libraries. The structure has to be created as following:

• Some Python libraries installed pre-compiled code (cpython). In case you build you layer on a different OS like MacOS, you you’ll get the following error message when executing the function:

ImportError: /var/task/cryptography/hazmat/bindings/_constant_time.so: invalid ELF header

To avoid such error, it’s better to create the layer on a EC2 with AWS Linux OS.