Module initialization error in AWS Lambda

[tmilicic]

When trying to use snappy inside AWS Lambda I am getting:
module initialization error: [Errno 2] No such file or directory: '/var/task/__pycache__/_cffi__x762f05ffx6bf5342b.c'

[martindurant]

I'm afraid I know nothing about installation in Lambda, why it might be different from other architectures. With luck, other people will be able to comment.

[tmilicic]

Ok thank you, I will no longer need this since I am just gonna use EC2 instance but it would be interesting to see why is this happening.

[andrix]

@tmilicic I don't have the opportunity to use AWS Lambda to debug it, if you have the chance you can post the results here. I believe that others can help too, with more debug information.

In the meantime, I will close the issue until having further information.

[pez5001]

I'm also running into this issue in AWS Lambda, managed to isolate it to just the snappy import.

This is an AWS Lambda function running python 3.6, with python-snappy==0.5.1 installed.
Interestingly, I can't reproduce it on the amazon linux instance we use to build the lambda package, (including when I set the working directory to read-only), but only when I deploy it and invoke it in Lambda.
I was able to hack in to my entry point a way to print the exception out to cloudwatch (otherwise the bug blows up the process before logging can even get set up) so please excuse the formatting, but here's the exception that gets raised when I try to call import snappy

[Errno 30] Read-only file system: '/var/task/__pycache__/_cffi__x70d2a14ex6bf5342b.c'
Traceback (most recent call last):
File "/var/task/snappy.py", line 47, in <module>
from _snappy import UncompressError, compress, decompress, \
ImportError: libsnappy.so.1: cannot open shared object file: No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/task/lambda_handler.py", line 9, in lambda_handler
import snappy
File "/var/task/snappy.py", line 50, in <module>
from snappy_cffi import UncompressError, compress, decompress, \
File "/var/task/snappy_cffi.py", line 169, in <module>
''', libraries=["snappy"])
File "/var/task/cffi/api.py", line 444, in verify
lib = self.verifier.load_library()
File "/var/task/cffi/verifier.py", line 103, in load_library
self._write_source()
File "/var/task/cffi/verifier.py", line 192, in _write_source
with open(self.sourcefilename, "w") as fp:
OSError: [Errno 30] Read-only file system: '/var/task/__pycache__/_cffi__x70d2a14ex6bf5342b.c'

So it looks like cffi is trying to write out that c file into the read-only directory the Lambda function gets run in, which is the source of the permission denied error.

What's especially interesting to me is I've been using snappy successfully in Lambda for a while, with an existing function that's python-snappy==0.5 in python 2.7.

[pez5001]

Actually, it looks like this isn't a bug with this library at all, rather a quirk of implementing in Lambda.

This line in the error output is the clue:
ImportError: libsnappy.so.1: cannot open shared object file: No such file or directory

Including that library in the root of the code package deployed to Lambda solved the issue - it must be that if that's not present, cffi will try to build it itself.