Improvement on AWS Python Parallel processing for Lambdas
This post was started after I read https://aws.amazon.com/blogs/compute/parallel-processing-in-python-with-aws-lambda/ by Oz Akan It turns out that article had a mistake in it which our team found through testing. And that is in the article as it appeared in May 2019, it had the "join" before the "recv" which is backward. https://docs.python.org/3.7/library/multiprocessing.html I'm noting that up front because I know most of you won't read this whole post and instead skim it for the bits you actually think you need. Using python on AWS with the lambda functions is one of the many alternative languages that AWS supports. There are a few cavets to getting high performance from your scripts. One is that multiprocessing is not fully supported the way it is in CPython. If you want to use multiprocessing.Queue or multiprocessing.Pool on AWS Lambda, you are going to get the exception: 1 2 3 4 [Errno 38] Function not implemented: OSErr...