Ok, lets see if we can make this script a little better. To do that we are going to exploit the use of Co-routines using the Python 3 library async and await . The allows the code to start all the work it can at whatever time it wants to, and then as each "awaited" task pauses for i/o to let another task have control of the process. We've already split the work into as many processes as we have on our host computer, now we will keep all of them busy as possible. I'm using the asyncio.wait(...) function because we probably want to run this as a lambda and we'll want to stop the work before our allotted time runs out and report the results we do have, as well as report that we ran out of time. So that our future selves can either increase the max allotted time, or report it as an error. ''' Sample code to show how to use Multiprocessing and Co-routines. ''' import time from multiprocessing import Process, Pipe, cpu_count import asyncio ...
Posts
Improvement on AWS Python Parallel processing for Lambdas
- Get link
- X
- Other Apps
This post was started after I read https://aws.amazon.com/blogs/compute/parallel-processing-in-python-with-aws-lambda/ by Oz Akan It turns out that article had a mistake in it which our team found through testing. And that is in the article as it appeared in May 2019, it had the "join" before the "recv" which is backward. https://docs.python.org/3.7/library/multiprocessing.html I'm noting that up front because I know most of you won't read this whole post and instead skim it for the bits you actually think you need. Using python on AWS with the lambda functions is one of the many alternative languages that AWS supports. There are a few cavets to getting high performance from your scripts. One is that multiprocessing is not fully supported the way it is in CPython. If you want to use multiprocessing.Queue or multiprocessing.Pool on AWS Lambda, you are going to get the exception: 1 2 3 4 [Errno 38] Function not implemented: OSErr...