-
-
Notifications
You must be signed in to change notification settings - Fork 33.9k
bpo-46799: Improve multiprocessing shared_memory ShareableList #31467
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
Hello, and thanks for your contribution! I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA). CLA MissingOur records indicate the following people have not signed the CLA: For legal reasons we need all the people listed to sign the CLA before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue. If you have recently signed the CLA, please wait at least one business day You can check yourself to see if the CLA has been received. Thanks again for the contribution, we look forward to reviewing it! |
|
@tcl326 Could you sign the new CLA by clicking |
| :class:`multiprocessing.shared_memory.ShareableList` performance by merging | ||
| the area in shared memory dedicated to offsets and packing formats together. | ||
| This allows a single :func:`struct.unpack_from` call to retrieve both the | ||
| offset and the packing format of a sinlge entry Fix UnicodeDecodeError with |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
small typo sinlge
This PR improves ShareableList's memory usage and performance by consolidating the metadata of each items into a single area in the shared memory block. This allow us to fetch the metadata needed to read and set item using a single
struct.unpack_fromcall. Additionally, this MR also removesself._allocated_offsetsthe offsets can be obtained directly from the shared memory and there is no need to keep a copy in ShareableList. Further more,self._allocated_offsetscan has a significant memory footprint when the number of items in the list is large. I am seeing a significant savings in memory footprint and increase in performance in a benchmark where a ShareableList with a list of 10,000,000 zeros is forked by 5 child processes, and each child precess iterate through the list on their own.Additionally, this PR also include the fix to #26328, and an additional fix where we properly allocate 8 bytes of memory for a string or byte array of size 8. Previously, we would've allocated 16 bytes for a string of size 8 or byte array of size 8.
Here is the resulting memory usage plotted using https://github.com/pythonprofilers/memory_profiler
Memory consumption of the current ShareableList in multiprocessing

Memory consumption of the ShareableList proposed in this MR

https://bugs.python.org/issue46799