Skip to content

Conversation

@tcl326
Copy link

@tcl326 tcl326 commented Feb 21, 2022

This PR improves ShareableList's memory usage and performance by consolidating the metadata of each items into a single area in the shared memory block. This allow us to fetch the metadata needed to read and set item using a single struct.unpack_from call. Additionally, this MR also removes self._allocated_offsets the offsets can be obtained directly from the shared memory and there is no need to keep a copy in ShareableList. Further more, self._allocated_offsets can has a significant memory footprint when the number of items in the list is large. I am seeing a significant savings in memory footprint and increase in performance in a benchmark where a ShareableList with a list of 10,000,000 zeros is forked by 5 child processes, and each child precess iterate through the list on their own.

Additionally, this PR also include the fix to #26328, and an additional fix where we properly allocate 8 bytes of memory for a string or byte array of size 8. Previously, we would've allocated 16 bytes for a string of size 8 or byte array of size 8.

Here is the resulting memory usage plotted using https://github.com/pythonprofilers/memory_profiler

  • Memory consumption of the current ShareableList in multiprocessing
    shareablelist

  • Memory consumption of the ShareableList proposed in this MR
    fixedshareablelist

https://bugs.python.org/issue46799

@the-knights-who-say-ni
Copy link

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA).

CLA Missing

Our records indicate the following people have not signed the CLA:

@tcl326

For legal reasons we need all the people listed to sign the CLA before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

If you have recently signed the CLA, please wait at least one business day
before our records are updated.

You can check yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

@hugovk hugovk added the performance Performance or resource usage label Feb 21, 2022
@terryjreedy terryjreedy changed the title bpo-46799: Improve ShareableList memory utilization and Performance bpo-46799: Improve multiprocessing shared_memory ShareableList Feb 25, 2022
@ghost
Copy link

ghost commented Mar 7, 2023

All commit authors signed the Contributor License Agreement.
CLA signed

@arhadthedev
Copy link
Member

@tcl326 Could you sign the new CLA by clicking not signed button in the cpython-cla-bot's message, please?

:class:`multiprocessing.shared_memory.ShareableList` performance by merging
the area in shared memory dedicated to offsets and packing formats together.
This allows a single :func:`struct.unpack_from` call to retrieve both the
offset and the packing format of a sinlge entry Fix UnicodeDecodeError with
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small typo sinlge

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants