A bug in the multiprocessing module

[eshkrig]

BPO	38195
Nosy	@pitrou, @applio, @iritkatriel
Files	testqueue.py: Example of a freezing program

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-09-17.06:51:09.943>
labels = ['extension-modules', 'type-bug', '3.9']
title = 'A bug in the multiprocessing module'
updated_at = <Date 2022-01-25.17:48:43.191>
user = 'https://bugs.python.org/eshkrig'

bugs.python.org fields:

activity = <Date 2022-01-25.17:48:43.191>
actor = 'eshkrig'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2019-09-17.06:51:09.943>
creator = 'eshkrig'
dependencies = []
files = ['48609']
hgrepos = []
issue_num = 38195
keywords = []
message_count = 3.0
messages = ['352613', '411500', '411645']
nosy_count = 4.0
nosy_names = ['pitrou', 'davin', 'eshkrig', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38195'
versions = ['Python 3.9']

[eshkrig]

Hi!

I found a bug in the multiprocessing module.

The documentation (https://docs.python.org/3.6/library/multiprocessing.html) for the Queue.get method says:
"If timeout is a positive number, it blocks at most timeout seconds ..."

But the Queue.get method freezes if the process filling the queue terminates while reading data.

In the attachment is an example of a program (testqueue.py) that freezes after ~ 5-30 seconds.
After pressing <control>-<c>, the output is as follows:
Traceback (most recent call last):
  File "./testqueue.py", line 45, in <module>
    result = ps_queues[cpu].get(timeout=2)
  File "/usr/lib64/python3.6/multiprocessing/queues.py", line 108, in get
    res = self._recv_bytes()
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 411, in _recv_bytes
    return self._recv(size)
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt

Please fix it.

[iritkatriel]

Python 3.6 is no longer maintained and this doesn't freeze on my Mac on 3.11.

Please create a new issue if you are seeing the freeze on a current python version (>= 3.9) and specify which system you are running it on.

[eshkrig]

$ cat /etc/os-release
NAME=Gentoo
ID=gentoo
PRETTY_NAME="Gentoo/Linux"
ANSI_COLOR="1;32"
HOME_URL="https://www.gentoo.org/"
SUPPORT_URL="https://www.gentoo.org/support/"
BUG_REPORT_URL="https://bugs.gentoo.org/"

$ uname -a
Linux lynx 5.16.2-gentoo-1 #1 SMP PREEMPT Fri Jan 21 10:38:04 MSK 2022 x86_64 AMD FX(tm)-8350 Eight-Core Processor AuthenticAMD GNU/Linux

$ python --version
Python 3.9.9

$ ./testqueue.py
queue.get() from pid 11371
queue.get() from pid 11372
^CTraceback (most recent call last):
  File "/home/john/main/123/./testqueue.py", line 45, in <module>
    result = ps_queues[cpu].get(timeout=2)
  File "/usr/lib/python3.9/multiprocessing/queues.py", line 117, in get
    res = self._recv_bytes()
  File "/usr/lib/python3.9/multiprocessing/connection.py", line 221, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib/python3.9/multiprocessing/connection.py", line 426, in _recv_bytes
    return self._recv(size)
  File "/usr/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt

[eshkrig]

$ cat /etc/os-release
NAME=Gentoo
ID=gentoo
PRETTY_NAME="Gentoo Linux"
ANSI_COLOR="1;32"
HOME_URL="https://www.gentoo.org/"
SUPPORT_URL="https://www.gentoo.org/support/"
BUG_REPORT_URL="https://bugs.gentoo.org/"
VERSION_ID="2.13"

$ uname -a
Linux lynx 6.3.6-gentoo-1 #1 SMP PREEMPT_DYNAMIC Fri Jun  9 12:39:50 MSK 2023 x86_64 AMD FX(tm)-8350 Eight-Core Processor AuthenticAMD GNU/Linux

$ python --version
Python 3.11.3

$ ./testqueue.py
queue.get() from pid 11623
queue.get() from pid 11624
queue.get() from pid 11625
^CTraceback (most recent call last):
  File "/home/john/main/123/./testqueue.py", line 45, in <module>
    result = ps_queues[cpu].get(timeout=2)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/queues.py", line 117, in get
    res = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/connection.py", line 215, in recv_bytes
    buf = self._recv_bytes(maxlength)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/connection.py", line 420, in _recv_bytes
    return self._recv(size)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/connection.py", line 378, in _recv
    chunk = read(handle, remaining)
            ^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

[eshkrig]

$ cat /etc/os-release
NAME=Gentoo
ID=gentoo
PRETTY_NAME="Gentoo Linux"
ANSI_COLOR="1;32"
HOME_URL="https://www.gentoo.org/"
SUPPORT_URL="https://www.gentoo.org/support/"
BUG_REPORT_URL="https://bugs.gentoo.org/"
VERSION_ID="2.15"

$ uname -a
Linux lynx 6.6.32-gentoo-1 #1 SMP PREEMPT_DYNAMIC Fri May 31 10:05:26 MSK 2024 x86_64 AMD FX(tm)-8350 Eight-Core Processor AuthenticAMD GNU/Linux

$ python --version
Python 3.12.3

$ ./testqueue.py
queue.get() from pid 13813
queue.get() from pid 13814
queue.get() from pid 13815
queue.get() from pid 13817
^CTraceback (most recent call last):
  File "/home/john/./testqueue.py", line 45, in <module>
    result = ps_queues[cpu].get(timeout=2)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/queues.py", line 117, in get
    res = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 437, in _recv_bytes
    return self._recv(size)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/multiprocessing/connection.py", line 395, in _recv
    chunk = read(handle, remaining)
            ^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt