multiprocessing.Process stuck on start method if subprocess used in process.�?#116249
Description
Bug report
Bug description:
In my project, where I found this bug it works a little differently than in this minimal, reproducible example.
In example bellow we can see that as line before process.start() as 'line after process.start() printing successfully. But after 16'th process started print we can't see even first line print of 17'th process run() function.
Note: My system has 32 cpu's.
In my project bug appears differently. We can't see 'line after process.start() print. It feels like thread from which we launched process after some other subprocess threads started, on start() goes into it run() function as it was normal function and prints 'line after process.start() only after subprocess finished.
https://github.com/KostromDan/Mods-Decompiler-GUI/blob/tk-to-PySide6-rewrite/MDGLogic/DeobfuscationMainThread.py#L70
Note: in code of this project I already replaced mutiprocessing to threading in DeobfuscationThread class to bypass this until fix released. If you want to reproduce just replace it back.
To make this script work:
- create
java_gradle_projectsfolder - unzip any java project which uses gradle and has gradlew.bat in main foolder in
java_gradle_projects/java_project
I used this java project https://github.com/architectury/architectury-api
import multiprocessing
import os.path
import shutil
import subprocess
import time
class SubprocessProcess(multiprocessing.Process):
def __init__(self, i):
super().__init__()
self.i = i
def run(self):
print(f'started {self.i}')
cmd = subprocess.Popen(['gradlew.bat', 'build'], cwd=os.path.join('java_gradle_projects', f'project_{self.i}'),
shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
cmd.wait()
print(f'finished {self.i}')
if __name__ == '__main__':
project_count = 32
print('removing old projects')
for project in os.listdir('java_gradle_projects'):
if project.startswith('project_'):
shutil.rmtree(os.path.join('java_gradle_projects', project))
print('creating new projects')
for i in range(project_count):
shutil.copytree(os.path.join('java_gradle_projects', 'java_project'),
os.path.join('java_gradle_projects', f'project_{i}'))
for i in range(project_count):
process = SubprocessProcess(i)
print(f'line before process.start() i={i}')
process.start()
print(f'line after process.start() i={i}')
time.sleep(0.1)output
removing old projectscreating new projects
line before process.start() i=0
line after process.start() i=0
started 0
line before process.start() i=1
line after process.start() i=1
started 1
line before process.start() i=2
line after process.start() i=2
started 2
line before process.start() i=3
line after process.start() i=3
started 3
line before process.start() i=4
line after process.start() i=4
started 4
line before process.start() i=5
line after process.start() i=5
started 5
line before process.start() i=6
line after process.start() i=6
started 6
line before process.start() i=7
line after process.start() i=7
started 7
line before process.start() i=8
line after process.start() i=8
started 8
line before process.start() i=9
line after process.start() i=9
started 9
line before process.start() i=10
line after process.start() i=10
started 10
line before process.start() i=11
line after process.start() i=11
started 11
line before process.start() i=12
line after process.start() i=12
started 12
line before process.start() i=13
line after process.start() i=13
started 13
line before process.start() i=14
line after process.start() i=14
started 14
line before process.start() i=15
line after process.start() i=15
started 15
line before process.start() i=16
line after process.start() i=16
started 16
line before process.start() i=17
line after process.start() i=17
line before process.start() i=18
line after process.start() i=18
line before process.start() i=19
line after process.start() i=19
line before process.start() i=20
line after process.start() i=20
line before process.start() i=21
line after process.start() i=21
line before process.start() i=22
line after process.start() i=22
line before process.start() i=23
line after process.start() i=23
line before process.start() i=24
line after process.start() i=24
line before process.start() i=25
line after process.start() i=25
line before process.start() i=26
line after process.start() i=26
line before process.start() i=27
line after process.start() i=27
line before process.start() i=28
line after process.start() i=28
line before process.start() i=29
line after process.start() i=29
line before process.start() i=30
line after process.start() i=30
line before process.start() i=31
line after process.start() i=31
If we will replace multiprocessing.Process with threading.Thread all will work as expected.
Bug can't be reproduced if we replace ['gradlew.bat', 'build'] with ['ping', '1.1.1.1', '-n', '100'].
CPython versions tested on:
3.12
Operating systems tested on:
Windows