bpo-22490: Remove __PYVENV_LAUNCHER__ from environment during launch #9516

ronaldoussoren · 2018-09-23T14:43:19Z

Alternative patch for bpo-22490 that removes the shell environment variable PYVENV_LAUNCHER during interpreter launch because it is only used to communicate between the stub executable in Mac/Tools/pythonw.c and the real interpreter.

This avoids problems when launching the "real" interpreter without going through the stub executable.

https://bugs.python.org/issue22490

… macOS This changeset removes the environment varialbe "__PYVENV_LAUNCHER__" during interpreter launch as it is only needed to communicate between the stub executable in framework installs and the actual interpreter. Leaving the environment variable present may lead to misbehaviour when launching other scripts.

jaraco

Looks good to me. I think it would be nice if, since #9498 didn't fail any tests, there were a documented way (even if just in the ticket), one could verify that the behavior is working as intended (a test that would fail with #9498).

gvanrossum · 2018-09-23T20:58:28Z

I would love to know how to repro the original bug so I can demonstrate this actually fixes it.

Here's what I think should be the repro:

/Library/Frameworks/Python.framework/Versions/3.8/bin/python3  -c 'import subprocess; p = subprocess.Popen([".mypy/venv/bin/python3","-m","mypy.dmypy","-h"]).communicate()'

The setup is that .mypy/venv is a virtualenv that has mypy.dmypy installed. This is what failed for the user who reported this.

Unfortunately this does not fail for me (with a framework build from master installed). Would I have to do the install via brew? (I would have to learn about locally testing modified brew recipes, I suppose.)

gvanrossum

The patch LG from a code review POV.

ned-deily · 2019-03-24T19:08:07Z

@ronaldoussoren ping?

…ld-system-reviewers,rstewart The __PYVENV_LAUNCHER__ is set on macos to indicate which python to use. Sadly, keeping this set confuses pip in virtualenvs. This corresponds to python/cpython#9516. Differential Revision: https://phabricator.services.mozilla.com/D59853

…ld-system-reviewers,rstewart The __PYVENV_LAUNCHER__ is set on macos to indicate which python to use. Sadly, keeping this set confuses pip in virtualenvs. This corresponds to python/cpython#9516. Differential Revision: https://phabricator.services.mozilla.com/D59853 --HG-- extra : moz-landing-system : lando

…ld-system-reviewers,rstewart The __PYVENV_LAUNCHER__ is set on macos to indicate which python to use. Sadly, keeping this set confuses pip in virtualenvs. This corresponds to python/cpython#9516. Differential Revision: https://phabricator.services.mozilla.com/D59853 UltraBlame original commit: ed91bca1f4a669454801390a1e5432fa2cd9561e

mattip · 2020-02-27T15:12:30Z

We think this is borking creation of a PyPy virtualenv using cpython host on macOS and is was very difficult to detect, so it would be nice to clean this up. For a test, you could try to install pypy 7.3.0 using cpython, see https://foss.heptapod.net/pypy/pypy/issues/3175. Although now that we know we will probably fix it for PyPy's next release.

Mac/Tools/pythonw.c

Co-Authored-By: Nicola Soranzo <[email protected]>

jaraco · 2020-03-22T18:31:13Z

The test failure is also happening in master, so unrelated to this change.

miss-islington · 2020-03-22T18:31:49Z

Thanks @ronaldoussoren for the PR, and @jaraco for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7, 3.8.
🐍🍒⛏🤖

bedevere-bot · 2020-03-22T18:32:12Z

GH-19110 is a backport of this pull request to the 3.8 branch.

miss-islington · 2020-03-22T18:32:13Z

Sorry, @ronaldoussoren and @jaraco, I could not cleanly backport this to 3.7 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 044cf94f610e831464a69a8e713dad89878824ce 3.7

…ythonGH-9516) * bpo-22490: Remove "__PYVENV_LAUNCHER__" from the shell environment on macOS This changeset removes the environment varialbe "__PYVENV_LAUNCHER__" during interpreter launch as it is only needed to communicate between the stub executable in framework installs and the actual interpreter. Leaving the environment variable present may lead to misbehaviour when launching other scripts. * Actually commit the changes for issue 22490... * Correct typo Co-Authored-By: Nicola Soranzo <[email protected]> * Run make patchcheck Co-authored-by: Jason R. Coombs <[email protected]> Co-authored-by: Nicola Soranzo <[email protected]> (cherry picked from commit 044cf94) Co-authored-by: Ronald Oussoren <[email protected]>

…aunch (pythonGH-9516) * bpo-22490: Remove "__PYVENV_LAUNCHER__" from the shell environment on macOS This changeset removes the environment varialbe "__PYVENV_LAUNCHER__" during interpreter launch as it is only needed to communicate between the stub executable in framework installs and the actual interpreter. Leaving the environment variable present may lead to misbehaviour when launching other scripts. * Actually commit the changes for issue 22490... * Correct typo Co-Authored-By: Nicola Soranzo <[email protected]> * Run make patchcheck Co-authored-by: Jason R. Coombs <[email protected]> Co-authored-by: Nicola Soranzo <[email protected]>. (cherry picked from commit 044cf94) Co-authored-by: Ronald Oussoren <[email protected]>

bedevere-bot · 2020-03-22T18:47:19Z

GH-19111 is a backport of this pull request to the 3.7 branch.

…H-9516) (GH-19110) * bpo-22490: Remove "__PYVENV_LAUNCHER__" from the shell environment on macOS This changeset removes the environment varialbe "__PYVENV_LAUNCHER__" during interpreter launch as it is only needed to communicate between the stub executable in framework installs and the actual interpreter. Leaving the environment variable present may lead to misbehaviour when launching other scripts. * Actually commit the changes for issue 22490... * Correct typo Co-Authored-By: Nicola Soranzo <[email protected]> * Run make patchcheck Co-authored-by: Jason R. Coombs <[email protected]> Co-authored-by: Nicola Soranzo <[email protected]> (cherry picked from commit 044cf94) Co-authored-by: Ronald Oussoren <[email protected]> Co-authored-by: Ronald Oussoren <[email protected]>

…aunch (GH-9516) (GH-19111) * bpo-22490: Remove "__PYVENV_LAUNCHER__" from the shell environment on macOS This changeset removes the environment varialbe "__PYVENV_LAUNCHER__" during interpreter launch as it is only needed to communicate between the stub executable in framework installs and the actual interpreter. Leaving the environment variable present may lead to misbehaviour when launching other scripts. * Actually commit the changes for issue 22490... * Correct typo Co-Authored-By: Nicola Soranzo <[email protected]> * Run make patchcheck Co-authored-by: Jason R. Coombs <[email protected]> Co-authored-by: Nicola Soranzo <[email protected]>. (cherry picked from commit 044cf94) Co-authored-by: Ronald Oussoren <[email protected]> Co-authored-by: Ronald Oussoren <[email protected]>

…ythonGH-9516) * bpo-22490: Remove "__PYVENV_LAUNCHER__" from the shell environment on macOS This changeset removes the environment varialbe "__PYVENV_LAUNCHER__" during interpreter launch as it is only needed to communicate between the stub executable in framework installs and the actual interpreter. Leaving the environment variable present may lead to misbehaviour when launching other scripts. * Actually commit the changes for issue 22490... * Correct typo Co-Authored-By: Nicola Soranzo <[email protected]> * Run make patchcheck Co-authored-by: Jason R. Coombs <[email protected]> Co-authored-by: Nicola Soranzo <[email protected]>

Exactly how this all works isn't clear, but since the fix in python/cpython#9516 was to simply hide the environment variable from the interpreter it seems reasonably safe. Fixes #27377.

ronaldoussoren added 2 commits Sep 23, 2018

Actually commit the changes for issue 22490...

9e04423

ronaldoussoren requested a review from gpshead as a code owner Sep 23, 2018

the-knights-who-say-ni added the CLA signed label Sep 23, 2018

bedevere-bot added the awaiting merge label Sep 23, 2018

jaraco approved these changes Sep 23, 2018

View changes

ronaldoussoren added needs backport to 3.6 needs backport to 3.7 labels Sep 23, 2018

gvanrossum approved these changes Sep 24, 2018

View changes

nsoranzo mentioned this pull request Oct 17, 2018

Scripts installed with wrong shebang under subprocess pypa/virtualenv#845

Closed

nsoranzo mentioned this pull request Oct 17, 2018

Allow selection of python version when starting managed Galaxy galaxyproject/planemo#874

Merged

gpshead removed the needs backport to 3.6 label Dec 26, 2018

madsmtm mentioned this pull request Jul 20, 2019

Install executables in virtual environments with flit install --python takluyver/flit#274

Closed

gpshead added the needs backport to 3.8 label Sep 28, 2019

itsayellow mentioned this pull request Jan 12, 2020

FileNotFoundError for pipx_shared.pth with pipx install --python and pyenv pypa/pipx#334

Closed

irvinlim mentioned this pull request Feb 1, 2020

Blacklist __PYVENV_LAUNCHER__ env var in subprocess calls sjkingo/virtualenv-api#46

Merged

cjolowicz mentioned this pull request Feb 21, 2020

RuntimeError: failed to find interpreter for Builtin discover of python_spec='python3.8' pypa/virtualenv#1643

Closed

6 tasks

cjolowicz mentioned this pull request Feb 21, 2020

Prevent sys.prefix from leaking into child process on macOS pypa/virtualenv#1648

Merged

4 tasks

gaborbernat mentioned this pull request Mar 10, 2020

conda commands as Python3 subprocesses inside a virtualenv fail under macOS pypa/virtualenv#1704

Closed

nsoranzo reviewed Mar 10, 2020

View changes

Mac/Tools/pythonw.c Outdated Show resolved Hide resolved

Correct typo

e06d8ce

Co-Authored-By: Nicola Soranzo <[email protected]>

jaraco requested review from vsajip and as code owners Mar 22, 2020

Run make patchcheck

1679408

jaraco merged commit 044cf94 into python:master Mar 22, 2020
8 of 9 checks passed

bedevere-bot removed the awaiting merge label Mar 22, 2020

bedevere-bot removed the needs backport to 3.8 label Mar 22, 2020

miss-islington assigned jaraco Mar 22, 2020

bedevere-bot removed the needs backport to 3.7 label Mar 22, 2020

jonchang mentioned this pull request Mar 23, 2020

Please test/review Python for issue 22490 Homebrew/homebrew-core#52046

Closed

excitedleigh mentioned this pull request May 15, 2020

__PYVENV_LAUNCHER__ being set creates unexpected problems for child Python instances xonsh/xonsh#3576

Closed

uranusjr mentioned this pull request Jul 19, 2020

shebang of script installed in system site packages venv points to system python in Mac OS Homebrew pypa/pip#8600

Closed

jsirois mentioned this pull request Sep 28, 2020

Binary location and interpreter selection issues pantsbuild/pants#10855

Closed

jsirois mentioned this pull request Sep 28, 2020

No such file for interpreter's binary name pantsbuild/pex#1009

Closed

jsirois mentioned this pull request Sep 28, 2020

Release 2.1.17 pantsbuild/pex#1013

Closed

4 tasks

gaborbernat mentioned this pull request Oct 16, 2020

tox is fooled by __PYVENV_LAUNCHER__ tox-dev/tox#1689

Open

foolip mentioned this pull request Apr 23, 2021

Virtualenv-provided pip3 doesn't work on macOS web-platform-tests/wpt#27377

Closed

foolip mentioned this pull request Apr 23, 2021

Unset __PYVENV_LAUNCHER__ on macOS to avoid virtualenv weirdness web-platform-tests/wpt#28663

Merged

python / cpython Public

bpo-22490: Remove __PYVENV_LAUNCHER__ from environment during launch #9516

bpo-22490: Remove __PYVENV_LAUNCHER__ from environment during launch #9516

ronaldoussoren commented Sep 23, 2018 •

edited by bedevere-bot

jaraco left a comment

gvanrossum commented Sep 23, 2018

gvanrossum left a comment

ned-deily commented Mar 24, 2019

mattip commented Feb 27, 2020

jaraco commented Mar 22, 2020

miss-islington commented Mar 22, 2020

bedevere-bot commented Mar 22, 2020

miss-islington commented Mar 22, 2020

bedevere-bot commented Mar 22, 2020

python / cpython Public

bpo-22490: Remove __PYVENV_LAUNCHER__ from environment during launch #9516

bpo-22490: Remove __PYVENV_LAUNCHER__ from environment during launch #9516

Conversation

ronaldoussoren commented Sep 23, 2018 • edited by bedevere-bot

jaraco left a comment

gvanrossum commented Sep 23, 2018

gvanrossum left a comment

ned-deily commented Mar 24, 2019

mattip commented Feb 27, 2020

jaraco commented Mar 22, 2020

miss-islington commented Mar 22, 2020

bedevere-bot commented Mar 22, 2020

miss-islington commented Mar 22, 2020

bedevere-bot commented Mar 22, 2020

ronaldoussoren commented Sep 23, 2018 •

edited by bedevere-bot