bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_ca… #29384

hroncok · 2021-11-03T11:37:14Z

Consider the following directory structure:

.
└── PATH1
    └── namespace
        └── sub1
            └── __init__.py

And both PATH1 and PATH2 in sys path:

$ PYTHONPATH=PATH1:PATH2 python3.11
>>> import namespace
>>> import namespace.sub1
>>> namespace.__path__
_NamespacePath(['.../PATH1/namespace'])
>>> ...

While this interpreter still runs, PATH2/namespace/sub2 is created:

.
├── PATH1
│   └── namespace
│       └── sub1
│           └── __init__.py
└── PATH2
    └── namespace
        └── sub2
            └── __init__.py

The newly created module cannot be imported:

>>> ...
>>> namespace.__path__
_NamespacePath(['.../PATH1/namespace'])
>>> import namespace.sub2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'namespace.sub2'

Calling importlib.invalidate_caches() now newly allows to import it:

>>> import importlib
>>> importlib.invalidate_caches()
>>> namespace.__path__
_NamespacePath(['.../PATH1/namespace'])
>>> import namespace.sub2
>>> namespace.__path__
_NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace'])

This was not previously possible.

https://bugs.python.org/issue45703

Automerge-Triggered-By: GH:encukou

Doc/library/importlib.rst

…ches() Consider the following directory structure: . └── PATH1 └── namespace └── sub1 └── __init__.py And both PATH1 and PATH2 in sys path: $ PYTHONPATH=PATH1:PATH2 python3.11 >>> import namespace >>> import namespace.sub1 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> ... While this interpreter still runs, PATH2/namespace/sub2 is created: . ├── PATH1 │ └── namespace │ └── sub1 │ └── __init__.py └── PATH2 └── namespace └── sub2 └── __init__.py The newly created module cannot be imported: >>> ... >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'namespace.sub2' Calling importlib.invalidate_caches() now newly allows to import it: >>> import importlib >>> importlib.invalidate_caches() >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace']) This was not previously possible.

vstinner · 2021-11-10T10:22:26Z

cc @jaraco

pythonGH-29384) Consider the following directory structure: . └── PATH1 └── namespace └── sub1 └── __init__.py And both PATH1 and PATH2 in sys path: $ PYTHONPATH=PATH1:PATH2 python3.11 >>> import namespace >>> import namespace.sub1 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> ... While this interpreter still runs, PATH2/namespace/sub2 is created: . ├── PATH1 │ └── namespace │ └── sub1 │ └── __init__.py └── PATH2 └── namespace └── sub2 └── __init__.py The newly created module cannot be imported: >>> ... >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'namespace.sub2' Calling importlib.invalidate_caches() now newly allows to import it: >>> import importlib >>> importlib.invalidate_caches() >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace']) This was not previously possible.

miss-islington · 2021-12-07T14:45:05Z

Thanks @hroncok for the PR 🌮🎉.. I'm working now to backport this PR to: 3.10.
🐍🍒⛏🤖

miss-islington · 2021-12-07T14:45:07Z

Sorry @hroncok, I had trouble checking out the 3.10 backport branch.
Please backport using cherry_picker on command line.
cherry_picker ae1965ccb4b1fad63fab40fe8805d1b8247668d3 3.10

miss-islington · 2021-12-07T14:45:09Z

Thanks @hroncok for the PR 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

pythonGH-29384) Consider the following directory structure: . └── PATH1 └── namespace └── sub1 └── __init__.py And both PATH1 and PATH2 in sys path: $ PYTHONPATH=PATH1:PATH2 python3.11 >>> import namespace >>> import namespace.sub1 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> ... While this interpreter still runs, PATH2/namespace/sub2 is created: . ├── PATH1 │ └── namespace │ └── sub1 │ └── __init__.py └── PATH2 └── namespace └── sub2 └── __init__.py The newly created module cannot be imported: >>> ... >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'namespace.sub2' Calling importlib.invalidate_caches() now newly allows to import it: >>> import importlib >>> importlib.invalidate_caches() >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace']) This was not previously possible. (cherry picked from commit ae1965c) Co-authored-by: Miro Hrončok <[email protected]>

bedevere-bot · 2021-12-07T14:45:25Z

GH-29964 is a backport of this pull request to the 3.9 branch.

…date_ca… (pythonGH-29384) Consider the following directory structure: . └── PATH1 └── namespace └── sub1 └── __init__.py And both PATH1 and PATH2 in sys path: $ PYTHONPATH=PATH1:PATH2 python3.11 >>> import namespace >>> import namespace.sub1 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> ... While this interpreter still runs, PATH2/namespace/sub2 is created: . ├── PATH1 │ └── namespace │ └── sub1 │ └── __init__.py └── PATH2 └── namespace └── sub2 └── __init__.py The newly created module cannot be imported: >>> ... >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'namespace.sub2' Calling importlib.invalidate_caches() now newly allows to import it: >>> import importlib >>> importlib.invalidate_caches() >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace']) This was not previously possible. (cherry picked from commit ae1965c) Co-authored-by: Miro Hrončok <[email protected]>

pythonGH-29384) Consider the following directory structure: . └── PATH1 └── namespace └── sub1 └── __init__.py And both PATH1 and PATH2 in sys path: $ PYTHONPATH=PATH1:PATH2 python3.11 >>> import namespace >>> import namespace.sub1 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> ... While this interpreter still runs, PATH2/namespace/sub2 is created: . ├── PATH1 │ └── namespace │ └── sub1 │ └── __init__.py └── PATH2 └── namespace └── sub2 └── __init__.py The newly created module cannot be imported: >>> ... >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'namespace.sub2' Calling importlib.invalidate_caches() now newly allows to import it: >>> import importlib >>> importlib.invalidate_caches() >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace']) This was not previously possible.

bedevere-bot · 2022-01-26T15:43:18Z

GH-30922 is a backport of this pull request to the 3.10 branch.

…date_cache (GH-29384) (GH-30922) Consider the following directory structure: . └── PATH1 └── namespace └── sub1 └── __init__.py And both PATH1 and PATH2 in sys path: $ PYTHONPATH=PATH1:PATH2 python3.11 >>> import namespace >>> import namespace.sub1 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> ... While this interpreter still runs, PATH2/namespace/sub2 is created: . ├── PATH1 │ └── namespace │ └── sub1 │ └── __init__.py └── PATH2 └── namespace └── sub2 └── __init__.py The newly created module cannot be imported: >>> ... >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'namespace.sub2' Calling importlib.invalidate_caches() now newly allows to import it: >>> import importlib >>> importlib.invalidate_caches() >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace']) This was not previously possible.

…ate_cache (pythonGH-29384) (pythonGH-30922) Consider the following directory structure: . └── PATH1 └── namespace └── sub1 └── __init__.py And both PATH1 and PATH2 in sys path: $ PYTHONPATH=PATH1:PATH2 python3.11 >>> import namespace >>> import namespace.sub1 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> ... While this interpreter still runs, PATH2/namespace/sub2 is created: . ├── PATH1 │ └── namespace │ └── sub1 │ └── __init__.py └── PATH2 └── namespace └── sub2 └── __init__.py The newly created module cannot be imported: >>> ... >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'namespace.sub2' Calling importlib.invalidate_caches() now newly allows to import it: >>> import importlib >>> importlib.invalidate_caches() >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace']) This was not previously possible.

…ate_cache (GH-29384) (GH-30922) (GH-31076) Consider the following directory structure: . └── PATH1 └── namespace └── sub1 └── __init__.py And both PATH1 and PATH2 in sys path: $ PYTHONPATH=PATH1:PATH2 python3.11 >>> import namespace >>> import namespace.sub1 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> ... While this interpreter still runs, PATH2/namespace/sub2 is created: . ├── PATH1 │ └── namespace │ └── sub1 │ └── __init__.py └── PATH2 └── namespace └── sub2 └── __init__.py The newly created module cannot be imported: >>> ... >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'namespace.sub2' Calling importlib.invalidate_caches() now newly allows to import it: >>> import importlib >>> importlib.invalidate_caches() >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace']) This was not previously possible. Co-Authored-By: Miro Hrončok <[email protected]> Automerge-Triggered-By: GH:encukou

…ate_cache (pythonGH-29384) (pythonGH-30922) (pythonGH-31076) Consider the following directory structure: . └── PATH1 └── namespace └── sub1 └── __init__.py And both PATH1 and PATH2 in sys path: $ PYTHONPATH=PATH1:PATH2 python3.11 >>> import namespace >>> import namespace.sub1 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> ... While this interpreter still runs, PATH2/namespace/sub2 is created: . ├── PATH1 │ └── namespace │ └── sub1 │ └── __init__.py └── PATH2 └── namespace └── sub2 └── __init__.py The newly created module cannot be imported: >>> ... >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'namespace.sub2' Calling importlib.invalidate_caches() now newly allows to import it: >>> import importlib >>> importlib.invalidate_caches() >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace']) >>> import namespace.sub2 >>> namespace.__path__ _NamespacePath(['.../PATH1/namespace', '.../PATH2/namespace']) This was not previously possible. Co-Authored-By: Miro Hrončok <[email protected]> Automerge-Triggered-By: GH:encukou

the-knights-who-say-ni added the CLA signed label Nov 3, 2021

hroncok force-pushed the invalidate_namespacepath_cache branch 2 times, most recently from d5e7171 to 397a07e Compare Nov 3, 2021

AlexWaygood reviewed Nov 3, 2021

View changes

Doc/library/importlib.rst Outdated Show resolved Hide resolved

hroncok force-pushed the invalidate_namespacepath_cache branch from 397a07e to 09ff56e Compare Nov 3, 2021

hroncok force-pushed the invalidate_namespacepath_cache branch from 09ff56e to f79ced4 Compare Nov 3, 2021

This comment has been minimized.

Sign in to view

hroncok marked this pull request as ready for review Nov 3, 2021

encukou approved these changes Nov 23, 2021

View changes

bedevere-bot added the awaiting merge label Nov 23, 2021

encukou added the 🤖 automerge PR will be merged once it's been approved and all CI passed label Nov 23, 2021

miss-islington merged commit ae1965c into python:main Nov 23, 2021
13 checks passed

bedevere-bot removed the awaiting merge label Nov 23, 2021

hroncok deleted the invalidate_namespacepath_cache branch Nov 23, 2021

encukou added needs backport to 3.9 needs backport to 3.10 labels Dec 7, 2021

miss-islington self-assigned this Dec 7, 2021

bedevere-bot removed the needs backport to 3.9 label Dec 7, 2021

bedevere-bot removed the needs backport to 3.10 label Jan 26, 2022

bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_ca… #29384

bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_ca… #29384

hroncok commented Nov 3, 2021 •

edited by miss-islington

This comment has been minimized.

vstinner commented Nov 10, 2021

miss-islington commented Dec 7, 2021

miss-islington commented Dec 7, 2021

miss-islington commented Dec 7, 2021

bedevere-bot commented Dec 7, 2021

bedevere-bot commented Jan 26, 2022

bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_ca… #29384

bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_ca… #29384

Conversation

hroncok commented Nov 3, 2021 • edited by miss-islington

This comment has been minimized.

vstinner commented Nov 10, 2021

miss-islington commented Dec 7, 2021

miss-islington commented Dec 7, 2021

miss-islington commented Dec 7, 2021

bedevere-bot commented Dec 7, 2021

bedevere-bot commented Jan 26, 2022

hroncok commented Nov 3, 2021 •

edited by miss-islington