Intra-package References Documentation Incomplete

[ADataGman]

BPO	35927
Nosy	@brettcannon, @terryjreedy, @ncoghlan, @ericsnowcurrently
Files	sound.zip: zip of code sample copied from documentation

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-02-07.04:38:30.393>
labels = ['3.8', 'type-bug', '3.7', 'docs']
title = 'Intra-package References Documentation Incomplete'
updated_at = <Date 2019-02-14.00:38:19.737>
user = 'https://bugs.python.org/ADataGman'

bugs.python.org fields:

activity = <Date 2019-02-14.00:38:19.737>
actor = 'ADataGman'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2019-02-07.04:38:30.393>
creator = 'ADataGman'
dependencies = []
files = ['48109']
hgrepos = []
issue_num = 35927
keywords = []
message_count = 7.0
messages = ['335002', '335104', '335105', '335126', '335303', '335385', '335486']
nosy_count = 6.0
nosy_names = ['brett.cannon', 'terry.reedy', 'ncoghlan', 'docs@python', 'eric.snow', 'ADataGman']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35927'
versions = ['Python 3.7', 'Python 3.8']

[ADataGman]

Attempting to follow https://docs.python.org/3.6/tutorial/modules.html#intra-package-references I was unable to recreate the intra-package reference as described.

"For example, if the module sound.filters.vocoder needs to use the echo module in the sound.effects package, it can use from sound.effects import echo."

Creating the file structure described in https://docs.python.org/3.6/tutorial/modules.html#packages, with empty __init__.py files at all levels, or with __all__ defined as containing relevant file names, results in "No module named 'sound'". If I try to run this using "from ..effects import echo" then it results in "attempted relative import beyond top-level package".

At least one other user has run into this issue with this stack overflow post: https://stackoverflow.com/questions/53109627/python-intra-package-reference-doesnt-work-at-all

[terryjreedy]

3.6 only gets security patches.

I cannot comment on relative imports as I never use them, but the absolute imports should work and do for me. I just added to idlelib.idle_test a file a.py containing 'from idlelib.idle_test import htest' and it works find. The idlelib and idle_test __init__.py files are not empty, but that should not make a difference. In fact, many of the test_xyz modules already import from the non-test utility modules in the same directory.

The error messsage suggests that you did not put the top level package directory, 'sound', in a directory in sys.path, as you must. What directory contains your sound package and what is your sys.path? I suspect that this should be closed as 'not a bug'.

[terryjreedy]

When reporting problems, copy and paste into the report the complete exception traceback and error message. For import problems, also copy and paste sys.path. If you disagree that this should be closed, please do so.

[ADataGman]

Here's the full error log I receive when using the code I submitted with
both ..effects and sound.effects. It also includes a print out of the sys
path as run from vocoder.

I tried running this in 3.7.2 and still get the same errors. It seems like
it's already on the sys path? I'm not saying there is a bug in python
itself, it's just not at all clear from the documentation that what is
described as being a way to access sibling packages actually works.

Given the attached code sample, as a newb I'd greatly appreciate an example
of how to take that sample and turn it into functioning code with whatever
modifiers is necessary. Ultimately whatever changes are made I'd like to
see reflected in the documentation for the intra-reference section.

PS C:\sound> &
C:/Users/user/AppData/Local/Programs/Python/Python37/python.exe
c:/sound/filters/vocoder.py
Traceback (most recent call last):
  File "c:/sound/filters/vocoder.py", line 7, in <module>
    vocoder.pp()
  File "c:/sound/filters/vocoder.py", line 4, in pp
    from ..effects import echo
ValueError: attempted relative import beyond top-level package
PS C:\sound> &
C:/Users/user/AppData/Local/Programs/Python/Python37/python.exe
c:/sound/filters/vocoder.py
Traceback (most recent call last):
  File "c:/sound/filters/vocoder.py", line 7, in <module>
    vocoder.pp()
  File "c:/sound/filters/vocoder.py", line 4, in pp
    from sound.effects import echo
ModuleNotFoundError: No module named 'sound'
PS C:\sound> &
C:/Users/user/AppData/Local/Programs/Python/Python37/python.exe
c:/sound/filters/vocoder.py
['c:\\sound\\filters',
'C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37\\python37.zip',
'C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37\\DLLs',
'C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37\\lib',
'C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37',
'C:\\Users\\user\\AppData\\Roaming\\Python\\Python37\\site-packages',
'C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages']

On Fri, Feb 8, 2019 at 2:31 PM Terry J. Reedy <report@bugs.python.org>
wrote:

Terry J. Reedy <tjreedy@udel.edu> added the comment:

When reporting problems, copy and paste into the report the complete
exception traceback and error message. For import problems, also copy and
paste sys.path. If you disagree that this should be closed, please do so.

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue35927\>

---

[ncoghlan]

As discussed in http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html#executing-the-main-module-twice and https://www.python.org/dev/peps/pep-0395/#why-are-my-imports-broken the cryptic error message here is due to the relative import being resolved based on the module name "__main__", since the submodule was executed directly, rather than the intended "sound.filter.vocoder" name that it would have when imported or executed via the "-m" switch.

It's spectacularly unobvious what's going on if you're not deeply familiar with the intricacies of main module execution, but I haven't had any great ideas for practical ways to improve the error message.

I'm not sure we currently have access to the required info at the point this error is raised, but perhaps it would be enough to add the name of the module doing the import to the error message as a reminder that directly executing scripts inside packages does weird things to import resolution?

Something like:

"Attempted relative import beyond top-level package '__main__'"

[terryjreedy]

AData> You log and output show that you problem is as described: package 'sound' is in directory "C:/" and "C:/" is not is sys.path. Possible remedies:

1. Add "C:/" to sys.path at the top of each module that imports sound. This is a nuisance.
2. Move the sound package to 'C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages'. This can also a nuisance.
3. Add a file named 'C.pth' and containing "C:/" to the site-packages. ('pth' abbreviates 'path'.) This makes C:/ an extension of site-packages. But all the non-python stuff in C: likely makes this a bad idea.
   4 (3 improved). Add a directory such as "C:/pyprojects", make the contents of the .pth file "C:/pyprojects", and move sound to pyprojects.
   This is what I have done. When I install a new version of Python, I copy the .pth file to the new site-packeges directory. Works great.

[ADataGman]

Terry, thank you for the detailed list of alternatives on how to approach
this, that's helpful in how to tackle this issue in my current development.

Nick, for the relative imports of relative imports I think the error
message is fine. It makes sense. It doesn't necessarily give you a lot of
hints on where to go next unfortunately, but it does explain why it doesn't
work.

For the absolute imports, a hint like, the module must be discoverable in
sys.path or the like if you wanted to make this clearer.

My biggest concern was just that the way the documentation is written for
https://docs.python.org/3.6/tutorial/modules.html#intra-package-references
. It implies that the code will run without any hint of needing to be
either run from a higher level, or a work around used like Terry suggested.
Now that I've spent a few days researching and reading more on my own, and
with help from you and Terry, I understand what's happening and have
options. But for others just reading the docs and not already familiar,
it's not clear why the current example wouldn't work when run from the
sibling level.

I'd like to see the documentation for intra-package references updated to
clearly state that the code is being run from a level above. "For example,
if the module sound.filters.vocoder needs to use the echo module in the
sound.effects package, it can use from sound.effects import echo." only
works if you run it from the sound module. Which is fine! It's just not
clear to a newb that it's being executed from the sound package rather than
the filters package. Perhaps modify
https://docs.python.org/3.6/tutorial/modules.html#packages to include a
"sound.py" at the top level folder, and in the explanation for the
intra-package reference, just mention that the example quoted above runs
from that new file.

If there is a optimal way to modify the path for adding in a sibling level
package, please include that. Otherwise just clarify that the example
relies on the code being run from something from another level up.

On Tue, Feb 12, 2019 at 11:35 PM Terry J. Reedy <report@bugs.python.org>
wrote:

Terry J. Reedy <tjreedy@udel.edu> added the comment:

AData> You log and output show that you problem is as described: package
'sound' is in directory "C:/" and "C:/" is not is sys.path. Possible
remedies:

1. Add "C:/" to sys.path at the top of each module that imports sound.
   This is a nuisance.
2. Move the sound package to
   'C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages'.
   This can also a nuisance.
3. Add a file named 'C.pth' and containing "C:/" to the site-packages.
   ('pth' abbreviates 'path'.) This makes C:/ an extension of site-packages.
   But all the non-python stuff in C: likely makes this a bad idea.
   4 (3 improved). Add a directory such as "C:/pyprojects", make the contents
   of the .pth file "C:/pyprojects", and move sound to pyprojects.
   This is what I have done. When I install a new version of Python, I copy
   the .pth file to the new site-packeges directory. Works great.

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue35927\>

---

[zahlman]

For completeness:

This issue is not version-specific - neither in terms of how it works, nor in terms of what the documentation says. Differences in the current 3.11 doc are not meaningful to the current discussion as far as I can tell.

The "attempted relative import beyond top-level package" error has two causes. One is the same cause as the "No module named 'sound'" error. Both the relative import from ..effects import echo and the absolute import from sound.effects import echo require the root of the sound package to be on the module search path (exposed as sys.path). Since the sound folder was placed directly in C:\, that means Python must be told to look within C:\ for modules, or else it will not be able to find the sound package (represented implicitly by the sound folder and optionally explicitly by sound/__init__.py).

This is not a special case - it works the same way for the standard library (except for built-ins implemented in C and available automatically) and for third-party libraries (that's one of the main reasons why you need pip to install them: to put them in the appropriate location, compatible with the default sys.path, so that they will be found).

The reason, presumably, that this part of the tutorial doesn't talk about setting up sys.path to contain the package is that it is already explained earlier on the page - in section 6.1.2:

When a module named spam is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations:

• The directory containing the input script (or the current directory when no file is specified).
• PYTHONPATH (a list of directory names, with the same syntax as the shell variable PATH).
• The installation-dependent default.

The current documentation adds a few minor details to this, reflecting actual small changes to how the system works. However, I agree that this is not very explicit. I think it should explain more clearly that modules not listed on sys.path will fail to import, and that such a problem should be resolved by taking care to ensure sys.path is properly configured (ideally, with an opinionated first-to-last-resort set of steps to try, starting with cding to the right place and ending with in-code sys.path modification). Overall, the tutorial here feels too much like ordinary documentation and not enough like a tutorial to me; it is talking too much about how the functionality works, and not enough about what to do in common cases.

---

The second problem with the relative import attempt is that it doesn't work from the main module. This is, and was, already covered by section 6.4.2:

Note that relative imports are based on the name of the current module. Since the name of the main module is always "__main__", modules intended for use as the main module of a Python application must always use absolute imports.

However, this can also be worked around using the -m switch, which IMO deserves a mention here.

Incidentally, this problem is described and answered in great detail by several popular questions on Stack Overflow. It's hard to identify a canonical at the moment, but the pithily-named Relative imports for the billionth time seems to be the most popular and contains all the necessary information (it just... has too many answers, and is a bit hard to follow).