Python launcher invoked with command "py" fails to start process

[kai2nenobu]

Bug report

When I invoke a python launcher of version 3.11.0 with "py" command, a python launcher fails to start process.

Step to reproduce

1. Install python 3.11.0 with py launcher for all users.

2. Invoke "py" in command prompt.

3. I expect to launch a python interpreter, but got a error as below

   >"py"
   Unable to create process using 'C:\Python311\py"':  The system cannot find the file specified.

   Debug output when enable PYLAUNCHER_DEBUG

   >"py"
   argv0: py
   version: 3.11.0
   SearchInfo.originalCmdLine: "py"
   SearchInfo.restOfCmdLine:
   SearchInfo.executablePath: (null)
   SearchInfo.scriptFile: (null)
   SearchInfo.executable: py"
   SearchInfo.executableArgs: (null)
   SearchInfo.company: (null)
   SearchInfo.tag: (null)
   SearchInfo.oldStyleTag: False
   SearchInfo.lowPriorityTag: False
   SearchInfo.allowDefaults: False
   SearchInfo.allowExecutableOverride: False
   SearchInfo.windowed: False
   SearchInfo.list: False
   SearchInfo.listPaths: False
   SearchInfo.help: False
    -V:3.11          C:\Python311\python.exe
    -V:3.10          C:\Python310\python.exe
    -V:3.9           C:\Python39\python.exe
    -V:3.7           C:\Python37\python.exe
   env.company: PythonCore
   env.tag: 3.11
   # about to run: C:\Python311\py"
   Unable to create process using 'C:\Python311\py"':  The system cannot find the file specified.

I think this error seems to be similar to #95285. If I invoke py or "py.exe", a python interpreter starts well as I expected.

>py
Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

>"py.exe"
Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

In version 3.10.8, all of "py", py and "py.exe" start a python interpreter correctly.

Your environment

• CPython versions tested on: 3.11.0
• Operating system and architecture: Windows 10 (10.0.19045.2310)/amd64

Linked PRs

• gh-99442: Fix handling in py.exe launcher when argv[0] does not include a file extension #99542
• [3.11] gh-99442: Fix handling in py.exe launcher when argv[0] does not include a file extension (GH-99542) #99579

[kai2nenobu]

This error causes a failure to build MSI installer between version 3.7 and 3.10. (Example of GitHub Actions build failure in my personal project).

This is because WiX invokes "py" -m pip install -U blurb during build for exe.wixproj and recent GitHub-Hosted runner uses Python Launcher of version 3.11.0.

[pfmoore]

Confirmed, I am seeing the same issue (see the linked rust issue report above).

Command: "py" "-V"
argv0: py
version: 3.11.0
SearchInfo.originalCmdLine: "py" -V
SearchInfo.restOfCmdLine:  -V
SearchInfo.executablePath: (null)
SearchInfo.scriptFile: (null)
SearchInfo.executable: py"
SearchInfo.executableArgs: (null)
SearchInfo.company: (null)
SearchInfo.tag: (null)
SearchInfo.oldStyleTag: False
SearchInfo.lowPriorityTag: False
SearchInfo.allowDefaults: False
SearchInfo.allowExecutableOverride: False
SearchInfo.windowed: False
SearchInfo.list: False
SearchInfo.listPaths: False
SearchInfo.help: False
 -V:3.11          C:\Users\Gustav\AppData\Local\Programs\Python\Python311\python.exe
 -V:3.10          C:\Users\Gustav\AppData\Local\Programs\Python\Python310\python.exe
 -V:3.9           C:\Users\Gustav\AppData\Local\Programs\Python\Python39\python.exe
 -V:3.8           C:\Users\Gustav\AppData\Local\Programs\Python\Python38\python.exe
env.company: PythonCore
env.tag: 3.11
# about to run: C:\Users\Gustav\AppData\Local\Programs\Python\Python311\py" -V
Unable to create process using 'C:\Users\Gustav\AppData\Local\Programs\Python\Python311\py" -V': The system cannot find the file specified.

[eryksun]

In parseCommandLine(), the value of end points to just past the closing double quote if there's no "." found in the tail component. findArgumentEnd() could be modified to exclude the double quote from the end, since the double quote at the end isn't actually part of the command-line argument. It's a syntax character for argument parsing.

cpython/PC/launcher2.c

Lines 565 to 580 in 88385b8

	const wchar_t *tail = findArgumentEnd(search->originalCmdLine, -1);
	const wchar_t *end = tail;
	search->restOfCmdLine = tail;
	while (--tail != search->originalCmdLine) {
	if (*tail == L'.' && end == search->restOfCmdLine) {
	end = tail;
	} else if (*tail == L'\\' || *tail == L'/') {
	++tail;
	break;
	}
	}
	if (tail == search->originalCmdLine && tail[0] == L'"') {
	++tail;
	}
	// Without special cases, we can now fill in the search struct
	int tailLen = (int)(end ? (end - tail) : wcsnlen_s(tail, MAXLEN));

---

A couple of issues in findArgumentLength(), which could also be resolved here without creating a new issue for them:

• If the buffer doesn't start with a double quote, it needs to search for white space -- at least tab ("\t") in addition to space (" ").
• Since findArgumentLength() is used by findArgumentEnd(), which is only ever called to parse the first argument of the command line, it should not implement backslash escaping of double quotes. When parsing the first argument of the command line, both CreateProcessW() and CommandLineToArgvW() enter quote mode if the first character is a double quote, in which case the argument ends at the next double quote, regardless of whether it's followed by whitespace. That's a bit simpler than the C runtime, but even the C runtime doesn't implement escaping of double quotes in the first argument.

[pfmoore]

The fix appears to have addressed part of my issue, but I'm still getting the same error in a slightly different case. I haven't yet been able to reduce the test case to something smaller, but if I use the just command line utility, with a justfile as follows:

set windows-shell := ["pwsh.exe", "-NoLogo", "-Command"]

test_py:
    #!py
    print(12)

then running just test_py gives the error:

Unable to create process using 'C:\Users\Gustav\AppData\Local\Temp\justc2pOnD\py C:\Users\Gustav\AppData\Local\Temp\justc2pOnD\test_py': The system cannot find the file specified.

If I add some debugging prints, I can see that the command being run was "py" "C:\Users\Gustav\AppData\Local\Temp\justc2pOnD\test_py".

[zooba]

Thanks for the quick extra test! This one looks like it's probably shebang parsing - can you include the whole output you get with PYLAUNCHER_DEBUG set?

[zooba]

Oh, this is probably the deliberate change of behaviour to make shebang lines only search PATH if you use /usr/bin/env in them.

If it falls through to the default case, we don't let you search for arbitrary executables. It will only look adjacent to the script file. The old behaviour was never documented, and isn't safe (the new behaviour is only mildly safer, but at least you can scan for usr/bin/env and recognise when it's going to happen, rather than trying to prove a negative). The new documentation reads:

Shebang lines that do not match any of these patterns are treated as Windows paths that are absolute or relative to the directory containing the script file. This is a convenience for Windows-only scripts, such as those generated by an installer, since the behavior is not compatible with Unix-style shells. These paths may be quoted, and may include multiple arguments, after which the path to the script and any additional arguments will be appended.

[eryksun]

#!py is a strange shebang, asking the launcher to recursively run itself. There should be a check in _useShebangAsExecutable() to prevent recursively running the launcher, as you did for the "#!/usr/bin/env" case in searchPath(). Or modify checkShebang() to always check for this. Otherwise #!py will blow up if the launcher is named "py.exe", and the script is beside it.

[pfmoore]

#!py is a strange shebang, asking the launcher to recursively run itself.

It is, and I don't love it, but most other approaches for my use case have downsides. I will say that the old launcher went into a hard recursive loop on this, locking my machine with masses of processes. That's extremely bad, and avoiding that is definitely worthwhile. Obviously, from what I'm doing here, you can tell that I'd prefer #!py to be treated the same as no shebang (i.e. run the script with Python)... And indeed, that's what apparently happens in a normal script. If I have test.py containing

#!py
print("Hello")

and I run it as py test.py, then it does indeed print "Hello"... So all I'm really asking is that the same behaviour works in the case we're triggering here (where the command line has "py" quoted).

The reason I'm doing this at all is that I'm using just, which lets you define actions starting with #! which it processes as a shebang. Just's processing is built in, and (ignoring details) it looks for the executable on PATH. So this shebang will cause just to run the action as a script using py.exe. But then, py re-interprets the shebang, leading to this issue.

The problem is that I don't want to hard-code a path for the interpreter in the shebang line, and the only way to run Python that's guaranteed to be on PATH is via py.exe.

I'm not even sure who I should be asking for help with this. In isolation, the behaviour of both just and py are reasonable. And if there was a python command on PATH, I'd use that. But the combination leaves no really good options.

[eryksun]

The new launcher resolves #!py relative to the script directory. Currently, this leads to an endless loop if the script is in the same directory as the "py.exe" launcher, but generally that's not the case. On Unix, #!py is resolved relative to the current working directory instead of the script directory.

Does just support passing command-line options in the shebang? If the shebang is #!py -3, then the launcher started with -3 won't read the shebang.

Alternatively, does just support Unix templates such as #!/usr/bin/env py? The launcher's implementation of the "env" command will search PATH, but it will disregard the shebang if the search resolves to itself.

---

On a related note, the launcher supports user-defined commands such as "py" in the [commands] section of "py.ini". The new launcher requires user-defined commands in a shebang to be qualified by one of the Unix template paths: "/usr/bin/env ", "/usr/bin/", or "/usr/local/bin/". A bare command is only supported if it starts with "python", which, among other possibilities, can override a registered installation for a "python[x[.y]]" command. With the old launcher, shebang commands that started with "python" were always reserved for registered installations, but user-defined commands that didn't start with "python" could be used without qualification, such as #!py instead of #!/usr/bin/py. I think the new semantics are an improvement, overall.

[pfmoore]

All of those are great ideas, thanks! I'll give them a try when I have some free time.

Overall, it does seem a bit non-obvious to find something that works, which is a shame, but I don't think there's an easy resolution for that. Long-term, we should ideally have a unified way of running Python on all platforms, but I doubt that will happen soon...

[pfmoore]

Actually, I had a few moments. Just's implementation of #!/usr/bin/env is problematic, because it relies on Cygwin (specifically the cygpath command) being available. That's not something I can rely on.

But using #!py -3 works fine, so that seems like an ideal fix. I'll submit a documentation request to the just project to suggest that form when creating recipes defined in Python on Windows (I can't think of a cross-platform way that will work, unfortunately).

[pfmoore]

And indeed, that's what apparently happens in a normal script.

Actually, it appears that the fix for this issue has broken that case 🙁

At this point, I'm frustrated by this whole thing. The new behaviour (treating the shebang as relative to the script location) is a change from the behaviour described in PEP 397:

Otherwise the command is assumed to be directly ready to execute - ie. a fully-qualified path (or a reference to an executable on the PATH) optionally followed by arguments.

which was (as far as I am aware) introduced without discussion, and which is not included in the "What's new in 3.11" documentation. If I'm wrong, please point me to the discussion of this new behaviour. It's quite possible that I would have approved the change if I'd been asked about it in advance, and I'm definitely not suggesting that the new behaviour is wrong as such - but I do think that we should have provided better support and information for people who might have been relying on the old behaviour.

Now that this has already happened, I'm not sure what, if anything, should be done. My feeling is that an "after the fact" update to the what's new document, offering migration advice in the "Porting to 3.11" section for people who were using the old behaviour, is probably the best thing to do. But I'm open to other suggestions (although I'm not keen on "do nothing", as you might have guessed 🙂).

[zooba]

Yeah, there were certainly no updates made to the PEP, and we based the changes on what was in the documentation (worth noting that the PEP has a number of inconsistencies in this area, and is also dated with respect to what's considered secure practices).

The discussion unfortunately seems to be in #98732, hidden under resolved comments (one of the reasons I always disliked the plan of moving everything to GitHub, but I didn't do this on purpose ;) ). FWIW, I found this by looking at NEWS for "launcher" and going to the one that was pretty clearly about changing the shebang handling.

I still think it's highly defensible to drop the CWD search, but happy to argue about dropping the PATH search for unqualified names. There are already fixes coming in 3.11.1 to get closer to old behaviour, so no harm in bringing back more. Let's do it on a new issue though.

At this point, I'm frustrated by this whole thing.

My sincere apologies for this. I know it's hard to keep up with all the changes going on in all the different places, but it's equally hard to know whose opinion ought to be sought on every issue. It's also frustrating for me to find that every change I make frustrates someone else for not being consulted (you're not the first), even if the change was accidental ;) , but I'm not sure what the solution is other than to stop contributing. 🤷‍♂️ That would be a childish response, but so would be pinging everyone on every discussion and waiting for quorum before doing anything. I'm open to ideas, and happy to discuss offline (with Paul, that is, not any random person who happens by here).