(cherry picked from commit 9977629)

#1597)

(cherry picked from commit f6e863d)

…H-1595) (#1605)

Warnings emitted when compile a regular expression now always point
to the line in the user code.  Previously they could point into inners
of the re module if emitted from inside of groups or conditionals..
(cherry picked from commit c7ac728)

(cherry picked from commit 1bd7d29)

…H-1612) (GH-1618)

Catch the Windows socket WSAEINVAL error (code 10022) in imaplib and
poplib on shutdown(SHUT_RDWR): An invalid operation was attempted

This error occurs sometimes on SSL connections.
(cherry picked from commit 83a2c28)

test_thread: setUp() now uses support.threading_setup() and
support.threading_cleanup() to wait until threads complete to avoid
random side effects on following tests.

Co-Authored-By:  Grzegorz Grzywacz <[email protected]>
(cherry picked from commit 79ef7f8)

…1515) (#1626)

compiled out-of-tree (builddir != srcdir). (see also bpo-15366)
(cherry picked from commit dbdea62)

…1627)

Under *spawn* and *forkserver* start methods, SimpleQueue.empty() could
raise AttributeError due to not setting _poll in __setstate__.

test_is_alive_after_fork() now joins directly the thread to avoid the
following warning added by bpo-30357:

Warning -- threading_cleanup() failed to cleanup 0 threads
after 2 sec (count: 0, dangling: 21)

Use also a different exit code to catch generic exit code 1.
(cherry picked from commit f8d05b3)

Workaround for a regrtest bug.

…names. (GH-1652) (#1674)

Based on patch by Eryk Sun.
(cherry picked from commit d896985)

Based on patches by Duane Griffin and Tim Mitchell.
(cherry picked from commit 753bca3)

(cherry picked from commit 906f533)

(cherry picked from commit 8175547)

…H-146) (#1750)

(cherry picked from commit d618c8c)

Ran the docstrings through spell checker, and fixed spelling issues.

(cherry picked from commit 21b3e04)

…1747)

(cherry picked from commit 93fc20b)

The library does not enforce compliance with the HTTP protocol,
so violations are not technically disallowed. Extend the stream's
description to avoid suggesting that intentional protocol violations are
not supported.
(cherry picked from commit a083c8e)

…oop on exc (GH-1683) (#1816)

* bpo-30414: multiprocesing.Queue._feed do not break from main loop on exc

Queue background running thread was not handling exceptions correctly.
Any exception occurred inside thread (putting unpickable object) cause
feeder to finish running. After that every message put into queue is
silently ignored.

* bpo-30414: multiprocesing.Queue._feed do not break from main loop on exc

Queue background running thread was not handling exceptions correctly.
Any exception occurred inside thread (putting unpickable object) cause
feeder to finish running. After that every message put into queue is
silently ignored.
(cherry picked from commit bc50f03)

…) (#1829)

Unlikely to the simplejson module, json.JSONDecodeError doesn't accept the end argument.
(cherry picked from commit 5becf38)

Also document that some attributes may be None.
(cherry picked from commit 12d6b5d)

#1288)

(cherry picked from commit 9616a82)

…#1857)

(cherry picked from commit e405d4b)

…1869)

The data model section of the language reference was written well
before the zero-argument form of super() was added.

To avoid giving the impression that they're doing something
unusual, this updates the description of `__new__` and `__init__`
to use the zero-argument form.

Patch by Cheryl Sabella.
(cherry picked from commit 12b1c18)

(cherry picked from commit eba68e2)

(cherry picked from commit f34c685)

(cherry picked from commit a5aa72a)

GH-1895)

(cherry picked from commit 56ddfd2)

…H-1896)

The "iterable iterable" phrasing created confusion between the term
reference and the parameter name.

This simplifies the phrasing to just use the parameter name
without linking directly to the term definition.
(cherry picked from commit 08e2f35)

(cherry picked from commit 0737ee2)

…sses (#1676) (#1902)

…H-344) (GH-1914)

(cherry picked from commit cdb89cd)

Initially the macOS builds are allowed to fail until such time that they can be determined to be stable and not add an unacceptable amount of time to the overall Travis-passing process.
(cherry picked from commit 21c2dd7)

… being raised (GH-1920)

We get `ERROR_BAD_NETPATH` (53) on AppVeyor which is translated to
ENOENT (2).
(cherry picked from commit 7a99625)

Also on the short-lived `buildbot-custom` branch.
(cherry picked from commit d3bedf3)

Update the code example in Functions and Methods section
Remove objtype argument in MethodType
(cherry picked from commit 1bced56)

…H-1944)

* bpo-30538: Update count() in Functional HOWTO
* bpo-30538: Update enumerate() arguments in Functional HOWTO
(cherry picked from commit 9be4ff3)

Also weakens the 'should this be run?' regex to allow all builds when .travis.yml changes.
(cherry picked from commit c53b13b)

(cherry picked from commit 167e0fc)

(cherry picked from commit 48fb766)

Clarify that `two-pass` buffer can only be dumped once, and it prints out all text sent to it during all processing, even from Clinic blocks *after* the dumping point.

* Update config.{guess,sub} from gnu.org..
(cherry picked from commit 6f46683)

If pass a server_hostname= that fails IDNA decoding to SSLContext.wrap_socket or SSLContext.wrap_bio, then the SSLContext object had a spurious Py_DECREF called on it, eventually leading to segfaults.
(cherry picked from commit 65ece7c)

…2000)

* Fix bpo-30584

* Adding a comment mentionning the bpo and explaining what is the identifier

* Add Denis Osipov to Misc/ACKS
(cherry picked from commit 897bba7)

On Windows, subprocess.Popen.communicate() now also ignore EINVAL
on stdin.write() if the child process is still running but closed the
pipe.
(cherry picked from commit d52aa31)

(cherry picked from commit 824f687)

Avoid a deadlock when the waiter who is about to take the lock is
cancelled

Issue #27585

(cherry picked from commit b4e5fee)

…2048)

…H-2053)

This will allow for centralized management of the Codecov config to prevent skew as well as easier management going forward.

Closes python/core-workflowGH-81.
(cherry picked from commit 11ffb45)

…es (GH-1536) (GH-2056)

Document that assertNotRegexpMatches is a deprecated alias for assertNotRegex.
(cherry picked from commit 74921ed)

(cherry picked from commit 42e3acd)

(cherry picked from commit 34792d2)

…e with certain servers (GH-409) (#2063)

* bpo-29406: asyncio SSL contexts leak sockets after calling close with certain servers (#409)

(cherry picked from commit a608d2d)

* [3.5] bpo-29406: asyncio SSL contexts leak sockets after calling close with certain servers (GH-409)

* asyncio SSL contexts leak sockets after calling close with certain servers

* cleanup _shutdown_timeout_handle on _fatal_error.
(cherry picked from commit a608d2d)

…H-2066)

Mention that fnmatchcase does not call normcase, and fnmatch does.
(cherry picked from commit e5f6e86)

Also updates checked-in line endings on some files.

This PR contains two updates to typing module:

- Support ContextManager on all versions (original PR by Jelle Zijlstra).
- Add generic AsyncContextManager..
(cherry picked from commit 29fda8d)

…called. (#2110)

…ing close with certain servers (GH-409) (#2063)" (#2113)

This reverts commit 1395c58.

…2080) (GH-2093)

(cherry picked from commit 0afbabe)

Also fixes some line endings missed in GH-840 backport.

import json before using json module
(cherry picked from commit 1dbce04)

…H-2034) (#2122)

The traceback no longer displayed for SystemExit raised in a callback registered by atexit.
(cherry picked from commit 3fd54d4)

(cherry picked from commit dc980df)

Move up the discussion about 'with' keyword, so it appears earlier in the document.
(cherry picked from commit bd4e9e0)

(cherry picked from commit d79c1d4)

Add a test to check the current MAGIC_NUMBER against the
expected number for the release if the current release is
at candidate or final level. On test failure, describe to
the developer the procedure for changing the magic number.

This ensures that pre-merge CI will automatically pick up
on magic number changes in maintenance releases (and
explain why those are problematic), rather than relying on
all core developers to be aware of the implications of
such changes.

…2159) (#2167)

* bpo-24484: Avoid race condition in multiprocessing cleanup

The finalizer registry can be mutated while inspected by multiprocessing
at process exit.

* Use test.support.start_threads()

* Add Misc/NEWS.
(cherry picked from commit 1eb6c00)

Replace platform with platforms.
(cherry picked from commit 4ebf03d)

…2173) (GH-2197)

(cherry picked from commit 184bd82)

The public cyrus.andrew.cmu.edu IMAP server (port 993) doesn't accept
TLS connection using our self-signed x509 certificate. Remove the two
tests which are already skipped.

* bpo-29591: Upgrade Modules/expat to libexpat 2.2

* bpo-29591: Restore Python changes on expat

* bpo-29591: Remove expat config of unsupported platforms

Remove the configuration (Modules/expat/*config.h) of unsupported
platforms:

* Amiga
* MacOS Classic on PPC32
* Open Watcom

* bpo-29591: Remove useless XML_HAS_SET_HASH_SALT

The XML_HAS_SET_HASH_SALT define of Modules/expat/expat.h became
useless since our local expat copy was upgrade to expat 2.1 (it's now
expat 2.2.0).

(cherry picked from commit 23ec4b5)

) (#2216)

(cherry picked from commit 0d32218)

…4 and v6. (GH-879) (#2218)

the original logic was just comparing the network address
but this is wrong because if the network address is equal then
we need to compare the ip address for breaking the tie

add more ip_interface comparison tests.
(cherry picked from commit 7bd8d3e)

…1663)

…2222)

In rare circumstances PyImport_Import() could return NULL without raising
an error.
(cherry picked from commit 145541c)

* Rewrite code to pass slaveargs from the master process to worker
  processes: reuse the same code of the Python master branch
* Move code to initialize tests in a new setup_tests() function,
  similar change was done in the master branch
* In a worker process, call setup_tests() with the namespace built
  from slaveargs to initialize correctly tests

Before this change, warm_caches() was not called in worker processes
because the setup was done before rebuilding the namespace from
slaveargs. As a consequence, the huntrleaks feature was unstable. For
example, test_zipfile reported randomly false positive on reference
leaks.

(cherry picked from commit 258bfc4)

bpo-23890: unittest.TestCase.assertRaises() now manually breaks a
reference cycle to not keep objects alive longer than expected.
(cherry picked from commit bbd3cf8)

…H-2231)

`PyModule_New()` now refers to `PyModule_NewObject()`
(cherry picked from commit 2d0afef)

…icy (GH-595) (GH-2234)

If max_line_length=None is specified while using the Compat32 policy,
it is no longer ignored..
(cherry picked from commit b459f74)

…#2247)

This reverts commit ab4ea09.

…to regrtest (#2250)

* bpo-30540: regrtest: add --matchfile option

* Add a new option taking a filename to get a list of test names to
  filter tests.
* support.match_tests becomes a list.
* Modify run_unittest() to accept to match the whole test identifier,
  not just a part of a test identifier.

For example, the following command only runs test_access()
of the FileTests class of test_os:

$ ./python -m test -v -m test.test_os.FileTests.test_access test_os

* bpo-30523: regrtest: Add --list-cases option

* Add --list-cases option to regrtest
* Add get_abs_module() function, use it in list_cases()
* Add ns mandatory positional argument to runtest()
  and runtest_inner()
* Add file optional parameter to printlist()

Co-Authored-By: Louie Lu <[email protected]>

…m master to 3.5 (#2279)

* bpo-30383: regrtest: prepend testdir to sys.path

* bpo-30383: Backport test_regrtest

* regrtest: rename --slow option to --slowest

The old --slow syntax is still accepted.

* regrtest: add a single oneliner summary

Example: "Tests result: SUCCESS"

* test_regrtest: add test_coverage()

regrtest now also displays the number of successful tests when
coverage is used.

* test_regrtest: add test_crashed()

Handle correctly crashing test: account the crash has a failed test,
but continue to run other tests.

* regrtest: backport --list-tests feature

* regrtest: backport --fromfile enhancements

* regrtest: backport displaying progress enhancements

* test_regrtest: backport test_randseed()

* regrtest: Fix --coverage on Windows

Don't ignore any directory anymore. Change backported from master.

…n vcvarsall.bat (#2252) (#2281)

* Fixes build scripts to find msbuild.exe and stop relying on vcvarsall.bat
Also fixes bdist_wininst.vcxproj to use correct version in generated name.

…) (#2272)

* [3.5] bpo-29887: Test normalization now fails if download fails (GH-905)

* test_normalization fails if download fails

bpo-29887. The test is still skipped if "-u urlfetch" option is not
passed to regrtest (python3 -m test -u urlfetch test_normalization).

* Fix ResourceWarning in test_normalization

bpo-29887: Fix ResourceWarning in test_normalization if tests are
interrupted by CTRL+c.
(cherry picked from commit 722a3af)

* bpo-29887: test_normalization handles PermissionError (#1196)

Skip test_normalization.test_main() if download raises a permission
error.
(cherry picked from commit d13d547)

* no f-strings :(

… (#2290)

The current regex based splitting produces a wrong result. For example::

  http://abc#@def

Web browsers parse that URL as ``http://abc/#@def``, that is, the host
is ``abc``, the path is ``/``, and the fragment is ``#@def``.
(cherry picked from commit 90e01e5)

splithost() expects an URL starting with "//" not with "http://".

…ext module. (GH-2266) (#2298)

They now always return bytes.

Updated the gettext documentation.
(cherry picked from commit 26cb465)

New file: Modules/expat/siphash.h.
(cherry picked from commit 5ff7132)

CPython workflow is changing!  We're going to start using "blurb"
to manage Misc/NEWS entries:
    https://github.com/python/core-workflow
(This will be a big win for release managers, honest.)

This checkin simply populates the "Misc/NEWS.d" subdirectory tree
so that people can start putting their news entries in there.
No other changes (yet).

When a class and its subclass are present, the latter is skipped.
(cherry picked from commit 6580c19)

… (GH-2341)

(cherry picked from commit b066edf)

…2352)

(cherry picked from commit 32cb968)

… (#2350)

* bpo-30726: PCbuild _elementtree: remove duplicate defines (#2348)

bpo-30726, bpo-29591: libexpat 2.2.1 of Modules/expat/ now uses
a winconfig.h configuration file which already defines:

* XML_NS
* XML_DTD
* BYTEORDER=1234
* XML_CONTEXT_BYTES=1024
* HAVE_MEMMOVE

Remove these defines from PCbuild/_elementtree.vcxproj to prevent
compiler warnings.

Co-Authored-By: Jeremy Kloth <[email protected]>
(cherry picked from commit c8fb58b)

* bpo-30726: Fix elementtree warnings on Windows due to expat upgrade (#2319)

* bpo-30726: Fix elementtree warnings on Windows

Caused by usage of `getenv` which should be safe. And a few integer
truncations which should also be ok.

* bpo-30726: Don't ignore libexpat warnings which haypo intends to fix upstream

(cherry picked from commit 87c6555)

…s on Windows. (GH-2325) (#2361)

Prevent passing other invalid environment variables and command arguments..
(cherry picked from commit d174d24)

…ge() (GH-2268) (GH-2365)

Bug didn't manifest itself when importing a module with source as .py files are always the first on the search path. The issue only showed up in bytecode-only packages where the calculated file path would be ``__init__.py/__init__.pyc``.

Patch by Alexandru Ardelean.
(cherry picked from commit c38e32a)

defiend -> defined
(cherry picked from commit cab4692)

) (#2379)

(cherry picked from commit 0ee32c1)

)

(cherry picked from commit 70cb187)

… names (GH-2382) (#2392)

in `os.putenv()` and `os.spawn*()`..
(cherry picked from commit 7770394)

bpo-30764, bpo-29335: test_child_terminated_in_stopped_state() of
test_subprocess now uses support.SuppressCrashReport() to prevent the
creation of a core dump on FreeBSD.
(cherry picked from commit cdee3f1)

… not to (GH-2403) (#2419)

* bpo-30765: Avoid blocking when PyThread_acquire_lock() is asked not to lock

This is especially important if PyThread_acquire_lock() is called reentrantly
(for example from a signal handler).

* Update 2017-06-26-14-29-50.bpo-30765.Q5iBmf.rst

* Avoid core logic when taking the mutex failed.
(cherry picked from commit f84ac42)

* bpo-30523: regrtest --list-cases --match (#2401)

* regrtest --list-cases now supports --match and --match-file options.
  Example: ./python -m test --list-cases -m FileTests test_os
* --list-cases now also sets support.verbose to False to prevent
  messages to stdout when loading test modules.
* Add support._match_test() private function.
(cherry picked from commit ace56d5)

* bpo-30764: regrtest: add --fail-env-changed option (#2402)

* bpo-30764: regrtest: change exit code on failure

* Exit code 2 if failed tests ("bad")
* Exit code 3 if interrupted

* bpo-30764: regrtest: add --fail-env-changed option

If the option is set, mark a test as failed if it alters the
environment, for example if it creates a file without removing it.

(cherry picked from commit 63f54c6)

* bpo-30776: reduce regrtest -R false positives (#2422)

* Change the regrtest --huntrleaks checker to decide if a test file
  leaks or not. Require that each run leaks at least 1 reference.
* Warmup runs are now completely ignored: ignored in the checker test
  and not used anymore to compute the sum.
* Add an unit test for a reference leak.

Example of reference differences previously considered a failure
(leak) and now considered as success (success, no leak):

    [3, 0, 0]
    [0, 1, 0]
    [8, -8, 1]
(cherry picked from commit 48b5c42)

…2447)

…ng(). (GH-2285) (GH-2443)  (#2448)

And use it instead of PyUnicode_AsWideCharString() if appropriate.

_PyUnicode_AsWideCharString(unicode) is like PyUnicode_AsWideCharString(unicode, NULL), but
raises a ValueError if the wchar_t* string contains null characters.
(cherry picked from commit e613e6a).
(cherry picked from commit 0edffa3)

Clarify that the break statement breaks out of the innermost enclosing for or while loop.
(cherry picked from commit 36fc896)

…H-2302) (#2463)

Based on patch by Victor Stinner.

Add private C API function _PyUnicode_AsUnicode() which is similar to
PyUnicode_AsUnicode(), but checks for null characters..
(cherry picked from commit f7eae0a)

Forgetting to call Process.join() can keep some resources alive.
(cherry picked from commit a79f8fa)

…H-2470) (#2472)

* Clear potential ref cycle between Process and Process target

Besides Process.join() not being called, this was an indirect cause of bpo-30775.
The threading module already does this.

* Add issue reference.
(cherry picked from commit 79d37ae)

…H-2493) (#2498)

* bpo-30807: signal.setitimer() may disable the timer by mistake

* Add NEWS blurb
(cherry picked from commit 729780a)

bpo-11798, bpo-16662, bpo-16935, bpo-30813: Skip
test_discover_with_module_that_raises_SkipTest_on_import() and
test_discover_with_init_module_that_raises_SkipTest_on_import() of
test_unittest when hunting reference leaks using regrtest.
(cherry picked from commit e4f9a2d)

* bpo-30280: asyncio now cleans up threads

asyncio base TestCase now uses threading_setup() and
threading_cleanup() of test.support to cleanup threads.

* asyncio: Fix TestBaseSelectorEventLoop cleanup

bpo-30280: TestBaseSelectorEventLoop of
test.test_asyncio.test_selector_events now correctly closes the event
loop: cleanup its executor to not leak threads.

Don't override the close() method of the event loop, only override
the_close_self_pipe() method.

(cherry picked from commit b903067)

GH-2533)

Improve the description of the high and low parameters for set_write_buffer_limits.
Also fixed a small grammatical issue.
(cherry picked from commit 5200a7c)

… from 3.6 to 3.5 (#2540)

* [3.6] bpo-29512, bpo-30776: Backport regrtest enhancements from master to 3.6 (#2513)

* bpo-29512: Add test.bisect, bisect failing tests (#2452)

Add a new "python3 -m test.bisect" tool to bisect failing tests.

It can be used to find which test method(s) leak references, leak
files, etc.
(cherry picked from commit 84d9d14)

* bpo-30776: regrtest: reduce memleak false positive (#2484)

Only report a leak if each run leaks at least one memory block.
(cherry picked from commit beeca6e)

(cherry picked from commit a3ca94d)

* bpo-30764: Fix regrtest --fail-env-changed --forever (#2536) (#2539)

--forever now stops if a fail changes the environment.
(cherry picked from commit 5e87592)
(cherry picked from commit 4132adb)

…er it (GH-2409). (#2557)

(cherry picked from commit 8a8d285)

FXpy doesn't have a Python 3 port and it only
supports Python 2.2 and older versions.

Reported by Alex Walters on [email protected].

(cherry picked from commit d3ed287)

(cherry picked from commit 7526cad)

* bpo-30855: Trying to fix test_use on Windows.

Avoid possible weird behavior of WideInt convertion.
"winfo id" always returns string hexadecimal representation.

(cherry picked from commit b9d6724)

* bpo-30855: Trying to fix test_use on Windows.

(cherry picked from commit 29a2f7c)

(cherry picked from commit 76c567e)

(cherry picked from commit be5ebe5)

Leading whitespace was incorrectly dropped during folding of certain lines in the _header_value_parser's folding algorithm.  This makes the whitespace handling code consistent.

If history-length is set in .inputrc, and the history file is double the
history size (or more), history_get(N) returns NULL, and python
segfaults. Fix that by checking for NULL return value.

It seems that the root cause is incorrect handling of bigger history in
readline, but Python should not segfault even if readline returns
unexpected value.

This issue affects only GNU readline. When using libedit emulation
system history size option does not work.

multiprocessing.Queue.join_thread() now waits until the thread
completes, even if the thread was started by the same process which
created the queue.

Fix the following warning which occurs randomly when running
test_handle_called_with_mp_queue of test_logging.QueueListenerTest:

Warning -- threading_cleanup() failed to cleanup -1 threads after 4 sec (count: 0, dangling: 1)
(cherry picked from commit 3b69d91)

Handle getattr(copy, 'deepcopy') error in _elementtree module
initialization.
(cherry picked from commit b136f11)

…hen (GH-2634) (#2657)

called with bytes-like argument.
(cherry picked from commit 1180e5a)

…object.c. (GH-2623) (#2659)

Based on patch by Victor Stinner.
(cherry picked from commit 64e461b)

)

(cherry picked from commit f523255)

…json (GH-2690) (#2693)

encoder and decoder.
(cherry picked from commit d3aaa2f)