C API: Consider adding a public PyList_Extend() function

[vstinner]

Feature or enhancement

The private _PyList_Extend() function has been removed in Python 3.13: see PR #108451.

@scoder asked what is the replacement for this removed function.

The obvious replacement is PyObject_CallMethod(list, "extend", "O", arg): call the list.extend() method. But it's slower, PyObject_CallMethod() has to get the method and decode the "O" format string.

Another replacement is PyList_SetSlice(L, PY_SSIZE_T_MAX, PY_SSIZE_T_MAX, arg) which is less straightforward, but it's efficient.

I propose adding a public PyList_Extend() function. The list type is commonly used in C extensions, it's a convenient API to create a collection when the length is not known is advance.

I don't think that performance is really the problem here since PyList_SetSlice(L, PY_SSIZE_T_MAX, PY_SSIZE_T_MAX, arg) is available. The problem is more to more the function easier to discover and make the API easier to use. Also, it should help people to migrate away from the removed function.

There is already a public PyDict_Update() API for the dict type, it is part of the limited C API.

If we add a function for the list type, should we also add PySet_Update() "for completeness"? The private _PySet_Update() was removed in Python 3.13.

cc @serhiy-storchaka @encukou

---

A code search on _PyList_Extend in PyPI top 5,000 projects (2023-07-04) found 4 projects using this function:

• Cython (0.29.36)
• catboost (1.2)
• mypy (1.4.1)
• pyrsistent (0.19.3)

Logs:

PYPI-2023-07-04/catboost-1.2.tar.gz: catboost-1.2/catboost_all_src/contrib/tools/cython/Cython/Compiler/Builtin.py: BuiltinMethod("extend",  "TO",   "r", "__Pyx_PyList_Extend",
PYPI-2023-07-04/catboost-1.2.tar.gz: catboost-1.2/catboost_all_src/contrib/tools/cython/Cython/Compiler/ExprNodes.py: extend_func = "__Pyx_PyList_Extend"
PYPI-2023-07-04/catboost-1.2.tar.gz: catboost-1.2/catboost_all_src/contrib/tools/cython/Cython/Utility/Optimize.c: static CYTHON_INLINE int __Pyx_PyList_Extend(PyObject* L, PyObject* v) {
PYPI-2023-07-04/catboost-1.2.tar.gz: catboost-1.2/catboost_all_src/contrib/tools/cython/Cython/Utility/Optimize.c: PyObject* none = _PyList_Extend((PyListObject*)L, v);

PYPI-2023-07-04/Cython-0.29.36.tar.gz: Cython-0.29.36/Cython/Compiler/Builtin.py: BuiltinMethod("extend",  "TO",   "r", "__Pyx_PyList_Extend",
PYPI-2023-07-04/Cython-0.29.36.tar.gz: Cython-0.29.36/Cython/Compiler/ExprNodes.py: extend_func = "__Pyx_PyList_Extend"
PYPI-2023-07-04/Cython-0.29.36.tar.gz: Cython-0.29.36/Cython/Utility/Optimize.c: static CYTHON_INLINE int __Pyx_PyList_Extend(PyObject* L, PyObject* v) {
PYPI-2023-07-04/Cython-0.29.36.tar.gz: Cython-0.29.36/Cython/Utility/Optimize.c: PyObject* none = _PyList_Extend((PyListObject*)L, v);

PYPI-2023-07-04/mypy-1.4.1.tar.gz: mypy-1.4.1/mypyc/lib-rt/dict_ops.c: PyObject *res = _PyList_Extend((PyListObject *)list, view);
PYPI-2023-07-04/mypy-1.4.1.tar.gz: mypy-1.4.1/mypyc/lib-rt/dict_ops.c: PyObject *res = _PyList_Extend((PyListObject *)list, view);
PYPI-2023-07-04/mypy-1.4.1.tar.gz: mypy-1.4.1/mypyc/lib-rt/dict_ops.c: PyObject *res = _PyList_Extend((PyListObject *)list, view);
PYPI-2023-07-04/mypy-1.4.1.tar.gz: mypy-1.4.1/mypyc/lib-rt/list_ops.c: return _PyList_Extend((PyListObject *)o1, o2);

PYPI-2023-07-04/pyrsistent-0.19.3.tar.gz: pyrsistent-0.19.3/pvectorcmodule.c: PyObject *retVal = _PyList_Extend((PyListObject *)self->appendList, args);

Cython uses the function to implement its own __Pyx_PyList_Extend() API:

static CYTHON_INLINE int __Pyx_PyList_Extend(PyObject* L, PyObject* v) {
#if CYTHON_COMPILING_IN_CPYTHON
    PyObject* none = _PyList_Extend((PyListObject*)L, v);
    if (unlikely(!none))
        return -1;
    Py_DECREF(none);
    return 0;
#else
    return PyList_SetSlice(L, PY_SSIZE_T_MAX, PY_SSIZE_T_MAX, v);
#endif
}

[serhiy-storchaka]

PyDict_Update() is not trivial. PyList_Extend() is trivial, literally one-liner.

#define PyList_Extend(list, arg) PyList_SetSlice((list), PY_SSIZE_T_MAX, PY_SSIZE_T_MAX, (arg))

I don't mind adding it as a macro. Together with PyList_Clear().

#define PyList_Clear(list) PyList_SetSlice((list), 0, PY_SSIZE_T_MAX, NULL)