bpo-38794: setup: support linking openssl statically

[ghost]

Currently, it's easy to compile CPython even on old distributions like RHEL5, RHEL6, Ubuntu14.04 etc. except for ssl module.

This PR adds a new ./configure option --with-openssl-static which makes it easy to compile statically against OpenSSL so CPython with ssl module can be easily compiled on systems with OpenSSL <1.0.2 (you usually don't want to install newer openssl as system libary nor mess with rpath/set LD_LIBRARY_PATH every time you run python).

With this change, if --with-openssl-static is not set everything behaves like before.

When/if this PR is merged, autoreconf needs to be run to regenerate bundled ./configure script etc.. I am not sure what's the recommended process to run autoreconf (version, parameters, ...) so I have not comitted the generated files.

Installing CPython including ssl on system as old as RHEL5 with this option only takes (after installing required build dependencies from rhel5 repositories and installing libffi(-devel) rpm):

wget https://www.openssl.org/source/openssl-1.0.2t.tar.gz
tar xf openssl-1.0.2t.tar.gz
cd openssl-1.0.2t
./config --openssldir=/etc/pki/tls -fPIC
make

wget https://www.python.org/ftp/python/3.7.5/Python-3.7.5.tgz
tar xf Python-3.7.5.tgz
cd Python-3.7.5
./configure --with-openssl-static=path_to_just_compiled_ssl --prefix=prefix_path
make
make install

https://bugs.python.org/issue38794

[ghost]

The travis build is failing because autoreconf has not run.

See output line 1319:
clang -pthread -shared -L/home/travis/multissl/openssl/1.1.1d/lib -L/home/travis/multissl/openssl/1.1.1d/lib -L/home/travis/multissl/openssl/1.1.1d/lib -I/home/travis/multissl/openssl/1.1.1d/include build/temp.linux-x86_64-3.9/home/travis/build/python/cpython/Modules/_ssl.o @OPENSSL_STATIC_ROOT@/libssl.a @OPENSSL_STATIC_ROOT@/libcrypto.a -L/home/travis/multissl/openssl/1.1.1d/lib -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -o build/lib.linux-x86_64-3.9/_ssl.cpython-39-x86_64-linux-gnu.so

After autoreconf OPENSSL_STATIC_ROOT would be defined as "" and then ssl extension would be compiled just like before this change.

[brandtbucher]

Thanks for the PR @luv!

[brandtbucher]

Please run autoreconf... don't worry, it's part of the workflow for PRs like these.

This should also have a NEWS entry. It can just be something like:

Support linking OpenSSL statically with a new ``--with-openssl-static`` configure option.

[brandtbucher]

Looks good to me.

[brandtbucher]

CC @tiran

[ned-deily]

Has anyone looked at just using Modules/Setup to provide this? We often forget about it but one of the reasons it's still there is to do non-standard configurations like this that aren't handled by setup.py. If there is some problem with defining it there, we should first look at fixing that before adding yet another option to configure.ac.

[ghost]

@ned-deily Sounds good and if that option is easily discoverable it solves the problem just as well.

However I am not putting in the effort to reimplement this using Modules/Setup after jumping through all the hoops (no problem here) to get my PR approved only to get it rejected for dubious reasons by another maintainer. Not to mention that almost a year ago I asked that maintainer to close this PR with a proper explanation if he is not happy about adding this functionality. It also makes me wonder if that maintainer opposes all programs written in go because they link libraries statically and docker and all other containers because they dare to bundle openssl as well.

Anyway, you need to find someone else to do that work now.

[tiran]

Static linking for ssl and hashlib module has been proposed multiple times. Python will not have official support for static linking of OpenSSL. There are just too many ways it can go wrong and I lack both interest and time to support the feature. For example this implementation will likely break 3rd party extension with symbol conflicts.

You can either use Modules/Setup.local approach or the undocumented/unsupported hack in 3.10.