Shared Hosting — /lib64/libc.so.6: version GLIBC_2.14 not found


Shared Hosting — /lib64/libc.so.6: version GLIBC_2.14 not found

29/04/2018, by Justin Cook



Shared Hosting — /lib64/libc.so.6: version GLIBC_2.14 not found

Have you been in a situation with limited access lacking compilers and other common utilities, and all you need is that one thing that’s so close yet so far away?

Some clients prefer using CPanel hosting solutions as it has been around over two decades and is used by many a web developer. While it is perfectly fine in many situations, your ability to extend the environment can become painful as you may encounter blockers seemingly impossible to overcome if you are in a shared environment with zealous security.

Recently, one client switched providers. They assumed the toolchain would be portable. However, the new provider had a chroot Python implementation with no read access to the libraries. Therefore, the virtualenv was no longer rebuildable, and for another reason, not even usable. They were seemingly blocked. This was unfortunate as the new provider’s environment had far better performance and support. This is where we had to get involved.

The issue was that the entire environment was copied, but the Python vitualenv would not run. Upon inspection, the error was the following:

[~]# python
python: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by python)

This is a Glibc version mismatch. The virualenv was copied from an installation built against a newer version. There is more than one way to correct this [1], but the most obvious and straightforward is building against a compatible version of Glibc that supports all the symbol versions necessary for the binary and shared objects.

The first thing we should look at is the Glibc version [2] on the shared host:

[~]# /lib64/libc.so.6
GNU C Library stable release version 2.5, by Roland McGrath et al.
Copyright (C) 2006 Free Software Foundation, Inc.
...

If you are surprised, you are not alone! In 2018, this is a very old version given Red Hat Enterprise Linux 7 ships glibc-2.17 and Red Hat Enterprise Linux 6 sports glibc-2.12. The latest version at the time of writing is glibc-2.27 [3]. This could end up very complicated, but let’s see just how much work will be necessary. After all, we only had one error complaining about GLIBC_2.14!

We need to check the Glibc symbols of each shared object and binary file to determine what is not found and the version we can build against. Let’s start with creating our development environment to diagnose.

$ mkdir centos7 ; cd centos7 ; vagrant init jhcook/centos7 ; vagrant up
...
centos7$ vagrant ssh
[[email protected] ~]$ sudo yum install -y git gcc zlib-devel readline-devel bzip2-devel sqlite-devel openssl-devel
...
[[email protected] ~]$ curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
# Follow the instructions provided to update your shell profile then update your environment with `exit` and `vagrant ssh`
...
[[email protected] ~]$ pyenv install 2.7.14
...
[[email protected] ~]$ pyenv global 2.7.14
[[email protected] ~]$ pip install virtualenv
...
[[email protected] ~]$ pip install --upgrade pip
...

Now you have the build tools and Python 2.7.14 installed with the latest packages necessary to investigate. Since this is exactly what we need to install on the shared host, you can have a look at the Glibc symbols. Fingers crossed, we’ll create a nice and tidy ordered output of the Glibc symbols and the version built against.

[[email protected] ~]$ find .pyenv/versions/2.7.14 -name '*.so' -o -name python2.7 -type f -print0 | xargs -0 objdump -T | cut -c 20- | awk ‘/GLIBC_2.14/{print$4,$5}’ | sort | uniq
GLIBC_2.14 memcpy

That is one GLIBC_2.14 function! With a bit of Google, we came across a nice explanation [4] of the culprit and quicky discovered we should be able to build against glibc-2.12 which is provided in Red Hat Enterprise Linux 6. Although a lot of code has been written and modified since 2006, the ABI will be compatible, so we will be safe.

Let’s rinse and repeat with another box of that version. One thing to note, we use the venerable PyEnv [5] to create and manage multiple Python environments. But, when copying PyEnv versions across nodes, the build must come from the same user as it’s embedded deep in the build.

$ mkdir centos6 ; cd centos6 ; vagrant init jhcook/centos6 ; vagrant up
...
centos6$ vagrant ssh
[[email protected] ~]$ sudo yum install -y git gcc zlib-devel readline-devel bzip2-devel sqlite-devel openssl-devel
...
[[email protected] ~]$ sudo useradd remoteuser
...
[[email protected] ~]$ sudo su - remoteuser
[[email protected] ~]$ curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
# Follow the instructions provided to update your shell profile then update your environment with `exit` and `sudo su - remoteuser`
...
[[email protected] ~]$ pyenv install 2.7.14
...
[[email protected] ~]$ pyenv global 2.7.14
[[email protected] ~]$ pip install virtualenv
...
[[email protected] ~]$ pip install --upgrade pip
...

Now that’s out of the way, we should have a like-for-like Python environment. We can use our Bash foo and see what the difference is and if it will be supported by the ancient version of Glibc we must unfortunately deal with.

[[email protected] ~]$ find .pyenv/versions/2.7.14 -name '*.so' -o -name python2.7 -type f -print0 | xargs -0 objdump -T | cut -c 20- | awk '/memcpy/{print$4,$5}' | sort | uniq
GLIBC_2.2.5 memcpy

Wow! memcpy symbol now indicates GLIBC_2.2.5, and we expected to see this given what we found [4]. If you look further, you will see all versions are less than 2.5 and therefore should be supported by the ancient glibc-2.5. All we have to do is transfer the build across to the shared host and see what we come up with.

[[email protected] ~]$ cd .pyenv/versions
[[email protected] versions]$ tar zcvf ~/pyenv-2.7.14.tar.gz 2.7.14
...
[[email protected] versions]$ exit
[[email protected] ~]# mv ~remoteuser/pyenv-2.7.14.tar.gz /vagrant/mv ~remoteuser/pyenv-2.7.14.tar.gz /vagrant/
[[email protected] ~]# exit
[[email protected] ~]$ exit
$ scp pyenv-2.7.14.tar.gz [email protected]:
...
$ ssh [email protected]
[email protected] [~]# cd .pyenv/versions
[email protected] [~/.pyenv/versions]# tar zxvf ~/pyenv-2.7.14.tar.gz
...
[email protected] [~/.pyenv/versions]# cd ; pyenv global 2.7.14
[email protected] [~]# virtualenv awsVE
New python executable in /home/remoteuser/awsVE/bin/python2.7
Also creating executable in /home/remoteuser/awsVE/bin/python
Installing setuptools, pip, wheel...done.
[email protected] [~]# . awsVE/bin/activate
(awsVE) [email protected] [~]# pip install awscli
Collecting awscli
...
Successfully installed PyYAML-3.12 awscli-1.15.10 botocore-1.10.10 colorama-0.3.7 docutils-0.14 futures-3.2.0 jmespath-0.9.3 pyasn1-0.4.2 python-dateutil-2.7.2 rsa-3.4.2 s3transfer-0.1.13 six-1.11.0

Result!

Lastly, it would be smart to test the Python build on the remote host. But, given resource constraints, that may prove difficult.

[email protected] [~]# python -m test -j1
== CPython 2.7.14 (default, Apr 27 2018, 21:34:31) [GCC 4.4.7 20120313 (Red Hat 4.4.7-18)]
==   Linux-3.12.18-clouder0-x86_64-with-glibc2.3 little-endian
==   /tmp/test_python_12484
== CPU count: 1
Run tests in parallel using 1 child processes
Traceback (most recent call last):
  File "/home/remoteuser/.pyenv/versions/2.7.14/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/remoteuser/.pyenv/versions/2.7.14/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/remoteuser/.pyenv/versions/2.7.14/lib/python2.7/test/__main__.py", line 3, in 
    regrtest.main_in_temp_cwd()
  File "/home/remoteuser/.pyenv/versions/2.7.14/lib/python2.7/test/regrtest.py", line 2024, in main_in_temp_cwd
    main()
  File "/home/remoteuser/.pyenv/versions/2.7.14/lib/python2.7/test/regrtest.py", line 802, in main
    display_progress(test_index, text)
  File "/home/remoteuser/.pyenv/versions/2.7.14/lib/python2.7/test/regrtest.py", line 646, in display_progress
    load_avg_1min = os.getloadavg()[0]
OSError: Load averages are unobtainable

References
1. https://software.intel.com/en-us/articles/how-to-use-the-intel-cluster-checker-v3-sdk-on-a-cluster-using-multiple-linux-distributions
2. http://ask.xmodulo.com/check-glibc-version-linux.html
3. https://sourceware.org/glibc/wiki/Glibc%20Timeline
4. https://snorfalorpagus.net/blog/2016/07/17/compiling-python-extensions-for-old-glibc-versions/
5. https://github.com/pyenv/pyenv
6. Shmuel Csaba Otto Traian [CC BY-SA 3.0 or GFDL], from Wikimedia Commons

Facebook Comments

#development #glibc version not found #limited access #python

SECNIX SYSTEMS

www.secnix.com

CONTACT

Front Suite, First Floor,
131 High Street,
Teddington,
Middlesex,
England,
TW11 8HH

info [ @ ] secnix.com
+44 (0)3338 802382