Shared Hosting — /lib64/libc.so.6: version GLIBC_2.14 not found
29/04/2018, by Justin Cook
Shared Hosting — /lib64/libc.so.6: version GLIBC_2.14 not found
Have you been in a situation with limited access lacking compilers and other common utilities, and all you need is that one thing that’s so close yet so far away?
Some clients prefer using CPanel hosting solutions as it has been around over two decades and is used by many a web developer. While it is perfectly fine in many situations, your ability to extend the environment can become painful as you may encounter blockers seemingly impossible to overcome if you are in a shared environment with zealous security.
Recently, one client switched providers. They assumed the toolchain would be portable. However, the new provider had a chroot Python implementation with no read access to the libraries. Therefore, the virtualenv was no longer rebuildable, and for another reason, not even usable. They were seemingly blocked. This was unfortunate as the new provider’s environment had far better performance and support. This is where we had to get involved.
The issue was that the entire environment was copied, but the Python vitualenv would not run. Upon inspection, the error was the following:
[~]# python python: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by python)
This is a Glibc version mismatch. The virualenv was copied from an installation built against a newer version. There is more than one way to correct this [1], but the most obvious and straightforward is building against a compatible version of Glibc that supports all the symbol versions necessary for the binary and shared objects.
The first thing we should look at is the Glibc version [2] on the shared host:
[~]# /lib64/libc.so.6 GNU C Library stable release version 2.5, by Roland McGrath et al. Copyright (C) 2006 Free Software Foundation, Inc. ...
If you are surprised, you are not alone! In 2018, this is a very old version given Red Hat Enterprise Linux 7 ships glibc-2.17 and Red Hat Enterprise Linux 6 sports glibc-2.12. The latest version at the time of writing is glibc-2.27 [3]. This could end up very complicated, but let’s see just how much work will be necessary. After all, we only had one error complaining about GLIBC_2.14!
We need to check the Glibc symbols of each shared object and binary file to determine what is not found and the version we can build against. Let’s start with creating our development environment to diagnose.
$ mkdir centos7 ; cd centos7 ; vagrant init jhcook/centos7 ; vagrant up ... centos7$ vagrant ssh [vagrant@centos7-vbox ~]$ sudo yum install -y git gcc zlib-devel readline-devel bzip2-devel sqlite-devel openssl-devel ... [vagrant@centos7-vbox ~]$ curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash # Follow the instructions provided to update your shell profile then update your environment with `exit` and `vagrant ssh` ... [vagrant@centos7-vbox ~]$ pyenv install 2.7.14 ... [vagrant@centos7-vbox ~]$ pyenv global 2.7.14 [vagrant@centos7-vbox ~]$ pip install virtualenv ... [vagrant@centos7-vbox ~]$ pip install --upgrade pip ...
Now you have the build tools and Python 2.7.14 installed with the latest packages necessary to investigate. Since this is exactly what we need to install on the shared host, you can have a look at the Glibc symbols. Fingers crossed, we’ll create a nice and tidy ordered output of the Glibc symbols and the version built against.
[vagrant@centos7-vbox ~]$ find .pyenv/versions/2.7.14 -name '*.so' -o -name python2.7 -type f -print0 | xargs -0 objdump -T | cut -c 20- | awk ‘/GLIBC_2.14/{print$4,$5}’ | sort | uniq
GLIBC_2.14 memcpy
That is one GLIBC_2.14 function! With a bit of Google, we came across a nice explanation [4] of the culprit and quicky discovered we should be able to build against glibc-2.12 which is provided in Red Hat Enterprise Linux 6. Although a lot of code has been written and modified since 2006, the ABI will be compatible, so we will be safe.
Let’s rinse and repeat with another box of that version. One thing to note, we use the venerable PyEnv [5] to create and manage multiple Python environments. But, when copying PyEnv versions across nodes, the build must come from the same user as it’s embedded deep in the build.
$ mkdir centos6 ; cd centos6 ; vagrant init jhcook/centos6 ; vagrant up ... centos6$ vagrant ssh [vagrant@centos6-vbox ~]$ sudo yum install -y git gcc zlib-devel readline-devel bzip2-devel sqlite-devel openssl-devel ... [vagrant@centos6-vbox ~]$ sudo useradd remoteuser ... [vagrant@centos6-vbox ~]$ sudo su - remoteuser [remoteuser@centos6-vbox ~]$ curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash # Follow the instructions provided to update your shell profile then update your environment with `exit` and `sudo su - remoteuser` ... [remoteuser@centos6-vbox ~]$ pyenv install 2.7.14 ... [remoteuser@centos6-vbox ~]$ pyenv global 2.7.14 [remoteuser@centos6-vbox ~]$ pip install virtualenv ... [remoteuser@centos6-vbox ~]$ pip install --upgrade pip ...
Now that’s out of the way, we should have a like-for-like Python environment. We can use our Bash foo and see what the difference is and if it will be supported by the ancient version of Glibc we must unfortunately deal with.
[remoteuser@centos6-vbox ~]$ find .pyenv/versions/2.7.14 -name '*.so' -o -name python2.7 -type f -print0 | xargs -0 objdump -T | cut -c 20- | awk '/memcpy/{print$4,$5}' | sort | uniq
GLIBC_2.2.5 memcpyWow! memcpy symbol now indicates GLIBC_2.2.5, and we expected to see this given what we found [4]. If you look further, you will see all versions are less than 2.5 and therefore should be supported by the ancient glibc-2.5. All we have to do is transfer the build across to the shared host and see what we come up with.
[remoteuser@centos6-vbox ~]$ cd .pyenv/versions [remoteuser@centos6-vbox versions]$ tar zcvf ~/pyenv-2.7.14.tar.gz 2.7.14 ... [remoteuser@centos6-vbox versions]$ exit [root@centos6-vbox ~]# mv ~remoteuser/pyenv-2.7.14.tar.gz /vagrant/mv ~remoteuser/pyenv-2.7.14.tar.gz /vagrant/ [root@centos6-vbox ~]# exit [vagrant@centos6-vbox ~]$ exit $ scp pyenv-2.7.14.tar.gz [email protected]: ... $ ssh [email protected] remoteuser@sharedhost [~]# cd .pyenv/versions remoteuser@sharedhost [~/.pyenv/versions]# tar zxvf ~/pyenv-2.7.14.tar.gz ... remoteuser@sharedhost [~/.pyenv/versions]# cd ; pyenv global 2.7.14 remoteuser@sharedhost [~]# virtualenv awsVE New python executable in /home/remoteuser/awsVE/bin/python2.7 Also creating executable in /home/remoteuser/awsVE/bin/python Installing setuptools, pip, wheel...done. remoteuser@sharedhost [~]# . awsVE/bin/activate (awsVE) remoteuser@sharedhost [~]# pip install awscli Collecting awscli ... Successfully installed PyYAML-3.12 awscli-1.15.10 botocore-1.10.10 colorama-0.3.7 docutils-0.14 futures-3.2.0 jmespath-0.9.3 pyasn1-0.4.2 python-dateutil-2.7.2 rsa-3.4.2 s3transfer-0.1.13 six-1.11.0
Result!
Lastly, it would be smart to test the Python build on the remote host. But, given resource constraints, that may prove difficult.
remoteuser@remotehost [~]# python -m test -j1
== CPython 2.7.14 (default, Apr 27 2018, 21:34:31) [GCC 4.4.7 20120313 (Red Hat 4.4.7-18)]
== Linux-3.12.18-clouder0-x86_64-with-glibc2.3 little-endian
== /tmp/test_python_12484
== CPU count: 1
Run tests in parallel using 1 child processes
Traceback (most recent call last):
File "/home/remoteuser/.pyenv/versions/2.7.14/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/home/remoteuser/.pyenv/versions/2.7.14/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/remoteuser/.pyenv/versions/2.7.14/lib/python2.7/test/__main__.py", line 3, in
regrtest.main_in_temp_cwd()
File "/home/remoteuser/.pyenv/versions/2.7.14/lib/python2.7/test/regrtest.py", line 2024, in main_in_temp_cwd
main()
File "/home/remoteuser/.pyenv/versions/2.7.14/lib/python2.7/test/regrtest.py", line 802, in main
display_progress(test_index, text)
File "/home/remoteuser/.pyenv/versions/2.7.14/lib/python2.7/test/regrtest.py", line 646, in display_progress
load_avg_1min = os.getloadavg()[0]
OSError: Load averages are unobtainable References
1. https://software.intel.com/en-us/articles/how-to-use-the-intel-cluster-checker-v3-sdk-on-a-cluster-using-multiple-linux-distributions
2. http://ask.xmodulo.com/check-glibc-version-linux.html
3. https://sourceware.org/glibc/wiki/Glibc%20Timeline
4. https://snorfalorpagus.net/blog/2016/07/17/compiling-python-extensions-for-old-glibc-versions/
5. https://github.com/pyenv/pyenv
6. Shmuel Csaba Otto Traian [CC BY-SA 3.0 or GFDL], from Wikimedia Commons
Facebook Comments