编程,作为生物信息学的一个基础性技能,是任何一个生信工程师都无法绕开话题。也许有些人还在纠结 Perl 和 Python 到底应该学习哪一个,但作为目前最火最流行的编程语言 Python 还是非常值得尝试的。它不但可以进行文本处理,在统计、网站、游戏、爬虫、数据可视化等方面也有非常强大的应用,比起曾经的 Perl 真的强大和全面很多,且比 Perl 更容易入手。不管从长远发展,还是短期需要,学会 Python,看懂 Perl (或者先学 Python,后学 Perl) 应该是每一个生信工程必备的基础技能之一。
工欲善其事,必先利其器。关于 Python 安装教程在网上一搜一大把,但总感觉不够全面,尤其对于中间出现的一些问题的解决方法不尽如人意。鉴于此,本文基于 CentOS Linux 对 Python 的源码编译安装进行了一下简单的总结,记录如下。
1. 安装 Python-3.7.3
1.1 安装环境
os : CentOS Linux release 6.5 (x86_64) gcc: 4.8.5 20150623
1.2 安装步骤
如果您拥有 root 权限,请执以下依赖安装:
如果您没有 root 权限,可以参考《手把手教你在 Linux 源码安装最新版本的 R》一文,手动一个个去解决以上的依赖。
1.2.1 ssl 依赖
python3 需要引用 openssl
模块,但是 CentOS 需要的 openssl 版本最低为 1.0.2,而 CentOS 默认的为 1.0.1(CentOS-6.x 通过 yum
源安装的 openssl 的最高版本是 1.0.1),所以需要手动更新 openssl。
# 下载 wget http://www.openssl.org/source/openssl-1.1.1.tar.gz # 解压缩 tar -zxvf openssl-1.1.1.tar.gz # 进入目录安装 cd openssl-1.1.1 # 进行配置下,自定义 ./config --prefix=$HOME/soft-repos/openssl shared zlib # 编译并安装 make && make install # 配置到用户环境变量,随处使用 echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/soft-repos/openssl/lib" >> $HOME/.bashrc # 是环境变量配置生效 source $HOME/.bashrc
请注意:
- openssl 编译(config)的时候 必须要加上 shared 参数,否者源码安装 Python 即使添加了
--with-openssl
的自定义路径,依然会导致Could not build the ssl module!
报错! - 从 https://www.openssl.org/source/ 下载的源码 openssl-1.0.2s、openssl-1.0.2m,包括 CentOS-7.5 使用
yum
安装的最高版本的 openssl-1.0.2k 目前发现依然会导致Could not build the ssl module
,建议从 https://www.openssl.org/source/old/ 下载 1.1.1 的源码编译安装。
1.2.2 ctypes 依赖
在 CentOS 6.x 安装 libffi-devel
的时候出现以下问题:
$ yum install -y libffi-devel Loaded plugins: product-id, refresh-packagekit, search-disabled-repos, security, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Setting up Install Process cdrom | 4.1 kB 00:00 ... No package libffi-devel available. Error: Nothing to do
可以使用下面的方法安装:
[root@log01 ~]# rpm -ivh http://mirror.centos.org/centos/6/os/x86_64/Packages/libffi-devel-3.0.5-3.2.el6.x86_64.rpm Retrieving http://mirror.centos.org/centos/6/os/x86_64/Packages/libffi-devel-3.0.5-3.2.el6.x86_64.rpm warning: /var/tmp/rpm-tmp.V9ihbu: Header V3 RSA/SHA256 Signature, key ID c105b9de: NOKEY Preparing... ########################################### [100%] 1:libffi-devel ########################################### [100%] [root@log01 ~]# rpm -qa|grep libffi libffi-3.0.5-3.2.el6.x86_64 libffi-devel-3.0.5-3.2.el6.x86_64
源码方法安装如下:
$ wget ftp://sourceware.org/pub/libffi/libffi-3.2.1.tar.gz $ tar zvxf libffi-3.2.1.tar.gz $ ./configure --prefix=/Bioinfo/Pipeline/SoftWare/LibDependence/libffi-3.2.1 $ make $ make install
1.2.3 pygraphviz 依赖
$ /Bioinfo/Pipeline/SoftWare/Python-3.7.3/bin/pip3 install pygraphviz Collecting pygraphviz Using cached https://files.pythonhosted.org/packages/7e/b1/d6d849ddaf6f11036f9980d433f383d4c13d1ebcfc3cd09bc845bda7e433/pygraphviz-1.5.zip Installing collected packages: pygraphviz Running setup.py install for pygraphviz ... error Complete output from command /Bioinfo/Pipeline/SoftWare/Python-3.7.3/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-_zdjdg0j/pygraphviz/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-g0mz7lrq/install-record.txt --single-version-externally-managed --compile: running install Trying dpkg Failed to find dpkg Trying pkg-config Package libcgraph was not found in the pkg-config search path. Perhaps you should add the directory containing `libcgraph.pc' to the PKG_CONFIG_PATH environment variable No package 'libcgraph' found Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-install-_zdjdg0j/pygraphviz/setup.py", line 93, in <module> tests_require=['nose>=1.3.7', 'doctest-ignore-unicode>=0.1.2', 'mock>=2.0.0'], File "/Bioinfo/Pipeline/SoftWare/Python-3.7.3/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup return distutils.core.setup(**attrs) File "/Bioinfo/Pipeline/SoftWare/Python-3.7.3/lib/python3.7/distutils/core.py", line 148, in setup dist.run_commands() File "/Bioinfo/Pipeline/SoftWare/Python-3.7.3/lib/python3.7/distutils/dist.py", line 966, in run_commands self.run_command(cmd) File "/Bioinfo/Pipeline/SoftWare/Python-3.7.3/lib/python3.7/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/tmp/pip-install-_zdjdg0j/pygraphviz/setup_commands.py", line 44, in modified_run self.include_path, self.library_path = get_graphviz_dirs() File "/tmp/pip-install-_zdjdg0j/pygraphviz/setup_extra.py", line 162, in get_graphviz_dirs include_dirs, library_dirs = _try_configure(include_dirs, library_dirs, _pkg_config) File "/tmp/pip-install-_zdjdg0j/pygraphviz/setup_extra.py", line 117, in _try_configure i, l = try_function() File "/tmp/pip-install-_zdjdg0j/pygraphviz/setup_extra.py", line 72, in _pkg_config output = S.check_output(['pkg-config', '--libs-only-L', 'libcgraph']) File "/Bioinfo/Pipeline/SoftWare/Python-3.7.3/lib/python3.7/subprocess.py", line 395, in check_output **kwargs).stdout File "/Bioinfo/Pipeline/SoftWare/Python-3.7.3/lib/python3.7/subprocess.py", line 487, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['pkg-config', '--libs-only-L', 'libcgraph']' returned non-zero exit status 1. ---------------------------------------- Command "/Bioinfo/Pipeline/SoftWare/Python-3.7.3/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-_zdjdg0j/pygraphviz/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-g0mz7lrq/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-_zdjdg0j/pygraphviz/
参考:《Installation:fatal error: 'graphviz/cgraph.h' file not found》
$ wget https://graphviz.gitlab.io/pub/graphviz/stable/SOURCES/graphviz.tar.gz $ tar zvxf graphviz.tar.gz $ cd graphviz-2.40.1 $ ./configure --prefix=/Bioinfo/Pipeline/SoftWare/graphviz-2.40.1 $ make && make install
推荐把安装好的 graphviz 添加到环境变量,这样可以避免运行过程中出现:"pygraphviz/graphviz_wrap.c:2987:29: fatal error: graphviz/cgraph.h: No such file or directory" 无法找到头文件的异常。
export PKG_CONFIG_PATH=/Bioinfo/Pipeline/SoftWare/graphviz-2.40.1/lib/pkgconfig:$PKG_CONFIG_PATH export LD_LIBRARY_PATH=/Bioinfo/Pipeline/SoftWare/graphviz-2.40.1/lib:$LD_LIBRARY_PATH export C_INCLUDE_PATH=/Bioinfo/Pipeline/SoftWare/graphviz-2.40.1/include:$C_INCLUDE_PATH export CPLUS_INCLUDE_PATH=/Bioinfo/Pipeline/SoftWare/graphviz-2.40.1/include:$CPLUS_INCLUDE_PATH
如果 graphviz 添加到环境变量, pygraphviz
的 python 包可以参考下面的方法安装:
$ /Bioinfo/Pipeline/SoftWare/Python-3.7.3/bin/pip3 install --global-option=build_ext --global-option="-I/Bioinfo/Pipeline/SoftWare/graphviz-2.40.1/include" --global-option="-L/Bioinfo/Pipeline/SoftWare/graphviz-2.40.1/lib" pygraphviz /Bioinfo/Pipeline/SoftWare/Python-3.7.3/lib/python3.7/site-packages/pip/_internal/commands/install.py:207: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options. cmdoptions.check_install_build_global(options) Collecting pygraphviz Using cached https://files.pythonhosted.org/packages/7e/b1/d6d849ddaf6f11036f9980d433f383d4c13d1ebcfc3cd09bc845bda7e433/pygraphviz-1.5.zip Installing collected packages: pygraphviz Running setup.py install for pygraphviz ... done Successfully installed pygraphviz-1.5
1.3 编译安装 Python-3.7.3
第一,下载 Python 源码,解压。
# 官网下载地址 https://www.python.org/downloads wget https://www.python.org/ftp/python/2.7.15/Python-2.7.15.tgz wget https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tgz # 解压到指定目录 tar zvxf Python-3.7.3.tgz -C /usr/local/src
第二,进入解压的源码路径,编译 Python 源码。
export PKG_CONFIG_PATH=/Bioinfo/Pipeline/SoftWare/LibDependence/libffi-3.2.1/lib/pkgconfig:/Bioinfo/Pipeline/SoftWare/graphviz-2.40.1/lib/pkgconfig:$PKG_CONFIG_PATH export LD_LIBRARY_PATH=/Bioinfo/Pipeline/SoftWare/LibDependence/libffi-3.2.1/lib64:/Bioinfo/Pipeline/SoftWare/graphviz-2.40.1/lib:$LD_LIBRARY_PATH $ ./configure \ --enable-optimizations \ --prefix=/Bioinfo/Pipeline/SoftWare/Python-3.7.3 \ --with-openssl=/Bioinfo/Pipeline/SoftWare/LibDependence/openssl-1.1.1 \ CC=/Bioinfo/Pipeline/SoftWare/gcc-4.8.5/bin/gcc \ CXX=/Bioinfo/Pipeline/SoftWare/gcc-4.8.5/bin/c++ \ LDFLAGS="-L/Bioinfo/Pipeline/SoftWare/LibDependence/libffi-3.2.1/lib64 -L/Bioinfo/Pipeline/SoftWare/graphviz-2.40.1/lib" \ CPPFLAGS="-I/Bioinfo/Pipeline/SoftWare/graphviz-2.40.1/include" \ PKG_CONFIG_PATH="/Bioinfo/Pipeline/SoftWare/LibDependence/libffi-3.2.1/lib/pkgconfig:/Bioinfo/Pipeline/SoftWare/graphviz-2.40.1/lib/pkgconfig"
--enable-optimizations
是优化选项(LTO,PGO 等)加上这个 flag 编译后,性能有 10% 左右的优化,但是这会明显的增加编译时间。建议使用这个参数;--prefix
声明安装路径;- 安装多个 python 的版本,如果不开启
--enable-shared
,指定不同路径即可。当开启--enable-shared
时,默认只有一个版本的 python。 - python 3 编译可以使用
--with-openssl=DIR
指定 OpenSSL 安装路径进行编译的方式解决 OpenSSL 依赖,否则make
过程可能出错。
$ make ...... The following modules found by detect_modules() in setup.py, have been built by the Makefile instead, as configured by the Setup files: _abc atexit pwd time Failed to build these modules: _ctypes _hashlib _ssl Could not build the ssl module! Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host(). LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381 ......
make
过程如果出现ModuleNotFoundError: No module named '_ctypes'
或者INFO: Could not locate ffi libs and/or headers
参考:https://groups.google.com/forum/#!topic/comp.lang.python/npv-wzmytzo
- 如果指定
--with-openssl=DIR
依然无法解决 ssl 模块的问题,可以参考修改 Modules/Setup.dist 文件(默认这块是注释的,放开注释即可。这块功能是开启 SSL 模块,不然会出现安装完毕后,提示找不到 ssl 模块的错误)再执行 configure,修改内容如下:
# Socket module helper for SSL support; you must comment out the other # socket line above, and possibly edit the SSL variable: SSL=/usr/local/ssl _ssl _ssl.c \ -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \ -L$(SSL)/lib -lssl -lcrypto
第三,Makefile 生后依次在当前路径执行编译和安装命令。
make && make install
第四,安装完成。以上命令执行完毕,且无报错的情况下,我们将默认 python 换将切换至 3.7.3(一般不建议替换,个人建议把自定义安装的 Python bin 路径添加到 PATH 环境变量即可):
# 替换系统自带的 python(不建议) mv /usr/bin/python /usr/bin/python2 ln -s /Bioinfo/Pipeline/SoftWare/Python-3.7.3/bin/python3 /usr/bin/python # 添加新 Python 到 PATH 环境变量(建议) echo "export PATH=/Bioinfo/Pipeline/SoftWare/Python-3.7.3/bin:$PATH" >>~/.bashrc source ~/.bashrc
运行命令 python -V
,查看是否出现 3.7.3 的版本,出现即为安装成功。
1.4. 安装 pip+setuptools
# 下载 setuptools 和 pip 安装程序 curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py python get-pip.py
至此,CentOS Linux release 6.5 下的 python-3.7.3 全部安装完成。如果在安装过程中出现其他的报错,建议把 error 关键信息直接复制到 Google 进行检索,参考其他人的解决方法。
2. 安装 Python-3.6.9
2.1 安装环境
- 系统:CentOS Linux release 7.7.1908 (Core)
- GCC:gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
2.2 No module named '_bz2'
在 CentOS 7 中安装 Python-3.6.9 中发现 make 过程中一直提示:“ModuleNotFoundError: No module named '_bz2'”,尽管 sudo yum install bzip2 bzip2-devel
已经安装了 bzip2 的依赖,问题还是不得其解。最后参考 stackoverflow 上的《Correctly building local python3, with bz2 support》,终于解决问题,下面记录一下。
2.3 手动安装 bzip2
wget https://nchc.dl.sourceforge.net/project/bzip2/bzip2-1.0.6.tar.gz tar zvxf bzip2-1.0.6.tar.gz cd bzip2-1.0.6 make -f Makefile_libbz2_so # 这一步是生成 libbz2.so.1.0.6 的动态库文件 make make install PREFIX=/usr/local/software/bzip2-1.0.6 cp libbz2.so.1.0.6 /usr/local/software/bzip2-1.0.6/lib/
几点说明:
- bzip2 的官网 http://www.bzip.org/ 已经把 bzip2 的源码下载链接放到了 SourceForge,网络上一些从 http://www.bzip.org/1.0.6/bzip2-1.0.6.tar.gz 下载的做法已经失效。
- SourceForge 上 bzip2 的最新版本还是 1.0.6(Last modified 2018-11-3),更高级版本的 bzip2 我也不知道是否存在,也不知道能不能用。
2.4 编译安装 Python-3.6.9
$ wget https://www.python.org/ftp/python/3.6.9/Python-3.6.9.tgz $ tar zvxf Python-3.6.9.tgz $ cd Python-3.6.9 $ ./configure --enable-optimizations --prefix=/usr/local/software/python-3.6 CFLAGS="-I/usr/local/software/bzip2-1.0.6/include" LDFLAGS="-L/usr/local/software/bzip2-1.0.6/lib" $ make $ make install
- Python-3.6.9 中的
./configure --help
中没有--with-openssl
参数!有点神奇,我也不知道原因。 - 安装完成可以用
from _bz2 import BZ2Compressor, BZ2Decompressor
测试一下_bz2
是否可用。
F. 参考资料
- 行者无疆-ITer,《python2.7 源码编译安装》, 博客园
- Scott Frazer,《How do I compile Python 3.4 with custom OpenSSL?》, Stack Overflow