
.. blogpost::
    :title: Install Python 3.7 and many packages on Linux Debian 9
    :keywords: python 3.7, debian
    :date: 2018-12-29
    :categories: python
    :lid: blog-2018-python37-linux2

    Same steps four months later:
    :ref:`Install Python 3.7 and many packages on Linux Debian 9 <blog-2018-python37-linux>`.

    ::

        apt-get install sudo -y
        apt-get update

    Install everything to generate documentation with :epkg:`Sphinx`:

    ::

        wget https://github.com/jgm/pandoc/releases/download/2.5/pandoc-2.5-1-amd64.deb
        dpkg -i pandoc-2.5-1-amd64.deb
        apt-get install -y texlive texlive-latex-extra texlive-xetex texlive-lang-european texlive-lang-french
        apt-get install -y dvipng p7zip-full inkscape graphviz git uuid-dev
        apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm
        apt-get install -y libncurses5-dev libncursesw5-dev xz-utils tk-dev libsecret-1-dev libffi-dev curl
        apt-get install -y liblzma-doc liblzma-dev libgdbm-dev libsqlite3-dev libdb5.3-dev nano
        apt-get install -y libssl-dev zlib1g-dev libncurses5-dev libncursesw5-dev libreadline-dev libsqlite3-dev
        apt-get install -y libgdbm-dev libdb5.3-dev libbz2-dev libexpat1-dev liblzma-dev libffi-dev
        apt-get install -y libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev
        apt-get install -y unrar libstdc++

    *MKL*, see
    `MKL for .deb-based systems: An easy recipe <https://github.com/eddelbuettel/mkl4deb>`_.

    ::

        cd /tmp
        wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB
        apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB
        sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list'
        apt-get update
        apt-get install intel-mkl-64bit-2018.2-046

        update-alternatives --install /usr/lib/x86_64-linux-gnu/libblas.so     \
                    libblas.so-x86_64-linux-gnu      /opt/intel/mkl/lib/intel64/libmkl_rt.so 50
        update-alternatives --install /usr/lib/x86_64-linux-gnu/libblas.so.3   \
                    libblas.so.3-x86_64-linux-gnu    /opt/intel/mkl/lib/intel64/libmkl_rt.so 50
        update-alternatives --install /usr/lib/x86_64-linux-gnu/liblapack.so   \
                    liblapack.so-x86_64-linux-gnu    /opt/intel/mkl/lib/intel64/libmkl_rt.so 50
        update-alternatives --install /usr/lib/x86_64-linux-gnu/liblapack.so.3 \
                    liblapack.so.3-x86_64-linux-gnu  /opt/intel/mkl/lib/intel64/libmkl_rt.so 50

        echo "/opt/intel/lib/intel64"     >  /etc/ld.so.conf.d/mkl.conf
        echo "/opt/intel/mkl/lib/intel64" >> /etc/ld.so.conf.d/mkl.conf
        ldconfig

        echo "MKL_THREADING_LAYER=GNU" >> /etc/environment

    Install a newer version of :epkg:`cmake`:

    ::

        wget https://cmake.org/files/v3.13/cmake-3.13.2.tar.gz
        tar xzf cmake-3.13.2.tar.gz
        cd cmake-3.13.2
        ./configure
        make
        make install

    Install and make :epkg:`Python`,
    it must be installed in ``/usr/local`` otherwise
    :epkg:`virtualenv` fails to create a vitual
    environment.
    See `Building Python <https://docs.python.org/3/using/unix.html#building-python>`_:

    ::

        cd /usr/local
        wget https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tgz
        tar xzf Python-3.7.2.tgz
        cd Python-3.7.2
        ./configure --enable-optimizations --with-ensurepip=install
        make
        make altinstall
        chmod +x python3.7
        curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
        ./python get-pip.py
        cd ..
        cd ~

    Some package require a different build because they need shared libraries:

    ::

        ./configure --enable-optimizations --with-ensurepip=install --enable-shared --prefix=/opt/bin

    About ``--enable-shared``, see
    `Relocation R_X86_64_32S against '_Py_NotImplementedStruct' can not be used when making a shared object; recompile with -fPIC
    <https://stackoverflow.com/questions/42582712/relocation-r-x86-64-32s-against-py-notimplementedstruct-can-not-be-used-when>`_,
    `How to configure python with --enable-shared on Ubuntu 18.04
    <https://stackoverflow.com/questions/53463903/how-to-configure-python-with-enable-shared-on-ubuntu-18-04>`_.
    Option ``--enable-shared --prefix=/opt/python`` is needed when to embed *python*
    interpreter in a C application, see
    `Embedding the Python interpreter
    <https://pybind11.readthedocs.io/en/stable/compiling.html#embedding-the-python-interpreter>`_.
    Environment variable ``LD_LIBRARY_PATH`` must be set:

    ::

        export LD_LIBRARY_PATH=/opt/bin

    Add python to ``PATH``:

    ::

        export PATH=/usr/local/Python-3.7.2:$PATH

    Install packages:

    ::

        curl https://raw.githubusercontent.com/sdpython/pyquickhelper/master/requirements.txt -o requirements.txt
        python3.7 -c "from pip._internal import main;main('install -r requirements.txt'.split())"
        python3.7 -c "from pip._internal import main;main('uninstall -y jyquickhelper'.split())"

    Intall :epkg:`java`, :epkg:`Jenkins`, :epkg:`ufw`:

    ::

        apt-get install -y default-jre default-jdk daemon psmisc
        wget https://pkg.jenkins.io/debian/binary/jenkins_2.156_all.deb
        dpkg -i jenkins_2.156_all.deb
        apt-get install -y ufw

    Allows connection to port 8080 (:epkg:`Jenkins` default port):

    ::

        ufw enable
        ufw allow ssh
        ufw allow 8080/tcp

    Go to ``http://<ip_address>:8080`` and follow the instructions,
    install also extensions
    `Build-timeout Plugin <https://wiki.jenkins.io/display/JENKINS/Build-timeout+Plugin>`_,
    `Console Column Plugin <https://wiki.jenkins.io/display/JENKINS/Console+Column+Plugin>`_,
    `Next Executions <https://wiki.jenkins.io/display/JENKINS/Next+Executions>`_,
    `Collapsing Console Sections Plugin <https://wiki.jenkins.io/display/JENKINS/Collapsing+Console+Sections+Plugin>`_,
    `Credentials Plugin <https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Plugin>`_.
    If you need to manually restart :epkg:`Jenkins`, go to
    `How to restart Jenkins manually? <https://stackoverflow.com/questions/8072700/how-to-restart-jenkins-manually>`_.
    Fix dependencies:

    ::

        apt --fix-broken install

    Additional dependencies :epkg:`unrar`, :epkg:`nodejs`:

    ::

        curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
        apt-get install -y nodejs

    Additional dependencies *geos*, *geopandas*
    (current version of *pyproj* does not compile
    for Python 3.7, the latest sources must be used):

    ::

        apt-get install -y libproj-dev proj-bin proj-data libc6 libgeos-dev python-gdal gdal-bin
        export PYPROJMAJVER=13
        export PYPROJVERSION=5.2.0-1
        wget http://es.archive.ubuntu.com/ubuntu/pool/universe/p/proj/libproj${PYPROJMAJVER}_${PYPROJVERSION}_amd64.deb
        dpkg -i libproj${PYPROJMAJVER}_${PYPROJVERSION}_amd64.deb
        wget http://es.archive.ubuntu.com/ubuntu/pool/universe/p/proj/libproj-dev_${PYPROJVERSION}_amd64.deb
        dpkg -i libproj-dev_${PYPROJVERSION}_amd64.deb
        apt-get install -y libav-tools protobuf-compiler libprotoc-dev libcurl4-openssl-dev libssl-dev
        apt-get install -y libhdf5-dev libblosc-dev liblzo2-dev libappindicator3-1 fonts-liberation
        apt --fix-broken install
        wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
        dpkg -i google-chrome-stable_current_amd64.deb
        apt-get install -y chromium-driver
        apt-get install -y portaudio19-dev libblas-dev liblapack-dev libsuitesparse-dev
        apt-get install -y libasound2-dev libx11-dev xvfb pulseaudio python-pygame libunwind8

    Other missing dependencies:

    ::

        pip3.7 install cython
        pip3.7 install git+https://github.com/jswhit/pyproj.git
        pip3.7 install geopandas descartes
        pip3.7 install tables nltk statsmodels cartopy imbalanced-learn seaborn gensim  networkx
        pip3.7 install bqplot memory_profiler xarray folium antlr4-python3-runtime
        pip3.7 install category_encoders --no-deps
        pip3.7 install torchvision chainer fcn
        pip3.7 install spacy
        pip3.7 install xlwt xlrd wikipedia pycurl xlsxwriter pyinstrument wptools flask dask bcolz
        pip3.7 install sphinxcontrib.blockdiag sphinx_bootstrap_theme moviepy mutagen vprof solar-theme guzzle_sphinx_theme sphinx_modern_theme_modified sphinxjp.themes.basicstrap sphinx_readable_theme
        pip3.7 install pyspark qgrid yahoo_historical py4j bayespy lightgbm
        pip3.7 install version_information xgboost natsort pivottablejs prettytable
        pip3.7 install colormap easydev dbfread quandl seaborn requests-file requests-ftp lxml opencv-python python-pptx
        pip3.7 install http://www.xavierdupre.fr/enseignement/setup/fairtest-0.1-py3-none-any.whl
        pip3.7 install reportlab sympy vispy cufflinks brythonmagic pydy seasonal
        pip3.7 install treeinterpreter tutormagic pympler selenium httplib2
        pip3.7 install tpot pygal google-api-python-client datashader PyQt5
        pip3.7 install pyecharts pygal-maps-fr pythreejs vega yahoo-historical lifelines fast-histogram geoplotlib
        pip3.7 install missingno mpl-scatter-density mizani lml macropy3 plotnine geopy zarr
        pip3.7 install jsonpickle forestci psycopg2 pypi-server jupyterlab
        pip3.7 install imageio pyaudio pygame pybind11
        pip3.7 install git+https://github.com/cvxopt/cvxopt.git@dev
        pip3.7 install mpl-finance
        pip3.7 install git+https://github.com/sdpython/minepy.git
        pip3.7 install git+https://github.com/sdpython/pyPdf --upgrade --no-deps

    ::

        pip3.7 install matplotlib pandas numpy scikit-learn scipy statsmodels --upgrade

    ::

        python3.7 -c "import imageio;imageio.plugins.ffmpeg.download()"

    For :epkg:`spacy`:

    ::

        pip3.7 install http://www.xavierdupre.fr/enseignement/setup/fr_core_news_sm-2.0.0-py3-none-any.whl

    For module :epkg:`imageio`, the second instruction must be executed
    with every distinct user once:

    Install :epkg:`clang` 6 (for :epkg:`pythonnet`):

    ::

        echo "deb http://ftp.debian.org/debian stretch-backports main" | sudo tee /etc/apt/sources.list.d/clang-stable.list
        apt-get update
        apt-get install -y clang clang-6.0 clang-3.8
        alias clang='clang-6.0'

    Install :epkg:`mono` (for :epkg:`pythonnet`, skip?):

    ::

        apt-get install -y apt-transport-https
        apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
        echo "deb https://download.mono-project.com/repo/debian stable-stretch main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
        apt-get update
        apt-get install -y mono-devel ca-certificates-mono

    And for :epkg:`dotnet`
    (see `Install .NET Core SDK on Linux Debian 9 <https://www.microsoft.com/net/download/linux-package-manager/debian9/sdk-current>`_):

    ::

        wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg
        mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/
        wget -q https://packages.microsoft.com/config/debian/9/prod.list
        mv prod.list /etc/apt/sources.list.d/microsoft-prod.list
        chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg
        chown root:root /etc/apt/sources.list.d/microsoft-prod.list

        apt-get update
        apt-get install -y dotnet-sdk-2.2

    Install :epkg:`pythonnet`:

    ::

        pip3.7 install git+https://github.com/rkern/line_profiler.git
        git clone -b master https://github.com/pythonnet/pythonnet.git
        cd pythonnet
        python3.7 setup.py install --xplat

    Deal with permissions issues for nugets (wait for some nuget to be installed):

    ::

        chown -R jenkins:jenkins /tmp/NuGetScratch/

    For :epkg:`ML.net`
    (see `Building ML.NET on Linux and macOS <https://github.com/dotnet/machinelearning/blob/master/docs/building/unix-instructions.md>`_):

    ::

        apt-get update

    You might need to run ``chmod 777 -R /tmp/NuGetScratch;chmod 777 -R /tmp/NuGet``.
    I got the following issue
    `Restore task fails in an multi user scenario <https://github.com/NuGet/Home/issues/3897>`_.

    For :epkg:`onnx`:

    ::

        pip3.7 install onnx

    **MKL** and **deep learning**
    The machine I used does not have GPU.
    Missing :epkg:`MKL` :
    `MKL for .deb-based systems: An easy recipe <https://github.com/eddelbuettel/mkl4deb>`_.
    About :epkg:`pyopencl` ...

    ::

        apt-get install -y beignet-opencl-icd ocl-icd-opencl-dev libpocl-dev nvidia-cuda-toolkit
        pip3.7 install pyopencl pycuda tf-nightly azure

    To reboot:

    ::

        reboot

    Stores password with :epkg:`keyring`.
    First switch to :epkg:`Jenkins` user:

    ::

        su
        su - jenkins

    Then:

    ::

        import keyring
        import os
        keyring.set_password("web", "pyquickhelper,user", "...")
        keyring.set_password("web", "pyquickhelper,pwd", "...")
        mkdir -p /var/lib/jenkins/workspace/local_pypi/local_pypi_server

    About *Jenkins*:

    * `How do I change the Jenkins service account user?
      <https://community.talend.com/t5/Migration-Configuration-and/How-do-I-change-the-Jenkins-service-account-user/ta-p/45164>`_

    A few useful lines maybe:

    ::

        nano /etc/init.d/jenkins
        chown -R dupre:dupre /var/lib/jenkins
        chown -R dupre:dupre /var/cache/jenkins
        chown -R dupre:dupre /var/log/jenkins

        chown -R jenkins:jenkins /var/lib/jenkins
        chown -R jenkins:jenkins /var/cache/jenkins
        chown -R jenkins:jenkins /var/log/jenkins

        /etc/init.d/jenkins restart

    For :epkg:`spacy`, to download the ressources
    for a particular language:

    ::

        python3.7 -m spacy download en

    Server FTP,
    look into `How to Configure VSFTPD FTPS with SSL/TLS on Ubuntu 18.04 <https://linoxide.com/linux-how-to/configure-vsftpd-sftp-ubuntu/>`_.
