Jupyterが立ち上げるだけでプログラミング環境ができあがる各種の便利なDockerイメージを出してくれているが、GPUが使えない。ので、無理やり使えるようにしてみた。 Dockerfileの継承元のFROMを書き換える必要があったので強引な感じになった。 以下ではcudaのDockerイメージを元にしているが、jupyterを元にcudaをインストールしたほうが完了までの時間が短いはず。
なお、TensorflowがGPUの使えるJupyter入りのDockerイメージをリリースしてくれている。なので、Jupyter本家に拘らなければこんなことをする必要はほぼない…が、まあせっかく書いたので。
適当に継ぎ接ぎしてDockerfileを作る
mkdir ~/Documents/cuda-based-jupyter-datascience-notebook cd ~/Documents/cuda-based-jupyter-datascience-notebook wget https://raw.githubusercontent.com/jupyter/docker-stacks/master/base-notebook/Dockerfile -O base-notebook wget https://raw.githubusercontent.com/jupyter/docker-stacks/master/minimal-notebook/Dockerfile -O minimal-notebook wget https://raw.githubusercontent.com/jupyter/docker-stacks/master/scipy-notebook/Dockerfile -O scipy-notebook wget https://raw.githubusercontent.com/jupyter/docker-stacks/master/datascience-notebook/Dockerfile -O datascience-notebook cat << "EOF" > Dockerfile # Merging nvidia/cuda and jupyter/datascience-notebook DockerFiles. FROM nvidia/cuda:8.0-cudnn6-devel-ubuntu16.04 MAINTAINER rubyu EOF cat << "EOF" >> Dockerfile # jupter/base-notebook EOF sed -e '1,8 s/^/# /g' base-notebook >> Dockerfile cat << "EOF" >> Dockerfile # jupter/minimal-notebook EOF sed -e '1,6 s/^/# /g' minimal-notebook >> Dockerfile cat << "EOF" >> Dockerfile # jupter/scipy-notebook EOF sed -e '1,5 s/^/# /g' scipy-notebook >> Dockerfile cat << "EOF" >> Dockerfile # jupter/datascience-notebook EOF sed -e '1,5 s/^/# /g' datascience-notebook >> Dockerfile cat << "EOF" >> Dockerfile USER $NB_USER RUN jupyter notebook --generate-config && \ jupyter serverextension enable --py jupyterlab --sys-prefix && \ pip install environment_kernels && \ echo "c.NotebookApp.kernel_spec_manager_class = 'environment_kernels.EnvironmentKernelSpecManager'" >> ~/.jupyter/jupyter_notebook_config.py EOF
最後のRUNの部分がJupyter Labとconda createで作る仮想環境のための設定。
Dockerイメージ作成に必要なファイルをダウンロードしておく
wget https://github.com/jupyter/docker-stacks/raw/master/base-notebook/fix-permissions -O fix-permissions wget https://github.com/jupyter/docker-stacks/raw/master/base-notebook/jupyter_notebook_config.py -O jupyter_notebook_config.py wget https://github.com/jupyter/docker-stacks/raw/master/base-notebook/start-notebook.sh -O start-notebook.sh wget https://github.com/jupyter/docker-stacks/raw/master/base-notebook/start-singleuser.sh -O start-singleuser.sh wget https://github.com/jupyter/docker-stacks/raw/master/base-notebook/start.sh -O start.sh sudo chmod ugo+x fix-permissions sudo chmod ugo+x jupyter_notebook_config.py sudo chmod ugo+x start-notebook.sh sudo chmod ugo+x start-singleuser.sh sudo chmod ugo+x start.sh
実行できる権限がないとダメ。ここではwgetしているが、普通はgit cloneするのでハマることはないはず。
コンテナを立ち上げる
sudo nvidia-docker build -t rubyu/datascience-notebook-gpu:1.0 . sudo nvidia-docker run -d --name notebook-gpu -p 8888:8888 --restart=always -v /storage/samba:/home/jovyan rubyu/datascience-notebook-gpu:1.0 start.sh jupyter lab --NotebookApp.token='XXXX'
これで8888ポートでJupyter Labが起動している。
本題には関係ない話ですが、Jupyterのhome(/home/jovyan
)をまるごとホストのディレクトリに割り当てて、かつそれをSamba経由でアクセスできるようにするのが好きです。Windowsで処理する必要がある時にいちいち転送する必要がなくなるので。なんやかんやでGUIはWindowsが使いやすい(個人の感想です)
アタッチしてconda createで仮想環境を作る
docker exec -it notebook-gpu /bin/bash conda create -n your-new-envronment python=3.5 jupyter tensorflow-gpu && conda clean -tipsy
作成された仮想環境はJupyterから見えて、カーネルとして選択することができる。
Dockerfile
# Merging nvidia/cuda and jupyter/datascience-notebook DockerFiles. FROM nvidia/cuda:8.0-cudnn6-devel-ubuntu16.04 MAINTAINER rubyu # jupter/base-notebook # # Copyright (c) Jupyter Development Team. # # Distributed under the terms of the Modified BSD License. # # # Ubuntu 16.04 (xenial) from 2017-07-23 # # https://github.com/docker-library/official-images/commit/0ea9b38b835ffb656c497783321632ec7f87b60c # FROM ubuntu@sha256:84c334414e2bfdcae99509a6add166bbb4fa4041dc3fa6af08046a66fed3005f # # MAINTAINER Jupyter Project <jupyter@googlegroups.com> USER root # Install all OS dependencies for notebook server that starts but lacks all # features (e.g., download as all possible file formats) ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && apt-get -yq dist-upgrade \ && apt-get install -yq --no-install-recommends \ wget \ bzip2 \ ca-certificates \ sudo \ locales \ fonts-liberation \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \ locale-gen # Install Tini RUN wget --quiet https://github.com/krallin/tini/releases/download/v0.10.0/tini && \ echo "1361527f39190a7338a0b434bd8c88ff7233ce7b9a4876f3315c22fce7eca1b0 *tini" | sha256sum -c - && \ mv tini /usr/local/bin/tini && \ chmod +x /usr/local/bin/tini # Configure environment ENV CONDA_DIR=/opt/conda \ SHELL=/bin/bash \ NB_USER=jovyan \ NB_UID=1000 \ NB_GID=100 \ LC_ALL=en_US.UTF-8 \ LANG=en_US.UTF-8 \ LANGUAGE=en_US.UTF-8 ENV PATH=$CONDA_DIR/bin:$PATH \ HOME=/home/$NB_USER ADD fix-permissions /usr/local/bin/fix-permissions # Create jovyan user with UID=1000 and in the 'users' group # and make sure these dirs are writable by the `users` group. RUN useradd -m -s /bin/bash -N -u $NB_UID $NB_USER && \ mkdir -p $CONDA_DIR && \ chown $NB_USER:$NB_GID $CONDA_DIR && \ fix-permissions $HOME && \ fix-permissions $CONDA_DIR USER $NB_USER # Setup work directory for backward-compatibility RUN mkdir /home/$NB_USER/work && \ fix-permissions /home/$NB_USER # Install conda as jovyan and check the md5 sum provided on the download site ENV MINICONDA_VERSION 4.3.21 RUN cd /tmp && \ wget --quiet https://repo.continuum.io/miniconda/Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh && \ echo "c1c15d3baba15bf50293ae963abef853 *Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh" | md5sum -c - && \ /bin/bash Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh -f -b -p $CONDA_DIR && \ rm Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh && \ $CONDA_DIR/bin/conda config --system --prepend channels conda-forge && \ $CONDA_DIR/bin/conda config --system --set auto_update_conda false && \ $CONDA_DIR/bin/conda config --system --set show_channel_urls true && \ $CONDA_DIR/bin/conda update --all --quiet --yes && \ conda clean -tipsy && \ fix-permissions $CONDA_DIR # Install Jupyter Notebook and Hub RUN conda install --quiet --yes \ 'notebook=5.2.*' \ 'jupyterhub=0.8.*' \ 'jupyterlab=0.28.*' \ && conda clean -tipsy && \ fix-permissions $CONDA_DIR USER root EXPOSE 8888 WORKDIR $HOME # Configure container startup ENTRYPOINT ["tini", "--"] CMD ["start-notebook.sh"] # Add local files as late as possible to avoid cache busting COPY start.sh /usr/local/bin/ COPY start-notebook.sh /usr/local/bin/ COPY start-singleuser.sh /usr/local/bin/ COPY jupyter_notebook_config.py /etc/jupyter/ RUN fix-permissions /etc/jupyter/ # Switch back to jovyan to avoid accidental container runs as root USER $NB_USER # jupter/minimal-notebook # # Copyright (c) Jupyter Development Team. # # Distributed under the terms of the Modified BSD License. # # FROM jupyter/base-notebook # # MAINTAINER Jupyter Project <jupyter@googlegroups.com> USER root # Install all OS dependencies for fully functional notebook server RUN apt-get update && apt-get install -yq --no-install-recommends \ build-essential \ emacs \ git \ inkscape \ jed \ libsm6 \ libxext-dev \ libxrender1 \ lmodern \ pandoc \ python-dev \ texlive-fonts-extra \ texlive-fonts-recommended \ texlive-generic-recommended \ texlive-latex-base \ texlive-latex-extra \ texlive-xetex \ vim \ unzip \ && apt-get clean && \ rm -rf /var/lib/apt/lists/* # Switch back to jovyan to avoid accidental container runs as root USER $NB_USER # jupter/scipy-notebook # # Copyright (c) Jupyter Development Team. # # Distributed under the terms of the Modified BSD License. # FROM jupyter/minimal-notebook # # MAINTAINER Jupyter Project <jupyter@googlegroups.com> USER root # libav-tools for matplotlib anim RUN apt-get update && \ apt-get install -y --no-install-recommends libav-tools && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* USER $NB_USER # Install Python 3 packages # Remove pyqt and qt pulled in for matplotlib since we're only ever going to # use notebook-friendly backends in these images RUN conda install --quiet --yes \ 'nomkl' \ 'ipywidgets=7.0*' \ 'pandas=0.19*' \ 'numexpr=2.6*' \ 'matplotlib=2.0*' \ 'scipy=0.19*' \ 'seaborn=0.7*' \ 'scikit-learn=0.18*' \ 'scikit-image=0.12*' \ 'sympy=1.0*' \ 'cython=0.25*' \ 'patsy=0.4*' \ 'statsmodels=0.8*' \ 'cloudpickle=0.2*' \ 'dill=0.2*' \ 'numba=0.31*' \ 'bokeh=0.12*' \ 'sqlalchemy=1.1*' \ 'hdf5=1.8.17' \ 'h5py=2.6*' \ 'vincent=0.4.*' \ 'beautifulsoup4=4.5.*' \ 'protobuf=3.*' \ 'xlrd' && \ conda remove --quiet --yes --force qt pyqt && \ conda clean -tipsy && \ # Activate ipywidgets extension in the environment that runs the notebook server jupyter nbextension enable --py widgetsnbextension --sys-prefix && \ fix-permissions $CONDA_DIR # Install facets which does not have a pip or conda package at the moment RUN cd /tmp && \ git clone https://github.com/PAIR-code/facets.git && \ cd facets && \ jupyter nbextension install facets-dist/ --sys-prefix && \ rm -rf facets && \ fix-permissions $CONDA_DIR # Import matplotlib the first time to build the font cache. ENV XDG_CACHE_HOME /home/$NB_USER/.cache/ RUN MPLBACKEND=Agg python -c "import matplotlib.pyplot" && \ fix-permissions /home/$NB_USER USER $NB_USER # jupter/datascience-notebook # # Copyright (c) Jupyter Development Team. # # Distributed under the terms of the Modified BSD License. # FROM jupyter/scipy-notebook # # MAINTAINER Jupyter Project <jupyter@googlegroups.com> USER root # R pre-requisites RUN apt-get update && \ apt-get install -y --no-install-recommends \ fonts-dejavu \ gfortran \ gcc && apt-get clean && \ rm -rf /var/lib/apt/lists/* # Julia dependencies # install Julia packages in /opt/julia instead of $HOME ENV JULIA_PKGDIR=/opt/julia RUN . /etc/os-release && \ echo "deb http://ppa.launchpad.net/staticfloat/juliareleases/ubuntu $VERSION_CODENAME main" > /etc/apt/sources.list.d/julia.list && \ apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3D3D3ACC && \ apt-get update && \ apt-get install -y --no-install-recommends \ julia && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* && \ # Show Julia where conda libraries are \ echo "push!(Libdl.DL_LOAD_PATH, \"$CONDA_DIR/lib\")" >> /usr/etc/julia/juliarc.jl && \ # Create JULIA_PKGDIR \ mkdir $JULIA_PKGDIR && \ chown $NB_USER $JULIA_PKGDIR && \ fix-permissions $JULIA_PKGDIR USER $NB_USER # R packages including IRKernel which gets installed globally. RUN conda config --system --add channels r && \ conda install --quiet --yes \ 'rpy2=2.8*' \ 'r-base=3.3.2' \ 'r-irkernel=0.7*' \ 'r-plyr=1.8*' \ 'r-devtools=1.12*' \ 'r-tidyverse=1.0*' \ 'r-shiny=0.14*' \ 'r-rmarkdown=1.2*' \ 'r-forecast=7.3*' \ 'r-rsqlite=1.1*' \ 'r-reshape2=1.4*' \ 'r-nycflights13=0.2*' \ 'r-caret=6.0*' \ 'r-rcurl=1.95*' \ 'r-crayon=1.3*' \ 'r-randomforest=4.6*' && \ conda clean -tipsy && \ fix-permissions $CONDA_DIR # Add Julia packages # Install IJulia as jovyan and then move the kernelspec out # to the system share location. Avoids problems with runtime UID change not # taking effect properly on the .local folder in the jovyan home dir. RUN julia -e 'Pkg.init()' && \ julia -e 'Pkg.update()' && \ julia -e 'Pkg.add("HDF5")' && \ julia -e 'Pkg.add("Gadfly")' && \ julia -e 'Pkg.add("RDatasets")' && \ julia -e 'Pkg.add("IJulia")' && \ # Precompile Julia packages \ julia -e 'using HDF5' && \ julia -e 'using Gadfly' && \ julia -e 'using RDatasets' && \ julia -e 'using IJulia' && \ # move kernelspec out of home \ mv $HOME/.local/share/jupyter/kernels/julia* $CONDA_DIR/share/jupyter/kernels/ && \ chmod -R go+rx $CONDA_DIR/share/jupyter && \ rm -rf $HOME/.local && \ fix-permissions $JULIA_PKGDIR $CONDA_DIR/share/jupyter USER $NB_USER RUN jupyter notebook --generate-config && \ jupyter serverextension enable --py jupyterlab --sys-prefix && \ pip install environment_kernels && \ echo "c.NotebookApp.kernel_spec_manager_class = 'environment_kernels.EnvironmentKernelSpecManager'" >> ~/.jupyter/jupyter_notebook_config.py