Jupyterが立ち上げるだけでプログラミング環境ができあがる各種の便利なDockerイメージを出してくれているが、GPUが使えない。ので、無理やり使えるようにしてみた。 Dockerfileの継承元のFROMを書き換える必要があったので強引な感じになった。 以下ではcudaのDockerイメージを元にしているが、jupyterを元にcudaをインストールしたほうが完了までの時間が短いはず。
なお、TensorflowがGPUの使えるJupyter入りのDockerイメージをリリースしてくれている。なので、Jupyter本家に拘らなければこんなことをする必要はほぼない…が、まあせっかく書いたので。
適当に継ぎ接ぎしてDockerfileを作る
mkdir ~/Documents/cuda-based-jupyter-datascience-notebook
cd ~/Documents/cuda-based-jupyter-datascience-notebook
wget https://raw.githubusercontent.com/jupyter/docker-stacks/master/base-notebook/Dockerfile -O base-notebook
wget https://raw.githubusercontent.com/jupyter/docker-stacks/master/minimal-notebook/Dockerfile -O minimal-notebook
wget https://raw.githubusercontent.com/jupyter/docker-stacks/master/scipy-notebook/Dockerfile -O scipy-notebook
wget https://raw.githubusercontent.com/jupyter/docker-stacks/master/datascience-notebook/Dockerfile -O datascience-notebook
cat << "EOF" > Dockerfile
# Merging nvidia/cuda and jupyter/datascience-notebook DockerFiles.
FROM nvidia/cuda:8.0-cudnn6-devel-ubuntu16.04
MAINTAINER rubyu
EOF
cat << "EOF" >> Dockerfile
# jupter/base-notebook
EOF
sed -e '1,8 s/^/# /g' base-notebook >> Dockerfile
cat << "EOF" >> Dockerfile
# jupter/minimal-notebook
EOF
sed -e '1,6 s/^/# /g' minimal-notebook >> Dockerfile
cat << "EOF" >> Dockerfile
# jupter/scipy-notebook
EOF
sed -e '1,5 s/^/# /g' scipy-notebook >> Dockerfile
cat << "EOF" >> Dockerfile
# jupter/datascience-notebook
EOF
sed -e '1,5 s/^/# /g' datascience-notebook >> Dockerfile
cat << "EOF" >> Dockerfile
USER $NB_USER
RUN jupyter notebook --generate-config && \
jupyter serverextension enable --py jupyterlab --sys-prefix && \
pip install environment_kernels && \
echo "c.NotebookApp.kernel_spec_manager_class = 'environment_kernels.EnvironmentKernelSpecManager'" >> ~/.jupyter/jupyter_notebook_config.py
EOF
最後のRUNの部分がJupyter Labとconda createで作る仮想環境のための設定。
Dockerイメージ作成に必要なファイルをダウンロードしておく
wget https://github.com/jupyter/docker-stacks/raw/master/base-notebook/fix-permissions -O fix-permissions wget https://github.com/jupyter/docker-stacks/raw/master/base-notebook/jupyter_notebook_config.py -O jupyter_notebook_config.py wget https://github.com/jupyter/docker-stacks/raw/master/base-notebook/start-notebook.sh -O start-notebook.sh wget https://github.com/jupyter/docker-stacks/raw/master/base-notebook/start-singleuser.sh -O start-singleuser.sh wget https://github.com/jupyter/docker-stacks/raw/master/base-notebook/start.sh -O start.sh sudo chmod ugo+x fix-permissions sudo chmod ugo+x jupyter_notebook_config.py sudo chmod ugo+x start-notebook.sh sudo chmod ugo+x start-singleuser.sh sudo chmod ugo+x start.sh
実行できる権限がないとダメ。ここではwgetしているが、普通はgit cloneするのでハマることはないはず。
コンテナを立ち上げる
sudo nvidia-docker build -t rubyu/datascience-notebook-gpu:1.0 . sudo nvidia-docker run -d --name notebook-gpu -p 8888:8888 --restart=always -v /storage/samba:/home/jovyan rubyu/datascience-notebook-gpu:1.0 start.sh jupyter lab --NotebookApp.token='XXXX'
これで8888ポートでJupyter Labが起動している。
本題には関係ない話ですが、Jupyterのhome（/home/jovyan）をまるごとホストのディレクトリに割り当てて、かつそれをSamba経由でアクセスできるようにするのが好きです。Windowsで処理する必要がある時にいちいち転送する必要がなくなるので。なんやかんやでGUIはWindowsが使いやすい（個人の感想です）
アタッチしてconda createで仮想環境を作る
docker exec -it notebook-gpu /bin/bash conda create -n your-new-envronment python=3.5 jupyter tensorflow-gpu && conda clean -tipsy
作成された仮想環境はJupyterから見えて、カーネルとして選択することができる。
Dockerfile
# Merging nvidia/cuda and jupyter/datascience-notebook DockerFiles.
FROM nvidia/cuda:8.0-cudnn6-devel-ubuntu16.04
MAINTAINER rubyu
# jupter/base-notebook
# # Copyright (c) Jupyter Development Team.
# # Distributed under the terms of the Modified BSD License.
#
# # Ubuntu 16.04 (xenial) from 2017-07-23
# # https://github.com/docker-library/official-images/commit/0ea9b38b835ffb656c497783321632ec7f87b60c
# FROM ubuntu@sha256:84c334414e2bfdcae99509a6add166bbb4fa4041dc3fa6af08046a66fed3005f
#
# MAINTAINER Jupyter Project <jupyter@googlegroups.com>
USER root
# Install all OS dependencies for notebook server that starts but lacks all
# features (e.g., download as all possible file formats)
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && apt-get -yq dist-upgrade \
&& apt-get install -yq --no-install-recommends \
wget \
bzip2 \
ca-certificates \
sudo \
locales \
fonts-liberation \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \
locale-gen
# Install Tini
RUN wget --quiet https://github.com/krallin/tini/releases/download/v0.10.0/tini && \
echo "1361527f39190a7338a0b434bd8c88ff7233ce7b9a4876f3315c22fce7eca1b0 *tini" | sha256sum -c - && \
mv tini /usr/local/bin/tini && \
chmod +x /usr/local/bin/tini
# Configure environment
ENV CONDA_DIR=/opt/conda \
SHELL=/bin/bash \
NB_USER=jovyan \
NB_UID=1000 \
NB_GID=100 \
LC_ALL=en_US.UTF-8 \
LANG=en_US.UTF-8 \
LANGUAGE=en_US.UTF-8
ENV PATH=$CONDA_DIR/bin:$PATH \
HOME=/home/$NB_USER
ADD fix-permissions /usr/local/bin/fix-permissions
# Create jovyan user with UID=1000 and in the 'users' group
# and make sure these dirs are writable by the `users` group.
RUN useradd -m -s /bin/bash -N -u $NB_UID $NB_USER && \
mkdir -p $CONDA_DIR && \
chown $NB_USER:$NB_GID $CONDA_DIR && \
fix-permissions $HOME && \
fix-permissions $CONDA_DIR
USER $NB_USER
# Setup work directory for backward-compatibility
RUN mkdir /home/$NB_USER/work && \
fix-permissions /home/$NB_USER
# Install conda as jovyan and check the md5 sum provided on the download site
ENV MINICONDA_VERSION 4.3.21
RUN cd /tmp && \
wget --quiet https://repo.continuum.io/miniconda/Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh && \
echo "c1c15d3baba15bf50293ae963abef853 *Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh" | md5sum -c - && \
/bin/bash Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh -f -b -p $CONDA_DIR && \
rm Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh && \
$CONDA_DIR/bin/conda config --system --prepend channels conda-forge && \
$CONDA_DIR/bin/conda config --system --set auto_update_conda false && \
$CONDA_DIR/bin/conda config --system --set show_channel_urls true && \
$CONDA_DIR/bin/conda update --all --quiet --yes && \
conda clean -tipsy && \
fix-permissions $CONDA_DIR
# Install Jupyter Notebook and Hub
RUN conda install --quiet --yes \
'notebook=5.2.*' \
'jupyterhub=0.8.*' \
'jupyterlab=0.28.*' \
&& conda clean -tipsy && \
fix-permissions $CONDA_DIR
USER root
EXPOSE 8888
WORKDIR $HOME
# Configure container startup
ENTRYPOINT ["tini", "--"]
CMD ["start-notebook.sh"]
# Add local files as late as possible to avoid cache busting
COPY start.sh /usr/local/bin/
COPY start-notebook.sh /usr/local/bin/
COPY start-singleuser.sh /usr/local/bin/
COPY jupyter_notebook_config.py /etc/jupyter/
RUN fix-permissions /etc/jupyter/
# Switch back to jovyan to avoid accidental container runs as root
USER $NB_USER
# jupter/minimal-notebook
# # Copyright (c) Jupyter Development Team.
# # Distributed under the terms of the Modified BSD License.
#
# FROM jupyter/base-notebook
#
# MAINTAINER Jupyter Project <jupyter@googlegroups.com>
USER root
# Install all OS dependencies for fully functional notebook server
RUN apt-get update && apt-get install -yq --no-install-recommends \
build-essential \
emacs \
git \
inkscape \
jed \
libsm6 \
libxext-dev \
libxrender1 \
lmodern \
pandoc \
python-dev \
texlive-fonts-extra \
texlive-fonts-recommended \
texlive-generic-recommended \
texlive-latex-base \
texlive-latex-extra \
texlive-xetex \
vim \
unzip \
&& apt-get clean && \
rm -rf /var/lib/apt/lists/*
# Switch back to jovyan to avoid accidental container runs as root
USER $NB_USER
# jupter/scipy-notebook
# # Copyright (c) Jupyter Development Team.
# # Distributed under the terms of the Modified BSD License.
# FROM jupyter/minimal-notebook
#
# MAINTAINER Jupyter Project <jupyter@googlegroups.com>
USER root
# libav-tools for matplotlib anim
RUN apt-get update && \
apt-get install -y --no-install-recommends libav-tools && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
USER $NB_USER
# Install Python 3 packages
# Remove pyqt and qt pulled in for matplotlib since we're only ever going to
# use notebook-friendly backends in these images
RUN conda install --quiet --yes \
'nomkl' \
'ipywidgets=7.0*' \
'pandas=0.19*' \
'numexpr=2.6*' \
'matplotlib=2.0*' \
'scipy=0.19*' \
'seaborn=0.7*' \
'scikit-learn=0.18*' \
'scikit-image=0.12*' \
'sympy=1.0*' \
'cython=0.25*' \
'patsy=0.4*' \
'statsmodels=0.8*' \
'cloudpickle=0.2*' \
'dill=0.2*' \
'numba=0.31*' \
'bokeh=0.12*' \
'sqlalchemy=1.1*' \
'hdf5=1.8.17' \
'h5py=2.6*' \
'vincent=0.4.*' \
'beautifulsoup4=4.5.*' \
'protobuf=3.*' \
'xlrd' && \
conda remove --quiet --yes --force qt pyqt && \
conda clean -tipsy && \
# Activate ipywidgets extension in the environment that runs the notebook server
jupyter nbextension enable --py widgetsnbextension --sys-prefix && \
fix-permissions $CONDA_DIR
# Install facets which does not have a pip or conda package at the moment
RUN cd /tmp && \
git clone https://github.com/PAIR-code/facets.git && \
cd facets && \
jupyter nbextension install facets-dist/ --sys-prefix && \
rm -rf facets && \
fix-permissions $CONDA_DIR
# Import matplotlib the first time to build the font cache.
ENV XDG_CACHE_HOME /home/$NB_USER/.cache/
RUN MPLBACKEND=Agg python -c "import matplotlib.pyplot" && \
fix-permissions /home/$NB_USER
USER $NB_USER
# jupter/datascience-notebook
# # Copyright (c) Jupyter Development Team.
# # Distributed under the terms of the Modified BSD License.
# FROM jupyter/scipy-notebook
#
# MAINTAINER Jupyter Project <jupyter@googlegroups.com>
USER root
# R pre-requisites
RUN apt-get update && \
apt-get install -y --no-install-recommends \
fonts-dejavu \
gfortran \
gcc && apt-get clean && \
rm -rf /var/lib/apt/lists/*
# Julia dependencies
# install Julia packages in /opt/julia instead of $HOME
ENV JULIA_PKGDIR=/opt/julia
RUN . /etc/os-release && \
echo "deb http://ppa.launchpad.net/staticfloat/juliareleases/ubuntu $VERSION_CODENAME main" > /etc/apt/sources.list.d/julia.list && \
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3D3D3ACC && \
apt-get update && \
apt-get install -y --no-install-recommends \
julia && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* && \
# Show Julia where conda libraries are \
echo "push!(Libdl.DL_LOAD_PATH, \"$CONDA_DIR/lib\")" >> /usr/etc/julia/juliarc.jl && \
# Create JULIA_PKGDIR \
mkdir $JULIA_PKGDIR && \
chown $NB_USER $JULIA_PKGDIR && \
fix-permissions $JULIA_PKGDIR
USER $NB_USER
# R packages including IRKernel which gets installed globally.
RUN conda config --system --add channels r && \
conda install --quiet --yes \
'rpy2=2.8*' \
'r-base=3.3.2' \
'r-irkernel=0.7*' \
'r-plyr=1.8*' \
'r-devtools=1.12*' \
'r-tidyverse=1.0*' \
'r-shiny=0.14*' \
'r-rmarkdown=1.2*' \
'r-forecast=7.3*' \
'r-rsqlite=1.1*' \
'r-reshape2=1.4*' \
'r-nycflights13=0.2*' \
'r-caret=6.0*' \
'r-rcurl=1.95*' \
'r-crayon=1.3*' \
'r-randomforest=4.6*' && \
conda clean -tipsy && \
fix-permissions $CONDA_DIR
# Add Julia packages
# Install IJulia as jovyan and then move the kernelspec out
# to the system share location. Avoids problems with runtime UID change not
# taking effect properly on the .local folder in the jovyan home dir.
RUN julia -e 'Pkg.init()' && \
julia -e 'Pkg.update()' && \
julia -e 'Pkg.add("HDF5")' && \
julia -e 'Pkg.add("Gadfly")' && \
julia -e 'Pkg.add("RDatasets")' && \
julia -e 'Pkg.add("IJulia")' && \
# Precompile Julia packages \
julia -e 'using HDF5' && \
julia -e 'using Gadfly' && \
julia -e 'using RDatasets' && \
julia -e 'using IJulia' && \
# move kernelspec out of home \
mv $HOME/.local/share/jupyter/kernels/julia* $CONDA_DIR/share/jupyter/kernels/ && \
chmod -R go+rx $CONDA_DIR/share/jupyter && \
rm -rf $HOME/.local && \
fix-permissions $JULIA_PKGDIR $CONDA_DIR/share/jupyter
USER $NB_USER
RUN jupyter notebook --generate-config && \
jupyter serverextension enable --py jupyterlab --sys-prefix && \
pip install environment_kernels && \
echo "c.NotebookApp.kernel_spec_manager_class = 'environment_kernels.EnvironmentKernelSpecManager'" >> ~/.jupyter/jupyter_notebook_config.py
