Set-up¶
This section describes the set-up of the GoMSS Nowcast system deployment on the nemo2 host in the cluster.mathstat.dal.ca HPC cluster.
The GoMSS Nowcast project nowcast-vm repository contains repo contains a Vagrant virtual machine configuration that emulates the deployment on nemo2. The VM is intended for development and testing of elements of the nowcast system. The default Vagrant virtualbox provider is used, so it should be possible to use this VM configuration on any machine with Vagrant installed and sufficient resources to run a 512 Mb Ubuntu 14.04 LTS VM. The VM has been extensively used on 2015-vintage laptops running Ubuntu 16.04.
Create Directory Trees¶
All of the nowcast system files are stored in directory trees that start with nowcast-sys/
.
There are 2 such trees that were created with:
$ mkdir -p /home/dlatornell/nowcast-sys/
$ mkdir -p /data1/dlatornell/nowcast-sys/
/home/dlatornell/nowcast-sys/
holds the NEMO-3.6 and XIOS code and executables.
/data1/dlatornell/nowcast-sys/
holds the Python code that runs the system,
the system configuration files,
log files,
and directories of forcing and results files.
Add sub-trees for the atmospheric and open boundaries forcing files:
$ mkdir -p /data1/dlatornell/nowcast-sys/forcing/atmospheric/GEM2.5/downloaded
$ mkdir -p /data1/dlatornell/nowcast-sys/forcing/open_boundaries/PSY4/downloaded
Install and Build XIOS¶
Note
The cluster.mathstat.dal.ca HPC cluster compilers are node-locked on the nemo host, so the steps in this section must be completed on nemo, not nemo2.
Check out the XIOS 1.0 source code from the SVN repository:
$ cd /home/dlatornell/nowcast-sys
$ svn co -r 703 http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/branchs/xios-1.0 XIOS-1.0
Clone the GoMSS Nowcast XIOS-ARCH repository beside the XIOS-1.0
checkout:
$ cd /home/dlatornell/nowcast-sys
$ hg clone ssh://hg@bitbucket.org/gomss-nowcast/xios-arch XIOS-ARCH
Symlink ARCH
files for the Dal cluster from the XIOS-ARCH clone into XIOS-1.0/arch/
:
$ cd /home/dlatornell/nowcast-sys/XIOS-1.0/arch
$ ln -s ../../XIOS-ARCH/DAL/arch-MATHSTAT_CLUSTER.env
$ ln -s ../../XIOS-ARCH/DAL/arch-MATHSTAT_CLUSTER.fcm
$ ln -s ../../XIOS-ARCH/DAL/arch-MATHSTAT_CLUSTER.path
Build XIOS:
$ cd /home/dlatornell/nowcast-sys/XIOS-1.0
$ ./make_xios --arch MATHSTAT_CLUSTER --job 16 2>&1 | tee make.log
The --arch MATHSTAT_CLUSTER selects the correct build configuration for the cluster.mathstat.dal.ca HPC cluster and compilers.
The --job 16 option distributes the build across 16 processors to speed things up.
The 2>&1 | tee make.log suffix captures stderr and stdout together and stores them in make.log
as well as displaying them on the screen.
Install and Build NEMO-3.6¶
Note
The cluster.mathstat.dal.ca HPC cluster compilers are node-locked on the nemo host, so the steps in this section must be completed on nemo, not nemo2.
Check out the NEMO 3.6 source code from the SVN repository:
$ cd /home/dlatornell/nowcast-sys
$ svn --username dlatornell co -r 5659 \
http://forge.ipsl.jussieu.fr/nemo/svn/branches/2015/nemo_v3_6_STABLE/NEMOGCM NEMO-3.6
Clone the GoMSS Nowcast NEMO-ARCH repository beside the NEMO-3.6
checkout:
$ cd /home/dlatornell/nowcast-sys
$ hg clone ssh://hg@bitbucket.org/gomss-nowcast/nemo-arch NEMO-ARCH
Create a NEMO-3.6/ARCH/DAL/
directory and symlink ARCH
files for the Dal cluster from the NEMO-ARCH clone into NEMO-3.6/ARCH/DAL/
:
$ mkdir -p /home/dlatornell/nowcast-sys/NEMO-3.6/ARCH/DAL
$ cd /home/dlatornell/nowcast-sys/NEMO-3.6/ARCH/DAL
$ ln -s ../../../NEMO-ARCH/DAL/arch-MATHSTAT_CLUSTER_MPI.fcm
Clone the GoMSS-NEMO-config repository into the NEMO-3.6/CONFIG/
directory:
$ cd NEMO-3.6/CONFIG/
$ hg clone ssh://hg@bitbucket.org/gomss-nowcast/gomss-nemo-config GoMSS_NOWCAST
Add the line GoMSS_NOWCAST OPA_SRC to the NEMO-3.6/CONFIG/cfg.txt
file
to inform the NEMO FCM build system that the GoMSS_NOWCAST configuration
exists:
$ echo GoMSS_NOWCAST OPA_SRC >> cfg.txt
Build NEMO for the GoMSS_NOWCAST configuration, prefixing the ./makenemo command with a shell variable set to the absolute path of the already built XIOS-1.0:
$ XIOS_HOME=$(cd ../../XIOS-1.0/; pwd) \
./makenemo -n GoMSS_NOWCAST -m MATHSTAT_CLUSTER_MPI -j16 2>&1 | tee make.log
In the command above,
$(cd ../../../XIOS-1.0/; pwd) is a convenient bash idiom for transforming a relative path to the XIOS-1.0/
directory into an absolute path.
The -m MATHSTAT_CLUSTER selects the correct build configuration for the cluster.mathstat.dal.ca HPC cluster and compilers.
The -j16 option distributes the build across 16 processors to speed things up.
The 2>&1 | tee make.log suffix captures stderr and stdout together and stores them in make.log
as well as displaying them on the screen.
Install Matlab M_Map Toolkit¶
The Matlab scripts that the nowcast.workers.make_boundary_conditions
worker uses depends on the Matlab M_Map toolkit.
It is downloaded and installed with:
$ cd /data1/dlatornell/
$ wget https://www.eoas.ubc.ca/~rich/m_map1.4.tar.gz
$ cd /data1/dlatornell/nowcast-sys/
$ tar xvzf ../m_map1.4.tar.gz
Install Miniconda¶
The Python components of the nowcast system use a Conda environment to isolate Python package installation. Miniconda3 provides the conda package manager and environment management tools. It is downloaded and installed with:
$ DATA1=/data1/dlatornell
$ MINICONDA_INSTALLER=http://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ wget -q ${MINICONDA_INSTALLER} -O ${DATA1}/miniconda.sh
$ bash ${DATA1}/miniconda.sh -b -p ${DATA1}/miniconda
$ CONDA_BIN=${DATA1}/miniconda/bin
$ echo export PATH=${CONDA_BIN}:\${PATH} >> ${HOME}/.bashrc
The last line prepends the path to the directory containing the conda binaries to PATH
to that the conda command will be available.
Use:
$ source ~/.bashrc
to make that PATH
change take effect immediately.
Note
If the commands in your ~/.bashrc
file are not taking effect when you ssh into nemo2,
you probably need to create a ~/.bash_profile
file containing:
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
or add those lines to an existing ~/.bash_profile
file.
Create nowcast-env Conda Environment¶
Create a conda environment called nowcast-env that contains the packages required to run the system:
$ CONDA=${CONDA_BIN}/conda
$ NOWCAST_SYS=/data1/dlatornell/nowcast-sys
$ NOWCAST_ENV=${NOWCAST_SYS}/nowcast-env
$ PIP=${NOWCAST_ENV}/bin/pip
$ ${CONDA} create \
--channel defaults --channel gomss-nowcast --channel conda-forge \
--prefix ${NOWCAST_ENV} python=3 pip \
arrow attrs basemap bottleneck circus cmocean dask pyyaml pyzmq requests \
schedule xarray
$ ${PIP} install raven python-hglib scour
$ echo source activate ${NOWCAST_ENV} >> ${HOME}/.bashrc
The last line append a line to ~/.bashrc
that causes the nowcast-env environment to be activated whenever source ~/.bashrc is run.
Install NEMO_Nowcast and GoMSS_Nowcast Packages¶
During development the NEMO_Nowcast and GoMSS_Nowcast packages are cloned from Bitbucket so that changes can easily be pulled into the deployment environment:
$ cd ${NOWCAST_SYS}
$ hg clone ssh://hg@bitbucket.org/salishsea/nemo-cmd NEMO-Cmd
$ hg clone ssh://hg@bitbucket.org/43ravens/nemo_nowcast NEMO_Nowcast
$ hg clone ssh://hg@bitbucket.org/gomss-nowcast/gomss_nowcast GoMSS_Nowcast
Please see Getting the NEMO_Nowcast Code and Getting the GoMSS_Nowcast Code for more details.
Install the packages via symlinks from the cloned repos using the --editable option of the pip install command:
$ ${PIP} install --editable NEMO-Cmd
$ ${PIP} install --editable NEMO_Nowcast
$ ${PIP} install --editable GoMSS_Nowcast
Once the system reaches stable production status those package installations will be replaced with release packages from the Software Distribution Channel:
$ ${CONDA} update --channel gomss-nowcast nemo-cmd nemo_nowcast gomss_nowcast
Install Motu Client Package¶
The Python 2.7 motu-client-python package is used by the nowcast.workers.download_psy4
worker to download the Mercator Ocean 1/12° ocean forecast model product.
Set up a Python 2.7 conda environment for it to operate in,
download it,
and install it with:
$ MOTU_ENV=${NOWCAST_SYS}/motu-client-env
$ conda create --prefix ${MOTU_ENV} python=2.7 pip
$ $MOTU_ENV/bin/python2.7 -m pip install motu-client
Set Up System Logging Directory¶
The system logging files are stored in the ${NOWCAST_SYS}/logs/nowcast/
directory.
$ NOWCAST_LOGS=${NOWCAST_SYS}/logs/nowcast
$ mkdir -p ${NOWCAST_LOGS}
Environment Variables¶
The nowcast system uses several environment variables to pass configuration information into the system:
NOWCAST_ENV
: the path to the nowcast-env conda environmentNOWCAST_CONFIG
: the directory in which the system configuration files are storedNOWCAST_YAML
: the path and name of the nowcast system configuration fileNOWCAST_LOGS
: the directory in which the system log files are storedMOTU_CREDENTIALS
: the colon-separated username and password for the CMEMS service from which the Mercator Ocean 1/12° ocean forecast model products are downloadedSENTRY_DSN
: the (optional) data source name (DSN) URL to use for exception logging on the Sentry web serviceSLACK_DAILY_PROGRESS
: the (optional) incoming webhook URL to use for daily progress notifications to a Slack channel
To ensure that these environment variables are set at all times when we are working with the nowcast system we use the conda environment activate/deactivate hooks feature.
The environment variables are set upon activation of the nowcast-env environment by creating an activate hook shell script in ${NOWCAST_ENV}/etc/conda/activate.d/
:
$ mkdir -p ${NOWCAST_ENV}/etc/conda/activate.d
$ cat << EOF > ${NOWCAST_ENV}/etc/conda/activate.d/envvars.sh
export NOWCAST_ENV=${NOWCAST_ENV}
export NOWCAST_CONFIG=${NOWCAST_ENV}/GoMSS_Nowcast/config
export NOWCAST_YAML=${NOWCAST_ENV}/GoMSS_Nowcast/config/nowcast.yaml
export NOWCAST_LOGS=${NOWCAST_LOGS}
export MOTU_CREDENTIALS=...
export SENTRY_DSN=...
export SLACK_DAILY_PROGRESS=...
EOF
Note
The MOTU_CREDENTIALS
environment variable value is not included here because it should not be made public.
It should be set to a colon-separated username and password;
e.g. username:password.
The username and password are obtained by registering an account on the Copernicus Marine Environment Monitoring Service (CMEMS service).
Note
The SENTRY_DSN
environment variable value is not included here because it should not be made public.
Please see the Logging section of the NEMO_Nowcast package docs for details.
Note
The SLACK_DAILY_PROGRESS
environment variable value is not included here because it should not be made public.
Deletion of the environment variables upon deactivation of the environment is done by creating a deactivate hook shell script in ${NOWCAST_ENV}/etc/conda/deactivate.d/
:
$ mkdir -p ${NOWCAST_ENV}/etc/conda/deactivate.d
$ cat << EOF > ${NOWCAST_ENV}/etc/conda/deactivate.d/envvars.sh
unset NOWCAST_ENV
unset NOWCAST_CONFIG
unset NOWCAST_YAML
unset NOWCAST_LOGS
unset MOTU_CREDENTIALS
unset SENTRY_DSN
unset SLACK_DAILY_PROGRESS
EOF