Set-up

This section describes the set-up of the GoMSS Nowcast system deployment on the nemo2 host in the cluster.mathstat.dal.ca HPC cluster.

The GoMSS Nowcast project nowcast-vm repository contains repo contains a Vagrant virtual machine configuration that emulates the deployment on nemo2. The VM is intended for development and testing of elements of the nowcast system. The default Vagrant virtualbox provider is used, so it should be possible to use this VM configuration on any machine with Vagrant installed and sufficient resources to run a 512 Mb Ubuntu 14.04 LTS VM. The VM has been extensively used on 2015-vintage laptops running Ubuntu 16.04.

Create Directory Trees

All of the nowcast system files are stored in directory trees that start with nowcast-sys/. There are 2 such trees that were created with:

$ mkdir -p /home/dlatornell/nowcast-sys/
$ mkdir -p /data1/dlatornell/nowcast-sys/

/home/dlatornell/nowcast-sys/ holds the NEMO-3.6 and XIOS code and executables.

/data1/dlatornell/nowcast-sys/ holds the Python code that runs the system, the system configuration files, log files, and directories of forcing and results files.

Add sub-trees for the atmospheric and open boundaries forcing files:

$ mkdir -p /data1/dlatornell/nowcast-sys/forcing/atmospheric/GEM2.5/downloaded
$ mkdir -p /data1/dlatornell/nowcast-sys/forcing/open_boundaries/PSY4/downloaded

Install and Build XIOS

Note

The cluster.mathstat.dal.ca HPC cluster compilers are node-locked on the nemo host, so the steps in this section must be completed on nemo, not nemo2.

Check out the XIOS 1.0 source code from the SVN repository:

$ cd /home/dlatornell/nowcast-sys
$ svn co -r 703 http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/branchs/xios-1.0 XIOS-1.0

Clone the GoMSS Nowcast XIOS-ARCH repository beside the XIOS-1.0 checkout:

$ cd /home/dlatornell/nowcast-sys
$ hg clone ssh://hg@bitbucket.org/gomss-nowcast/xios-arch XIOS-ARCH

Symlink ARCH files for the Dal cluster from the XIOS-ARCH clone into XIOS-1.0/arch/:

$ cd /home/dlatornell/nowcast-sys/XIOS-1.0/arch
$ ln -s ../../XIOS-ARCH/DAL/arch-MATHSTAT_CLUSTER.env
$ ln -s ../../XIOS-ARCH/DAL/arch-MATHSTAT_CLUSTER.fcm
$ ln -s ../../XIOS-ARCH/DAL/arch-MATHSTAT_CLUSTER.path

Build XIOS:

$ cd /home/dlatornell/nowcast-sys/XIOS-1.0
$ ./make_xios --arch MATHSTAT_CLUSTER --job 16 2>&1 | tee make.log

The --arch MATHSTAT_CLUSTER selects the correct build configuration for the cluster.mathstat.dal.ca HPC cluster and compilers. The --job 16 option distributes the build across 16 processors to speed things up. The 2>&1 | tee make.log suffix captures stderr and stdout together and stores them in make.log as well as displaying them on the screen.

Install and Build NEMO-3.6

Note

The cluster.mathstat.dal.ca HPC cluster compilers are node-locked on the nemo host, so the steps in this section must be completed on nemo, not nemo2.

Check out the NEMO 3.6 source code from the SVN repository:

$ cd /home/dlatornell/nowcast-sys
$ svn  --username dlatornell co -r 5659 \
    http://forge.ipsl.jussieu.fr/nemo/svn/branches/2015/nemo_v3_6_STABLE/NEMOGCM NEMO-3.6

Clone the GoMSS Nowcast NEMO-ARCH repository beside the NEMO-3.6 checkout:

$ cd /home/dlatornell/nowcast-sys
$ hg clone ssh://hg@bitbucket.org/gomss-nowcast/nemo-arch NEMO-ARCH

Create a NEMO-3.6/ARCH/DAL/ directory and symlink ARCH files for the Dal cluster from the NEMO-ARCH clone into NEMO-3.6/ARCH/DAL/:

$ mkdir -p /home/dlatornell/nowcast-sys/NEMO-3.6/ARCH/DAL
$ cd /home/dlatornell/nowcast-sys/NEMO-3.6/ARCH/DAL
$ ln -s ../../../NEMO-ARCH/DAL/arch-MATHSTAT_CLUSTER_MPI.fcm

Clone the GoMSS-NEMO-config repository into the NEMO-3.6/CONFIG/ directory:

$ cd NEMO-3.6/CONFIG/
$ hg clone ssh://hg@bitbucket.org/gomss-nowcast/gomss-nemo-config GoMSS_NOWCAST

Add the line GoMSS_NOWCAST OPA_SRC to the NEMO-3.6/CONFIG/cfg.txt file to inform the NEMO FCM build system that the GoMSS_NOWCAST configuration exists:

$ echo GoMSS_NOWCAST OPA_SRC >> cfg.txt

Build NEMO for the GoMSS_NOWCAST configuration, prefixing the ./makenemo command with a shell variable set to the absolute path of the already built XIOS-1.0:

$ XIOS_HOME=$(cd ../../XIOS-1.0/; pwd) \
    ./makenemo -n GoMSS_NOWCAST -m MATHSTAT_CLUSTER_MPI -j16 2>&1 | tee make.log

In the command above, $(cd ../../../XIOS-1.0/; pwd) is a convenient bash idiom for transforming a relative path to the XIOS-1.0/ directory into an absolute path. The -m MATHSTAT_CLUSTER selects the correct build configuration for the cluster.mathstat.dal.ca HPC cluster and compilers. The -j16 option distributes the build across 16 processors to speed things up. The 2>&1 | tee make.log suffix captures stderr and stdout together and stores them in make.log as well as displaying them on the screen.

Install Matlab M_Map Toolkit

The Matlab scripts that the nowcast.workers.make_boundary_conditions worker uses depends on the Matlab M_Map toolkit. It is downloaded and installed with:

$ cd /data1/dlatornell/
$ wget https://www.eoas.ubc.ca/~rich/m_map1.4.tar.gz
$ cd /data1/dlatornell/nowcast-sys/
$ tar xvzf ../m_map1.4.tar.gz

Install Miniconda

The Python components of the nowcast system use a Conda environment to isolate Python package installation. Miniconda3 provides the conda package manager and environment management tools. It is downloaded and installed with:

$ DATA1=/data1/dlatornell
$ MINICONDA_INSTALLER=http://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ wget -q ${MINICONDA_INSTALLER} -O ${DATA1}/miniconda.sh
$ bash ${DATA1}/miniconda.sh -b -p ${DATA1}/miniconda
$ CONDA_BIN=${DATA1}/miniconda/bin
$ echo export PATH=${CONDA_BIN}:\${PATH} >> ${HOME}/.bashrc

The last line prepends the path to the directory containing the conda binaries to PATH to that the conda command will be available.

Use:

$ source ~/.bashrc

to make that PATH change take effect immediately.

Note

If the commands in your ~/.bashrc file are not taking effect when you ssh into nemo2, you probably need to create a ~/.bash_profile file containing:

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

or add those lines to an existing ~/.bash_profile file.

Create nowcast-env Conda Environment

Create a conda environment called nowcast-env that contains the packages required to run the system:

$ CONDA=${CONDA_BIN}/conda
$ NOWCAST_SYS=/data1/dlatornell/nowcast-sys
$ NOWCAST_ENV=${NOWCAST_SYS}/nowcast-env
$ PIP=${NOWCAST_ENV}/bin/pip
$ ${CONDA} create \
  --channel defaults --channel gomss-nowcast --channel conda-forge \
  --prefix ${NOWCAST_ENV} python=3 pip \
  arrow attrs basemap bottleneck circus cmocean dask pyyaml pyzmq requests \
  schedule xarray
$ ${PIP} install raven python-hglib scour
$ echo source activate ${NOWCAST_ENV} >> ${HOME}/.bashrc

The last line append a line to ~/.bashrc that causes the nowcast-env environment to be activated whenever source ~/.bashrc is run.

Install NEMO_Nowcast and GoMSS_Nowcast Packages

During development the NEMO_Nowcast and GoMSS_Nowcast packages are cloned from Bitbucket so that changes can easily be pulled into the deployment environment:

$ cd ${NOWCAST_SYS}
$ hg clone ssh://hg@bitbucket.org/salishsea/nemo-cmd NEMO-Cmd
$ hg clone ssh://hg@bitbucket.org/43ravens/nemo_nowcast NEMO_Nowcast
$ hg clone ssh://hg@bitbucket.org/gomss-nowcast/gomss_nowcast GoMSS_Nowcast

Please see Getting the NEMO_Nowcast Code and Getting the GoMSS_Nowcast Code for more details.

Install the packages via symlinks from the cloned repos using the --editable option of the pip install command:

$ ${PIP} install --editable NEMO-Cmd
$ ${PIP} install --editable NEMO_Nowcast
$ ${PIP} install --editable GoMSS_Nowcast

Once the system reaches stable production status those package installations will be replaced with release packages from the Software Distribution Channel:

$ ${CONDA} update --channel gomss-nowcast nemo-cmd nemo_nowcast gomss_nowcast

Install Motu Client Package

The Python 2.7 motu-client-python package is used by the nowcast.workers.download_psy4 worker to download the Mercator Ocean 1/12° ocean forecast model product. Set up a Python 2.7 conda environment for it to operate in, download it, and install it with:

$ MOTU_ENV=${NOWCAST_SYS}/motu-client-env
$ conda create --prefix ${MOTU_ENV} python=2.7 pip
$ $MOTU_ENV/bin/python2.7 -m pip install motu-client

Set Up System Logging Directory

The system logging files are stored in the ${NOWCAST_SYS}/logs/nowcast/ directory.

$ NOWCAST_LOGS=${NOWCAST_SYS}/logs/nowcast
$ mkdir -p ${NOWCAST_LOGS}

Environment Variables

The nowcast system uses several environment variables to pass configuration information into the system:

  • NOWCAST_ENV: the path to the nowcast-env conda environment

  • NOWCAST_CONFIG: the directory in which the system configuration files are stored

  • NOWCAST_YAML: the path and name of the nowcast system configuration file

  • NOWCAST_LOGS: the directory in which the system log files are stored

  • MOTU_CREDENTIALS: the colon-separated username and password for the CMEMS service from which the Mercator Ocean 1/12° ocean forecast model products are downloaded

  • SENTRY_DSN: the (optional) data source name (DSN) URL to use for exception logging on the Sentry web service

  • SLACK_DAILY_PROGRESS: the (optional) incoming webhook URL to use for daily progress notifications to a Slack channel

To ensure that these environment variables are set at all times when we are working with the nowcast system we use the conda environment activate/deactivate hooks feature.

The environment variables are set upon activation of the nowcast-env environment by creating an activate hook shell script in ${NOWCAST_ENV}/etc/conda/activate.d/:

$ mkdir -p ${NOWCAST_ENV}/etc/conda/activate.d
$ cat << EOF > ${NOWCAST_ENV}/etc/conda/activate.d/envvars.sh
export NOWCAST_ENV=${NOWCAST_ENV}
export NOWCAST_CONFIG=${NOWCAST_ENV}/GoMSS_Nowcast/config
export NOWCAST_YAML=${NOWCAST_ENV}/GoMSS_Nowcast/config/nowcast.yaml
export NOWCAST_LOGS=${NOWCAST_LOGS}
export MOTU_CREDENTIALS=...
export SENTRY_DSN=...
export SLACK_DAILY_PROGRESS=...
EOF

Note

The MOTU_CREDENTIALS environment variable value is not included here because it should not be made public. It should be set to a colon-separated username and password; e.g. username:password. The username and password are obtained by registering an account on the Copernicus Marine Environment Monitoring Service (CMEMS service).

Note

The SENTRY_DSN environment variable value is not included here because it should not be made public. Please see the Logging section of the NEMO_Nowcast package docs for details.

Note

The SLACK_DAILY_PROGRESS environment variable value is not included here because it should not be made public.

Deletion of the environment variables upon deactivation of the environment is done by creating a deactivate hook shell script in ${NOWCAST_ENV}/etc/conda/deactivate.d/:

$ mkdir -p ${NOWCAST_ENV}/etc/conda/deactivate.d
$ cat << EOF > ${NOWCAST_ENV}/etc/conda/deactivate.d/envvars.sh
unset NOWCAST_ENV
unset NOWCAST_CONFIG
unset NOWCAST_YAML
unset NOWCAST_LOGS
unset MOTU_CREDENTIALS
unset SENTRY_DSN
unset SLACK_DAILY_PROGRESS
EOF