GoMSS Nowcast System Workers

Process Flow

The system workflow looks like:

Work flow of preparation for and execution of the daily runs.

The green boxes in the figure above are workers, Python processes that are launched by the system to do a particular job and terminate when that job is finished. The workers are defined in Python modules in the nowcast.workers and nemo_nowcast.workers namespaces.

The Scheduler is a long-running process that periodically checks the system clock and launches workers when their scheduled time to run is reached.

Please see NEMO Nowcast Framework Architecture for a description of the system architecture.

Workers

download_weather

GoMSS NEMO nowcast weather model products download worker.

Download the netCDF files from the Environment Canada GEM 2.5km HRDPS research model forecast produced by Luc Fillion’s group for a specified date; (defaults to today).

nowcast.workers.download_weather.main()

Set up and run the worker.

For command-line usage see:

python -m nowcast.workers.download_weather --help

download_psy4

GoMSS NEMO nowcast worker to download Mercator Ocean PSY4 model fields.

A slice of the Mercator Ocean PSY4V1R2 daily average model product fields are downloaded for use in the preparation of boundary condition files containing u & v currents, sea surface height, salinity, and temperature.

nowcast.workers.download_psy4.main()

Set up and run the worker.

For command-line usage see:

python -m nowcast.workers.download_psy4 --help

make_weather_forcing

GoMSS NEMO nowcast worker to assemble a weather forcing file from a collection of HRDPS forecast files.

Concatenate the contents of the gzipped hourly netCDF files from the Environment Canada GEM 2.5km HRDPS research model forecast for a specified date; (defaults to today) to create a weather forcing file for NEMO.

The concatenated file is a netCDF-4/HDF5 format file with LZ-deflated variables to reduce its size. The number of hours in the forcing file is the same as the number of hours in the forecast (see the weather:download:forecast hrs value in the system configuration file).

The final hour of the previous day’s forecast is used as the 0th hour of weather forcing to avoid restart artifacts from the HRDPS product.

The precipitation is transformed from the cumulative metres of equivalent water to flux in kg/m^2/s. The 0th hour flux is calculated from the final 2 hours accummulation inthe previous day’s forecast.

Solid phase precipitation flux is calculated by assuming that precipitation at grid points where the value of the 2m air temperature (tair) is less than 273.15 Kelvin is snow. The calculated snow file is merged into the dataset created from the HRDPS forecast files to produce the NEMO forcing file.

nowcast.workers.make_weather_forcing.main()

Set up and run the worker.

For command-line usage see:

python -m nowcast.workers.make_weather_forcing --help

make_boundary_conditions

GoMSS NEMO nowcast worker to run Matlab scripts that generate daily boundary conditions files from downloaded Mercator Ocean PSY4 model fields.

nowcast.workers.make_boundary_conditions.main()

Set up and run the worker.

For command-line usage see:

python -m nowcast.workers.make_boundary_conditions --help

run_nemo

GoMSS NEMO nowcast worker that prepares the YAML run description file and bash run script for a nowcast, and launches the run.

nowcast.workers.run_nemo.main()

Set up and run the worker.

For command-line usage see:

python -m nowcast.workers.run_nemo --help

watch_nemo

GoMSS NEMO nowcast worker that monitors and reports on the progress of a NEMO run.

nowcast.workers.watch_nemo.main()

Set up and run the worker.

For command-line usage see:

python -m nowcast.workers.watch_nemo --help

make_plots

GoMSS NEMO nowcast worker that produces model results visualization images for the web site from run results.

nowcast.workers.make_plots.main()

Set up and run the worker.

For command-line usage see:

python -m nowcast.workers.make_plots --help

clear_checklist

NEMO_Nowcast framework clear_checklist worker.

Send a message to the nowcast system manager requesting that it clear its system state checklist.

This worker is normally launched in automation at the end of a nowcast processing cycle (e.g. end of the day), just prior to launching the nemo_nowcast.workers.rotate_logs worker.

It can also be launched from the command-line by the nowcast administrator as necessary for system maintenance.

nemo_nowcast.workers.clear_checklist.main()

Set up and run the worker.

For command-line usage see:

python -m nemo_nowcast.workers.clear_checklist --help

rotate_logs

NEMO_Nowcast framework rotate_logs worker.

Iterate through the nowcast system logging handlers, calling the doRollover() method on any that are instances of logging.handlers.RotatingFileHandler.

This worker is normally launched in automation at the end of a nowcast processing cycle (e.g. end of the day).

It can also be launched from the command-line by the nowcast administrator as necessary for system maintenance.

nemo_nowcast.workers.rotate_logs.main()

Set up and run the worker.

For command-line usage see:

python -m nemo_nowcast.workers.rotate_logs --help

next_workers Module

Functions to calculate lists of workers to launch after previous workers end their work.

Function names must be of the form after_worker_name().

nowcast.next_workers.after_clear_checklist(msg, config, checklist)

Calculate the list of workers to launch after the clear_checklist worker ends.

Parameters

msg (nemo_nowcast.message.Message) – Nowcast system message.

Returns

Worker(s) to launch next

Return type

list

nowcast.next_workers.after_download_psy4(msg, config, checklist)

Calculate the list of workers to launch after the download_psy4 worker ends.

Parameters

msg (nemo_nowcast.message.Message) – Nowcast system message.

Returns

Worker(s) to launch next

Return type

list

nowcast.next_workers.after_download_weather(msg, config, checklist)

Calculate the list of workers to launch after the download_weather worker ends.

Parameters

msg (nemo_nowcast.message.Message) – Nowcast system message.

Returns

Worker(s) to launch next

Return type

list

nowcast.next_workers.after_make_boundary_conditions(msg, config, checklist)

Calculate the list of workers to launch after the make_boundary_conditions worker ends.

Parameters

msg (nemo_nowcast.message.Message) – Nowcast system message.

Returns

Worker(s) to launch next

Return type

list

nowcast.next_workers.after_make_plots(msg, config, checklist)

Calculate the list of workers to launch after the make_plots worker ends.

Parameters

msg (nemo_nowcast.message.Message) – Nowcast system message.

Returns

Worker(s) to launch next

Return type

list

nowcast.next_workers.after_make_weather_forcing(msg, config, checklist)

Calculate the list of workers to launch after the make_weather_forcing worker ends.

Parameters

msg (nemo_nowcast.message.Message) – Nowcast system message.

Returns

Worker(s) to launch next

Return type

list

nowcast.next_workers.after_rotate_logs(msg, config, checklist)

Calculate the list of workers to launch after the rotate_logs worker ends.

Parameters

msg (nemo_nowcast.message.Message) – Nowcast system message.

Returns

Worker(s) to launch next

Return type

list

nowcast.next_workers.after_run_nemo(msg, config, checklist)

Calculate the list of workers to launch after the run_nemo worker ends.

Parameters

msg (nemo_nowcast.message.Message) – Nowcast system message.

Returns

Worker(s) to launch next

Return type

list

nowcast.next_workers.after_watch_nemo(msg, config, checklist)

Calculate the list of workers to launch after the watch_nemo worker ends.

Parameters

msg (nemo_nowcast.message.Message) – Nowcast system message.

Returns

Worker(s) to launch next

Return type

list

Results Figures Modules

The modules in the nowcast.figures namespace are used by the nowcast.workers.make_plots worker to produce the figures that are published to the web from each run. The figures are stored in the /data1/dlatornell/nowcast-sys/figures/ directory tree.

nowcast.figures.shared Module

A collection of constants and functions for use by figure modules in the nowcast.figures namespace(s).

Note

These constants and functions are intended for use only by nowcast.figures modules in order to avoid coupling between figure generation code and code used for other purposes. If you find that you want to use one of these functions outside of those namespaces please talk refactor the function into a tools package.

nowcast.figures.shared.MAP_PARAMS = {'full domain': namespace(ll_lat=37.425, ll_lon=-68.3, lon_0_offset=391.8, meridians=numpy.arange, parallels=numpy.arange, projection='lcc', ur_lat=49.1, ur_lon=-53.45)}

Dictionary of map parameters namespace objects. Items are intended for use as map_params arguments in nowcast.figures.shared.projected_lon_lat_axes() calls. Each item must have the following attributes:

projection

(str) Map projection to use. Please see https://matplotlib.org/basemap/users/mapsetup.html for the list of available projections. Look for the value of the projection argument in the Basemap() calls in the code examples.

ll_lon

(float or int) Longitude of the lower left corner of the map region.

ur_lon

(float or int) Longitude of the upper right corner of the map region.

ll_lat

(float or int) Latitude of the lower left corner of the map region.

ur_lat

(float or int) Latitude of the upper right corner of the map region.

lon_0_offset

(float or int) Central longitude offset. This is a tuning value that is used in combination with the map region corner values above to position the map region withing the plotting axes. Unfortunately, there is no evident algorithm to find its value. The recommended process is to make trial-and-error plots of a a model variable field and the bathymetry, starting with large map region, and gradually reducing the size of the map region and adjusting the lon_0_offset value.

meridians

(numpy.ndarray) Array of longitudes to mark with grid lines and axis tick labels.

parallels

(numpy.ndarray) Array of latitudes to mark with grid lines and axis tick labels.

nowcast.figures.shared.localize_time(data_array, time_coord='time_counter', local_datetime=None)

Offset data_array times to account for local time zone difference from UTC and add tz_name attribute to data_array.

Note

This function is intended for use just before presentation/output of data_array. It is strongly recommended to do all date/time calculations in UTC to avoid time change issues.

Parameters

• data_array (xarray.DataArray or xarray.Dataset) – Data array or dataset object to adjust time values of.

• time_coord (str) – Optional name of the time coordinate.

• local_datetime (arrow.Arrow) – Optional timezone-aware local date/time to use as basis to calculate offset from UTC. The 1st element of data_array is used when local_datetime is None.

nowcast.figures.shared.projected_lon_lat_axes(ax, map_params, bathy, theme, meridians_labels=(False, False, True, True), parallels_labels=(True, True, False, False))

Use Basemap (https://matplotlib.org/basemap/) to calculate transformed coordinates for ax according to the projection and other parameters in map_params.

The (x, y) arrays that are returned are the transformed coordinates to use for plotting on ax.

ax will be labelled with longitudes and latitudes on the sides indicated by meridians_labels and parallels_labels and it will show a longitude/latitude grid.

Parameters

• ax (matplotlib.axes.Axes) – Axes to calculate transformed coordinates for.

• map_params (types.SimpleNamespace) – Namespace of map parameters to use for calculation of transformed coordinates. Please see nowcast.figures.shared.MAP_PARAMS for definitions of the namespace attributes, and examples.

• bathy (xarray.Dataset) –

  A bathymetry dataset that provides the longitude and latitudes arrays on which variables to be plotted are calculated. The longitude and latitude variables names in the dataset are assumed to be nav_lon and nav_lat.

  Note

  This dataset must be neither masked nor scaled. That is accomplished by using the mask_and_scale=False argument in xarray.open_dataset().

• theme (nowcast.figures.website_theme) – Module-like object that defines the style elements for the figure. See nowcast.figures.website_theme for an example.

• meridians_labels (4-tuple of booleans (top, bottom, left, right)) – Axes spines on which to show longitude labels.

• parallels_labels (4-tuple of booleans (top, bottom, left, right)) – Axes spines on which to show latitude labels.

Returns

Transformed x and y coordinates to use for plotting on ax.

Return type

2-tuple of numpy.ndarray

nowcast.figures.website_theme Module

Colour, fonts, and utility functions that define the look and style of figures for the https://gomss.ocean.dal.ca/ web pages.

nowcast.figures.website_theme.COLOURS = {'axes': {'facecolor': '#dbdee1'}, 'axis': {'labels': 'black', 'spines': 'black', 'ticks': 'black'}, 'cbar': {'label': 'black', 'tick labels': 'black'}, 'coastline': 'black', 'contour lines': {'1000m isobath': 'white'}, 'figure': {'facecolor': 'white'}, 'land': '#8b7765', 'text': {'axis': 'black', 'figure title': 'black'}}

Colours of various figure elements; the dict key(s) should be descriptive enough to identify the element to which the colour applies.

nowcast.figures.website_theme.FONTS = {'axis': matplotlib.font_manager.FontProperties, 'axis small': matplotlib.font_manager.FontProperties, 'cbar': {'label': matplotlib.font_manager.FontProperties, 'tick labels': matplotlib.font_manager.FontProperties}, 'figure title': matplotlib.font_manager.FontProperties}

Font properties of various figure text elements; the top level dict keys should be descriptive enough to identify the element to which the font properties apply.

nowcast.figures.website_theme.SITE_BACKGROUND_COLOUR = 'white'

The salishsea.eos.ubc.ca/nemo/ pages background colour, from the https://bootswatch.com/superhero/ theme.

nowcast.figures.website_theme.set_axis_colors(ax)

Set the colours of axis labels, ticks, and spines.

Parameters

ax (matplotlib.axes.Axes) – Axes object to be formatted.

nowcast.figures Figure Modules

nowcast.figures.day_avg_tracer Module

Produce a figure that shows surface values of a tracer field for the full GoMSS model domain. The values are day averages. They are displayed as filled colour contours. The axes grid and tick labels are an angled lon/lat grid using a Lambert Conformal Conic map projection.

Testing notebook for this module is https://nbviewer.jupyter.org/urls/bitbucket.org/gomss-nowcast/gomss_nowcast/raw/default/notebooks/figures/TestDayAvgTracer.ipynb

Development notebook for this module is https://nbviewer.jupyter.org/urls/bitbucket.org/gomss-nowcast/gomss_nowcast/raw/default/notebooks/figures/DevelopDayAvgTracer.ipynb

nowcast.figures.day_avg_tracer.make_figure(results_archive, run_date, var, cmap, bathy, figsize=(16, 9), theme=<module 'nowcast.figures.website_theme' from '/home/docs/checkouts/readthedocs.org/user_builds/gomss-nowcast-system/checkouts/latest/nowcast/figures/website_theme.py'>)

Plot colour contours of day averaged tracer variable surface values for the full GoMSS model domain on an angled lon/lat grid using a Lambert Conformal Conic map projection.

Parameters

• results_archive (pathlib.Path) – Path of directory tree in which NEMO model results are stored.

• run_date (arrow.Arrow) – Run date to produce the figure for.

• var (str) – Name of NEMO results variable to display in figure.

• cmap (matplotlib.colors.ListedColormap) – Colour map to use for filled contours in figure.

• bathy (xarray.Dataset) – GoMSS NEMO model bathymetry.

• figsize (2-tuple) – Figure size (width, height) in inches.

• theme – Module-like object that defines the style elements for the figure. See nowcast.figures.website_theme for an example.

Returns

matplotlib.figure.Figure