Operations, Maintenance, and Monitoring

Cold Start

The long-running processes in the nowcast system framework, the message broker, the manager, and the scheduler, are managed by the circus process manager tool.

Start the nowcast system with:

$ source activate /misc/data1/dlatornell/nowcast-sys/nowcast-env
(/misc/data1/dlatornell/nowcast-sys/nowcast-env)$ circusd --daemon $NOWCAST_CONFIG/circus.ini

circusd monitors the long-running processes and restarts them if they crash or are shutdown accidentally.

System Management

circusctl is the command-line interface for interacting with the processes that are running under circusd. Start it with:

$ source activate /misc/data1/dlatornell/nowcast-sys/nowcast-env
(/misc/data1/dlatornell/nowcast-sys/nowcast-env)$ circusctl

See the circusctl man page, or use the help command within circusctl to get information on the available commands. A few that are useful:

  • list to get a comma-separated list of the processes that circusd is managing

  • status to see their status

  • stop to stop a process; e.g. stop scheduler

  • start to start a stopped process; e.g. start scheduler

  • restart to stop and restart a process; e.g. restart scheduler

  • signal hup to send a HUP signnal to a process, which will cause it to reload its configuration from the NOWCAST_YAML file that the process was started with; e.g. signal hup manager. This is the way to communicate nowcast system configuration changes to the long-running processes.

  • quit to stop all of the processes and shutdown circusd

Use ctrl-c to exit from circusctl.