What are we covering?
A Rose suite is a runnable collection of one or more Rose application configurations, plus any miscellaneous scripts and resources.
A Rose suite hierarchy can look a bit like this:
rose-suite.conf # Rose suite configuration file suite.rc # cylc suite configuration file app/ # App configurations: hello_world/ # hello_world app: rose-app.conf # App configuration file meta/ # Metadata configuration for app: rose-meta.conf # Metadata configuration file opt/ # Optional configuration files: rose-app-hi.conf # Optional configuration file recon/ # recon app: rose-app.conf # App configuration file file/ # Non-Rose-format files: INITHELLOENV # Non-Rose-format file bin/ # Suite scripts and executables: hello.sh # A script meta/ # Metadata for the Rose suite configuration: rose-meta.conf # Metadata configuration file
There are (only) 5 different types of Rose configuration files:
There is also a cylc file, which is in a slightly different INI-based format - suite.rc. This holds cylc suite dependencies and configuration.
[command] default=hello.exe [env] !!MAX_TARGETS=5 [!namelist:hello]
! or !! in front of a section ([!env], [!!env]) or option (!MAX_TARGETS, !!MAX_TARGETS) means it is ignored and will not be output when the configuration is run. A single ! reflects where a user has edited a file to comment out an entry whereas !! shows where a metadata operation has commented out an entry.
Discovery information about the suite:
access-list=bob helen mary owner=fred project=refresh-Q-2013 sub-project=improve-B title=Run XYZ for A, with improved B
N.B. this is a special case of no (or
root
) section.
Holds suite-specific settings (more in a later part of the documentation):
[jinja2:suite.rc] COMPUTE_HOST=blueberrypi [file:app/extract_and_process_blueberries] source=fcm:berry_tr/standard_apps/extract/blueberries@25678 [file:etc/pastry.yum] source=/project/baking/pastry.yum
This configures a specific application in the suite, which is usually a scientific model:
# Holds bash commands to run (e.g. 'rsync X/ Y/', 'run.exe') [command] # The usual command: default=throw_snowball.exe # A different command, run using the option --command-key=recon: recon=reconfigure_snowball.exe # Environment variables set before the command is run [env] MAX_TARGETS=5
# A file that we want to be constructed. [file:local_orography] # The file to create a symlink to. source=/project/snowball/ctldata/orog_devon # mode can also be auto (usually =copy), or mkdir for dirs. mode=symlink # Another file - this time built from our namelist. [file:snow_nl] source=namelist:run_snow # Our namelist [namelist:run_snow] snow_consistency='good' slush_fraction=0.1 care_about_shape=.false.
This provides metadata for a configuration, and is used to drive the config editor GUI.
[namelist:run_snow] help=This section holds options for configuring the snow =retrieval and snowball morphology. [namelist:run_snow=snow_consistency] title=Snow Consistency values='too dry', 'too icy', 'good'
[namelist:run_snow=slush_fraction] description=This specifies the maximum acceptable mass fraction of slush. title=Slush Max Acceptable Fraction [namelist:run_snow=care_about_shape] help=If .true., we'll attempt to make the snowball =like this: = =http://mrhonner.files.wordpress.com/2011/01/snowball-1.jpg = =If .false., it may be a bit messy - this depends =on namelist:run_snow=snow_consistency. type=logical
This is the site/user configuration file, that controls how Rose tools behave.
An example site configuration snippet:
[rose-suite-run] hosts=my-cylc-server1 my-cylc-server2
and a user one:
[rose-config-edit] should-show-ignored=True
(Different format)
This is the cylc configuration file - we'll discuss it properly in a later part of the documentation.
#!jinja2 [scheduling] [[dependencies]] graph = """ tea_start => boil_kettle & get_teabag & \ get_milk boil_kettle & get_teabag => pour_kettle get_milk => pour_milk pour_kettle & pour_milk => stir """ ...
The suite and application configurations both allow the user to construct/install files (and directories) for using at runtime. This gets done at the start before the actual suite or app is run.
Typically, you'd use this to fetch resources that your apps or executables might use when they run. This uses sections that look like [file:TARGET] which accept particular options (source, and optionally mode and checksum) to build a TARGET.
This is most often seen in an app's rose-app.conf where you want to build a file that contains a few namelists:
[file:christmas_tree.nl] source=namelist:baubles namelist:tinsel namelist:fairy_lights
where the namelists are specified in the same rose-app.conf.
However, you can retrieve files from other locations such as Subversion repositories, elsewhere in the filesystem, and via ssh.
If you make the entries in source for a given file:TARGET look like filesystem paths, the app or suite will try to copy these into TARGET. For example, to copy a file /data/stuff/pi_to_a_million_places to a TARGET within the app runtime directory of constants/pi, you would write:
[file:constants/pi] source=/data/stuff/pi_to_a_million_places
If this file is a bit large to copy over, you
could consider asking for a symbolic link instead
of a copy. Use mode=symlink+ to
ensure that the target of the symbolic link exists.
Use mode=symlink if the target of the
symbolic link is not guaranteed to exist at the
time the symbolic link is created. (E.g. the
target may be a file only available in a alternate
compute node.)
[file:constants/pi] mode=symlink+ source=/data/stuff/pi_to_a_trillion_places [file:matter/dark] mode=symlink source=/data/matter/dark
You can also use directories as targets/sources (trailing slashes optional):
[file:constants/] source=/data/stuff/
You may want to retrieve a resource at the start of a suite from a Subversion repository. In that case, you might have this in your rose-suite.conf:
[file:share/icecreams.dat] source=svn://host/nice_things_svn/trunk/icecreams.dat@100
This would install a file icecreams.dat in the share/ directory of the running suite.
You can find out more at the File Creation Reference.
There is a convenient command line utility for accessing Rose configuration information: rose config.
You can view your current site and user configuration by running:
(shell)$
rose config
or to drill down to e.g. the rose-suite-run hosts value:
(shell)$
rose config rose-suite-run hosts
rose config also works for other Rose configuration files (and any configuration files that have similar formats as a Rose configuration file). For example:
(shell)$
rose config --file=/path/to/rose-app.conf env
will print the option-value pairs in the env section in the rose-app.conf file. To print a specific option value, you might run something like:
(shell)$
rose config --file=/path/to/rose-app.conf env SPAM_NUMBER
You can also retrieve metadata values for a Rose configuration setting - for example, running this in an app directory:
(shell)$
rose config --meta env=SPAM_NUMBER help
returns the help (if any) for the SPAM_NUMBER option in the env section.
Once you have a suite with some of these files set up, you can run it on the command line by invoking rose suite-run from within the suite directories. You can see the available options by typing rose help suite-run.
Some useful options for rose suite-run are:
Some useful commands to know about once you have got your suite running:
cylc gui for a suite called
NAME. The rose wrapper uses the base
name of $PWD as NAME in the
absence of the --name=NAME
option.rose
suite-run (for suite and job hosts) for
the suite in each NAME argument. If no
NAME is specified, use the base name
of $PWD as NAME.See the CLI documentation for details of further options and the Quick Reference for a list of commonly used Rose commands.
Rosie uses Subversion/FCM as the version control system. Each suite is a separate FCM project with its own trunk and branches. A simplified branch naming convention is used, e.g. a/a/0/0/1/trunk, a/a/0/0/1/my_branch.
You can work on a suite exactly as you would work on source code managed under FCM.
Rosie provides utilities for copying suites, which is useful when basing a suite on a standard template or forking a project.
However, when proposing or testing changes to a
suite, it's usually better to use a branch
(fcm branch-create).
To examine the ancestry of a suite - what suite
it was copied from and what suites have been copied
from it - you can use rosie graph
SUITEID to get a graphical view.
The config editor does not interface with Subversion/FCM - saving changes purely modifies your working copy, just like a text editor. Changes should be committed from the command line using FCM commands.
rosie go is the suite manager GUI.
It acts as a hub for all your suite work.
Demo! If viewing offline, see the Rosie Go Guide.
rosie lookup queries the Rosie
database (through a web API) to find suites. For
example, running:
(shell)$
rosie lookup bob test
would basically return all suites that had "bob" and "test" in their rose-suite.info files. You can also run more targeted queries such as:
(shell)$
rosie lookup -Q owner eq bob and title contains test
rosie ls lists the suites currently
checked out in a user's roses/
directory.
Points covered:
rosie go and command line
equivalents