Suite configurations should be portable between users (at least at the same site). E.g.: another user should be able to run the same suite:
Input configurations should be programming language neutral.
The life cycles of application configurations in a suite may differ from that of the suite.
The configurations are independent of the utilities. For example, the configuration metadata for the suite and application configurations will drive the Rose config editor GUI, but will not be bound or restricted by it.
A configuration in Rose is normally represented by a directory with the following:
We have added the following conventions into our INI format:
[] are placed directly under the
root level.[!SECTION]) or key=value pair (i.e.
!key=value) denotes an ignored setting.
[[hello], [hello]],
[hello [world] and beyond] should all be
errors on parsing.$NAME or ${NAME} for
environment variable substitution.E.g.
# This is line 1 of the comment for this file.
# This is line 2 of the comment for this file.
# This comment will be ignored.
# This is a comment for section-1.
[section-1]
# This is a comment for key-1.
key-1=value 1
# This comment will be ignored.
# This is line 1 of the comment for key-2.
# This is line 2 of the comment for key-2.
key-2=value 2 line 1
value 2 line 2
# This is a comment for key-3.
key-3=value 3 line 1
= value 3 line 2 has leading identation.
=
= value 3 line 3 is blank. This is line 4.
# section-2 is user-ignored.
[!section-2]
key-4=value 4
# ...
[section-3]
# key-5 is program ignored.
!!key-5=value 5
In this document, the shorthand
SECTION=KEY=VALUE is used to represent a
KEY=VALUE pair in a [SECTION]
of an INI format file.
In a Rose configuration directory, we can add an opt/ sub-directory for optional configuration files. Optional configuration files contain additional configuration, which can be selected at run time to override the configuration in the main rose-${TYPE}.conf file. The name of each optional configuration should follow the syntax rose-${TYPE}-${KEY}.conf, where ${KEY} is a short name to describe the override functionality of the optional configuration file.
A root level opts=KEY ... setting in the main configuration will tell the run time program to load the relevant optional configurations in the opt/ sub-directory at run time. Individual Rose utilities may also read optional configuration keys from environment variables and/or command line options.
Where multiple $KEY settings are given, the optional configurations are applied in that order - for example, a setting:
opts=ketchup mayonnaise
implies loading the optional configuration rose-app-ketchup.conf and then the optional configuration rose-app-mayonnaise.conf, which may override the previous one.
By default, a Rose command will fail if an optional configuration file is missing. However, if you put the optional configuration key in brackets, then the optional configuration file is allowed to be missing. E.g.:
opts=ketchup (mayonnaise)
In the above example, rose-app-mayonnaise.conf can be missing.
Some Rose utilities (e.g. rose suite-run, rose task-run, rose app-run, etc) allow optional configurations to be selected at run time using:
--opt-conf-key=KEY or -O KEY.
See reference of individual commands for detail.
Note that by default optional configurations must exist
else an error will be raised. To specify an optional
configuration which may be missing write the name of the
configuration inside parenthesis (e.g. (foo)).
Metadata utilities such as rose
app-upgrade and rose macro treat
each main + optional configuration as a separate entity
to be transformed, upgraded, or validated. Use cases
with more than one optional configuration are not
handled.
When transforming or upgrading, each optional configuration is treated separately and re-created after the transform as a functional difference from the main upgraded configuration.
The logic for transforming or upgrading a main configuration C with optional configurations O1 and O2 into a new main configuration Ct and new optional configurations O1t and O2t can be represented like this:
C => Ct C + O1 => C1t C + O2 => C2t O1t = C1t - Ct O2t = C2t - Ct
A root level import=PATH1 PATH2... setting in the main configuration will tell Rose utilities to search for configurations at PATH1, PATH2 (and so on) and inherit configuration and files from them if found.
At the moment, use of this is only encouraged for configuration metadata.
Some Rose utilities (e.g. rose suite-run,
rose
task-run, rose app-run, etc)
allow you to re-define configuration settings at run
time using the
--define=[SECTION]NAME=VALUE or -D
[SECTION]NAME=VALUE options on the command line.
This would add new settings or override any settings
defined in the main and optional configurations.
E.g.:
# Set [env]FOO=foo, and [env]BAR=bar # (Overriding any original settings of [env]FOO or [env]BAR) rose task-run -D '[env]FOO=foo' -D '[env]BAR=bar' # Switch off [env]BAZ rose task-run -D '[env]!BAZ='
Aspects of some Rose utilities can be configured per
installation via the site configuration file and per
user via the user configuration file. Any configuration
in the site configuration overrides the default, and
any configuration in the user configuration overrides
the site configuration and the default. Rose expects
these files to be in the modified INI format described
above. Rose utilities search for its site configuration
at $ROSE_HOME/etc/rose.conf where
$ROSE_HOME/bin/rose is the location of the
rose command, and they search for the user
configuration at $HOME/.metomi/rose.conf
where $HOME is the home directory of the
current user.
Allowed settings in the site and user configuration files will be documented in a future version of this document. In the mean time, the settings are documented as comments in the etc/rose.conf.example file of each distribution of Rose.
You can also override many internal constants of the
rose config edit and rosie
go. To change the keyboard shortcut of the
Find Next action in the config editor to
F3, put the following lines in your user
config file, and the setting will apply the next time
you run rose config-edit:
[rose-config-edit] accel-find-next=F3
The configuration and functionality of a suite will usually be covered by the use of cylc. In which case, most of the suite configuration will live in the cylc suite.rc file. Otherwise, a suite is just a directory of files.
A suite directory may contain the following:
The suite install configuration file
rose-suite.conf should contain the
information on how to install the suite. It may have
the following sections and root level options:
[env][jinja2:suite.rc][jinja2:suite.rc] section. The
assignments will be inserted after the
#!jinja2 line of the installed
suite.rc file.[file:NAME]file:app/APP=source=LOCATION.It may have the following top level (no section) options:
metaroot-dir=LIST
root-dir=hpc*=$WORKDIR
=*=$DATADIR
In this example, rose suite-run of
a suite with name $NAME will create
~/cylc-run/$NAME as a symbolic link to
$DATADIR/cylc-run/$NAME/ on any machine,
except those with their hostnames matching
hpc*. In which case, it will create
~/cylc-run/$NAME as a symbolic link to
$WORKDIR/cylc-run/$NAME/.
root-dir{share}=LISTroot-dir{share/cycle}=LISTroot-dir{work}=LISTroot-dir-share=LISTroot-dir{share}=LIST.root-dir-work=LISTroot-dir{work}=LIST.The suite information file
rose-suite.info should contain the
information on identify and the purpose of the suite.
It has no sections, only KEY=VALUE pairs. The
owner, project and
title settings are compulsory. Otherwise,
any KEY=VALUE pairs can appear in this file. If the
name of a KEY ends with -list, the value is
expected to be a space-delimited list. The following
keys are known to have special meanings:
ownerprojecttitleaccess-listdescriptionsub-projectproject,
if applicable.The configuration of an application is represented by a directory. It may contain the following:
Files in this directory are copied to the working directory in run time.
Note: If there is a clash between a [file:*] section and a file under file/, the setting in the [file:*] section takes precedence. E.g. Suppose we have a file file/hello.txt. In the absence of [file:hello.txt], it will copy file/hello.txt to $PWD/hello.txt in run time. However, if we have a [file:hello.txt] section and a source=SOURCE setting, then it will install the file from SOURCE instead. If we have [!file:hello.txt], then the file will not be installed at all.
E.g. The application configuration directory may look like:
./bin/ ./rose-app.conf ./file/file1 ./file/file2 ./meta/rose-meta.conf ./opt/rose-app-extra1.conf ./opt/rose-app-extra2.conf ...
The rose-app.conf contains a serialised data structure that is an unordered map (sections=) of unordered maps (keys=values). There can also be keys=values without a section, at the top level. The sections and keys can be:
file-install-rootmetamode[command] section. See also
Running
Tasks > rose task-run > Built-in Applications
Selection[command][env][etc][file:NAME]file:app/APP=source=LOCATION.[namelist:NAME]source setting of a file. Each setting
in a namelist:NAME section is a KEY=VALUE pair
exactly like a normal Fortran namelist, but without
the trailing comma.[namelist:NAME(SORT-INDEX)]source setting of
a file to refer to all
[namelist:NAME(SORT-INDEX)] as
namelist:NAME(:), and the namelist
groups will be sorted alphanumerically by the
SORT-INDEX.[namelist:NAME]).[namelist:NAME{CATEGORY}] or
[namelist:NAME{CATEGORY}(SORT-INDEX)][namelist:NAME] as well as from
their specific category
[namelist:NAME{CATEGORY}].[poll]all-files: A list of space delimited list of file paths. This test passes only if all file paths in the list exist.
any-files: A list of space delimited list of file paths. This test passes if any file path in the list exists.
test: A shell command. This test passes if the command returns a 0 (zero) return code.
Normally, the all-files and
any-files tests both test for the
existence of file paths. If this is not enough,
e.g. you want to test for the existence of a string
in each file, you can specify a
file-test to do a grep.
E.g.:
all-files=file1 file2
file-test=test -e {} && grep -q 'hello' {}
At runtime, any {} pattern in the above would be
replaced with the name of the file. The above make sure that
both file1 and file2 exist and that
they both contain the string hello.
The above tests will only be performed once when the application runner starts. If a list of delays are added, the tests will be performed a number of times with delays between them. If the prerequisites are still not met after the number of delays, the application runner will fail with a time out. The list is a comma-separated list. The syntax looks like [n*][DURATION], where DURATION is an ISO 8601 duration such as PT5S (5 seconds) or PT10M (10 minutes), and n is an optional number of times to repeat it. E.g.:
# Default delays=0 # Poll 1 minute after the runner begins, repeat every minute 10 times delays=10*PT1M # Poll when runner begins, # repeat every 10 seconds 6 times, # repeat every minute 60 times, # repeat once after 1 hour delays=0,6*PT10S,60*PT1M,PT1H
See Running Tasks > rose task-run > Built-in Application: fcm_make.
See Running Tasks > rose task-run > Built-in Application: rose_ana and rose stem > Analysing output with rose_ana.
See Running Tasks > rose task-run > Built-in Application: rose_arch.
See Running Tasks > rose task-run > Built-in Application: rose_prune.
A [file:TARGET] section may have the
following settings:
sourceschemes setting described below.
Normally, a source that does not exist would trigger an error in run time. However, it may be useful to have an optional source for a file sometimes. In which case, the syntax source=(SOURCE) can be used to specify an optional source. E.g. source=namelist:foo (namelist:bar) would allow namelist:bar to be missing or ignored without an error.
checksummodeThe following is a list of all the supported usages:
mode=auto,source=mode=auto,source=SOURCEmode=auto,source=SOURCE-LISTmode=mkdirmode=symlink,source=SOURCEmode=symlink+,source=SOURCEThe root level schemes setting: While
the system would attempt to automatically detect the
scheme of a source, the name of the source can
sometimes be ambiguous. E.g. A URL with a
http scheme can be a path in a version
control system, or a path to a plain file. The root
level schemes setting can be used to help
the system to do the right thing. The general syntax of
the value of the root level schemes
setting looks like:
schemes=PATTERN-1=SCHEME-1
=PATTERN-2=SCHEME-2
E.g.:
schemes=hpc*:*=rsync
=http://host/svn-repos/*=svn
[file:foo.txt]
source=hpc1:/path/to/foo.txt
[file:bar.txt]
source=http://host/svn-repos/path/to/bar.txt
In this example, a URI matching the pattern
hpc*:* would use the rsync
scheme to pull the source to the current host, and a
URI matching the pattern
http://host/svn-repos/* would use the
svn scheme. For all other URIs, the system
will try to make an intelligent guess.
The system will always match a URI in the order as specified by the setting to avoid ambiguity.
The system has built-in support for the following schemes:
fsnamelistnamelist:NAME sections in the
configuration file.svnrsyncrsync
via ssh. A URI should have the form
HOST:PATH. (Note: If required, you can use the
User
setting in ~/.ssh/config to specify the user
ID for logging into HOST.)The application launcher will use the following logic to determine the root directory to install file targets with a relative path:
The rose-ana builtin application reads
information about which analysis steps it should perform
from the rose-app.conf file for that task.
Each of the section names (in square brackets) which
describe an analysis step must follow a particular format:
ana:. This is required
for rose-ana to recognise it as a valid section.FilePattern class
from the grepper module you would provide the
name grepper.FilePattern.The content within each of these sections consists of a series of key-value option pairs; just like other standard Rose apps. However the availability of options for a given section is specified and controlled by the class rather than the meta-data. This makes it easy to provide your own analysis modules without requiring changes to Rose itself.
Therefore you should consult the documentation or source code of the analysis module you wish to use for details of which options it supports. Additionally, some special treatment is applied to all options depending on what they contain:
$ they will be substituted for the
equivalent environment variable, if one is available.
{} the
option will be returned multiple times with the parentheses
substituted once for each argument passed to
rose task-run
The app may also define a configuration section;
[ana:config], whose key-value pairs define app-wide
settings that are passed through to the analysis classes. In
the same way that the task options are dependent on the class
definition; interpretation of the config options is
done by the class(es), so their documentation or source code
should be consulted for details.