Configuration Metadata

Rose User Guide: Configuration Metadata

Configuration Metadata

Introduction

What are we covering?

  • Configuration Metadata
  • rose config-edit
  • rose macro
  • rose app-upgrade, rose stem

Configuration Metadata

Metadata describes other Rose configuration settings - help, range, pattern, and so on.

We should have already seen some metadata in the Metadata Tutorial.

Metadata Location

Metadata for rose-suite.conf (and rose-app.conf) settings can be referenced by the meta flag in an app config, e.g.:

meta=rose-demo-upgrade/27.1

This usually references centrally-stored metadata, looked up using the meta-path user/site config setting. The example above, for a meta-path of /project/rose-meta/, would be stored at /project/rose-meta/rose-demo-upgrade/27.1/.

Metadata Locations (cont.)

We are not limited to a flat structure for our centrally-stored metadata. For example, if we wanted to take our example above and adopt a tree structure for all our rose-demo apps, the meta tag may be:

meta=rose-demo/upgrade/27.1

and so the metadata would be stored at /project/rose-meta/rose-demo/upgrade/27.1/.

Specifying Other Locations

You can store metadata in other locations and override where the meta flag points to, using:

  • The --meta-path=PATH option of relevant commands.
  • The ROSE_META_PATH environment variable.
  • The meta-path setting in site/user configurations.

Alternatively, you can store metadata in a meta/ subdirectory in a suite or an application, which will override the others (e.g. as in the Metadata Tutorial).

Metadata Import

Metadata can also inherit from other metadata, using a top-level import option. For example, if you had a rose-meta.conf file that began like this:

import=salt-and-vinegar/vn2.0 cheese-and-onion/vn3.0

then the metadata would include content from those two meta locations.

Metadata on Branches

If you wanted to develop metadata on a branch, using a working copy somewhere, you could test it by pointing meta-path at it (e.g. by running rose config-edit --meta-path=$HOME/branch/rose-meta/).

Advanced Configuration Metadata

The Metadata Tutorial covered some fundamental metadata usage - there is quite a lot more!

  • Logical expression syntax
  • More metadata options
  • Macros
  • Widgets

Logical Expression Syntax

There is a logical expression syntax for several metadata options. Here are some examples:

  • range: 
    [namelist:run_em=number_magnetic_poles]
range=this % 2 == 0 and this >= 2
  • trigger: 
    [namelist:run_snow=slush_fraction]
trigger=namelist:run_snow=num_slushballs: this > 0.5
  • warn-if: 
    [namelist:run_snow=ice_fraction]
warn-if=this >= namelist:run_snow=max_ok_ice_fraction or this < 0.2

Configuration Metadata - Logic Syntax

The syntax is a subset of Python with some extra metadata functionality - you can use standard Python operators e.g.:

(foo * bar + baz) / wibble >= wobble - flob ** qux

or

"spam" in foo and not bar or (baz and 5 < wibble <= wobble)

Where you would normally use a variable name, use an id to refer to the value of a setting (e.g. namelist:run_snow=max_ok_ice_fraction) or use this as a shorthand for the id of this setting.

Configuration Metadata - Logic Syntax Arrays

The syntax also has array element support via bracketed indices (e.g. this(2) or namelist:run_snow=monthly_snowfall(12)).

There are any and all functions (e.g. any(namelist:run_snow=monthly_snowfall > 10)).

Configuration Metadata - Logic Syntax Reference

For more information, see the reference page.

More Metadata Highlights

See Metadata Reference and the Advanced Metadata Tutorials.

compulsory
The most controversial metadata option! Setting it to true for a section means that the section should always be in the configuration, ignored or not. Setting it to true for an option in a section means that the option is required to be in the configuration, ignored or not, if and only if its section is in the configuration.

More Metadata Highlights - duplicate

duplicate
Allow duplicated copies of the setting. This is used for sections where there may be more than one with the same metadata - for example multiple namelist groups of the same name.
sort-key
Used to order and group pages and variables in the config editor.

More Metadata Highlights - value-titles

value-titles
This works with values see; Metadata Reference for further information. Allows you to set a title to display for each of the values in values rather than the actual value e.g. for values=0, 1 and value-titles=False, True "False" is displayed instead of the value "0" and "True" is displayed instead of the value "1".

More Metadata Highlights - value-hints

value-hints
Allows you to set hints to display for each of the suggested values for a variable, but still allows the user to provide their own override. This can be used as an auto-complete in rose edit using value-hints i.e. value-hints=pear,peach,plum as suggested fruits.

More Metadata Highlights - warn-if/fail-if

warn-if, fail-if
These use the full logical syntax we just looked at. Unlike range and trigger, they can use other variable ids. This makes them slow, so they are only run on request (config editor validate button/menu, rose macro). fail-if will cause an error if the expression evaluates true. warn-if is the same, but will flag up a warning instead of an error. For example: 
fail-if=any(namelist:test=ctrl_array % this == 0)

More Metadata Highlights - widget

macro
Used to associate a setting with a set of custom macros.
widget
Used to set a custom or different built-in widget for the setting (in the config editor).

Macros

Macros are written in Python and live with metadata under lib/python/macros/. There are three categories:

  • Validator macros - checking macros that can flag errors in your application configurations. Only used when metadata is not enough.
  • Transformer macros - changing macros that alter your application configurations. These are useful when applying a change that affects many settings - for example, a change in resolution or domain.

Macros continued

  • Upgrade macros - upgrade (or downgrade) application configurations

You can find out how to write macros in the Advanced Metadata Tutorials and the API documentation.

Widgets

You can replace widgets in the config editor with your own. These live with metadata under lib/python/widget/. For more details, see the API documentation and the Advanced Tutorial.

Icons!

You can set an icon in the metadata that is used in the config editor to distinguish configurations: put a file etc/icon.png in the metadata directory.

Rose Icon

rose config-edit

The config editor is the suite/application editor GUI.

Demo! If this isn't part of a course, see the Config Editor Guide.

rose macro

This is the CLI equivalent (together with a text editor) of the config editor. It applies Rose built-in macros to applications, plus any custom macros defined in the metadata.

When run from an application directory rose macro will run on the contained rose-app.conf file. When run from elsewhere within the suite's directory rose macro will run on all rose-app.conf files as well as the rose-suite.conf and rose-suite.info files.

rose macro Examples

The macros to be run can be specified in the following ways:

  • rose macro --fix - runs all internal transformer (fixer) macros.
  • rose macro --transform - runs all transformer macros.
  • rose macro --validate - runs all validator macros.
  • rose macro MY_MACRO - runs a macro called MY_MACRO.

In an example app in the rose distribution:

rose macro fib.FibonnaciChecker

produces:

[V] fib.FibonacciChecker: issues: 1
    env=INVALID_SEQUENCE=0, 1, 1, 2, 3, 5, 8, 14
        not the Fibonacci sequence

Upgrading

You can upgrade app configurations to use newer versions of metadata (corresponding to newer versions of the application code). This may result in some changes to the configuration (e.g. new or renamed variables).

rose app-upgrade

Upgrade macros are the mechanism for making these changes - they are a special form of macro. You can upgrade a configuration that has upgrade macros defined by running rose app-upgrade.

rose app-upgrade Example

For example, you might run:

rose app-upgrade vn2.0

which might give:

[U] Upgradevn1.4-vn2.0: changes: 2
    namelist:shoes=self_tying_shoelaces=2
        Added with value '2'
    =meta=shoe-control/vn2.0
        Upgraded from vn1.4 to vn2.0
Accept y/n (default n)?

rose stem

rose stem is a testing system for use with Rose. It provides a user-friendly way of defining source trees and tasks on the command line which are then passed by rose stem to the suite as Jinja2 variables. There is a User Guide section and an Advanced Tutorial for rose-stem.

Future of Rose Metadata

  • More work for us
  • More work for you
  • More fun for all