Rose Advanced Tutorial: suite.rc Templating

suite.rc Templating

Introduction

This part of the Rose user guide walks you through using templating and rose-suite.conf variables in your suite.rc, using Jinja2.

This allows you to collapse repeated configuration, and move commonly used settings to a central place. It is particularly useful for suites with many, very similar tasks.

Example

We'll demonstrate templating by using the finale of a firework display as an example.

Large firework displays can have sophisticated time-sensitive scheduling with a large number of similar 'tasks'.

Example suite.rc

We'll start out by looking at a suite.rc file that doesn't use any templating.

Create a new suite (or just a new directory somewhere - e.g. in your homespace) containing a blank rose-suite.conf and a suite.rc file that looks like this.

Example suite.rc Explained

This suite.rc has two families of tasks - tasks within a family are either identical (IGNITE family) or variations on a theme (DETONATE family). There are clear patterns in both the dependency graph and the [runtime] section.

We want to collapse most of this down with Jinja2.

Starting Out

The first thing to do is to mark the suite.rc as a jinja2 file by adding this shebang line to the top of the file:

#!jinja2

For Loops

Let's have a look at using a for loop in Jinja2.

The dependency graph for the ignition tasks follows a simple pattern for the first 16 tasks.

For Loop, Part 1

We can replace the lines:

    ignite_rocket_00 => \
    ignite_rocket_01 => \
    ignite_rocket_02 => \
    ...
    ignite_rocket_15 => \

with:

{%- for num in range(16) %}
           ignite_rocket_{{ num }} => \
{%- endfor %}

For Loop, Part 1 - Explanation

We've used Jinja2 blocks ({% to %}) to template the ignite_rocket... line 16 times, substituting ({{ to }}) a number num in the line each time.

When evaluated, it will produce something that is very nearly correct:

            ignite_rocket_0 => \
            ignite_rocket_1 => \
            ignite_rocket_2 => \
            ignite_rocket_3 => \
            ...

Variable Assignment

This doesn't have properly formatted number suffixes like the original text - the original formatting would sort correctly in cylc gui (_00, _01) and other output.

We can produce nicely formatted numbers by creating another variable in Jinja2, inside the for loop. Replace the for loop text with:

{%- for num in range(16) %}
{%- set num_label = '%02d' % num %}
           ignite_rocket_{{ num_label }} => \
{%- endfor %}

Whitespace Control

This would produce output like this:

           ignite_rocket_00 => \
           ignite_rocket_01 => \
           ignite_rocket_02 => \
           ...

For loop, part 2

We can template away the rest of the graph in exactly the same way. Replace ignite_rocket_16 & \ to ignite_rocket_28 & \ with:

{%- for num in range(16, 29) %}
{%- set num_label = '%02d' % num %}
                          ignite_rocket_{{ num_label }} & \
{%- endfor %}

However, this doesn't handle the special case of the ignite_rocket_29 line, and it feels redundant to have an almost duplicated loop below our first one.

If Blocks

Jinja2 supports if blocks, so we can actually change what we do based on the value of a Jinja2 variable. Replace the first and second loops and the ignite_rocket_29 line with:

{%- for num in range(30) %}
{%- set num_label = '%02d' % num %}
{%- if num <= 15 %}
            ignite_rocket_{{ num_label }} => \
{%- elif num == 29 %}
                          ignite_rocket_{{ num_label }}
{%- else %}
                          ignite_rocket_{{ num_label }} & \
{%- endif %}
{%- endfor %}

This will give us the correct output.

For Loops, part 3

We can also replace the last part of the dependency graph. Replace the whole ignite_rocket_00 => detonate_rocket_00 to ignite_rocket_29 => detonate_rocket_29 loop with:

{%- for num in range(30) %}
{%- set num_label = '%02d' % num %}
            ignite_rocket_{{ num_label }} => detonate_rocket_{{ num_label }}
{%- endfor %}

Finished graph

Your dependency graph configuration should now look like:

        graph = """
            start => \
{%- for num in range(30) %}
{%- set num_label = '%02d' % num %}
{%- if num <= 15 %}
            ignite_rocket_{{ num_label }} => \
{%- elif num == 29 %}
                          ignite_rocket_{{ num_label }}
{%- else %}
                          ignite_rocket_{{ num_label }} & \
{%- endif %}
{%- endfor %}
{%- for num in range(30) %}
{%- set num_label = '%02d' % num %}
            ignite_rocket_{{ num_label }} => detonate_rocket_{{ num_label }}
{%- endfor %}
            DETONATE:finish-all => stop
        """

Runtime Loops

We can also replace some of the [runtime] configuration - the ignite_rocket tasks are extremely repetitive, but the detonate_rocket tasks have some differences.

We'll start with the ignite_rocket tasks - replace the sections ignite_rocket_00 to ignite_rocket_29 with:

{%- for num in range(30) %}
{%- set num_label = '%02d' % num %}
    [[ignite_rocket_{{ num_label }}]]
        inherit = IGNITE
{%- endfor %}

Runtime Loops - Patterns

The detonate_rocket tasks are more complex, but have some repeating patterns based on the task number. SOUND alternates, and COLOUR_CODE repeatedly loops from 1 to 5.

This means we can extract this information from the task number with some Jinja2 magic.

Runtime Loops - Replacement

Replace the sections detonate_rocket_00 to detonate_rocket_29 with:

{%- for num in range(30) %}
{%- set num_label = '%02d' % num %}
    [[detonate_rocket_{{ num_label }}]]
        inherit = DETONATE
        [[[environment]]]
            COLOUR_CODE = {{ num % 5 + 1 }}
            SOUND = {{ ["BANG", "WHOOSH"][num % 2] }}
{%- endfor %}

Runtime Loops - Explanation

This uses the remainder operator of Jinja2/Python (%) to extract the correct number for COLOUR_CODE, and to get the correct list element for SOUND. N.B. Jinja2 supports quite a lot of Python syntax, but not all.

Variables

We can see that we've repeated range(30) a lot - it would be much better to move the 30 number into a variable to remove the hard-coding.

Replace all the occurrences of 30 with ROCKET_NUMBER, and put this line above the [cylc] section at the top of the file:

{%- set ROCKET_NUMBER = 30 %}

Variable Expressions

We should also replace some of the other hard-coded numbers - the first loop should become:

{%- for num in range(ROCKET_NUMBER) %}
{%- set num_label = '%02d' % num %}
{%- if num <= ROCKET_NUMBER / 2 %}
            ignite_rocket_{{ num_label }} => \
{%- elif num == ROCKET_NUMBER - 1 %}
                          ignite_rocket_{{ num_label }}
{%- else %}
                          ignite_rocket_{{ num_label }} & \
{%- endif %}
{%- endfor %}

Variable Expressions - Continued

We should also replace the other range(30) expressions.

List Variables

We can also move our SOUND list into a variable - replace the line:

            SOUND = {{ ["BANG", "WHOOSH"][num % 2] }}

with

            SOUND = {{ ROCKET_SOUNDS[num % 2] }}

We should also put a line at the top setting the variable, after the {%- set ROCKET_NUMBER line:

{%- set ROCKET_SOUNDS = ["BANG", "WHOOSH"] %}

suite.rc Standalone Draft

Your suite.rc file should now look like this.

This is a valid way of writing a suite.rc - but we could centralise the input variables even more.

rose-suite.conf

Jinja2 input variables, like the ones we've created, can be moved into the rose-suite.conf file for easy configurability and metadata support.

Move the variables into the rose-suite.conf file by removing the two {% set ROCKET lines and appending the following text to the rose-suite.conf file:

[jinja2:suite.rc]
ROCKET_NUMBER = 30
ROCKET_SOUNDS = ["BANG", "WHOOSH"]

Summary So Far

We have:

used for loops to collapse repeated content
used if blocks to control differences in content
centralised hardcoded settings and moved them to the rose-suite.conf file.

Adding the app

There's one thing missing from the suite before it can run successfully - the ROSE_TASK_APP for the DETONATE family of tasks. Create an app/detonate/ directory in your suite, and create a rose-app.conf file under that directory that looks like this.

Examining Template Output

We can look at our template output by getting cylc to pre-process the jinja2 out of our suite.rc, outputting it in a cylc-readable format. We first need to register and validate it.

We could use cylc register and cylc validate, but as we're (hopefully) going to run the suite in a minute, run:

rose suite-run -- --hold

to automatically do this and start the suite in a paused state.

cylc gui, cylc view -g -j

You can use the graph view in cylc gui to examine the dependency tree before you run the suite. You can also examine the text directly. Run:

cylc view -g -j $SUITE_NAME

where $SUITE_NAME is the name of the suite directory. Keep the terminal you've used open. You should get a text editor with the Jinja2-and-cylc-processed suite.rc, which should be functionally identical to our starting point - continuation lines may be collapsed.

Running the Suite

Start the suite running by clicking the relevant control in the open cylc gui program, then bring up your most recently used terminal window (probably the cylc view one).

You should see some output!

Optional Extras

If you like, you can try editing the suite to increase/decrease the number of tasks or change the output. You may want to add some metadata for your rose-suite.conf settings.