Rose User Guide: Suite Writing Tutorial

Suite Writing Tutorial

Introduction

This chapter of the Rose user guide walks you through creating a suite and applications from scratch, using a simple example. Usually, users will create suites from existing standard template suites.

You should already be familiar with the brief tour.

Example

We expect most applications to be configurations for Fortran programs - your suite will be built around their inputs and their inter-dependencies.

This example supposes:

We're on a sailing ship, making a passage
We're navigating using dead reckoning and some Fortran code

Starting Out

We need to analyse our program to figure out:

what it needs to run
when it needs to run

dead_reckoning.f90

Our Fortran program is dead_reckoning.f90, which (pretends to) calculate the new position of our ship.

Have a look at this file.

Analysing the Program

The inputs to dead_reckoning.f90 are:

Two environment variables, TIME_INTERVAL_HRS and POS_FPATH
A file report.nl with one namelist report_nl, containing an option to control verbosity, l_verbose
An input (and output) file located at $POS_FPATH that stores the latitude and longitude.

What It Needs to Run

When we run the compiled program, we'll need all these inputs to be present.

In this example, we'll run the compiled program as a proper Rose app. This means that environment variables can be set in the suite.rc or in our Rose app config. The report.nl namelist file should be generated via the app config.

The input/output file needs to be outside the app, stored in the suite directory somewhere.

When It Needs to Run

We want to run this program every 3 hours, on the hour - so there is a repeated dependency on the time or cycle.

We also want to build the program to begin with, so we need an fcm_make app that runs at the start.

Create a Blank Suite

To start making our suite, we need to do the following:

run rosie create
write some (made up) suite information in the text file
answer y at the prompt.

We then get a working copy of the new suite in the $HOME/roses/ directory.

In the new suite working copy, we see two files:

rose-suite.conf (empty)
rose-suite.info (information that we just filled in)

We need to add a lot more than this!

Install the Source

We'll need to store the source code with our suite.

Change directory to the suite and make a directory src/:

cd ~/roses/$SUITE_ID

mkdir src

Copy the source code from dead_reckoning.f90 into a file called src/dead_reckoning.f90.

Install the Source - results

Your suite should now look like this:

ls -R

.:
src  rose-suite.conf  rose-suite.info

./src:
dead_reckoning.f90

Add an App

Let's make an application configuration for our fortran program. We can call it whatever we like - let's choose navigate

Make a directory app/ in the root directory of the suite:

mkdir app

Now make a directory inside app/ called navigate:

mkdir app/navigate

App Explanation

This new directory is the container for our app configuration. It could hold a Rose configuration file, other files within an optional file/ subdirectory, scripts in a bin/ directory, and more.

In our example we'll only need to add a Rose configuration file, rose-app.conf, inside it.

Add an App Configuration

We need to cover the following inputs:

Environment TIME_INTERVAL_HRS and POS_FPATH
A namelist file report.nl with one namelist report_nl, containing a logical variable l_verbose
An input (and output) file called position that stores the latitude and longitude.

Add an App Configuration Continued

We've decided we'll setup the environment and the position file in the suite, so ignore them for now.

We'll now create our Rose configuration file for the navigate app.

Add an App Configuration Contents

Create a file called rose-app.conf in the app/navigate/ directory:

touch app/navigate/rose-app.conf

Now open it in a text editor and paste in the following configuration:

[command]
default=dead_reckoning.exe

[file:report.nl]
source = namelist:report_nl

[namelist:report_nl]
l_verbose=.true.

Add an App Configuration Explained

As with most app configs, this describes how to run a Unix command - in this case, a dead_reckoning.exe executable.

When the app is run, this will create our report.nl file with this inside:

&report_nl
l_verbose=.true.,
/

It will then invoke dead_reckoning.exe.

Tidy App Configuration

To ensure that the file is stored in the normal format, issue the command:

rose config-dump

This prettifies the format by handling whitespace and sorting in a consistent way. For example, the extra spaces around the = in the line source = namelist:report_nl will be removed.

Add a Build Application

We need to add an fcm_make application, in order to compile src/dead_reckoning.f90 into an executable that our navigate app can call.

Rose does some under-the-hood magic by default for apps whose name begins with fcm_make. This means that we don't need to instruct Rose how to invoke fcm make explicitly, so our app config will be minimal.

Add a Build Application - directory

Create a directory fcm_make_navigate/ in the app/ directory:

mkdir app/fcm_make_navigate

Create a rose-app.conf file in the new directory:

echo 'mode=fcm_make' >app/fcm_make_navigate/rose-app.conf

Add a Build Configuration

Unlike navigate, we don't need to put anything in the rose-app.conf configuration - Rose already knows how to run fcm.

However, we do need a configuration for fcm to read, so create a file/ directory under app/fcm_make_navigate/:

mkdir app/fcm_make_navigate/file

Add a Build Configuration Continued

Create an fcm-make.cfg file in the new directory:

touch app/fcm_make_navigate/file/fcm-make.cfg

We need to put some instructions for fcm in this file - open it in a text editor and paste in the following lines:

steps = build
build.source = $ROSE_SUITE_DIR/src/
build.target{task} = link

This is a minimal fcm configuration for building the program.

Explaining the fcm_make Magic

The Rose built-in fcm_make application knows to ask fcm make to build the executable within the share/ directory of our running suite so other tasks can use it. Our executable will be built in the share/fcm_make_navigate/build/bin/ directory.

Paths to this kind of directory will get automatically added by Rose for subsequent tasks. This means that we don't need to refer to this directory to use dead_reckoning.exe.

Suite Contents So Far

We should now have the following suite contents:

ls -R

.:
app  src  rose-suite.conf  rose-suite.info

./app:
fcm_make_navigate  navigate

./app/fcm_make_navigate:
file  rose-app.conf

(continued)

Suite Contents So Far Continued

./app/fcm_make_navigate/file:
fcm-make.cfg

./app/navigate:
rose-app.conf

./src:
dead_reckoning.f90

Suite Contents Explained

We have:

the source code, in src/dead_reckoning.f90.
the configuration for the build of the source code, app/fcm_make_navigate/.
the configuration for running the compiled source code, app/navigate/.

What Else Do We Need?

We've told Rose how to run the build via the fcm_make_navigate app config, and how to run the resultant executable via the navigate app config.

We now need to link them together and handle the rest of the inputs, via cylc.

Create a suite.rc

We need to set up our suite so that it runs our applications.

First, create the file suite.rc in the root directory of the suite (the same one as rose-suite.info and rose-suite.conf):

touch suite.rc

Now, we'll open the file in a text editor and insert some standard suite.rc content.

suite.rc Contents

#!jinja2
[cylc]
    UTC mode = True # Ignore DST
    abort if any task fails = True
[scheduling]
[runtime]
    [[root]]
        script = rose task-run --verbose
        [[[events]]]
            mail events = submission timeout, execution timeout

suite.rc Dependencies

That sets up our suite in the preferred way. Now we need to add some configuration to run our applications.

We know the following about our dependencies:

navigate should be run every 3 hours, and it should wait for the last instance to finish before it starts again.
fcm_make_navigate should be run once at the start.

Setup the suite.rc

We need a cycling suite that repeats every 3 hours, with a run once (R1) task (fcm_make_navigate) at the beginning.

suite.rc Content

Replace [scheduling] with the following lines:

[scheduling]
    initial cycle point = 20130601T00Z
    final cycle point = 20130603T00Z
    [[dependencies]]
        [[[R1]]]
            graph = fcm_make_navigate => navigate
        [[[PT3H]]]
            graph = navigate[-PT3H] => navigate

suite.rc Explanation

What we've just done:

We've set the suite to run from midnight, 1 June 2013 to midnight, 3 June 2013. This is just a model time - not anything to do with real time.
We've set the fcm_make_navigate task to run once at the initial cycle point (i.e. midnight, 1 June 2013).

suite.rc Explanation Continued

We've made navigate run every 3 hours.
We've configured the dependency between fcm_make_navigate and navigate.

clock triggering

If we were writing a real suite, we'd use the clock-trigger special tasks setting in cylc to make navigate trigger off the system wallclock time, and adjust the start/end dates to be more current. However, this doesn't matter for our example - and we don't want to wait 2 days for the suite to finish!

suite.rc Runtime

Remember we need the environment variable TIME_INTERVAL_HRS, POS_FPATH and the actual position file that POS_FPATH refers to, with latitude and longitude, for navigate to run.

We need to put this in the suite [runtime] configuration under a navigate sub section.

Runtime definitions

It's important to note that without runtime for each task, our suite will fail validation in rose suite-run. This is because rose suite-run uses strict cylc validation to make sure the dependency tasks are actually the tasks you intended to run.

You can switch this behaviour off by using the --no-strict option. In this example, we'll just put in runtime sections for each task.

suite.rc Empty Task Runtime

We need a blank runtime section for fcm_make_navigate - it doesn't need any extra configuration, beyond root.

Add this at the end of the suite.rc file:

    [[fcm_make_navigate]]

suite.rc Runtime Environment

We'll now add some runtime configuration for navigate.

Add these lines to the end of the suite.rc file:

    [[navigate]]
        [[[environment]]]
            POS_FPATH = $CYLC_SUITE_SHARE_PATH/position
            TIME_INTERVAL_HRS = 3

suite.rc Runtime Results

This sets the environment for the app navigate. POS_FPATH is using an environment variable that cylc provides, CYLC_SUITE_SHARE_PATH to define the file path to the lat/long position file.

TIME_INTERVAL_HRS is 3 because the app cycles every 3 hours.

suite.rc Special Task

We need to add something that makes our $POS_FPATH position file in the first place.

We won't bother with an app for this - we'll just write a little cylc task.

We'll add a task called write_start_position. Add it as a run-once (R1) task by replacing the graph for the [[[R1]]] section with:

        [[[R1]]]
            graph = """
                fcm_make_navigate => navigate
                write_start_position => navigate
            """

suite.rc Special Task Runtime

Finally, add these lines at the end of the suite.rc file:

    [[write_start_position]]
        script = echo '50.0 -3.0' > $CYLC_SUITE_SHARE_PATH/position

This initialises our location for the navigate app via a file (pointed to via $POS_FPATH). Our start coordinates are 50.0 north, 3.0 west. You can change these to another location if you like.

Checking the suite.rc

Your suite.rc should now look like this.

Version control

Store your changes under version control by running fcm add -c and fcm commit (or the corresponding svn commands) from the root of your suite working copy.

Summary So Far

So far we've:

analysed the Fortran program dead_reckoning.f90
created a blank suite with rosie create

Summary So Far (continued)

made an app configuration for dead_reckoning.f90, called navigate.
made a build app configuration for it, called fcm_make_navigate.
written a suite.rc file

Running Our Suite

Now our suite is ready to run.

Invoke rose suite-run:

rose suite-run

If everything has been setup successfully, cylc gui will launch with your running suite.

Finished Output

You can look at the finished output by running:

rose suite-log

The position will be written in the out file for each navigate task.

Visualising the Data

If you want a quick and easy way of visualising the output, try replacing the line:

  PRINT*, "New position, me hearties:",new_lat," ",new_long

in dead_reckoning.f90 with

  lat = (180.0/pi) * lat
  long = (180.0/pi) * long
  WRITE(*,'(A, F7.4, A, F7.4, A, F7.4, A, F7.4, A, F7.4, A, F7.4, A)') &
        "<a href='https://maps.googleapis.com/maps/api/staticmap?center=",&
        lat,",",long,"&zoom=7&size=600x300&markers=color:blue|label:A|",&
        lat,",",long,"&markers=color:red|label:B|",new_lat,",",new_long,&
        "''>Ye chart!</a>"

You'll now get a useful link in your navigate task output. Expect a wiggly course!

Note: If you are viewing the job log via Rose Bush, try switching on Tags mode by clicking on the Tags at the top right hand corner of the page.

More Cycling

Our suite has a single cycling period of 3 hours, but we could have other time definitions in the same suite. Let's add a task called take_sun_sight that runs at 12 each day. This will correct our latitude.

Adding Alternate Dependency

Add these lines below [[dependencies]] in your suite.rc file:

        [[[T12]]]
            graph = navigate => take_sun_sight

This will run after the navigate task from that cycle time finishes.

Extra Override Dependency

In order to make the navigate task wait for our new take_sun_sight task, we'll need to add some extra configuration for the 15 hour cycle - add the following lines in the same way as you did for [[[T12]]]:

        [[[T15]]]
            graph = take_sun_sight[-PT3H] => navigate

Adding a Script

It isn't necessary to put all scripts in an app or in the script in the suite.rc - you can put scripts in the bin/ directory of a suite, and they will be added to cylc's path.

Adding a Script (continued)

Change directory to the root of your suite, and create a bin/ directory:

mkdir bin

Create an empty file in the bin/ directory:

touch bin/sun_sight

Open this file with a text editor.

Editing a Script

Paste the following text into the sun_sight file:

#!/usr/bin/env python

import random
import sys


if __name__ == "__main__":
    random.seed()
    with open(sys.argv[1], "r") as f:
        (lat, long) = f.read().split()
    lat = float(lat) + random.uniform(-0.05, 0.05)
    print "Yarr! Our corrected position be {0}, {1}".format(lat, long)
    with open(sys.argv[1], "w") as f:
        f.write("{0} {1}\n".format(lat, long))

Referencing Scripts

Save the file. Make it executable by running:

chmod +x bin/sun_sight

We need to reference this script explicitly in the suite.rc for our take_sun_sight task - append these lines to the file:

    [[take_sun_sight]]
        script = sun_sight $CYLC_SUITE_SHARE_PATH/position

Results

Run the suite by invoking:

rose suite-run

Our extra task should run at 20130601T1200Z and 20130602T1200Z.

Summary

In this part we have:

added a task with a different cycling period
added a script in bin/

Next Steps

You may want to experiment with adding metadata to your navigate app, following the approach in the Metadata Development Guide.