Rose Bunch

rose_bunch is a built-in Rose app which allows multiple variants of a command to be run under a single job.

Purpose

Often, we want to run many instances of a command that differ only slightly from each other at the same time - an example would be where a command is run repeatedly with only its arguments changing.

Rather than creating multiple apps or optional configs to change the way a command is to be run, we can instead use the built-in rose_bunch application to run multiple command variants, in parallel, under a single job as defined by an application configuration.

Note, however, that for “embarrassingly parallel” code it would be better to alter the code rather than use rose_bunch to handle this for you.

Warning

It is important to note that when running your rose_bunch app under load balancing systems such as PBS or Slurm, you will need to set resource requests to reflect the resources required by running multiple commands at once.

For example, if a single command would require 1GB memory and the app is configured to run up to 4 commands at once then 4GB of memory should be requested.

Example

In this example we are going to create a suite that simulates the handling of landing planes at an airport. For a given plane the process of landing and unloading is the same: land, taxi to the terminal, unload passengers and get clear. We can refer to this as the “landing” routine. What differs between landings is the plane type, number of passengers carried and the resulting timings for each stage of the landing process.

Create a new Rose suite configuration:

mkdir -p ~/cylc-src/rose-bunch
cd ~/cylc-src/rose-bunch

Create a blank rose-suite.conf and a flow.cylc file that looks like this:

[scheduler]
    UTC mode = True # Ignore DST
[scheduling]
    [[graph]]
        R1 = lander
[runtime]
    [[root]]
        script = rose task-run
    [[lander]]

In the source directory create an app/ directory:

mkdir app

In the app directory create a lander/ directory:

cd app
mkdir lander

In the app/lander/ directory create a rose-app.conf file using your editor of choice and paste the following lines into it:

mode=rose_bunch

[bunch]
command-format=land %(class)s %(passengers)s

[bunch-args]
class=airbus concorde airbus cessna
passengers=40 20 30 2

This configuration will run a rose_bunch task that calls multiple instances of the land command, supplying arguments to each instance from the class and passengers entries under rose_bunch[bunch-args].

In the app/lander/ directory create a bin/ directory:

mkdir bin

Using your editor of choice, create a file named land under the bin directory and paste in these lines:

#!/usr/bin/env bash

CLASS=$1
PASSENGERS=$2

# Get settings
case $CLASS in
    airbus) LANDTIME=30; UNLOADRATE=8;;
    cessna) LANDTIME=20; UNLOADRATE=2;;
    concorde) LANDTIME=10; UNLOADRATE=4;;
esac

echo "[ $(rose date) ] $CLASS carrying $PASSENGERS passengers incoming"

# Land plane
echo "[ $(rose date) ] Approaching runway"
sleep $LANDTIME
echo "[ $(rose date) ] On the tarmac"

# Unload passengers
sleep $(($PASSENGERS / $UNLOADRATE))
echo "[ $(rose date) ] Unloaded"

# Clear terminal
sleep 10
echo "[ $(rose date) ] Clear of terminal"

This script captures the landing routine and expects two arguments: the plane type (its class) and the number of passengers it is carrying.

Finally, make the new land file executable by navigating into the bin directory of the lander app and running:

chmod +x land

Navigate to the top directory of your source workflow (where the flow.cylc and rose-suite.conf files can be found) and validate, install and run the workflow:

cylc validate .
cylc install
cylc play rose-bunch

Once the workflow has finished running and has shutdown, open Cylc Review to view its output (note that you can close the Cylc GUI at this point):

cylc review

Note

You can quickly view the workflow log by running cylc cat-log rose-bunch .

In the Cylc Review jobs page for your workflow you should be presented with a page containing a single row for the lander task, from which you can access its output. In that row you should see something like this:

Cylc Review view of output

In the Cylc Review entry you should see that the usual links are present for the task such as job.out, job.status etc. with the addition of two drop-down boxes: one for bunch.*.err and one for bunch.*.out. Rather than mixing the outputs from the multiple command invocations being run at once, rose_bunch directs their output to individual output files. So, for example, the output from running the command with the first set of parameters can be found in the bunch.0.out file, the second set in the bunch.1.out file etc. Examine these output files now to confirm that all four of the args combinations have been run and produced output.

Naming Invocations

While the different invocations of the command have their own output directed to indexed files, it can sometimes be difficult to quickly identify which file to look in for output. To aid this, rose_bunch supports naming command instances via the rose_bunch[bunch]names= option.

Returning to our source directory:

cd ~/cylc-src/rose-bunch

Open your app config (under app/lander/rose-app.conf) and add the following line under the rose_bunch[bunch] section:

names=BA123 Emirates345 BA007 PC456

We can now install and play this updated suite:

cylc validate ~/cylc-src/rose-bunch
cylc install rose-bunch
cylc play rose-bunch

Inspecting the job log directory for this latest run on Cylc Review, the bunch.*.err and bunch.*.out files now have the names you have configured rather than the previous bunch.0.out ... bunch.3.out.

Limiting Concurrent Invocations

In some situations we may need to limit the number of concurrently running command invocations - often as a result of resource limitations. Rather than batching up jobs into sets of N simultaneously running commands, rose_bunch apps can be configured to run as many commands as possible within some limit i.e. while N commands are running, if one of them finishes, don’t wait for the remaining N-1 jobs to finish before running the (N+1)th one.

In the case of our simulated airport we will pretend we only have two runways available at a time on which our planes can land. As such we need to limit the number of planes landing. We do this using the rose_bunch[bunch]pool-size= configuration option of the rose_bunch app:

cd ~/cylc-src/rose-bunch

Open your app config (under app/lander/rose-app.conf) and add the following line to the rose_bunch[bunch] section:

pool-size=2

We can now install and play this updated suite:

cylc validate ~/cylc-src/rose-bunch
cylc install rose-bunch
cylc play rose-bunch

Notice that this time round it takes longer for the task to run as it has been limited in the number of command variants it can run simultaneously.

As an example, when the BA007 invocation starts running you should see the line:

[INFO] BA007: added to pool

appear in the job output after a while whereas, when running without a rose_bunch[bunch]pool-size, the line will appear pretty quickly.

Summary

In this tutorial we have learnt how to configure a rose_bunch app to run a set of command variants under one job. We have learnt how to name the individual variants for convenience in examining the logs and how to limit the number of concurrently running commands.

Further options are listed in the rose_bunch documentation. These include configuring how to proceed following failure of an individual command invocation (rose_bunch[bunch]fail-mode=), automatically generating N command instances and enabling/disabling the app’s incremental mode.