In this section we will cover writing basic workflows in cylc.
suite.rc File Format¶
We refer to a Cylc workflow as a Cylc suite. A Cylc suite is a
directory containing a
suite.rc file. This configuration file is where
we define our workflow. The
suite.rc file uses a nested INI-based
- Comments start with a
- Settings are written as
key = valuepairs.
- Settings can be contained within sections.
- Sections are written inside square brackets i.e.
- Sections can be nested, by adding an extra square bracket with each level,
so a sub-section would be written
[[sub-section]], a sub-sub-section
[[[sub-sub-section]]], and so on.
# Comment [section] key = value [[sub-section]] another-key = another-value # Inline comment yet-another-key = """ A Multi-line String """
Throughout this tutorial we will refer to settings in the following format:
[section]- refers to the entire section.
[section]key- refers to a setting within the section.
[section]key=value- expresses the value of the setting.
[section][sub-section]another-key. Note we only use one set of square brackets with nested sections.
It is advisable to indent
suite.rc files. This indentation, however,
is ignored when the file is parsed so settings must appear before
[section] key = value # This setting belongs to the section. [[sub-section]] key = value # This setting belongs to the sub-section. # This setting belongs to the sub-section as indentation is ignored. # Always write settings before defining any sub-sections! key = value
suite.rc file format duplicate sections are additive, that is
to say the following two examples are equivalent:
[a] c = C [b] d = D [a] e = E
[a] c = C e = E [b] d = D
Settings, however, are not additive meaning that a duplicate setting will override an earlier value. The following two examples are also equivalent:
a = foo a = bar
a = bar
Task are represented as words and dependencies as arrows (
=>), so the
following text defines two tasks where
make_dough is dependent on
purchase_ingredients => make_dough
In a Cylc workflow this would mean that
make_dough would only run when
purchase_ingredients has succeeded. These dependencies can be chained together:
purchase_ingredients => make_dough => bake_bread => sell_bread
This line of text is referred to as a graph string. These graph strings can be combined to form more complex workflows:
purchase_ingredients => make_dough => bake_bread => sell_bread pre_heat_oven => bake_bread bake_bread => clean_oven
Graph strings can also contain “and” (
&) and “or” (
|) operators, for
instance the following lines are equivalent to the ones just above:
purchase_ingredients => make_dough pre_heat_oven & make_dough => bake_bread => sell_bread & clean_oven
The order in which lines appear in the graph section doesn’t matter, for instance the following examples are the same as each other:
foo => bar bar => baz
bar => baz foo => bar
[scheduling] [[dependencies]] graph = """ purchase_ingredients => make_dough pre_heat_oven & make_dough => bake_bread => sell_bread & clean_oven """
This is a minimal Cylc suite, in which we have defined a graph representing a workflow for Cylc to run. We have not yet provided Cylc with the scripts or binaries to run for each task. This will be covered later in the runtime tutorial.
Cylc provides a GUI for visualising graphs. It is run on the
command line using the
cylc graph <path> command where the path
is to the
suite.rc file you wish to visualise.
cylc graph will display a diagram similar to the ones you have
seen so far. The number
1 which appears below each task is the
cycle point. We will explain what this means in the next section.
A graph can be drawn in multiple ways, for instance the following two examples are equivalent:
The graph drawn by
cylc graph may vary slightly from one run to another
but the tasks and dependencies will always be the same.
In this practical we will create a new Cylc suite and write a graph for it to use.
Create a Cylc suite.
A Cylc suite is just a directory containing a
If you don’t have one already, create a
cylc-rundirectory in your user space i.e:
Within this directory create a new folder called
graph-introduction, which is to be our suite directory. Move into it:
mkdir ~/cylc-run/graph-introduction cd ~/cylc-run/graph-introduction
Inside this directory create a
suite.rcfile and paste in the following text:
[scheduling] [[dependencies]] graph = """ # Write graph strings here! """
Write a graph.
We now have a blank Cylc suite, next we need to define a workflow.
suite.rcfile to add graph strings representing the following graph:
cylc graphto visualise the workflow.
Once you have written some graph strings try using
cylc graphto display the workflow. Run the following command:
cylc graph .
cylc graphtakes the path to the suite as an argument. As we are inside the suite directory we can run
cylc graph ..
If the results don’t match the diagram above try going back to the suite.rc file and making changes.
In the top right-hand corner of the
cylc graphwindow there is a refresh button which will reload the GUI with any changes you have made.
There are multiple correct ways to write this graph. So long as what you see in
cylc graphmatches the above diagram then you have a correct solution.
Two valid examples:
foo & pub => bar => baz & wop baz => qux
foo => bar => baz => qux pub => bar => wop
The whole suite should look something like this:
[scheduling] [[dependencies]] graph = """ foo & pub => bar => baz & wop baz => qux """