In this section we will look at how to write cycling (repeating) workflows.
To make a workflow repeat we must tell Cylc three things:
- The recurrence
- How often we want the workflow to repeat.
- The initial cycle point
- At what cycle point we want to start the workflow.
- The final cycle point
- Optionally we can also tell Cylc what cycle point we want to stop the workflow.
Let’s take the bakery example from the previous section. Bread is produced in batches so the bakery will repeat this workflow for each batch of bread they bake. We can make this workflow repeat with the addition of three lines:
[scheduling] + cycling mode = integer + initial cycle point = 1 [[dependencies]] + [[[P1]]] graph = """ purchase_ingredients => make_dough pre_heat_oven & make_dough => bake_bread => sell_bread & clean_oven """
cycling mode = integersetting tells Cylc that we want our cycle points to be numbered.
initial cycle point = 1setting tells Cylc to start counting from 1.
P1is the recurrence. The graph within the
[[[P1]]]section will be repeated at each cycle point.
The first three cycles would look like this, with the entire workflow repeated at each cycle point:
Note the numbers under each task which represent the cycle point each task is in.
We’ve just seen how to write a workflow that repeats every cycle.
Cylc runs tasks as soon as their dependencies are met so cycles are not necessarily run in order. This could cause problems, for instance we could find ourselves pre-heating the oven in one cycle whist we are still cleaning it in another.
To resolve this we must add dependencies between the
cycles. We do this by adding lines to the graph. Tasks in the
previous cycle can be referred to by suffixing their name with
for example. So to ensure the
clean_oven task has been completed before
the start of the
pre_heat_oven task in the next cycle, we would write
the following dependency:
clean_oven[-P1] => pre_heat_oven
This dependency can be added to the suite by adding it to the other graph lines:
[scheduling] cycling mode = integer initial cycle point = 1 [[dependencies]] [[[P1]]] graph = """ purchase_ingredients => make_dough pre_heat_oven & make_dough => bake_bread => sell_bread & clean_oven + clean_oven[-P1] => pre_heat_oven """
The resulting suite would look like this:
Adding this dependency “strings together” the cycles, forcing them to run in order. We refer to dependencies between cycles as inter-cycle dependencies.
In the dependency the
[-P1] suffix tells Cylc that we are referring to a
task in the previous cycle. Equally
[-P2] would refer to a task two
Note that the
purchase_ingredients task has no arrows pointing at it
meaning that it has no dependencies. Consequently the
tasks will all run straight away. This could cause our bakery to run into
cash-flow problems as they would be purchasing ingredients well in advance
of using them.
To solve this, but still make sure that they never run out of ingredients, the bakery wants to purchase ingredients two batches ahead. This can be achieved by adding the following dependency:
[scheduling] cycling mode = integer initial cycle point = 1 [[dependencies]] [[[P1]]] graph = """ purchase_ingredients => make_dough pre_heat_oven & make_dough => bake_bread => sell_bread & clean_oven clean_oven[-P1] => pre_heat_oven + sell_bread[-P2] => purchase_ingredients """
This dependency means that the
purchase_ingredients task will run after
sell_bread task two cycles before.
[-P2] suffix is used to reference a task two cycles before. For the
first two cycles this doesn’t make sense as there was no cycle two cycles
before, so this dependency will be ignored.
Any inter-cycle dependencies stretching back to before the initial cycle point will be ignored.
In the previous examples we made the workflow repeat by placing the graph
[[[P1]]] section. Here
P1 is a recurrence meaning
repeat every cycle, where
P1 means every cycle,
P2 means every
other cycle, and so on. To build more complex workflows we can use multiple
[scheduling] cycling mode = integer initial cycle point = 1 [[dependencies]] [[[P1]]] # Repeat every cycle. graph = foo [[[P2]]] # Repeat every second cycle. graph = bar [[[P3]]] # Repeat every third cycle. graph = baz
By default recurrences start at the initial cycle point, however it
is possible to make them start at an arbitrary cycle point. This is done by
writing the cycle point and the recurrence separated by a forward slash
5/P3 means repeat every third cycle starting from cycle
The start point of a recurrence can also be defined as an offset from the
initial cycle point, e.g.
+P5/P3 means repeat every third cycle
starting 5 cycles after the initial cycle point.
If you have not completed the previous practical use the following code for
[scheduling] [[dependencies]] graph = """ foo & pub => bar => baz & wop baz => qux """
Create a new suite.
~/cylc-run/directory create a new (sub-)directory called
integer-cyclingand move into it:
mkdir ~/cylc-run/integer-cycling cd ~/cylc-run/integer-cycling
Copy the above code into a
suite.rcfile in that directory.
Make the suite cycle.
Add in the following lines.
[scheduling] + cycling mode = integer + initial cycle point = 1 [[dependencies]] + [[[P1]]] graph = """ foo & pub => bar => baz & wop baz => qux """
Visualise the suite.
Try visualising the suite using
cylc graph .
You can get Cylc graph to draw dotted boxes around the cycles by clicking the “Organise by cycle point” button on the toolbar:
cylc graphdisplays the first three cycles of a suite. You can tell
cylc graphto visualise the cycles between two points by providing them as arguments, for instance the following example would show all cycles between
cylc graph . 1 5 &
Add another recurrence.
Suppose we wanted the
quxtask to run every other cycle as opposed to every cycle. We can do this by adding another recurrence.
Make the following changes to your
[scheduling] cycling mode = integer initial cycle point = 1 [[dependencies]] [[[P1]]] graph = """ foo & pub => bar => baz & wop - baz => qux """ + [[[P2]]] + graph = """ + baz => qux + """
cylc graphto see the effect this has on the workflow.
Next we need to add some inter-cycle dependencies. We are going to add three inter-cycle dependencies:
wopfrom the previous cycle and
bazfrom the previous cycle and
fooevery odd cycle.
quxfrom the previous cycle and
fooevery even cycle.
Have a go at adding inter-cycle dependencies to your
suite.rcfile to make your workflow match the diagram below.
P2means every odd cycle.
2/P2means every even cycle.
[scheduling] cycling mode = integer initial cycle point = 1 [[dependencies]] [[[P1]]] graph = """ foo & pub => bar => baz & wop wop[-P1] => pub # (1) """ [[[P2]]] graph = """ baz => qux baz[-P1] => foo # (2) """ [[[2/P2]]] graph = """ qux[-P1] => foo # (3) """