Suicide triggers allow us to remove a task from the suite’s graph whilst the suite is running.
The main use of suicide triggers is for handling failures in the workflow.
Imagine a bakery which has a workflow that involves making cake.
make_cake_mixture => bake_cake => sell_cake
There is a 50% chance that the cake will turn out fine, and a 50% chance that it will get burnt. In the case that we burn the cake the workflow gets stuck.
In this event the
sell_cake task will be unable to run as it depends on
bake_cake. We would say that this suite has stalled.
When Cylc detects that a suite has stalled it sends you an email to let you
know that the suite has got stuck and requires human intervention to proceed.
In order to prevent the suite from entering a stalled state we need to handle
the failure of the
At the bakery if they burn a cake they eat it and make another.
The following diagram outlines this workflow with its two possible pathways,
:succeed in the event that
bake_cake is successful and
We can add this logic to our workflow using the
bake_cake => sell_cake bake_cake:fail => eat_cake
If you don’t specify a qualifier Cylc assumes you mean
:succeed so the
following two lines are equivalent:
foo => bar foo:succeed => bar
Why Do We Need To Remove Tasks From The Graph?¶
Create a new suite called
mkdir -p ~/cylc-run/suicide-triggers cd ~/cylc-run/suicide-triggers
Paste the following code into the
[scheduling] cycling mode = integer initial cycle point = 1 [[dependencies]] [[[P1]]] graph = """ make_cake_mixture => bake_cake => sell_cake bake_cake:fail => eat_cake """ [runtime] [[root]] script = sleep 2 [[bake_cake]] # Random outcome 50% chance of success 50% chance of failure. script = sleep 2; if (( $RANDOM % 2 )); then true; else false; fi
cylc gui and run the suite:
cylc gui suicide-triggers & cylc run suicide-triggers
The suite will run for three cycles then get stuck. You should see something
similar to the diagram below. As the
bake_cake task fails randomly what
you see might differ slightly. You may receive a “suite stalled” email.
The reason the suite stalls is that, by default, Cylc will run a maximum of three cycles concurrently. As each cycle has at least one task which hasn’t either succeeded or failed Cylc cannot move onto the next cycle.
For more information search
max active cycle points in the
Cylc User Guide.
You will also notice that some of the tasks (e.g.
eat_cake in cycle
in the above example) are drawn in a faded gray. This is because these tasks
have not yet been run in earlier cycles and as such cannot run.
Removing Tasks From The Graph¶
In order to get around these problems and prevent the suite from stalling we must remove the tasks that are no longer needed. We do this using suicide triggers.
A suicide trigger is written like a normal dependency but with an exclamation mark in-front of the task on the right-hand-side of the dependency meaning “remove the following task from the graph at the current cycle point.”
For example the following graph string would remove the task
from the graph if the task
foo were to succeed.
foo => ! bar
There are three cases where we would need to remove a task in the cake-making example:
bake_caketask succeeds we don’t need the
eat_caketask so should remove it.
bake_cake => ! eat_cake
bake_caketask fails we don’t need the
sell_caketask so should remove it.
bake_cake:fail => ! sell_cake
bake_caketask fails then we will need to remove it else the suite will stall. We can do this after the
eat_caketask has succeeded.
eat_cake => ! bake_cake
Add the following three lines to the suite’s graph:
bake_cake => ! eat_cake bake_cake:fail => ! sell_cake eat_cake => ! bake_cake
We can view suicide triggers in
cylc graph by un-selecting the
Ignore Suicide Triggers button in the toolbar. Suicide triggers
will then appear as dashed lines with circular endings. You should see
something like this:
If we wanted to make the cycles run in order we might write an inter-cycle dependency like this:
sell_cake[-P1] => make_cake_mixture
In order to handle the event that the
sell_cake task has been removed from
the graph by a suicide trigger we can write our dependency with an or
| like so:
eat_cake[-P1] | sell_cake[-P1] => make_cake_mixture
make_cake_mixture task from the next cycle will run after whichever
eat_cake tasks is run.
Add the following graph string to your suite.
eat_cake[-P1] | sell_cake[-P1] => make_cake_mixture
cylc gui and run the suite. You should see that if the
bake_cake task fails both it and the
sell_cake task disappear and
are replaced by the
Comparing “Regular” and “Suicide” Triggers¶
In Cylc “regular” and “suicide” triggers both work in the same way. For example
the following graph lines implicitly combine using an
foo => pub bar => pub
foo & bar => pub
Suicide triggers combine in the same way:
foo => !pub bar => !pub
foo & bar => !pub
This means that suicide triggers are treated as “invisible tasks” rather than as “events”. Suicide triggers can have pre-requisites just like a normal task.
The following sections outline examples of how to use suicide triggers.
A common use case where a
recover task is used to handle a task failure.
[scheduling] [[dependencies]] graph = """ # Regular graph. foo => bar # The fail case. bar:fail => recover # Remove the "recover" task in the success case. bar => ! recover # Remove the "bar" task in the fail case. recover => ! bar # Downstream dependencies. bar | recover => baz """ [runtime] [[root]] script = sleep 1 [[bar]] script = false
A workflow where sub-graphs of tasks are to be run in the success and or fail cases.
[scheduling] [[dependencies]] graph = """ # Regular graph. foo => bar # Success case. bar => tar & jar # Fail case. bar:fail => baz => jaz # Remove tasks from the fail branch in the success case. bar => ! baz & ! jaz # Remove tasks from the success branch in the fail case. bar:fail => ! tar & ! jar & ! par # Remove the bar task in the fail case. baz => ! bar # Downstream dependencies. tar | jaz => pub """ [runtime] [[root]] script = sleep 1 [[bar]] script = true
Triggering Based On Other States¶
In these examples we have been using suicide triggers to handle task failure. The suicide trigger mechanism works with other qualifiers as well for example:
foo:start => ! bar
Suicide triggers can also be used with custom outputs. In the following example
showdown produces one of three possible custom outputs,
[scheduling] [[dependencies]] graph = """ # The "regular" dependencies showdown:good => good showdown:bad => bad showdown:ugly => ugly good | bad | ugly => fin # The "suicide" dependencies for each case showdown:good | showdown:bad => ! ugly showdown:bad | showdown:ugly => ! good showdown:ugly | showdown:good => ! bad """ [runtime] [[root]] script = sleep 1 [[showdown]] # Randomly return one of the three custom outputs. script = """ SEED=$RANDOM if ! (( $SEED % 3 )); then cylc message 'The-Good' elif ! (( ( $SEED + 1 ) % 3 )); then cylc message 'The-Bad' else cylc message 'The-Ugly' fi """ [[[outputs]]] # Register the three custom outputs with cylc. good = 'The-Good' bad = 'The-Bad' ugly = 'The-Ugly'
An example of a workflow where there are no tasks which are dependent on the task to suicide trigger.
It is possible for a task to suicide trigger itself e.g:
foo:fail => ! foo
This is usually not recommended but in the case where there are no tasks dependent on the one to remove it is an acceptable approach.
[scheduling] [[dependencies]] graph = """ foo => bar => baz bar => pub # Remove the "pub" task in the event of failure. pub:fail => ! pub """ [runtime] [[root]] script = sleep 1 [[pub]] script = false