Polling
Polling allows you to check for some condition to be met prior to running the main command in an app without the need for additional entries in the dependencies graph.
For example, you might want to run a polling command to check for the existence of a particular file before running the main command which requires said file.
Example
Create a new Rose suite configuration:
mkdir -p ~/cylc-src/polling
cd ~/cylc-src/polling
Create a blank rose-suite.conf
and a flow.cylc
file that looks like this:
[scheduler]
UTC mode = True # Ignore DST
[scheduling]
[[graph]]
R1 = """
compose_letter => send_letter
bob => read_letter
"""
This is a simple workflow which consists of the following:
A
compose_letter
task.A
send_letter
task which is run once the letter is composed.A
bob
task which we will be using to poll with.A
read_letter
task which will run once the polling task is complete.
It will need some runtime. Add the following to your flow.cylc
file:
[runtime]
[[root]]
script = sleep 10
[[compose_letter]]
script = sleep 5; echo 'writing a letter to Bob...'
[[send_letter]]
env-script = eval `rose task-env`
script = """
sleep 5
echo 'Hello Bob' > $ROSE_DATA/letter.txt
sleep 10
"""
[[bob]]
script = rose task-run
[[read_letter]]
env-script = eval `rose task-env`
script = sleep 5; cat $ROSE_DATA/letter.txt
post-script = rm $ROSE_DATA/letter.txt
Adding Polling
In the source directory create an app
directory.
In the app
directory create a directory called bob
.
In the newly-created bob
directory, create a rose-app.conf
file.
Edit the rose-app.conf
file to look like this:
[poll]
delays=10*PT5S
test=test -e $ROSE_DATA/letter.txt
[command]
default=echo 'Ooh, a letter!'
We now have an app that does the following:
Has a polling
test
that checks for the existence of a file.Polls up to 10 times with 5 second delays between each attempt.
Prints a message once the polling test succeeds.
Note
The ordering of the [poll]
and [command]
sections is not important.
In practice, it may be preferable to have the [command]
section at
the top as that should contain the main command(s) being run by the app.
Save your changes and install and run the workflow using cylc install and cylc play:
cylc validate .
cylc install
cylc play polling
Notice that bob
finishes and triggers read_letter
before
send_letter
has completed. This is because the polling condition has
been met, allowing the main command in bob
to be run.
Improving The Polling
At present we have specified our own routine for testing for the existence
of a particular file using the test
option. However, Rose provides a
simpler method for doing this.
Edit the rose-app.conf
in your bob
app to look like the
following:
[poll]
delays=10*PT5S
all-files=$ROSE_DATA/letter.txt
[command]
default=echo 'Ooh, a letter!'
Polling is now making use of the all-files
option, which allows you to
specify a list of files to check the existence of. Save your changes, install
and run the workflow to confirm it still works.
Available Polling Types
Test and all-files are just two of the available polling options:
all-files
Tests if all of the files in a list exist.
any-files
Tests if any of the files in a list exist.
file-test
Changes the test used to evaluate the
any-files
andall-files
lists to a shell script to be run on each file (e.g.grep
). Passes if the command exits with a zero return code.test
Tests using a shell script, passes if the command exits with a zero return code. Note this is separate from the
all-files
,any-files
testing logic.
Tip
For more details see Application Configuration.
Possible Uses For Polling
Depending on your needs, possible uses for polling might include:
Checking for required output from a long-running task rather than waiting for the task to complete.
Monitoring output from another suite.
Checking if a file has required content before using it.