Pursuing & supporting
reproducible workflows
for all with
Oliver Sanders & Sadie Bartholomew
Modelling Infrastructure Support
Systems team,
Met Office
This is a simple workflow
This is a repeating workflow
This is a Cylc cycling workflow
This is an infinite cycling workflow
Cylc provides a moving window on an infinite workflow
- Workflows can be infinite
- Complex workflows scale
- Advanced cycling is possible
Workflows with multiple cycle intervals
| Arbitrary (integer) cycling | |
| Real time cycling | |
| Simulated time cycling | |
| Externally driven cycling |
This is a simple workflow definition
This is a Cycling workflow definition
System architecture
Submit jobs with:
- at
- background (no batch submission system)
- loadleveler
- lsf
- moab
- pbs
- sge
- slurm
| Live monitoring via GUI or CLI | |
| Manual intervention with running workflows supported | |
| Automatically re-submit failed jobs |
Usage & the Cylc Community
International
use [
]
& development [
]
Example: weather forecasting
research
real-time
cycling
seamless
(meta-scheduling)
transition
simulated
time
cycling
operations
At the
- demonstrated scaling to:
- ˜1000s of recurrences
- ˜10,000s of tasks
- ˜100,000s of dependencies
- meta-scheduling for our HPCs:
- overall estate with 460,672 CPUs/cores
- 15,600 TFLOPs peak performance
- > 500 consistently since June
- a general upward trend (ignoring noise)
Met Office workflows are often huge!
(by numbers of tasks, dependencies, recurrences...)
fully open-source, via
Python-powered core scheduling code
ecosystem & history
of development!
Workflow design choices include:
| single monolithic workflow | modularity
|
multiple connected workflows (inter-workflow triggering) |
| coarse | task granularity
|
fine |
| via in-built parameterisation | task generation
|
via templating (Jinja2) |
|
runtime inheritance: |
hierarchy branching, levels & interconnections |
|
The Future of Cylc
8
Same concept, revised tech stack:
|
Scheduling & runtime |
2
|
⟶ |
3
|
|
Data & network layer |
|
⟶ |
|
|
Web framework |
|
⟶ |
|
|
Front-end |
|
⟶ |
|
|
Documentation |
|
⟶ |
|
- Cylc-8.0.0 "coming soon" (optimistic estimate for release ~June 2020)
- Until then, please use Cylc 7.8.X (the latest, currently 7.8.4 ) which is thoroughly tested & supported
- Install for Linux via GitHub
Cylc development is supported by:
