What does this cover?

What does this part cover?

• suite design
• cylc capabilities
• cylc gui

What does this part cover? continued

• Rose Bush, and other Rose utilities
• Built-in applications - fcm_make, rose_ana, rose_prune...
• In Depth Usage

Dependency Syntax a | b

If task c depends on either task a or task b:

[scheduling]
    [[dependencies]]
        graph = a | b => c

If both task b and task c depend on task a:

[scheduling]
    [[dependencies]]
        graph = a => b & c

Dependency Syntax b & c

If task b only runs when task a fails:

[scheduling]
    [[dependencies]]
        graph = a:fail => b

If task b runs after task a starts:

[scheduling]
    [[dependencies]]
        graph = a:start => b

Dependency Syntax a[-PT6H]

In a cycling suite, with cycles every 6 hours, if task a depends on a previous cycle's task a succeeding:

[scheduling]
    [[dependencies]]
        [[[T00, T06, T12, T18]]]
            graph = a[-PT6H] => a

If task b depends on the previous cycle's task a failing:

[scheduling]
    [[dependencies]]
        [[[T00, T06, T12, T18]]]
            graph = a[-PT6H]:fail => b

Dependency Syntax a & ( b | c )

If task d depends on task a and either task b or task c succeeding:

[scheduling]
    [[dependencies]]
        graph = a & ( b | c ) => d

Dependency Syntax a => !b

If task b is not needed when task a succeeds, it can be suicide triggered:

[scheduling]
    [[dependencies]]
        graph = a => !b

Typically, b would be a failure recovery task that should never run if a succeeds, but should be run after a fails:

[scheduling]
    [[dependencies]]
        graph = """
            a => !b
            a:fail => b
        """

Dependency Syntax a => !b (2)

See the suicide triggering tutorial for more details.

Dependency Syntax a => FAMILY_B

If all the tasks in FAMILY_B should run when a succeeds:

[scheduling]
    [[dependencies]]
        graph = a => FAMILY_B

Dependency Syntax FAMILY_A:succeed-any => b

If task b should run when any task in the FAMILY_A family succeeds:

[scheduling]
    [[dependencies]]
        graph = FAMILY_A:succeed-any => b

If task b should run when all tasks in the FAMILY_A family have finished (whether succeeded or failed):

[scheduling]
    [[dependencies]]
        graph = FAMILY_A:finish-all => b

Dependency Reference

Full details of dependency configuration are in the cylc User Guide, including:

• more namespace (family) triggers - triggers based on groups
• clock-trigger tasks - tasks triggered at certain wallclock times
• max active cycle points - prevent too many cycles being active at any one time
• message triggers - triggers based on task output

Cylc Capabilities

cylc has some nice capabilities:

• inserting tasks at runtime
• limiting the number of tasks in a family running at any one time (queues)
• giving manual control over the status of tasks

Cylc Capabilities (Continued)

• automatically resubmitting tasks that have failed, optionally with different environments or commands (retries)
• automatically resubmitting tasks when submission fails (submission retries)
• restarting suites

Cylc Capabilities (Continued 2)

• broadcasting runtime changes to a particular task or cycle within a running suite (cylc broadcast)
• reload the suite.rc settings for a running suite

Environment

There are some useful environment variables that you can use in your suite.rc file and Rose applications:

• ROSE_SUITE_DIR, root directory of the running suite.
• ROSE_TASK_NAME, name of the suite task.
• ROSE_TASK_CYCLE_TIME, cycle time of the suite task.

Environment (2)

• ROSE_DATA, data directory of the suite.
• ROSE_DATAC, data directory for tasks in this cycle time.
• ROSE_ORIG_HOST, host that invoked 'rose suite-run'.

Environment (3)

• ROSE_TASK_LOG_ROOT, the path where log files for a task are stored without the file extension - if you create a file in this directory it will be associated with the task in the suite log viewer.

Environment Reference

For more, see rose task-env and the cylc User Guide. Note: to reference Rose environment variables in your suite.rc file, you'll need to use rose task-env in the env-script runtime setting (tutorial here).

cylc-run Directories

cylc keeps a runtime directory for each suite in ~/cylc-run/, which is updated when you run the suite.

It is a mirror of your suite directory structure and content, but with some added cylc directories and symbolic links (e.g. share/). These hold things like installed files, current task statuses, and task output.

cylc-run directories: roses vs cylc-run

cylc-run directories: share/, share/cycle/, work/

share/, share/cycle/ and work/ sub-directories:

• Usually symbolic links to data directories.
• share/ is intended for any files that need to be shared between tasks during the lifetime of the suite.
• share/cycle/ is intended for any files that need to be shared are regularly generated, archived and removed by cycling tasks.
• work/ contains task working directories, e.g. containing Rose application namelists and other temporary files

cylc-run directories: log/

log/, log.*/ sub-directories:

• The suite subdirectory contains the suite logs, including STDOUT and STDERR outputs.
• The job/ subdirectory contains the job scripts for submitting the tasks, their STDOUT and STDERR outputs.
• Rose adds the rose-suite.version file, which records any version control information about the suite, including any local modifications.

Suite Design

See Cylc Rose Suite Design Best Practice Guide.

Good Practice

• Having read the cylc and Rose documentation before writing the suite
• Using families where appropriate
• Condensing repeated settings or blocks with Jinja-2

Good Practice (continued)

• Single, well-commented suite.rc file
• Rerunnable tasks, doing one thing and doing it well
• Using Rose standard environments, utilities, standard suite templates, and the recommended event handlers.

Bad practice

• Skipping the cylc and Rose documentation
• Repeated runtime settings
• Obfuscating the suite.rc with Jinja2

Bad practice (continued)

• Overusing include files
• Magical, black box tasks with suite-level functionality
• Reinventing the (Rose) wheel

cylc gui

• Usually launched by rose suite-run or using the commands rose suite-gcontrol (rose sgc) or cylc gui SUITE-NAME.
• Three views - graph, dot (square now!), tree.
• Easy interface to start, pause (hold) and stop the suite.
• View task states, and job.out, job.err, etc files for task jobs.
• Poll and kill submitted and running task jobs.
• Manually trigger tasks and reset task states.

cylc gscan

• Monitor all your running suites on a set of suite hosts from a single GUI.
• Launch cylc gui for individual suites.

Rose Bush

Browse suite logs via HTTP.

View suite status on your web browser. Can be launched using rose suite-log.

Rose Built-in Applications

• fcm_make - built-in application to run fcm make
• rose_ana - an engine to perform analysis on program outputs
• rose_arch - archive output
• rose_prune - housekeep a cycling suite
• rose_bunch - run multiple command variants in parallel

Suite Utilities

• rose date - flexible way to manipulate date/time strings with or without an offset.
• rose host-select - configurable remote hosts selector

Tidying Up Runtime Directories

Run rose suite-clean SUITE_NAME [...] to delete suite output in the ~/cylc-run/SUITE_NAME/ directories. The command removes the actual locations of share/, share/cycle/ and work/ (which may be symbolic links), as well as the ~/cylc-run/SUITE_NAME/ directory structure in each job host of the latest run.

Tidying Up Rosie Suites

Run rosie delete SUITE_NAME to delete SUITE_NAME from the HEAD of its Rosie Subversion repository as well as the ~/roses/SUITE_NAME/ directory.

If you only want to remove the ~/roses/SUITE_NAME/ directory, you can run rm -rf ~/roses/SUITE_NAME or rosie delete --local-only SUITE_NAME.

Independent Learning

Next Steps:

• If you haven't already done so, go through the Suite Writing Tutorial.
• Have a look at some of the Advanced Suite Tutorials.
• Have a go at the Rose Quiz