Rose Arch
rose_arch
is a built-in Rose app that provides a generic
solution to the archiving of suite files.
Good Practice
Only archive the minimum files needed at each cycle of your suite. Run the archiving task before any housekeeping in the graph.
Example
Create a new Rose suite configuration:
mkdir -p ~/cylc-src/rose-arch-tutorial
Create a blank rose-suite.conf
and a flow.cylc
file that looks like this:
[scheduler]
UTC mode = True # Ignore DST
[[events]]
abort on workflow timeout = True
workflow timeout = PT1H
[scheduling]
[[graph]]
R1 = make_files => archive_files_rsync => archive_files_scp
[runtime]
[[root]]
env-script = eval $(rose task-env)
script = rose task-run
[[make_files]]
script = """
echo 'zip' >> $ROSE_DATAC/file_zip
echo 'solo' >> $ROSE_DATAC/file_solo
echo 'list1' >> $ROSE_DATA/file_list1
echo 'list2' >> $ROSE_DATA/file_list2
echo 'list3' >> $ROSE_DATA/file_list3
mkdir -p $ROSE_DATA/ARCHIVING || true
mkdir -p $ROSE_DATA/ARCHIVING/rename || true
"""
[[archive_files_rsync]]
[[archive_files_scp]]
In the source directory create an app/
directory:
mkdir app
In the app directory create an archive_files_rsync/
directory:
cd app
mkdir archive_files_rsync
In the app/archive_files_rsync/
directory create a
rose-app.conf
file. This example uses vi
, but please use your
editor of choice:
cd archive_files_rsync
vi rose-app.conf
Paste in the following lines:
mode=rose_arch
[env]
ARCH_TARGET=$ROSE_DATA/ARCHIVING
[arch]
command-format=rsync -a %(sources)s %(target)s
source-prefix=$ROSE_DATAC/
target-prefix=$ARCH_TARGET/
update-check=mtime+size
[arch:solo.file]
source=file_solo
[arch:files]
source=file_list1 file_list3
source-prefix=$ROSE_DATA/
[arch:dir]
source=file*
source-prefix=$ROSE_DATA/
[arch:file_zipped.tar]
source=file_zip
Move to the app/
directory:
cd ..
ls
The following should be returned:
archive_files_rsync
Create an archive_files_scp/
directory:
mkdir archive_files_scp
In the archive_files_scp/
directory create a rose-app.conf
file. This example uses vi
, but please use your editor of choice:
cd archive_files_scp
vi rose-app.conf
Paste in the following lines:
mode=rose_arch
[env]
ARCH_TARGET=$ROSE_DATA/ARCHIVING
[arch]
command-format=scp %(sources)s %(target)s
source-prefix=$ROSE_DATA/
target-prefix=$ARCH_TARGET/
update-check=mtime+size
[arch:rename/]
rename-format=%(cycle)s_%(tag)s_%(name)s
rename-parser=^.*list(?P<tag>.*)$
source=file_list?
Description
You have now created a suite that defines three tasks:
make_files
Sets up the files and
ARCHIVING/
directory forarchive_files_rsync/
andarchive_files_scp/
to “archive”, move, data to.archive_files_rsync
“Archives” (
rsync
’s) files to theARCHIVING/
folder in the$ROSE_DATA/
directory.archive_files_scp
“Archives” (
scp
’s) the renamed files and moves them to theARCHIVING/
folder in the$ROSE_DATA/
directory.
Save your changes and run the workflow:
cd ~/cylc-src/rose-arch-tutorial
cylc validate .
cylc install
cylc play rose-arch-tutorial
View the workflow output using cylc cat-log
and inspect the
output of the make_files
, archive_files_rsync
and archive_files_scp
tasks.
Results Of “Archiving”
Change to the $ROSE_DATA/ARCHIVING/
directory of the workflow i.e:
cd ~/cylc-run/rose-arch-tutorial/runN/share/data/ARCHIVING/
List the directory by typing:
ls
You should see the following returned:
dir file_zipped.tar files rename solo.file
Change directory to files/
and list the files:
cd files
ls
The following should be returned:
file_list1 file_list3
Change directory to ARCHIVING/dir/
and list the files:
cd ..
cd dir
ls
The following should be returned:
file_list1 file_list2 file_list3
Note
These were all of the files in the $ROSE_DATA/
directory.
Change directory to ARCHIVING/rename/
and list the files:
cd ..
cd rename
ls
The following should be returned:
1_1_file_list1 1_2_file_list2 1_3_file_list3
These are the renamed files.
Most users will have their own system or location that they wish to archive their data to. Here the example shown uses rsync and scp. Please refer your own site specific archiving solutions and seek site specific advice.
Arch Settings
Some settings that can be used are described below. See the rose_arch documentation for more information:
Above .tar
was used to compress the file. However, compress=gzip
can also be used. Note either of these commands can be used to compress a
file or a folder/directory.
In the above example a regular expression ‘reg exp’ was used by the
rename-parser
, for example, ^.*list(?P<tag>.*)$
, where:
^
= start of a string.$
= end of a string..
= any character.*
= greedy (all).?P<NAME>
= named group.
Note
rose arch
uses the Python flavor for regular expressions.
In the above example source was used to accept a list of glob patterns.
For example, file_list?
was used where the ?
relates to one unknown
character.
Note
These examples are just some possible examples and not a full list.
As well as rose_arch[arch]
and [arch:TARGET]
other options
can be provided to the app, for example:
[env]
Can be defined near the top of the app to allow an environment variable to be available to the
[arch:]
commands in the app.Also see
rose-app.conf[env]
and the suite example above.[poll]
Polling can be defined, and is often near the bottom of the app. This will allow the app to poll with a defined delay, e.g.
rose-app.conf[poll]delays=5
.[file:TARGET]
This option allows the user to, for example, make the directory
TARGET
, e.g.*[file:TARGET]mode=mkdir
.
For more information, see the rose_arch documentation.