Introduction

This part of the Rose user guide explains how to use the rose stem testing system, and the motivation behind it.

rose stem requires the use of FCM as it requires some of the version control information.

Motivation

Why do we test code?

Motivation II

Most people would answer something along the lines of "so we know it works".

However, this is really asking two related but separate questions.

• Does the code do what I meant it to do?
• Does the code do anything I didn't mean it to do?

Motivation III

Answering the first question may involve writing a bespoke test and checking the results. The second question can at least partially be answered by using an automated testing system which runs predefined tasks and presents the answers. rose stem is a system for doing this.

N.B. When writing tests for new code, they should be added to the testing system so that future developers can be confident that they haven't broken the new functionality.

Rose Stem

There are two components in rose stem:

• rose stem - the command line tool which executes an appropriate suite.
• rose ana - a Rose task app which can compare the result of a task against a control.

We will describe each in turn. It is intended that a test suite lives alongside the code in the same version-controlled project, which should encourage developers to update the test suite when they update the code. This means that the test suite will always be a valid test of the code it is accompanying.

Running a suite with rose stem

The rose stem command is essentially a wrapper to rose suite-run, which accepts some additional arguments and converts them to Jinja2 variables which the suite can interpret.

These arguments are:

• --source - specifies a source tree to include in a suite.
• --group - specifies a group of tasks to run.

A group is a set of Rose tasks which together test a certain configuration of a program.

The --source argument

The source argument provides a set of Jinja2 variables which can then be included in any compilation tasks in a suite. You can specify multiple --source arguments on the command line. For example:

rose stem --source=/path/to/workingcopy --source=fcm:other_project_tr@head

Each source tree is associated with a project (via an fcm command) when rose stem is run on the command line. This project name is then used in the construction of the Jinja2 variable names.

Source II

Each project has a Jinja2 variable SOURCE_FOO where FOO is the project name. This contains a space-separated list of all sourcetrees belonging to that project, which can then be given to an appropriate build task in the suite so it builds those source trees.

Similarly, a HOST_SOURCE_FOO variable is also provided. This is identical to SOURCE_FOO except any working copies have the local hostname prepended. This is to assist building on remote machines.

The first source specified must be a working copy which contains the rose stem suite. The suite is expected to be in a subdirectory named rose-stem off the top of the working copy. This source is used to generate three additional variables:

• SOURCE_FOO_BASE - the base directory of the project
• HOST_SOURCE_FOO_BASE - the base directory of the project with the hostname prepended if it is a working copy
• SOURCE_FOO_REV - the revision of the project (if any)

These settings override the variables in the rose-suite.conf file.

Source III

These should allow the use of configuration files which control the build process inside the working copy, e.g. you can refer to:

      {{HOST_SOURCE_FOO_BASE}}/fcm-make/configs/machine.cfg{{SOURCE_FOO_REV}}

If you omit the source argument, rose stem defaults to assuming

rose stem --source=.

viz. that the current directory is part of the working copy which should be added as a source tree.

Source IV

The project to which a source tree belongs is normally automatically determined using FCM commands. However, in the case where the source tree is not a valid FCM URL, or where you wish to assign it to another project, you can specify this using the --source argument:

rose stem --source=foo=/path/to/source

assigns the URL /path/to/source to the foo project, so the variables SOURCE_FOO and SOURCE_FOO_BASE will be set to /path/to/source.

The --group argument

The group argument is used to provide a Pythonic list of groups in the variable RUN_NAMES which can then be looped over in a suite to switch sets of tasks on and off.

Each --group argument adds another group to the list. For example:

rose stem --group=mygroup --group=myothergroup

runs two groups named mygroup and myothergroup with the current working copy. The suite will then interpret these into a set of tasks which build with the given source tree(s), run the program, and compare the output.

The --task argument

The --task argument is provided as a synonym for --group. Depending on how exactly the rose stem suite works users may find one of these arguments more intuitive to use than the other.

Comparing output with rose ana

Any task beginning with rose_ana_ will be interpreted by Rose as a rose ana task, and run through the rose ana task app.

A rose ana rose-app.conf file contains a series of blocks; each one describing a different analysis task to perform. A common task which rose ana is used for is to compare output contained in different files (e.g. from a new test versus previous output from a control). The analysis modules which provide these tasks are flexible and able to be provided by the user; however there is one built-in module inside rose ana itself.

rose ana II

An example based on the built-in grepper module:

  [ana:grepper.FilePattern(Compare data from myfile)]
  pattern='data value:(\d+)'
  files=/data/kgo/myfile
       =../run.1/myfile

This tells rose ana to scan the contents of the file ../run.1/myfile (which is relative to the rose ana task's work directory) and the contents of /data/kgo/myfile for the specified regular expression. Since the pattern contains a group (in parentheses) so it is the contents of this group which will be compared between the two files. The grepper.FilePattern analysis task can optionally be given a "tolerance" option for matching numeric values, but without it the matching is expected to be exact. If the pattern or group contents do not match the task will return a failure.

rose ana III

As well as sections defining analysis tasks, rose_ana apps allow for one additional section for storing global configuration settings for the app. Just like the tasks themselves these options and their effects are dependent on which analysis tasks are used by the app.

Therefore we will here present an example using the built-in grepper class. An app may begin with a section like this:

  [ana:config]
  grepper-report-limit=5
  skip-if-all-files-missing=.true.

rose ana IV

Each of these modifies the behaviour of grepper. The first option suppresses printed output for each analysis task once the specified number of lines have been printed (in this case 5 lines). The second option causes rose_ana to skip any grepper tasks which compare files in the case that both files do not exist.

Note that any options given to this section may instead be specified in the rose-ana section of the user or site configuration. In the case that the same configuration option appears in both locations the one contained in the app file will take precedence.

rose ana V

It is possible to add additional analysis modules to rose ana by placing an appropriately formatted python file in one of the following places (in order of precedence):

• The ana sub-directory of the rose ana app
• The ana sub-directory of the suite
• Any other directory which is accessible to the process running rose ana and is specified in the method-path variable in the rose ana section of the rose.conf file.

The only module provided with Rose can be found at lib/python/rose/apps/ana_builtin/grepper.py and it provides the following analysis tasks and options:

rose ana VI

• SingleCommandStatus: Runs a shell command, passing or failing depending on the exit status of that command.
  • files - a newline-separated list of filenames (optional) which may appear in the command.
  • command - the command to run; if it contains Python style format specifiers these will be expanded using the list of files above (if provided).
  • kgo_file - if the list of files above was provided gives the (0-based) index of the file holding the "kgo" or "control" output for use with the comparisons database (if active).

rose ana VII

• SingleCommandPattern: Runs a shell command, then passes or fails depending on the presence of a given regular expression in the command's stdout.
  • files - (optional) same as previous task.
  • command - same as previous task.
  • kgo_file - same as previous task.
  • pattern - the regular expression to search for in the stdout from the command.

rose ana VIII

• FilePattern: Compares the contents of files using a regular expression.
  • files - (optional) same as previous tasks.
  • kgo_file - same as previous tasks.
  • pattern - the regular expression to search for in the files. The expression should include one or more capture groups; each of these will be compared between the files any time the pattern occurs.
  • tolerance - by default the above comparisons will be compared exactly, but if this (optional) argument is specified they will be converted to float values and compared according to the given tolerance. If this tolerance ends in % it will be interpreted as a relative tolerance (otherwise absolute).

rose ana IX

• FileCommandPattern: Uses a regular expression to compares the contents of the stdout resulting from a given command applied to two or more files.
  • files - (optional) same as previous tasks.
  • kgo_file - same as previous tasks.
  • pattern - same as previous tasks.
  • tolerance - (optional) same as previous tasks.
  • command - the command to run; it should contain a Python style format specifier to be expanded using the list of files above.

rose ana X

The grepper analysis module also supports some configuration options (specified in the [ana:config] section of the app or in the [rose-ana] section of the Rose site/user configuration:

• grepper-report-limit: A numerical value giving the maximum number of informational output lines to print for each comparison. This is intended for cases where for example a pattern-matching comparison is expected to match many thousands of occurences in the given files; it may not be desirable to print the results of every comparison. After the given number of lines are printed a special message indicating that the rest of the output is truncated will be produced.
• skip-if-all-files-missing: Can be set to .true. or .false.; if active, any comparison done on files by grepper will be skipped if all of those files are non-existent. In this case the task will return as "skipped" rather than passed/failed.

rose ana XI

The format for analysis modules themselves is relatively simple; the easiest route to understanding how they should be arranged is likely to look at the built-in grepper module. But the key concepts are as follows. To be recognised as a valid analysis module, the Python file must contain at least one class which inherits and extends rose.apps.rose_ana.AnalysisTask, which will automatically populate itself with certain attributes; e.g:

from rose.apps.rose_ana import AnalysisTask

class CustomAnalysisTask(AnalysisTask):
    """My new custom analysis task."""
    def run_analysis(self):
        print self.options  # Dictionary of options (see next slide)
        if self.options["option1"] == "5":
            self.passed = True

rose ana XII

Assuming the above was saved in a file called custom.py and placed into a folder suitable for analysis modules this would allow a rose ana app to specify:

[ana:custom.CustomAnalysisTask(Example rose-ana test)]
option1 = 5
option2 = test of rose ana
option3 = .true.

Note that the custom part of the filename appears at the start of the ana entry, followed by the name of the desired class (in the style of Python's own namespacing). All options specified by the app-task will be processed by rose ana into a dictionary and attached to the running analysis class instance as the options attribute. Hopefully you can see that in this case the task would pass because option1 is set to 5 as required by the class.

rose ana XIII

The parent AnalysisTask class will also set several other attributes which may be of use when designing a new method:

• self.config - a dictionary containing any rose ana configuration options
• self.reporter - a reference to the rose.reporter.Reporter instance used by the parent app (for printing to stderr/stdout)
• self.kgo_db - a reference to the KGO database object created by the parent app (for adding entries to the database - see next section)
• self.popen - a reference to the rose.popen.RosePopener instance used by the parent app (for spawning subprocesses)

rose ana XIV

The parent class also provides the self.passed and self.skipped methods, which will default to False but should be set to True by the task to indicate how rose ana should proceed. (Note that the skipped state is only checked if the passed state is False.

The rose ana comparison database

In addition to performing the comparisons each of the rose ana tasks in the suite can be configured to append some key details about any comparisons performed to an sqlite database kept in the suite's log directory (at log/rose-ana-comparisons.db).

This is intended to provide a quick means to interrogate the suite for information about the status of any comparisons it has performed. There are 2 tables present in the suite which contain the following:

rose ana db II

• tasks (TABLE) Contains an entry for each rose ana task, using the following columns:
  • task_name (TEXT) The exact name of the rose ana task.
  • completed (INT) Set to 1 when the task starts performing its comparisons then updated to 0 when the task has completed (Note: task success is not related to the success/failed state of the comparisons).
  The intention of this table is to detect if any rose ana tasks have failed unexpectedly (or are still running).

rose ana db III

• comparisons Contains an entry for each individual comparison from every rose ana task, using the following columns:
  • comp_task (TEXT) The comparison task name - by convention this is usually the comparison section name from the app defintion (including the part inside the brackets).
  • kgo_file (TEXT) The full path to the file specified as the KGO file in the app definition.
  • suite_file (TEXT) The full path to the file specified as the active test output in the app definition.
  • status (TEXT) The status of the task (one of " OK ", "FAIL" or "WARN").
  • comparison (TEXT) Additional details which may be provided about the comparison
  The intention of this table is to provide a record of which files were compared by which tasks, how they were compared and what the result of the comparison was.

rose ana db IV

The database is entirely optional; by default is will not be produced; if it is required it can be activated by setting the kgo-database option in the rose-ana section of the Rose configuration to .true..

Note that the system does not provide any direct methods for working with or interrogating the database - since there could be various reasons for doing so, and there may be other suite-design factors to consider. Users are therefore expected to provide this functionality separately based on their specific needs.

Summary

From within a working copy, running rose stem is simple. Just run

rose stem --group=groupname

For further information about writing your own rose stem test suite, please see the advanced tutorial on the subject.