carputils

carputils evolved out of the old svn-hosted python scripts for running ready made developer tests and benchmarks. These test packages duplicated much of the infrastructure both within and between packages. When a need for running sets of standard tests and comparing them against known solutions arose, a new structure for the tests was designed with the following goals:

  • Minimal code should be shared between tests and between repositories of tests
  • Best practices for Python code should be followed, including organisation of shared functionality in a Python package, assisting the above goal
  • Tests should be able to specify standard simulations for comparison with a known solution with minimal effort
  • These tests should be runnable in bulk, and a report generated allowing the quick validation that changes to the code have not changed simulation results

Required Packages

carputils uses some additional common numerical libraries in Python for pre- and post-processing of simulations. To install, run the following pip command:

pip install numpy scipy matplotlib --user

If scipy installation fails due to not finding BLAS/LAPACK, run with the bash variables BLAS and LAPACK set to the path of the shared library file:

BLAS=/path/to/libblas.so LAPACK=/path/to/liblapack.so pip install scipy

If no system BLAS/LAPACK is available, it is easily built from source. Download the latest LAPACK from http://www.netlib.org/lapack/ and unpack it, and copy the configuration template:

tar xvf lapack-3.5.0.tgz
cd lapack-3.5.0
cp make.inc.example make.inc

Then, edit make.inc and add the options -fPIC and -m64 (on 64 bit) to OPTS and NOOPT. Build BLAS and LAPACK with:

make blaslib
make

The shared library files librefblas.a and liblapack.a are then generated in the root of the source, and their paths can be passed to pip as above as the BLAS and LAPACK variables respectively.


Regression Testing

carputils provides the carptests executable in the bin subfolder, while enables the automatic running of sets of tests and the generation of reports comparing the results against reference solutions.

Note

Consider adding the bin subfolder of carputils to your $PATH environment variable, to make carptests runnable from anywhere on the command line.

Structure of Repositories

The testing framework is composed of several git repositories:

  • carputils - A python package providing common functionality for all examples
  • The example repositories, which are each a set of python scripts which set up and run CARP simulations, including
    • devtests - Smaller CARP simulation examples
    • benchmarks - Larger CARP simulation examples
  • The reference solution repositories, which contain previously computed solutions to the above examples that are known to be correct, including
    • devtests-reference
    • benchmarks-reference

Why __init__.py?

The __init__.py files in devtests and benchmarks may seem odd, however they are required by the testing framework. carptests uses the python package hierarchy to find and import individual tests, so for that reason the example repos must be python packages. Python determines that a directory is a package if it contains a file named __init__.py. This can be used to control how the package is imported, however for our purposes we only need to add an empty __init__.py file in all directories from the top level of the package (devtests/__init__.py, for example) down to the level of the directory where there is a test we want to run.

Acquiring Existing References

To run regression tests, you will need to acquire the reference solutions. Make a directory to store them and clone the reference solution repositories there:

mkdir /data/carp-regression-reference
cd /data/carp-regression-reference
git clone git@bitbucket.org:carpusers/devtests-reference.git devtests
git clone git@bitbucket.org:carpusers/benchmarks-reference.git benchmarks

Make sure that the cloned respository name matches that of the corresponding test repositories above. Then, set the parent directory as the REGRESSION_REF setting in settings.yaml:

REGRESSION_REF: /data/carp-regression-reference

Optionally, set the REGRESSION_TEMP and REGRESSION_PKG settings in settings.yaml, as described in the file template.

Reference Repository Layout

carptests automatically determines where to find the reference solutions based on the REGRESSSION_REF setting in the carputils settings.yaml file, the name of the module that the test was found in, and the name of the test. To ensure that the reference solution files are exactly where they are expected to be, it is strongly encouraged that users generate the reference solution with the --refgen parameter to carptests, as described in Generating Reference Solutions. This not only ensures that the reference is stored in the right place, but also that it is generated with exactly the inputs parameters expected by the test.

Running Regression Tests

Running the executable without arguments will by default run all tests in the module(s) specified in REGRESSION_REF in settings.yaml:

carptests

A summary of the daily results can be found here.

To override this behaviour, specify one or more modules on the command line. For example, to run all mechanics examples in devtests and all tests in benchmarks, run:

carptests devtests.mechanics benchmarks

Tests can also have case-insensitive tags assigned. Tags are defined in the carputils.testing.tag module, and summarised in the automatic documentation. To run, for example, only fast tests taking less than 10 seconds:

carptests --tags fast

On completion of all tests, a report is generated summarising the results. For more details on the command line interface, run:

carptests --help

Generating Reference Solutions

To generate and store any files needed for testing in the relevant repository, run carptests with the --refgen optional argument. You will probably want to specify a single example to generate the reference solutions for to avoid overwriting the entire suite’s reference solutions. For example, to generate the reference solutions for the mechanics ring example in devtests (devtests/mechanics/ring), run:

carptests devtests.mechanics.ring --refgen

This will run the test(s) specified in the mechanics ring example and copy the files needed for comparison in the test to the correct location in the reference solution repository. A summary is generated of the run tests, including a list of the generated reference solution files:

===================================================================
                              SUMMARY
===================================================================

  Tests run .................. 1
  Reference files generated .. 1
  Runtime .................... 0:00:17.049971

  Files generated:
    /data/carp-test-reference/devtests/mechanics.ring.run/neumann/x.dynpt.gz

====================================================================

If you are satisfied that this generated solution is the ‘correct’ one, that others should compare against when running this test, you need to commit and push this new or modified file. In the above example:

cd /data/carp-test-reference/devtests/mechanics.ring.run/neumann/
git add x.dynpt.gz
git commit -m "Updating the mechanics ring test ref. solution"
git push # To push your commit to bitbucket and share with others

Defining New Tests

Note

The ‘simple bidomain’ example in the devtests repository (devtests/bidomain/simple) defines some simple tests and can be used as a starting point for new examples with tests.

The testing framework views the tests’ directory structure as a Python package and searches the package hierarchy to find and run tests which have been defined.

Tests are defined by assigning the variable __tests__ in the top level namespace of a run script, which must be a list of carputils.testing.Test objects. This can be placed anywhere in the source file outside a function or __name__ == '__main__' block, but it is suggested to put it immediately before the __name__ == '__main__' block, and after the definition of the run function described in examples:

from carputils import tools
from carputils.testing import Test

@tools.carpexample()
def run(args, job):
    pass # Placeholder

test1 = Test(...)
test2 = Test(...)
__tests__ = [test1, test2]

if __name__ == '__main__':
    run()

The carputils.testing.Test object takes at least two arguments:

  • The name of the test
  • The function to run to execute the simulation

Additionally, tests will usually take at least a third optional argument:

  • A list of command line options to run the test with.

So, if you were to normally run a test with the command line:

./run.py --experiment neumann --np 8

You might configure the test:

test1 = Test('neumann_np8', run, ['--experiment', 'neumann', '--np', 8])

The function to run (here just called run) should be as explained in examples:

@tools.carpexample()
def run(args, job):
    # Run the simulation
    pass # Placeholder

Adding Checks

You must also tell the framework how to validate the simulation output against a reference solution. Generally, you will do this by comparing an output file against a reference, and computing some kind of error norm.

The carputils.testing.checks module defines some functions for comparing igb files, carputils.testing.max_error() and carputils.testing.error_norm(). These functions calculate a scalar error value, which the testing framework compares against the test developer’s preassigned tolerance. A check is added to a test by:

from carputils.testing import max_error
test1.add_filecmp_test('x.dynpt.gz',  # Output file to compare
                       max_error,     # Error calc method
                       0.001)         # Error tolerance

These simple error methods should cover most use cases, though you may easily create your own by passing a function that takes the reference filename as the first argument and the temporary test output at runtime as the second argument, and returns an error value.

The error tolerance value is only used to determine test success. For more complex control of test success, you may alternatively pass as the tolerance argument a function which takes the return value of the error function and returns a boolean.

Important Considerations

  • The directory your test is in must be part of the Python package. To ensure this is the case, there must be an empty file called __init__.py in the same directory as your test’s run script, and any intermediate directories from the top level package down.

Reference Generation

Once your tests are defined, you will need to share any reference simulation results required for comparison in the tests. To do so, run the carptests executable with the --refgen parameter (probably with the module of your example specified to avoid running all examples) as described in Generating Reference Solutions.

Optional Arguments

Tags

Tests can optionally set one or more tags, for the purposes of categorisation of simulations. Standard tags are as described in carputils.testing.tag, but test developers can also add their own limited-use tags with the add method:

from carputils.testing import tag
tag.add('my_special_tag')

Tags are assigned to test by supplying a list of tags as the tags keyword argument:

test1 = Test('mytest', run, ['--argument'],
             tags=[tag.SERIAL, tag.FAST,
                   tag.MY_SPECIAL_TAG]) # tags are case insensitive

Tests can then be filtered at runtime with the --tags command line argument, as described in Running Regression Tests.

Reference Directory

If you want to validate multiple tests against the same reference solution, consider use of the refdir keyword argument.

By default, reference solutions are stored in the directory:

REGRESSION_REF/package/module/testname

where REGRESSION_REF is replaced by the value set in the carputils settings.yaml file. For example, for the test neumann in devtests/mechanics/ring/run.py, this will be:

REGRESSION_REF/devtests/mechanics.ring.run/neumann

To force a test of a different name to use the same reference solution, use the refdir argument to override the last part of the directory path:

test1 = Test('neumann', run, ['--argument'], refdir='other')
# Giving the ref dir REGRESSION_REF/devtests/mechanics.ring.run/other

You will probably want to disable automatic reference generation for all but one of the tests sharing a directory, to be sure which test the calculated reference is from:

test1.disable_reference_generation()

A key application of this is for running a number of similar tests, for example the same simulation with different numbers of CPUs:

__tests__ = []
for np in [1, 2, 4, 8, 16]:
    argv = ['--experiment', 'neumann', '--np', np]
    test = Test('neumann_np{0}'.format(np), run, argv, refdir='neumann')
    if np != 1:
        # Only generate reference for --np 1
        test.disable_reference_generation()
    __tests__.append(test)

Nightly Summary

Follow the link to see the results of the nightly runs.


Installation From Source

Mac OSX Source Install

While CME software has been successfully installed on a growing number of Apple systems, the installation process has not been streamlined or finalized. The current documentation may therefore not be complete or apply to all systems. Comments, questions, or suggestions for improvements are well appreciated by Gernot Plank and Karli Gillette.

System Requirements

First, installs of both homebrew and Xcode with command line tools are required.

  • homebrew:
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
  • Xcode: download from App Store. If working with an older OSX, Xcode can be downloaded from apple developer.
xcode-select --install

Now using homebrew install gcc and wget:

brew install gcc
brew install wget
brew install libpng

PETSc and CARPentry

Warning

Doing a source installation of CARPentry requires additional permissions. Please contact Karli Gillette before proceeding.

Decide where you are going to do the install of CARPentry and PETSc that requires multiple directories and packages. For example: $HOME/install

Set your PETSc environmental variables in your ~/.bashrc. (If necessary, create the .bashrc file):

export PETSC_DIR=$HOME/install/petsc
export PETSC_ARCH=osx-clang-debug2

Update your shell variables by typing source ~/.bashrc. Now download PETSc:

cd $HOME/install
git clone -b maint https://bitbucket.org/petsc/petsc petsc
cd petsc

Insert osx-clang-debug.py into the config directory of PETSc and run the configure scripts:

cp $HOME/Downloads/osx-clang-debug.py config
python config/osx-clang-debug.py
make PETSC_DIR=$PETSC_DIR PETSC_ARCH=$PETSC_ARCH all
make PETSC_DIR=$PETSC_DIR PETSC_ARCH=$PETSC_ARCH test

Continue with downloading the developer sources for CARP:

cd $HOME/install
svn co —username=`username` https://carpentry.medunigraz.at/carp-dcse-pt/branches/mechanics carp-dcse-pt
cd carp-dcse-pt

Patch some files:

  • Add switches.def to $HOME/install/carp-dcse-pt Note: you can set the flags to be different if needed for different packages.
  • Add osx_carp_patch to $HOME/install/carp-dcse-pt/CARP. Note: Unpackage using tar -xvf osx_carp_patch.tar.gz

Now build CARP:

make            # you will get an error about switches, don’t fret!
make svn_co     # updates/checks out related repositories
make gengetopt
make param
make redblack

Add the provided paths (gengetopt and param) to your PATH environment variable (~/.bashrc):

export PATH=$PATH:$HOME/install/carp-dcse-pt/gengetopt-2.22.6/src:$HOME/install/carp-dcse-pt/PrM/src

Update your terminal enviroment (source ~/.bashrc) and finish making CARP:

bash            # to update the shell
make all

If you get an error for python2:

  • Check to make sure you have a system-wide installation of python2. If the executable is not calling python2 from /usr/bin then create a symbolic link for it (ln -s python python2) or you can also do a source installation which gives you the python2 executable (suggestion from Christoph Augustin)
  • If you do not have a system-wide installation of python2, install a local one from source and make sure the executable is on your path.

carputils And Examples

The documentation for carputils installation is located here.

cd $HOME/install
git clone https://username@bitbucket.org:/carpusers/carputils.git
mkdir carp-examples
cd carp-examples
git clone https://username@bitbucket.org:carpusers/tutorials.git

Update your bashrc:

export PYTHONPATH=$PYTHONPATH:$HOME/install/carputils:$HOME/install/carp-examples
export TUTORIALS = $HOME/install/carp-examples/tutorials

Generate a settings files for carputils:

cd $HOME/install/carputils/bin
./cusettings $HOME/.config/carputils/settings.yaml

Go in and edit your settings.yaml file with the proper directories for CARP-CPU and MESHALYZER

CPU: $HOME/install/carpentry/bin
MESHALYZER_DIR: $HOME/install/meshalyzer

In addition to the carputils basic installation, following repositories might be beneficial to have too:

git clone git@bitbucket.org:carpusers/devtests.git
git clone git@bitbucket.org:carpusers/benchmarks.git
git clone git@bitbucket.org:carpusers/pvprocess.git

Installing a Local Python Environment

Many HPC systems use older operating systems with out of data Python distributions, so you may find yourself without the required Python 2.7 version. Follow the below instructions to build and install a local Python environment.

Firstly, get the latest Python source from https://www.python.org/downloads/. Get the latest Python 2 version - carputils does not run in Python 3. Then, make a separate directory to contain the installed Python distribution (including the binaries and any extra modules you will install), e.g.:

cd $HOME
mkdir python-distribution

Extract the Python source tarball and run the configure script, setting the --prefix and --exec-prefix options to the path of the directory created above:

tar xvf Python-2.7.10.tar.gz
cd Python-2.7.10
./configure --prefix $HOME/python-distribution --exec-prefix $HOME/python-distribution

Then, build and install Python to the specified folder:

make
make install

The built Python interpreter should now be present in $HOME/python-distribution/bin. To make this interpreter the default when running python on the command line, add the following line to the bottom of the file ~/.bashrc:

export PATH=$HOME/python-distribution/bin:$PATH

Finally, install the pip Python package manager to facilitate the easier installation of Python modules later. This is done with:

python -m ensurepip
pip install --upgrade pip # To update to latest version

reStructuredText References

As the rst syntax is typically not used on a daily basis, have a look at listed pages to update your knowledge:

  • docutils on sourcefourge.net

  • referencing figures

  • Start using a reStructuredText linter to inspect any rst-file before upload.

    Hint

    rst-lint may be installed with your Python installation manager - pip2 install restructuredtext-lint --user

  • To reliably embed a Youtube video open its Share options and copy and paste the Embed content into

    a raw directive as follows:

    .. raw:: html
    
          <iframe width="560" height="315" src="https://www.youtube.com/embed/gvkPgDWUWrU" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
    

    getting following result:

  • List of available admonitions:

    Attention

    .. attention::

    Caution

    .. caution::

    Danger

    .. danger::

    Error

    .. error::

    Hint

    .. hint::

    Important

    .. important::

    Note

    .. note::

    Tip

    .. tip::

    Warning

    .. warning::

  • Identation levels:

    ##################
    H1: document title
    ##################
    
    *********
    Sample H2
    *********
    
    Sample H3
    =========
    
    Sample H4
    ---------
    
    Sample H5
    ^^^^^^^^^
    
    Sample H6
    """""""""
    
    Here comes your text.