Skip to content

Run regression tests

The Python script regression_test.py runs a suite of regression simulations and compares the output to the reference output for each simulation. It is set up to run the simulations on an HPC cluster; however, you can run modify the script to run tests locally. However, the script was written with remote execution in mind and running it locally can be time-consuming. Running the entire regression suite locally will take several hours on average.

The regression scenarios, script, configuration file, and other relevant files are all in the Regression_ directory. Be aware that many of these tests, due to abnormally high or low values, will produce output that should not be considered scientifically accurate.

  1. Modify the configuration file, regression_test.cfg, for your environment, setting the values for the location of the working directory, input files, binary file, and cluster settings.

    • For local Windows simulations, set the values under [WINDOWS].
    • For local CentOS simulations, set the values under [POSIX]. Note that CentOS simulations are run locally by default and cannot be commissioned to an HPC cluster.
    • For simulations on IDM HPC clusters, no changes are necessary if your username and password are cached locally.
    • For simulations on your own HPC cluster, create [HPC-] and [ENVIRONMENT-] sections for your cluster that contain the same variables as shown for IDM HPC clusters.

  2. Select the suite of regression tests you want to run. This is indicated by a JSON file in the following format:

    1
2
3
4
5
6
7
    {
    "tests": [{
        "path": "Relative path to test directory."
    }, {
        "path": "Relative path to test directory."
    }]
}

    You can use one of the JSON files in the Regression directory or create your own. The sanity.json file is recommended for quickly testing a wide range of EMOD functionality.

  3. From the Regression directory, open a Command Prompt window and run the regression test script, regression_test.py. It requires the name of the regression suite (without the .json extension) and the relative path to Eradication.exe. For example:

    regression_test.py sanity ..\Eradication\x64\Release\Eradication.exe

    In addition, you may need to include the following optional arguments depending on your testing environment or how Eradication.exe was built.

    Argument Default Description
    --perf False Measure Eradication.exe performance. --hidegraphs

  4. Review the output and examine any failures.

    EMOD will output the standard error and logging files, StdErr.txt and StdOut.txt, produced from any simulation (see software-error-logging). In addition, regression_test.py will output time.txt under the regression test working directory and report_xxxx.xml under Regression/reports. The time report contains the EMOD version and total run time of the simulation. The regression report is in JUnit_ standard form and captures run information, including pass/fail/complete and time to complete.

    If a simulation completes saying the run passed but the channel order was different than the reference output, this is considered a pass. However, if any output completes but does not match the reference output, this is considered a failure and a matplotlib chart of the output will appear in a pop-up window. The chart will appear immediately after the simulation, before the entire suite of regression tests completes. You can manipulate the output of the charts, such as adjusting the scale of the plots, zooming or panning, and so forth, through the icons at the bottom of the chart window.

    Popup Chart

If any of the regression tests fail and you have not made any changes to the EMOD source code, visit https://github.com/orgs/EMOD-Hub/discussions or use the GitHub link at the top of the page to submit a new issue in the repo. Be sure to include any error information.