Testing and Continuous Integration

Legend

Designates a mandatory point (policy).

Designates a recommended point (guideline).

Use regression tests in the development process.

Rationale

Regression tests prevent new features from affecting prior functionality, and they allow developers to have more confidence in their changes when modifying code.

Ref

WSC 5.1

Provide a test suite for verifying correctness of build and installation.

Details

A series of tests (including unit tests) should be runnable by anyone after the package is built to verify a correct build. The installation should also provide tests – which could derive from or be the same as example codes – must be run and installed alongside the library for future verification.

Rationale

An executable test suite gives users confidence that the product gives correct results. It is also recommended that at least a significant subset of the tests be can be run in batch-only environments.

Ref

xSDK M2

Provide a development test suite along with the source code for external contributors.

Details

The Git repository must contain test problems an easy way to run them and verify correctness for developers who are working on remote systems. External contributors must also have access to any results from continuous integration pipelines run against their changes before they make a pull request.

Rationale

External contributors should be able to test there code to the same standard as the development team. This ensure easier integration of new contribution and transparency in the process.

Use continuous integration (CI).

Details

Example CI systems in use by RADIUSS products:

RAJA with Travis CI on GitHub. See .travis.yml

MFEM with AppVeyor on GitHub. See .appveyor.yml

Ascent with Azure Pipelines on GitHub. See azure-pipelines.yml

Rationale

CI helps to prevent integration problems by identifying issues as they arise instead of allowing them to compound. This process helps reduce rework, cost, and time. Examples of CI tools include Travis CI for projects hosted on GitHub, or GitLab CI for projects hosted on a GitLab instance such as LC-GitLab. Some products may use Azure Pipelines in the future.

Continuously check compatibility with different compilers and versions.

Details

During the development process, tests should include compilation across a matrix of compilers and compiler versions, including both open source and vendor-proprietary when available

Rationale

This helps to guard against platform-specific or compiler-specific errors.

Ensure code quality/health in an automated manner.

Details

Example tools:

Asan (AddressSanitizer)

clang-tidy

cppcheck

cpplint

Rationale

Code quality checkers help to prevent code constructs that may result in errors in future environments where the code may run. They also reduce noise in build output that may obfuscate real errors. Code style checkers can also help enforce consistent style across a project.

Ref

WSC 5.4