-Unit testing {#page_unittesting}
-============
-
-The main goal of unit tests in \Gromacs is to help developers while developing
-the code. They focus on testing functionality of a certain module or a group
-of closely related modules. They are designed for quick execution, such that
-they are easy to run after every change to check that nothing has been broken.
-
-Finding, building and running
-=============================
-
-As described in \ref page_codelayout, `src/gromacs/` is divided into modules,
-each corresponding to a subdirectory. If available, unit tests for that module
-can be found in a `tests/` subdirectory under the top-level module directory.
-Typically, tests for code in _file_.h in the module is in a corresponding
-`tests/`<em>file</em>`.cpp`. Not all files have corresponding tests, as it may not
-make sense to test that individual file in isolation. Focus of the tests is on
-functionality exposed outside the module. Some of the tests, in particular for
-higher-level modules, are more like integration tests, and test the
-functionality of multiple modules.
-Shared code used to implement the tests is in `src/external/gmock-1.7.0/` and
-`src/testutils/` (see below).
-
-The tests are built if `BUILD_TESTING=ON` (the default) and
-`GMX_BUILD_UNITTESTS=ON` (the default if `libxml2` is available) in CMake.
-Each module produces a separate unit test binary (_module_`-test`) under
-`bin/`, which can execute all the tests for that module.
-
-The tests can be executed in a few different ways:
- - Build the `test` target (e.g., `make test`):
- This runs all the tests using CTest. This includes also the regression
- tests if CMake has been told where to find them (regression tests are not
- discussed further on this page).
- If some of the tests fail, this only prints basic summary information (only
- a pass/fail status for each test binary or regression test class).
- You can execute the failing test binaries individually to get more
- information on the failure.
- Note that `make test` does not rebuild the test binaries if you have changed
- the source code, so you need to separately run `make` or `make tests`.
- The latter only builds the test binaries and their dependencies.
- - Build the `check` target (e.g., `make check`):
- This behaves the same as the `test` target, with a few extensions:
- 1. Test binaries are rebuilt if they are outdated before the tests are run.
- 2. If a test fails, the output of the test binary is shown.
- 3. If unit tests and/or regression tests are not available, a message is
- printed.
- - Directly executing a test binary. This provides the most useful output for
- diagnosing failures, and allows debugging test failures. The output
- identifies the individual test(s) that fail, and shows the results of all
- failing assertions. Some tests also add extra information to failing
- assertions to make it easier to identify the reason. It is possible to
- control which tests are run using command line options. Execute the binary
- with `-h` to get additional information.
-
-When executed using CTest, the tests produce XML output in
-`Testing/Temporary/`, containing the result of each test as well as failure
-messages. This XML is used by Jenkins for reporting the test status for
-individual tests. Note that if a test crashes or fails because of an assert or
-a gmx_fatal() call, no XML is produced for the binary, and Jenkins does not
-report anything for the test binary. The actual error is only visible in the
-console output.
-
-Unit testing framework
-======================
-
-The tests are written using [Google Test][], which provides a framework for
-writing unit tests and compiling them into a test binary. Most of the command
-line options provided by the test binaries are implemented by Google Test. See
-the [Google Test Primer][] for an introduction.
-Some tests also use [Google Mock][], which provides a framework for creating
-mock implementations of C++ classes. Both components are included in the
-source tree under `src/external/gmock-1.7.0/`, and are compiled as part of the
-unit test build.
-
-`src/testutils/` contains \Gromacs-specific shared test code. This includes
-a few parts:
- - CMake macros for declaring test binaries. These take care of providing the
- %main() method for the test executables and initializing the other parts of
- the framework, so that the test code in modules can focus on the actual
- tests. This is the only part of the framework that you need to know to be
- able to write simple tests: you can use `gmx_add_unit_test()` in CMake to
- create your test binary and start writing the actual tests right away.
- See `src/testutils/TestMacros.cmake` and existing CMake code for examples
- how to use them.
- - Generic test fixtures and helper classes. Functionality here includes
- locating test input files from the source directory and constructing
- temporary files (gmx::test::TestFileManager), adding custom command line
- options to the test binary (#GMX_TEST_OPTIONS), some custom test assertions
- for better exception and floating-point handling (testasserts.h), utilities
- for constructing command line argument arrays (gmx::test::CommandLine) and
- test fixtures for tests that need to test long strings for correctness
- (gmx::test::StringTestBase) and for tests that execute legacy code where
- `stdin` reading etc. cannot be easily mocked
- (gmx::test::IntegrationTestFixture).
- - Some classes and functions to support the above. This code is for internal
- use of the CMake machinery to build and set up the test binaries, and to
- customize Google Test to suit our environment.
- - Simple framework for building tests that check the results against reference
- data that is generated by the same test code. This can be used if it is not
- easy to verify the results of the code with C/C++ code alone, but manual
- inspection of the results is manageable.
-
- When the test using the framework is first executed, `-ref-data create` can
- be passed on command line to create the reference data (also options
- starting with double dashes are accepted).
- On later executions, the tests read the reference data and fail if the
- results are not the same. It is possible to update existing reference data
- with `-ref-data update`.
-
- The reference data is stored in XML files under
- `src/gromacs/`<em>module</em>`/tests/refdata/`. This part of the framework
- depends on `libxml2`. For inspecting the reference data in a browser, there
- are XSLT stylesheets that transform the XML files into HTML. Such custom
- transformations need to be written for each type of test if the output is
- not easy to check otherwise. Because of security features in browsers, the
- transformations may not work for all browsers. For the same reason, the
- XSLT files must be in the same folder as the XML files. For cases where the
- XSLT files are shared between multiple modules, `src/testutils/copy_xsl.sh`
- takes care to synchronize the files after a master copy is edited.
-
- The current reference data functionality is quite basic, but it can be extended
- if/when more control over, e.g., comparison tolerances is needed.
- See gmx::test::TestReferenceData for documentation and existing tests
- (e.g., the selection unit tests) for examples of how to use it.
-
-In addition to `src/testutils/`, some of the module test directories may
-provide reusable test code that is used in higher-level tests. For example,
-the \ref module_analysisdata module provides test fixtures, a mock
-implementation for gmx::AnalysisDataModuleInterface, and some helper classes
-that are also used in \ref module_trajectoryanalysis module tests.
-These cases are handled using CMake object libraries that are linked to all the
-test binaries that need them.
-
-Getting started with new tests
-==============================
-
-To start working with new tests, you should first read the [Google Test][]
-documentation to get a basic understanding of the testing framework, and read
-the above description to understand how the tests are organized in \Gromacs.
-It is not necessary to understand all the details, but an overall understanding
-helps to get started.
-
-Writing a basic test is straightforward, and you can look at existing tests for
-examples. The existing tests have a varying level of complexity, so here are
-some pointers to find tests that use certain functionality:
- - `src/gromacs/utility/tests/stringutil.cpp` contains very simple tests for
- functions (e.g., for gmx::formatString() and gmx::replaceAll()). These do
- not use any fancy functionality, only plain Google Test assertions.
- The only thing required for these tests is the `TEST()` macro and the block
- following it, plus headers required to make them compile.
- - The same file contains also simple tests using the reference framework to
- check line wrapping (the tests for gmx::TextLineWrapper). The test fixture
- for these tests is in `src/testutils/stringtest.h`/`.cpp`. The string test
- fixture also demonstrates how to add a custom command line option to the
- test binary to influence the test execution.
- - The \ref module_selection module tests contain more complex use of the
- reference framework. This is the code the reference framework was
- originally written for.
- `src/gromacs/selection/tests/selectioncollection.cpp` is the main file to
- look at.
- - For more complex tests that do not use the reference framework, but instead
- do more complex verification in code, you can look at
- `src/gromacs/selection/tests/nbsearch.cpp`.
- - For complex tests with mock-up classes and the reference framework, you can
- look at \ref module_analysisdata tests.
-
-Here are some things to keep in mind when working with the unit tests:
- - Try to keep the execution time for the tests as short as possible, while
- covering the most important paths in the code under test. Generally, tests
- should take seconds instead of minutes to run, so that no one needs to
- hesitate before running the tests after they have done some changes.
- Long-running tests should go somewhere else than in the unit test set.
- Note that Jenkins runs many of the tests under Valgrind, so heavy tests are
- going to slow down also that part of the verification.
- - Try to produce useful messages when a test assertion fails. The assertion
- message should tell what went wrong, with no need to run the _test itself_
- under a debugger (e.g., if the assertion is within a loop, and the loop
- index is relevant for understanding why the assertion fails, it should be
- included in the message). Even better if even a user can understand what
- goes wrong, but the main audience for the messages is the developer who
- caused the test to fail.
-
-[Google Test]: http://code.google.com/p/googletest/
-[Google Test Primer]: http://code.google.com/p/googletest/wiki/V1_7_Primer
-[Google Mock]: http://code.google.com/p/googlemock/