Finding, building and running
-----------------------------
-As described in :doc:`codelayout`, ``src/gromacs/`` is divided into modules,
+As described in :ref:`dev-source-layout`, ``src/gromacs/`` is divided into modules,
each corresponding to a subdirectory. If available, unit tests for that module
can be found in a ``tests/`` subdirectory under the top-level module directory.
Typically, tests for code in :file:`{file}.h` in the module is in a corresponding
functionality exposed outside the module. Some of the tests, in particular for
higher-level modules, are more like integration tests, and test the
functionality of multiple modules.
-Shared code used to implement the tests is in ``src/external/gmock-1.7.0/`` and
+Shared code used to implement the tests is in ``src/external/googletest/`` and
``src/testutils/`` (see below).
The tests are built if ``BUILD_TESTING=ON`` (the default) and
-``GMX_BUILD_UNITTESTS=ON`` (the default if ``libxml2`` is available) in CMake.
-Each module produces a separate unit test binary (:file:`{module}-test`) under
-``bin/``, which can execute all the tests for that module.
+``GMX_BUILD_UNITTESTS=ON`` (the default) in CMake. Each module
+produces at least one separate unit test binary
+(:file:`{module}-test`) under ``bin/``, which can execute tests for
+that module.
The tests can be executed in a few different ways:
3. If unit tests and/or regression tests are not available, a message is
printed.
-- Directly executing a test binary. This provides the most useful output for
- diagnosing failures, and allows debugging test failures. The output
- identifies the individual test(s) that fail, and shows the results of all
- failing assertions. Some tests also add extra information to failing
- assertions to make it easier to identify the reason. It is possible to
- control which tests are run using command line options. Execute the binary
- with ``-h`` to get additional information.
+- The implementation of ``make check`` calls CTest via the ``ctest`` binary
+ to run all the individual test binaries. More fine-grained control is available
+ there, e.g. filtering by test name or label, or increasing verbosity.
+- Directly executing a test binary. This provides the most useful
+ output for diagnosing failures, and allows debugging test failures.
+ The output identifies the individual test(s) that fail, and shows
+ the results of all failing assertions. Some tests also add extra
+ information to failing assertions to make it easier to identify the
+ reason. Some tests are skipped because they cannot run with the
+ number of MPI ranks or GPU devices detected. Explicit information
+ about such cases can be obtained by using the ``-echo-reasons`` flag
+ to the test binary. It is possible to control which tests are run
+ using command line options. Execute the binary with ``--help`` to
+ get additional information.
When executed using CTest, the tests produce XML output in
``Testing/Temporary/``, containing the result of each test as well as failure
-messages. This XML is used by Jenkins for reporting the test status for
+messages. This XML is used by GitLab CI for reporting the test status for
individual tests. Note that if a test crashes or fails because of an assert or
-a gmx_fatal() call, no XML is produced for the binary, and Jenkins does not
+a gmx_fatal() call, no XML is produced for the binary, and CI does not
report anything for the test binary. The actual error is only visible in the
console output.
the `Google Test Primer`_ for an introduction.
Some tests also use `Google Mock`_, which provides a framework for creating
mock implementations of C++ classes. Both components are included in the
-source tree under ``src/external/gmock-1.7.0/``, and are compiled as part of the
+source tree under ``src/external/googletest/``, and are compiled as part of the
unit test build.
``src/testutils/`` contains |Gromacs|-specific shared test code. This includes
for constructing command line argument arrays, and
test fixtures for tests that need to test long strings for correctness
and for tests that execute legacy code where
- `stdin` reading etc. cannot be easily mocked.
+ ``stdin`` reading etc. cannot be easily mocked.
__ doxygen-module-testutils_
- Simple framework for building tests that check the results against reference
data that is generated by the same test code. This can be used if it is not
easy to verify the results of the code with C/C++ code alone, but manual
- inspection of the results is manageable. The C++ API for this is also
- documented on the `Doxygen page for testutils`__.
+ inspection of the results is manageable. The general approach is
+ documented on the `Doxygen page on using the reference data`__.
- __ doxygen-module-testutils_
-
- When the test using the framework is first executed, ``-ref-data create`` can
- be passed on command line to create the reference data (also options
- starting with double dashes are accepted).
- On later executions, the tests read the reference data and fail if the
- results are not the same. It is possible to update existing reference data
- with ``-ref-data update``.
-
- The reference data is stored in XML files under
- :file:`src/gromacs/{module}/tests/refdata/`. This part of the framework
- depends on ``libxml2``. For inspecting the reference data in a browser, there
- are XSLT stylesheets that transform the XML files into HTML. Such custom
- transformations need to be written for each type of test if the output is
- not easy to check otherwise. Because of security features in browsers, the
- transformations may not work for all browsers. For the same reason, the
- XSLT files must be in the same folder as the XML files. For cases where the
- XSLT files are shared between multiple modules, ``src/testutils/copy_xsl.sh``
- takes care to synchronize the files after a master copy is edited.
-
- The current reference data functionality is quite basic, but it can be extended
- if/when more control over, e.g., comparison tolerances is needed.
+ __ doxygen-page-refdata_
In addition to ``src/testutils/``, some of the module test directories may
provide reusable test code that is used in higher-level tests. For example,
the ``src/gromacs/analysisdata/tests/`` provides test fixtures, a mock
-implementation for gmx::AnalysisDataModuleInterface, and some helper classes
+implementation for gmx::IAnalysisDataModule, and some helper classes
that are also used in ``src/gromacs/trajectoryanalysis/tests/``.
These cases are handled using CMake object libraries that are linked to all the
test binaries that need them.
+.. _gmx-make-new-tests:
+
Getting started with new tests
------------------------------
following it, plus headers required to make them compile.
- The same file contains also simple tests using the reference framework to
check line wrapping (the tests for ``gmx::TextLineWrapper``). The test fixture
- for these tests is in ``src/testutils/stringtest.h``/``.cpp``. The string test
+ for these tests is in ``src/testutils/include/testutils/stringtest.h``/``.cpp``. The string test
fixture also demonstrates how to add a custom command line option to the
test binary to influence the test execution.
- ``src/gromacs/selection/tests/`` contains more complex use of the
should take seconds instead of minutes to run, so that no one needs to
hesitate before running the tests after they have done some changes.
Long-running tests should go somewhere else than in the unit test set.
- Note that Jenkins runs many of the tests under Valgrind, so heavy tests are
- going to slow down also that part of the verification.
+ Note that CI will run the tests in several build configuration and
+ slow tests will significantly slow down the pipelines and can even cause
+ them to timeout.
- Try to produce useful messages when a test assertion fails. The assertion
message should tell what went wrong, with no need to run the *test itself*
under a debugger (e.g., if the assertion is within a loop, and the loop
.. _Google Test Primer: http://code.google.com/p/googletest/wiki/V1_7_Primer
.. _Google Mock: http://code.google.com/p/googlemock/
-.. include:: doxygen-links.rst
+.. include:: /fragments/doxygen-links.rst
+
+MPI tests
+---------
+
+If your test makes specific requirements on the number of MPI ranks,
+or needs a communicator as part of its implementation, then there are
+GROMACS-specific extensions that make normal-looking GoogleTests work
+well in these cases. Use ``GMX_TEST_MPI(RankRequirement)`` and declare
+the test with ``gmx_add_mpi_unit_test`` to teach ``CTest`` how to run
+the test regardless of whether the build is with thread-MPI or real
+MPI. See ``src/testutils/include/mpitest.h`` for details.
biod.pnpi.spb.ru / alexxy / gromacs.git / blobdiff