4 The repository contains DockerFiles and GitLab Runner configuration
5 files to support automated testing and documentation builds.
6 General information on configuring GitLab CI pipelines can be found
7 in the official `Gitlab documentation <https://docs.gitlab.com/ee/ci/yaml/>`_.
9 The GitLab CI configuration entry point is the :file:`.gitlab-ci.yml` file
10 at the root of the source tree.
11 Configuration templates are found in the files in the
12 :file:`admin/ci-templates/` directory.
14 Docker images used by GitLab Runner are available on `Docker Hub <https://hub.docker.com/u/gromacs>`__.
15 Images are (re)built manually using details in :file:`admin/containers`.
17 This documentation is incomplete, pending resolution of :issue:`3275`.
19 .. todo:: Expand this documentation to resolve :issue:`3275`
24 .. todo:: Discuss the distinct characteristics of |Gromacs| CI pipelines to relevant to job configuration.
26 .. todo:: Comment on the number of pipelines that can be or which are likely to be running at the same time.
30 Full automated testing is only available for merge requests originating from
31 branches of the main https://gitlab.com/gromacs/gromacs repository.
32 GitLab CI pipelines created for forked repositories will include fewer jobs
33 in the testing pipeline. Non-trivial merge requests may need to be issued
34 from a branch in the ``gromacs`` project namespace in order to receive
35 sufficient testing before acceptance.
40 At the root of the repository, :file:`.gitlab-ci.yml` defines the stages and
41 some default parameters, then includes files from :file:`admin/gitlab-ci/` to
42 define jobs to be executed in the pipelines.
44 Note that job names beginning with a period (``.``) are
45 `"hidden" <https://docs.gitlab.com/ee/ci/yaml/#hidden-keys-jobs>`_.
46 Such jobs are not directly eligible to run, but may be used as templates
47 via the `*extends* job property <https://docs.gitlab.com/ee/ci/yaml/#extends>`_.
52 Refer to https://docs.gitlab.com/ee/ci/yaml for complete documentation on
53 GitLab CI job parameters, but note the following GROMACS-specific conventions.
58 Used by several of our templates to prepend shell commands to
59 a job *script* parameter.
60 Avoid using *before-script* directly, and be cautious
61 about nested *extends* overriding multiple *before_script* definitions.
64 There is no global default, but jobs that build software will likely
65 set *cache*. To explicitly unset *cache* directives, specify a job
66 parameter of ``cache: {}``.
67 Refer to `GitLab docs <https://docs.gitlab.com/ee/ci/yaml/#cache>`__
68 for details. In particular, note the details of cache identity according
69 to `cache:key <https://docs.gitlab.com/ee/ci/yaml/#cachekey>`__
72 Part of the tool chain configuration. Instead of setting *image*
73 directly, *extend* a *.use_<toolchain>* template from
74 :file:`admin/gitlab-ci/global.gitlab-ci.yml`
80 *Job* parameters for controlling the circumstances under which jobs run.
81 (Some key words may have different meanings when occurring as elements
82 of other parameters, such as *archive:when*, to which this note is not
84 Instead of setting any of these directly in a job definition, try to use
85 one of the pre-defined behaviors (defined as ``.rules:<something>`` in
86 :file:`admin/gitlab-ci/rules.gitlab-ci.yml`).
87 Errors or unexpected behavior will occur if you specify more than one
88 *.rules:...* template, or if you use these parameters in combination
89 with a *.rules...* template.
90 To reduce errors and unexpected behavior, restrict usage of these controls
91 to regular job definitions (don't use in "hidden" or parent jobs).
92 Note that *rules* is not compatible with the older *only* and *except*
93 parameters. We have standardized on the (newer) *rules* mechanism.
96 Jobs that can only run in the |Gromacs| GitLab CI Runner infrastructure
97 should require the ``k8s-scilifelab`` tag.
98 These include jobs that specify Kubernetes configuration variables or
99 require special facilities, such as GPUs or MPI.
100 Note that the *tag* controls which Runners are eligible to take a job.
101 It does not affect whether the job is eligible for addition to a particular pipeline.
102 Additional *rules* logic should be used to make sure that jobs with the
103 ``k8s-scilifelab`` do not become eligible for pipelines launched outside
104 of the |Gromacs| project environment.
105 See, for instance, :term:`CI_PROJECT_NAMESPACE`
108 Many job definitions will add or override keys in *variables*.
109 Refer to `GitLab <https://docs.gitlab.com/ee/ci/yaml/#variables>`__
110 for details of the merging behavior. Refer to :ref:`variables` for local usage.
112 Schedules and triggers
113 ~~~~~~~~~~~~~~~~~~~~~~
115 Pipeline `schedules <https://gitlab.com/help/ci/pipelines/schedules>`__ are
116 configured through the GitLab web interface.
117 Scheduled pipelines may provide different variable definitions through the
118 environment to jobs that run under the ``schedules``
119 `condition <https://gitlab.com/help/ci/pipelines/schedules#using-only-and-except>`__.
121 Nightly scheduled pipelines run against ``master`` and *release* branches in
122 the GROMACS repository.
124 Running post-merge-acceptance pipelines
125 """""""""""""""""""""""""""""""""""""""
127 The Gitlab CI for |Gromacs| runs a set of jobs by default only after a MR has been
128 accepted and the resulting commit is included in the target branch if it is ``master``
129 or one of the *release* branches. Those jobs can be triggered manually using the
130 ``POST_MERGE_ACCEPTANCE`` input variable documented below when executing a new pipeline
131 through the Gitlab web interface.
136 In addition to the templates in the main job definition files,
137 common "mix-in" functionality and behavioral templates are defined in
138 :file:`admin/gitlab-ci/global.gitlab-ci.yml`.
139 For readability, some parameters may be separated into their own files, named
140 according to the parameter (e.g. :file:`rules.gitlab-ci.yml`).
142 Jobs beginning with ``.use-`` provide mix-in behavior, such as boilerplate for
143 jobs using a particular tool chain.
145 Jobs beginning with a `parameter <https://docs.gitlab.com/ee/ci/yaml>`__
146 name allow parameters to be set in a single place for common job characteristics.
147 If providing more than a default parameter value, the job name should be suffixed
148 by a meaningful descriptor and documented within
149 :file:`admin/gitlab-ci/global.gitlab-ci.yml`
156 1. Indicate the purpose of the job.
157 2. Indicate relationships between multi-stage tasks.
158 3. Distinguish jobs in the same stage.
159 4. Distinguish job definitions throughout the configuration.
161 Jobs may be reassigned to different stages over time, so including the stage
162 name in the job name is not helpful, generally. If tags like "pre" and "post,"
163 or "build" and "test" are necessary to distinguish phases of, say, "webpage,"
164 then such tags can be buried at the end of the job name.
166 Stylistically, it is helpful to use delimiters like ``:`` to distinguish the
167 basic job name from qualifiers or details. Also consider
168 `grouping jobs <https://docs.gitlab.com/ee/ci/pipelines/index.html#grouping-jobs>`__
172 Updating regression tests
173 ~~~~~~~~~~~~~~~~~~~~~~~~~
175 Changes in |Gromacs| that require changes in regression-tests are notoriously hard,
176 because a merge request that tests against the non-updated version of the
177 regression tests will necessarily fail, while updating regression tests while
178 the current change is not integrated into master, might cause other
179 merge request pipelines to fail.
181 The solution is a new regression-test branch or commit, uploaded to gitlab.
182 Then set that regression test branch with REGRESSIONTESTBRANCH or
183 the specific commit with REGRESSIONTESTCOMMIT when
184 running the specific pipeline that requires the regressiontest-update.
185 See below on how to set variables for specific pipelines.
190 The GitLab CI framework, GitLab Runner, plugins, and our own scripts set and
191 use several `variables <https://docs.gitlab.com/ee/ci/variables/README.html>`__.
193 Default values are available from the ``.variables:default`` definition in
194 :file:`admin/gitlab-ci/global.gitlab-ci.yml`.
195 Many of the mix-in / template jobs provide additional or overriding definitions.
196 Other variables may be set when making final job definitions.
198 Variables may control the behvior of GitLab-CI (those beginning with ``CI_``),
199 GitLab Runner and supporting infrastructure, or may be used by job definitions,
200 or passed along to the environment of executed commands.
202 *variables* keys beginning with ``KUBERNETES_`` relate to the GitLab Runner
203 `Kubernets executor <https://docs.gitlab.com/runner/executors/kubernetes.html#the-kubernetes-executor>`__
205 Other important variable keys are as follows.
209 Distinguishes pipelines created for repositories in the ``gromacs``
210 GitLab project space. May be used to pre-screen jobs to determine
211 whether |Gromacs| GitLab infrastructure is available to the pipeline
212 before the job is created.
214 COMPILER_MAJOR_VERSION
215 Integer version number provided by toolchain mix-in for convenience and
218 CMAKE_COMPILER_SCRIPT
219 CMake command line options for a tool chain. A definition is provided by
220 the mix-in toolchain definitions (e.g. ``.use-gcc8``) to be appended to
221 :command:`cmake` calls in a job's *script*.
224 Provide CMake command line arguments to define GROMACS MPI build options.
227 Read-only environment variable that can be checked to see if a job is
228 executing in a pipeline for preparing a tagged release.
229 Can be set when launching pipelines via the GitLab web interface.
230 For example, see *rules* mix-ins in :file:`admin/gitlab-ci/global.gitlab-ci.yml`.
233 List additional OS package requirements. Used in *before_script* for some
234 mix-in job definitions to install additional software dependencies. If
235 using such a job with *extends*, override this variable key with a
236 space-delimited list of packages (default: ``""``). Consider proposing a
237 patch to the base Docker images to include the dependency to reduce
238 pipeline execution time.
241 Use this branch of the regressiontests rather than master to allow for
242 merge requests that require updated regression tests with valid CI tests.
245 Use this commit to the regressiontests rather than the head on master to
246 allow for merge requests that require updated regression tests with
249 POST_MERGE_ACCEPTANCE
250 Read-only environment variable that indicates that only jobs scheduled to
251 run after a commit has been merged into its target branch should be executed.
252 Can be set to run pipelines through the web interface or as schedules.
253 For use please see the *rules* mix-ins in :file:`admin/gitlab-ci/global.gitlab-ci.yml`.
256 .. todo:: Define common variables.
257 ``BUILD_DIR``, ``INSTALL_DIR``, ``CACHE_FALLBACK_KEY``, ...
262 Variables for individual piplelines are set in the gitlab interface under
263 ``CI/CD``; ``Pipelines``. Then chose in the top right corner ``Run Piplelines``.
264 Under ``Run for``, the desired branch may be selected, and variables may be set