3 Getting good performance from :ref:`mdrun <gmx mdrun>`
4 ======================================================
6 Here we give an overview on the parallelization and acceleration schemes employed by |Gromacs|.
7 The aim is to provide an understanding of the underlying mechanisms that make |Gromacs| one of the
8 fastest molecular dynamics packages. The information presented
9 should help choosing appropriate parallelization options, run configuration,
10 as well as acceleration options to achieve optimal simulation performance.
13 The |Gromacs| build system and the :ref:`gmx mdrun` tool have a lot of built-in
14 and configurable intelligence to detect your hardware and make pretty
15 effective use of it. For a lot of casual and serious use of
16 :ref:`gmx mdrun`, the automatic machinery works well enough. But to get the
17 most from your hardware to maximize your scientific quality, read on!
19 Hardware background information
20 -------------------------------
21 Modern computer hardware is complex and heterogeneous, so we need to
22 discuss a little bit of background information and set up some
23 definitions. Experienced HPC users can skip this section.
28 A hardware compute unit that actually executes
29 instructions. There is normally more than one core in a
30 processor, often many more.
33 A special kind of memory local to core(s) that is much faster
34 to access than main memory, kind of like the top of a human's
35 desk, compared to their filing cabinet. There are often
36 several layers of caches associated with a core.
39 A group of cores that share some kind of locality, such as a
40 shared cache. This makes it more efficient to spread
41 computational work over cores within a socket than over cores
42 in different sockets. Modern processors often have more than
46 A group of sockets that share coarser-level locality, such as
47 shared access to the same memory without requiring any network
48 hardware. A normal laptop or desktop computer is a node. A
49 node is often the smallest amount of a large compute cluster
50 that a user can request to use.
53 A stream of instructions for a core to execute. There are many
54 different programming abstractions that create and manage
55 spreading computation over multiple threads, such as OpenMP,
56 pthreads, winthreads, CUDA, OpenCL, and OpenACC. Some kinds of
57 hardware can map more than one software thread to a core; on
58 Intel x86 processors this is called "hyper-threading", while
59 the more general concept is often called SMT for
60 "simultaneous multi-threading". IBM Power8 can for instance use
61 up to 8 hardware threads per core.
62 This feature can usually be enabled or disabled either in
63 the hardware bios or through a setting in the Linux operating
64 system. |Gromacs| can typically make use of this, for a moderate
65 free performance boost. In most cases it will be
66 enabled by default e.g. on new x86 processors, but in some cases
67 the system administrators might have disabled it. If that is the
68 case, ask if they can re-enable it for you. If you are not sure
69 if it is enabled, check the output of the CPU information in
70 the log file and compare with CPU specifications you find online.
72 thread affinity (pinning)
73 By default, most operating systems allow software threads to migrate
74 between cores (or hardware threads) to help automatically balance
75 workload. However, the performance of :ref:`gmx mdrun` can deteriorate
76 if this is permitted and will degrade dramatically especially when
77 relying on multi-threading within a rank. To avoid this,
78 :ref:`gmx mdrun` will by default
79 set the affinity of its threads to individual cores/hardware threads,
80 unless the user or software environment has already done so
81 (or not the entire node is used for the run, i.e. there is potential
83 Setting thread affinity is sometimes called thread "pinning".
86 The dominant multi-node parallelization-scheme, which provides
87 a standardized language in which programs can be written that
88 work across more than one node.
91 In MPI, a rank is the smallest grouping of hardware used in
92 the multi-node parallelization scheme. That grouping can be
93 controlled by the user, and might correspond to a core, a
94 socket, a node, or a group of nodes. The best choice varies
95 with the hardware, software and compute task. Sometimes an MPI
96 rank is called an MPI process.
99 A graphics processing unit, which is often faster and more
100 efficient than conventional processors for particular kinds of
101 compute workloads. A GPU is always associated with a
102 particular node, and often a particular socket within that
106 A standardized technique supported by many compilers to share
107 a compute workload over multiple cores. Often combined with
108 MPI to achieve hybrid MPI/OpenMP parallelism.
111 A proprietary parallel computing framework and API developed by NVIDIA
112 that allows targeting their accelerator hardware.
113 |Gromacs| uses CUDA for GPU acceleration support with NVIDIA hardware.
116 An open standard-based parallel computing framework that consists
117 of a C99-based compiler and a programming API for targeting heterogeneous
118 and accelerator hardware. |Gromacs| uses OpenCL for GPU acceleration
119 on AMD devices (both GPUs and APUs) and Intel integrated GPUs; NVIDIA
120 hardware is also supported.
123 A type of CPU instruction by which modern CPU cores can execute multiple
124 floating-point instructions in a single cycle.
127 Work distribution by parallelization in |Gromacs|
128 -------------------------------------------------
130 The algorithms in :ref:`gmx mdrun` and their implementations are most relevant
131 when choosing how to make good use of the hardware. For details,
132 see the :ref:`Reference Manual <gmx-reference-manual-rst>`. The most important of these are
134 .. _gmx-domain-decomp:
139 The domain decomposition (DD) algorithm decomposes the
140 (short-ranged) component of the non-bonded interactions into
141 domains that share spatial locality, which permits the use of
142 efficient algorithms. Each domain handles all of the
143 particle-particle (PP) interactions for its members, and is
144 mapped to a single MPI rank. Within a PP rank, OpenMP threads
145 can share the workload, and some work can be offloaded to a
146 GPU. The PP rank also handles any bonded interactions for the
147 members of its domain. A GPU may perform work for more than
148 one PP rank, but it is normally most efficient to use a single
149 PP rank per GPU and for that rank to have thousands of
150 particles. When the work of a PP rank is done on the CPU,
151 :ref:`mdrun <gmx mdrun>` will make extensive use of the SIMD
152 capabilities of the core. There are various
153 :ref:`command-line options <controlling-the-domain-decomposition-algorithm>`
154 to control the behaviour of the DD algorithm.
157 The particle-mesh Ewald (PME) algorithm treats the long-ranged
158 component of the non-bonded interactions (Coulomb and possibly also
159 Lennard-Jones). Either all, or just a subset of ranks may
160 participate in the work for computing the long-ranged component
161 (often inaccurately called simply the "PME"
162 component). Because the algorithm uses a 3D FFT that requires
163 global communication, its parallel efficiency gets worse as more ranks
164 participate, which can mean it is fastest to use just a subset
165 of ranks (e.g. one-quarter to one-half of the ranks). If
166 there are separate PME ranks, then the remaining ranks handle
167 the PP work. Otherwise, all ranks do both PP and PME work.
169 Parallelization schemes
170 -----------------------
172 |Gromacs|, being performance-oriented, has a strong focus on efficient parallelization.
173 There are multiple parallelization schemes available, therefore a simulation can be run on a
174 given hardware with different choices of run configuration.
176 .. _intra-core-parallelization:
178 Intra-core parallelization via SIMD: SSE, AVX, etc.
179 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
181 One level of performance improvement available in |Gromacs| is through the use of
182 ``Single Instruction Multiple Data (SIMD)`` instructions. In detail information
183 for those can be found under :ref:`SIMD support <gmx-simd-support>` in the installation
186 In |Gromacs|, SIMD instructions are used to parallelize the parts of the code with
187 the highest impact on performance (nonbonded and bonded force calculation,
188 PME and neighbour searching), through the use of hardware specific SIMD kernels.
189 Those form one of the three levels of non-bonded kernels that are available: reference or generic
190 kernels (slow but useful for producing reference values for testing),
191 optimized plain-C kernels (can be used cross-platform but still slow)
192 and SIMD intrinsics accelerated kernels.
194 The SIMD intrinsic code is compiled by the compiler.
195 Technically, it is possible to compile different levels of acceleration into one binary,
196 but this is difficult to manage with acceleration in many parts of the code.
197 Thus, you need to configure and compile |Gromacs| for the SIMD capabilities of the target CPU.
198 By default, the build system will detect the highest supported
199 acceleration of the host where the compilation is carried out. For cross-compiling for
200 a machine with a different highest SIMD instructions set, in order to set the target acceleration,
201 the ``-DGMX_SIMD`` CMake option can be used.
203 installation on multiple different machines, it is convenient to compile the analysis tools with
204 the lowest common SIMD instruction set (as these rely little on SIMD acceleration), but for best
205 performance :ref:`mdrun <gmx mdrun>` should be compiled be compiled separately with the
206 highest (latest) ``native`` SIMD instruction set of the target architecture (supported by |Gromacs|).
208 Recent Intel CPU architectures bring tradeoffs between the maximum clock frequency of the
209 CPU (ie. its speed), and the width of the SIMD instructions it executes (ie its throughput
210 at a given speed). In particular, the Intel ``Skylake`` and ``Cascade Lake`` processors
211 (e.g. Xeon SP Gold/Platinum), can offer better throughput when using narrower SIMD because
212 of the better clock frequency available. Consider building :ref:`mdrun <gmx mdrun>`
213 configured with ``GMX_SIMD=AVX2_256`` instead of ``GMX_SIMD=AVX512`` for better
214 performance in GPU accelerated or highly parallel MPI runs.
216 Some of the latest ARM based CPU, such as the Fujitsu A64fx, support the Scalable Vector Extensions (SVE).
217 Though SVE can be used to generate fairly efficient Vector Length Agnostic (VLA) code,
218 this is not a good fit for |Gromacs| (as the SIMD vector length assumed to be known at
219 CMake time). Consequently, the SVE vector length must be fixed at CMake time. The default
220 is to automatically detect the default vector length at CMake time
221 (via the ``/proc/sys/abi/sve_default_vector_length`` pseudo-file, and this can be changed by
222 configuring with ``GMX_SIMD_ARM_SVE_LENGTH=<len>``.
223 The supported vector lengths are 128, 256, 512 and 1024. Since the SIMD short-range non-bonded kernels
224 only support up to 16 floating point numbers per SIMD vector, 1024 bits vector length is only
225 valid in double precision (e.g. ``-DGMX_DOUBLE=on``).
226 Note that even if `mdrun` does check the SIMD vector length at runtime, running with a different
227 vector length than the one used at CMake time is undefined behavior, and `mdrun` might crash before reaching
228 the check (that would abort with a user-friendly error message).
230 Process(-or) level parallelization via OpenMP
231 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
233 |Gromacs| :ref:`mdrun <gmx mdrun>` supports OpenMP multithreading for all parts
234 of the code. OpenMP is enabled by default and
235 can be turned on/off at configure time with the ``GMX_OPENMP`` CMake variable
236 and at run-time with the ``-ntomp`` option (or the ``OMP_NUM_THREADS`` environment variable).
237 The OpenMP implementation is quite efficient and scales well for up to 12-24 threads on
238 Intel and 6-8 threads on AMD CPUs.
240 Node level parallelization via GPU offloading and thread-MPI
241 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
243 Multithreading with thread-MPI
244 ..............................
246 The thread-MPI library implements a subset of the MPI 1.1 specification,
247 based on the system threading support. Both POSIX pthreads and Windows threads are supported,
248 thus providing great portability to most UNIX/Linux and Windows operating systems.
249 Acting as a drop-in replacement for MPI, thread-MPI enables compiling and running :ref:`mdrun <gmx mdrun>`
250 on a single machine (i.e. not across a network) without MPI. Additionally, it not only provides a
251 convenient way to use computers with multicore CPU(s), but thread-MPI does in some
252 cases make :ref:`mdrun <gmx mdrun>` run slightly faster than with MPI.
254 Thread-MPI is included in the |Gromacs| source and it is the default parallelization since
255 version 4.5, practically rendering the serial :ref:`mdrun <gmx mdrun>` deprecated.
256 Compilation with thread-MPI is controlled by the ``GMX_THREAD_MPI`` CMake variable.
258 Thread-MPI is compatible with most :ref:`mdrun <gmx mdrun>` features and parallelization schemes,
259 including OpenMP, GPUs; it is not compatible with MPI and multi-simulation runs.
261 By default, the thread-MPI mdrun will use all available cores in the machine by starting
262 an appropriate number of ranks or OpenMP threads to occupy all of them. The number of
263 ranks can be controlled using the
264 ``-nt`` and ``-ntmpi`` options. ``-nt`` represents the total number of threads
265 to be used (which can be a mix of thread-MPI and OpenMP threads.
267 Hybrid/heterogeneous acceleration
268 .................................
270 Hybrid acceleration means distributing compute work between available CPUs and GPUs
271 to improve simulation performance. New non-bonded algorithms
272 have been developed with the aim of efficient acceleration both on CPUs and GPUs.
274 The most compute-intensive parts of simulations, non-bonded force calculation, as well
275 as possibly the PME, bonded force calculation and update and constraints can be
276 offloaded to GPUs and carried out simultaneously with remaining CPU work.
277 Native GPU acceleration is supported for the most commonly used algorithms in
279 For more information about the GPU kernels, please see the :ref:`Installation guide <gmx-gpu-support>`.
281 The native GPU acceleration can be turned on or off, either at run-time using the
282 :ref:`mdrun <gmx mdrun>` ``-nb`` option, or at configuration time using the ``GMX_GPU`` CMake variable.
284 To efficiently use all compute resource available, CPU and GPU computation is done simultaneously.
285 Overlapping with the OpenMP multithreaded bonded force and PME long-range electrostatic calculations
286 on the CPU, non-bonded forces are calculated on the GPU. Multiple GPUs, both in a single node as
287 well as across multiple nodes, are supported using domain-decomposition. A single GPU is assigned
288 to the non-bonded workload of a domain, therefore, the number GPUs used has to match the number
289 of of MPI processes (or thread-MPI threads) the simulation is started with. The available
290 CPU cores are partitioned among the processes (or thread-MPI threads) and a set of cores
291 with a GPU do the calculations on the respective domain.
293 With PME electrostatics, :ref:`mdrun <gmx mdrun>` supports automated CPU-GPU load-balancing by
294 shifting workload from the PME mesh calculations, done on the CPU, to the particle-particle
295 non-bonded calculations, done on the GPU. At startup a few iterations of tuning are executed
296 during the first 100 to 1000 MD steps. These iterations involve scaling the electrostatics cut-off
297 and PME grid spacing to determine the value that gives optimal CPU-GPU load balance. The cut-off
298 value provided using the :mdp:`rcoulomb` ``=rvdw`` :ref:`mdp` option represents the minimum
299 electrostatics cut-off the tuning starts with and therefore should be chosen as small as
300 possible (but still reasonable for the physics simulated). The Lennard-Jones cut-off ``rvdw``
301 is kept fixed. We don't allow scaling to shorter cut-off as we don't want to change ``rvdw``
302 and there would be no performance gain.
304 While the automated CPU-GPU load balancing always attempts to find the optimal cut-off setting,
305 it might not always be possible to balance CPU and GPU workload. This happens when the CPU threads
306 finish calculating the bonded forces and PME faster than the GPU the non-bonded force calculation,
307 even with the shortest possible cut-off. In such cases the CPU will wait for the GPU and this
308 time will show up as ``Wait GPU local`` in the cycle and timing summary table at the end
311 Parallelization over multiple nodes via MPI
312 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
314 At the heart of the MPI parallelization in |Gromacs| is the neutral-territory
315 :ref:`domain decomposition <gmx-domain-decomp>` with dynamic load balancing.
316 To parallelize simulations across multiple machines (e.g. nodes of a cluster)
317 :ref:`mdrun <gmx mdrun>` needs to be compiled with MPI which can be enabled using the ``GMX_MPI`` CMake variable.
319 .. _controlling-the-domain-decomposition-algorithm:
321 Controlling the domain decomposition algorithm
322 ..............................................
324 This section lists options that affect how the domain
325 decomposition algorithm decomposes the workload to the available
329 Can be used to set the required maximum distance for inter
330 charge-group bonded interactions. Communication for two-body
331 bonded interactions below the non-bonded cut-off distance always
332 comes for free with the non-bonded communication. Particles beyond
333 the non-bonded cut-off are only communicated when they have
334 missing bonded interactions; this means that the extra cost is
335 minor and nearly independent of the value of ``-rdd``. With dynamic
336 load balancing, option ``-rdd`` also sets the lower limit for the
337 domain decomposition cell sizes. By default ``-rdd`` is determined
338 by :ref:`gmx mdrun` based on the initial coordinates. The chosen value will
339 be a balance between interaction range and communication cost.
342 On by default. When inter charge-group bonded interactions are
343 beyond the bonded cut-off distance, :ref:`gmx mdrun` terminates with an
344 error message. For pair interactions and tabulated bonds that do
345 not generate exclusions, this check can be turned off with the
346 option ``-noddcheck``.
349 When constraints are present, option ``-rcon`` influences
350 the cell size limit as well.
351 Particles connected by NC constraints, where NC is the LINCS order
352 plus 1, should not be beyond the smallest cell size. A error
353 message is generated when this happens, and the user should change
354 the decomposition or decrease the LINCS order and increase the
355 number of LINCS iterations. By default :ref:`gmx mdrun` estimates the
356 minimum cell size required for P-LINCS in a conservative
357 fashion. For high parallelization, it can be useful to set the
358 distance required for P-LINCS with ``-rcon``.
361 Sets the minimum allowed x, y and/or z scaling of the cells with
362 dynamic load balancing. :ref:`gmx mdrun` will ensure that the cells can
363 scale down by at least this factor. This option is used for the
364 automated spatial decomposition (when not using ``-dd``) as well as
365 for determining the number of grid pulses, which in turn sets the
366 minimum allowed cell size. Under certain circumstances the value
367 of ``-dds`` might need to be adjusted to account for high or low
368 spatial inhomogeneity of the system.
372 Multi-level parallelization: MPI and OpenMP
373 ...........................................
375 The multi-core trend in CPU development substantiates the need for multi-level parallelization.
376 Current multiprocessor machines can have 2-4 CPUs with a core count as high as 64. As the memory
377 and cache subsystem is lagging more and more behind the multicore evolution, this emphasizes
378 non-uniform memory access (NUMA) effects, which can become a performance bottleneck. At the same
379 time, all cores share a network interface. In a purely MPI-parallel scheme, all MPI processes
380 use the same network interface, and although MPI intra-node communication is generally efficient,
381 communication between nodes can become a limiting factor to parallelization. This is especially
382 pronounced in the case of highly parallel simulations with PME (which is very communication
383 intensive) and with ``''fat''`` nodes connected by a slow network. Multi-level parallelism aims
384 to address the NUMA and communication related issues by employing efficient
385 intra-node parallelism, typically multithreading.
387 Combining OpenMP with MPI creates an additional overhead
388 especially when running separate multi-threaded PME ranks. Depending on the architecture,
389 input system size, as well as other factors, MPI+OpenMP runs can be as fast and faster
390 already at small number of processes (e.g. multi-processor Intel Westmere or Sandy Bridge),
391 but can also be considerably slower (e.g. multi-processor AMD Interlagos machines). However,
392 there is a more pronounced benefit of multi-level parallelization in highly parallel runs.
397 On CPU ranks, particle-particle (PP) and PME calculations are done in the same process one after
398 another. As PME requires all-to-all global communication, this is most of the time the limiting
399 factor to scaling on a large number of cores. By designating a subset of ranks for PME
400 calculations only, performance of parallel runs can be greatly improved.
402 OpenMP mutithreading in PME ranks is also possible.
403 Using multi-threading in PME can can improve performance at high
404 parallelization. The reason for this is that with N>1 threads the number of processes
405 communicating, and therefore the number of messages, is reduced by a factor of N.
406 But note that modern communication networks can process several messages simultaneously,
407 such that it could be advantageous to have more processes communicating.
409 Separate PME ranks are not used at low parallelization, the switch at higher parallelization
410 happens automatically (at > 16 processes). The number of PME ranks is estimated by mdrun.
411 If the PME load is higher than the PP load, mdrun will automatically balance the load, but
412 this leads to additional (non-bonded) calculations. This avoids the idling of a large fraction
413 of the ranks; usually 3/4 of the ranks are PP ranks. But to ensure the best absolute performance
414 of highly parallel runs, it is advisable to tweak this number which is automated by
415 the :ref:`tune_pme <gmx tune_pme>` tool.
417 The number of PME ranks can be set manually on the :ref:`mdrun <gmx mdrun>` command line using the ``-npme``
418 option, the number of PME threads can be specified on the command line with ``-ntomp_pme`` or
419 alternatively using the ``GMX_PME_NUM_THREADS`` environment variable. The latter is especially
420 useful when running on compute nodes with different number of cores as it enables
421 setting different number of PME threads on different nodes.
423 Running :ref:`mdrun <gmx mdrun>` within a single node
424 -----------------------------------------------------
426 :ref:`gmx mdrun` can be configured and compiled in several different ways that
427 are efficient to use within a single :term:`node`. The default configuration
428 using a suitable compiler will deploy a multi-level hybrid parallelism
429 that uses CUDA, OpenMP and the threading platform native to the
430 hardware. For programming convenience, in |Gromacs|, those native
431 threads are used to implement on a single node the same MPI scheme as
432 would be used between nodes, but much more efficient; this is called
433 thread-MPI. From a user's perspective, real MPI and thread-MPI look
434 almost the same, and |Gromacs| refers to MPI ranks to mean either kind,
435 except where noted. A real external MPI can be used for :ref:`gmx mdrun` within
436 a single node, but runs more slowly than the thread-MPI version.
438 By default, :ref:`gmx mdrun` will inspect the hardware available at run time
439 and do its best to make fairly efficient use of the whole node. The
440 log file, stdout and stderr are used to print diagnostics that
441 inform the user about the choices made and possible consequences.
443 A number of command-line parameters are available to modify the default
447 The total number of threads to use. The default, 0, will start as
448 many threads as available cores. Whether the threads are
449 thread-MPI ranks, and/or OpenMP threads within such ranks depends on
453 The total number of thread-MPI ranks to use. The default, 0,
454 will start one rank per GPU (if present), and otherwise one rank
458 The total number of OpenMP threads per rank to start. The
459 default, 0, will start one thread on each available core.
460 Alternatively, :ref:`mdrun <gmx mdrun>` will honor the appropriate system
461 environment variable (e.g. ``OMP_NUM_THREADS``) if set.
462 Note that the maximum number of OpenMP threads (per rank) is,
463 for efficiency reasons, limited to 64. While it is rarely beneficial to use
464 a number of threads higher than this, the GMX_OPENMP_MAX_THREADS CMake variable
465 can be used to increase the limit.
468 The total number of ranks to dedicate to the long-ranged
469 component of PME, if used. The default, -1, will dedicate ranks
470 only if the total number of threads is at least 12, and will use
471 around a quarter of the ranks for the long-ranged component.
474 When using PME with separate PME ranks,
475 the total number of OpenMP threads per separate PME rank.
476 The default, 0, copies the value from ``-ntomp``.
479 Can be set to "auto," "on" or "off" to control whether
480 :ref:`mdrun <gmx mdrun>` will attempt to set the affinity of threads to cores.
481 Defaults to "auto," which means that if :ref:`mdrun <gmx mdrun>` detects that all the
482 cores on the node are being used for :ref:`mdrun <gmx mdrun>`, then it should behave
483 like "on," and attempt to set the affinities (unless they are
484 already set by something else).
487 If ``-pin on``, specifies the logical core number to
488 which :ref:`mdrun <gmx mdrun>` should pin the first thread. When running more than
489 one instance of :ref:`mdrun <gmx mdrun>` on a node, use this option to to avoid
490 pinning threads from different :ref:`mdrun <gmx mdrun>` instances to the same core.
493 If ``-pin on``, specifies the stride in logical core
494 numbers for the cores to which :ref:`mdrun <gmx mdrun>` should pin its threads. When
495 running more than one instance of :ref:`mdrun <gmx mdrun>` on a node, use this option
496 to avoid pinning threads from different :ref:`mdrun <gmx mdrun>` instances to the
497 same core. Use the default, 0, to minimize the number of threads
498 per physical core - this lets :ref:`mdrun <gmx mdrun>` manage the hardware-, OS- and
499 configuration-specific details of how to map logical cores to
503 Can be set to "interleave," "pp_pme" or "cartesian."
504 Defaults to "interleave," which means that any separate PME ranks
505 will be mapped to MPI ranks in an order like PP, PP, PME, PP, PP,
506 PME, etc. This generally makes the best use of the available
507 hardware. "pp_pme" maps all PP ranks first, then all PME
508 ranks. "cartesian" is a special-purpose mapping generally useful
509 only on special torus networks with accelerated global
510 communication for Cartesian communicators. Has no effect if there
511 are no separate PME ranks.
514 Used to set where to execute the short-range non-bonded interactions.
515 Can be set to "auto", "cpu", "gpu."
516 Defaults to "auto," which uses a compatible GPU if available.
517 Setting "cpu" requires that no GPU is used. Setting "gpu" requires
518 that a compatible GPU is available and will be used.
521 Used to set where to execute the long-range non-bonded interactions.
522 Can be set to "auto", "cpu", "gpu."
523 Defaults to "auto," which uses a compatible GPU if available.
524 Setting "gpu" requires that a compatible GPU is available.
525 Multiple PME ranks are not supported with PME on GPU, so if a GPU is used
526 for the PME calculation -npme must be set to 1.
529 Used to set where to execute the bonded interactions that are part of the
530 PP workload for a domain.
531 Can be set to "auto", "cpu", "gpu."
532 Defaults to "auto," which uses a compatible CUDA GPU only when one
533 is available, a GPU is handling short-ranged interactions, and the
534 CPU is handling long-ranged interaction work (electrostatic or
535 LJ). The work for the bonded interactions takes place on the same
536 GPU as the short-ranged interactions, and cannot be independently
538 Setting "gpu" requires that a compatible GPU is available and will
542 Used to set where to execute update and constraints, when present.
543 Can be set to "auto", "cpu", "gpu."
544 Defaults to "auto," which currently always uses the CPU.
545 Setting "gpu" requires that a compatible CUDA GPU is available,
546 the simulation uses a single rank.
547 Update and constraints on a GPU is currently not supported
548 with mass and constraints free-energy perturbation, domain
549 decomposition, virtual sites, Ewald surface correction,
550 replica exchange, constraint pulling, orientation restraints
551 and computational electrophysiology.
554 A string that specifies the ID numbers of the GPUs that
555 are available to be used by ranks on each node. For example,
556 "12" specifies that the GPUs with IDs 1 and 2 (as reported
557 by the GPU runtime) can be used by :ref:`mdrun <gmx mdrun>`. This is useful
558 when sharing a node with other computations, or if a GPU that
559 is dedicated to a display should not be used by |Gromacs|.
560 Without specifying this parameter, :ref:`mdrun <gmx mdrun>`
561 will utilize all GPUs. When many GPUs are
562 present, a comma may be used to separate the IDs, so
563 "12,13" would make GPUs 12 and 13 available to :ref:`mdrun <gmx mdrun>`.
564 It could be necessary to use different GPUs on different
565 nodes of a simulation, in which case the environment
566 variable ``GMX_GPU_ID`` can be set differently for the ranks
567 on different nodes to achieve that result.
568 In |Gromacs| versions preceding 2018 this parameter used to
569 specify both GPU availability and GPU task assignment.
570 The latter is now done with the ``-gputasks`` parameter.
573 A string that specifies the ID numbers of the GPUs to be
574 used by corresponding GPU tasks on this node. For example,
575 "0011" specifies that the first two GPU tasks will use GPU 0,
576 and the other two use GPU 1. When using this option, the
577 number of ranks must be known to :ref:`mdrun <gmx mdrun>`, as well as where
578 tasks of different types should be run, such as by using
579 ``-nb gpu`` - only the tasks which are set to run on GPUs
580 count for parsing the mapping. See `Assigning tasks to GPUs`_
581 for more details. Note that ``-gpu_id`` and
582 ``-gputasks`` can not be used at the same time!
583 In |Gromacs| versions preceding 2018 only a single type
584 of GPU task ("PP") could be run on any rank. Now that there is some
585 support for running PME on GPUs, the number of GPU tasks
586 (and the number of GPU IDs expected in the ``-gputasks`` string)
587 can actually be 3 for a single-rank simulation. The IDs
588 still have to be the same in this case, as using multiple GPUs
589 per single rank is not yet implemented.
590 The order of GPU tasks per rank in the string is PP first,
591 PME second. The order of ranks with different kinds of GPU tasks
592 is the same by default, but can be influenced with the ``-ddorder``
593 option and gets quite complex when using multiple nodes.
594 Note that the bonded interactions for a PP task may
595 run on the same GPU as the short-ranged work, or on the CPU,
596 which can be controlled with the ``-bonded`` flag.
597 The GPU task assignment (whether manually set, or automated),
598 will be reported in the :ref:`mdrun <gmx mdrun>` output on
599 the first physical node of the simulation. For example:
603 gmx mdrun -gputasks 0001 -nb gpu -pme gpu -npme 1 -ntmpi 4
605 will produce the following output in the log file/terminal:
609 On host tcbl14 2 GPUs selected for this run.
610 Mapping of GPU IDs to the 4 GPU tasks in the 4 ranks on this node:
613 In this case, 3 ranks are set by user to compute PP work
614 on GPU 0, and 1 rank to compute PME on GPU 1.
615 The detailed indexing of the GPUs is also reported in the log file.
617 For more information about GPU tasks, please refer to
618 :ref:`Types of GPU tasks<gmx-gpu-tasks>`.
621 Allows choosing whether to execute the 3D FFT computation on a CPU or GPU.
622 Can be set to "auto", "cpu", "gpu.".
623 When PME is offloaded to a GPU ``-pmefft gpu`` is the default,
624 and the entire PME calculation is executed on the GPU. However,
625 in some cases, e.g. with a relatively slow or older generation GPU
626 combined with fast CPU cores in a run, moving some work off of the GPU
627 back to the CPU by computing FFTs on the CPU can improve performance.
629 .. _gmx-mdrun-single-node:
631 Examples for :ref:`mdrun <gmx mdrun>` on one node
632 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
638 Starts :ref:`mdrun <gmx mdrun>` using all the available resources. :ref:`mdrun <gmx mdrun>`
639 will automatically choose a fairly efficient division
640 into thread-MPI ranks, OpenMP threads and assign work
641 to compatible GPUs. Details will vary with hardware
642 and the kind of simulation being run.
648 Starts :ref:`mdrun <gmx mdrun>` using 8 threads, which might be thread-MPI
649 or OpenMP threads depending on hardware and the kind
650 of simulation being run.
654 gmx mdrun -ntmpi 2 -ntomp 4
656 Starts :ref:`mdrun <gmx mdrun>` using eight total threads, with two thread-MPI
657 ranks and four OpenMP threads per rank. You should only use
658 these options when seeking optimal performance, and
659 must take care that the ranks you create can have
660 all of their OpenMP threads run on the same socket.
661 The number of ranks should be a multiple of the number of
662 sockets, and the number of cores per node should be
663 a multiple of the number of threads per rank.
667 gmx mdrun -ntmpi 4 -nb gpu -pme cpu
669 Starts :ref:`mdrun <gmx mdrun>` using four thread-MPI ranks. The CPU
670 cores available will be split evenly between the ranks using OpenMP
671 threads. The long-range component of the forces are calculated on
672 CPUs. This may be optimal on hardware where the CPUs are relatively
673 powerful compared to the GPUs. The bonded part of force calculation
674 will automatically be assigned to the GPU, since the long-range
675 component of the forces are calculated on CPU(s).
679 gmx mdrun -ntmpi 1 -nb gpu -pme gpu -bonded gpu -update gpu
681 Starts :ref:`mdrun <gmx mdrun>` using a single thread-MPI rank that
682 will use all available CPU cores. All interaction types that can run
683 on a GPU will do so. This may be optimal on hardware where the CPUs
684 are extremely weak compared to the GPUs.
688 gmx mdrun -ntmpi 4 -nb gpu -pme cpu -gputasks 0011
690 Starts :ref:`mdrun <gmx mdrun>` using four thread-MPI ranks, and maps them
691 to GPUs with IDs 0 and 1. The CPU cores available will be split evenly between
692 the ranks using OpenMP threads, with the first two ranks offloading short-range
693 nonbonded force calculations to GPU 0, and the last two ranks offloading to GPU 1.
694 The long-range component of the forces are calculated on CPUs. This may be optimal
695 on hardware where the CPUs are relatively powerful compared to the GPUs.
699 gmx mdrun -ntmpi 4 -nb gpu -pme gpu -npme 1 -gputasks 0001
701 Starts :ref:`mdrun <gmx mdrun>` using four thread-MPI ranks, one of which is
702 dedicated to the long-range PME calculation. The first 3 threads offload their
703 short-range non-bonded calculations to the GPU with ID 0, the 4th (PME) thread
704 offloads its calculations to the GPU with ID 1.
708 gmx mdrun -ntmpi 4 -nb gpu -pme gpu -npme 1 -gputasks 0011
710 Similar to the above example, with 3 ranks assigned to calculating short-range
711 non-bonded forces, and one rank assigned to calculate the long-range forces.
712 In this case, 2 of the 3 short-range ranks offload their nonbonded force
713 calculations to GPU 0. The GPU with ID 1 calculates the short-ranged forces of
714 the 3rd short-range rank, as well as the long-range forces of the PME-dedicated
715 rank. Whether this or the above example is optimal will depend on the capabilities
716 of the individual GPUs and the system composition.
722 Starts :ref:`mdrun <gmx mdrun>` using GPUs with IDs 1 and 2 (e.g. because
723 GPU 0 is dedicated to running a display). This requires
724 two thread-MPI ranks, and will split the available
725 CPU cores between them using OpenMP threads.
729 gmx mdrun -nt 6 -pin on -pinoffset 0 -pinstride 1
730 gmx mdrun -nt 6 -pin on -pinoffset 6 -pinstride 1
732 Starts two :ref:`mdrun <gmx mdrun>` processes, each with six total threads
733 arranged so that the processes affect each other as little as possible by
734 being assigned to disjoint sets of physical cores.
735 Threads will have their affinities set to particular
736 logical cores, beginning from the first and 7th logical cores, respectively. The
737 above would work well on an Intel CPU with six physical cores and
738 hyper-threading enabled. Use this kind of setup only
739 if restricting :ref:`mdrun <gmx mdrun>` to a subset of cores to share a
740 node with other processes.
741 A word of caution: The mapping of logical CPUs/cores to physical
742 cores may differ between operating systems. On Linux,
743 ``cat /proc/cpuinfo`` can be examined to determine this mapping.
747 mpirun -np 2 gmx_mpi mdrun
749 When using an :ref:`gmx mdrun` compiled with external MPI,
750 this will start two ranks and as many OpenMP threads
751 as the hardware and MPI setup will permit. If the
752 MPI setup is restricted to one node, then the resulting
753 :ref:`gmx mdrun` will be local to that node.
755 .. _gmx-mdrun-multiple-nodes:
757 Running :ref:`mdrun <gmx mdrun>` on more than one node
758 ------------------------------------------------------
760 This requires configuring |Gromacs| to build with an external MPI
761 library. By default, this :ref:`mdrun <gmx mdrun>` executable is run with
762 ``gmx_mpi mdrun``. All of the considerations for running single-node
763 :ref:`mdrun <gmx mdrun>` still apply, except that ``-ntmpi`` and ``-nt`` cause a fatal
764 error, and instead the number of ranks is controlled by the
766 Settings such as ``-npme`` are much more important when
767 using multiple nodes. Configuring the MPI environment to
768 produce one rank per core is generally good until one
769 approaches the strong-scaling limit. At that point, using
770 OpenMP to spread the work of an MPI rank over more than one
771 core is needed to continue to improve absolute performance.
772 The location of the scaling limit depends on the processor,
773 presence of GPUs, network, and simulation algorithm, but
774 it is worth measuring at around ~200 particles/core if you
775 need maximum throughput.
777 There are further command-line parameters that are relevant in these
781 Defaults to "on." If "on," a simulation will
782 optimize various aspects of the PME and DD algorithms, shifting
783 load between ranks and/or GPUs to maximize throughput. Some
784 :ref:`mdrun <gmx mdrun>` features are not compatible with this, and these ignore
788 Can be set to "auto," "no," or "yes."
789 Defaults to "auto." Doing Dynamic Load Balancing between MPI ranks
790 is needed to maximize performance. This is particularly important
791 for molecular systems with heterogeneous particle or interaction
792 density. When a certain threshold for performance loss is
793 exceeded, DLB activates and shifts particles between ranks to improve
794 performance. If available, using ``-bonded gpu`` is expected
795 to improve the ability of DLB to maximize performance.
797 During the simulation :ref:`gmx mdrun` must communicate between all
798 PP ranks to compute quantities such as kinetic energy for log file
799 reporting, or perhaps temperature coupling. By default, this happens
800 whenever necessary to honor several :ref:`mdp options <mdp-general>`,
801 so that the period between communication phases is the least common
802 denominator of :mdp:`nstlist`, :mdp:`nstcalcenergy`,
803 :mdp:`nsttcouple`, and :mdp:`nstpcouple`.
805 Note that ``-tunepme`` has more effect when there is more than one
806 :term:`node`, because the cost of communication for the PP and PME
807 ranks differs. It still shifts load between PP and PME ranks, but does
808 not change the number of separate PME ranks in use.
810 Note also that ``-dlb`` and ``-tunepme`` can interfere with each other, so
811 if you experience performance variation that could result from this,
812 you may wish to tune PME separately, and run the result with ``mdrun
813 -notunepme -dlb yes``.
815 The :ref:`gmx tune_pme` utility is available to search a wider
816 range of parameter space, including making safe
817 modifications to the :ref:`tpr` file, and varying ``-npme``.
818 It is only aware of the number of ranks created by
819 the MPI environment, and does not explicitly manage
820 any aspect of OpenMP during the optimization.
822 Examples for :ref:`mdrun <gmx mdrun>` on more than one node
823 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
825 The examples and explanations for for single-node :ref:`mdrun <gmx mdrun>` are
826 still relevant, but ``-ntmpi`` is no longer the way
827 to choose the number of MPI ranks.
831 mpirun -np 16 gmx_mpi mdrun
833 Starts :ref:`gmx mdrun` with 16 ranks, which are mapped to
834 the hardware by the MPI library, e.g. as specified
835 in an MPI hostfile. The available cores will be
836 automatically split among ranks using OpenMP threads,
837 depending on the hardware and any environment settings
838 such as ``OMP_NUM_THREADS``.
842 mpirun -np 16 gmx_mpi mdrun -npme 5
844 Starts :ref:`gmx mdrun` with 16 ranks, as above, and
845 require that 5 of them are dedicated to the PME
850 mpirun -np 11 gmx_mpi mdrun -ntomp 2 -npme 6 -ntomp_pme 1
852 Starts :ref:`gmx mdrun` with 11 ranks, as above, and
853 require that six of them are dedicated to the PME
854 component with one OpenMP thread each. The remaining
855 five do the PP component, with two OpenMP threads
860 mpirun -np 4 gmx_mpi mdrun -ntomp 6 -nb gpu -gputasks 00
862 Starts :ref:`gmx mdrun` on a machine with two nodes, using
863 four total ranks, each rank with six OpenMP threads,
864 and both ranks on a node sharing GPU with ID 0.
868 mpirun -np 8 gmx_mpi mdrun -ntomp 3 -gputasks 0000
870 Using a same/similar hardware as above,
871 starts :ref:`gmx mdrun` on a machine with two nodes, using
872 eight total ranks, each rank with three OpenMP threads,
873 and all four ranks on a node sharing GPU with ID 0.
874 This may or may not be faster than the previous setup
875 on the same hardware.
879 mpirun -np 20 gmx_mpi mdrun -ntomp 4 -gputasks 00
881 Starts :ref:`gmx mdrun` with 20 ranks, and assigns the CPU cores evenly
882 across ranks each to one OpenMP thread. This setup is likely to be
883 suitable when there are ten nodes, each with one GPU, and each node
884 has two sockets each of four cores.
888 mpirun -np 10 gmx_mpi mdrun -gpu_id 1
890 Starts :ref:`gmx mdrun` with 20 ranks, and assigns the CPU cores evenly
891 across ranks each to one OpenMP thread. This setup is likely to be
892 suitable when there are ten nodes, each with two GPUs, but another
893 job on each node is using GPU 0. The job scheduler should set the
894 affinity of threads of both jobs to their allocated cores, or the
895 performance of :ref:`mdrun <gmx mdrun>` will suffer greatly.
899 mpirun -np 20 gmx_mpi mdrun -gpu_id 01
901 Starts :ref:`gmx mdrun` with 20 ranks. This setup is likely
902 to be suitable when there are ten nodes, each with two
903 GPUs, but there is no need to specify ``-gpu_id`` for the
904 normal case where all the GPUs on the node are available
907 Approaching the scaling limit
908 -----------------------------
910 There are several aspects of running a |Gromacs| simulation that are important as the number
911 of atoms per core approaches the current scaling limit of ~100 atoms/core.
913 One of these is that the use of ``constraints = all-bonds`` with P-LINCS
914 sets an artificial minimum on the size of domains. You should reconsider the use
915 of constraints to all bonds (and bear in mind possible consequences on the safe maximum for dt),
916 or change lincs_order and lincs_iter suitably.
918 Finding out how to run :ref:`mdrun <gmx mdrun>` better
919 ------------------------------------------------------
921 The Wallcycle module is used for runtime performance measurement of :ref:`gmx mdrun`.
922 At the end of the log file of each run, the "Real cycle and time accounting" section
923 provides a table with runtime statistics for different parts of the :ref:`gmx mdrun` code
924 in rows of the table.
925 The table contains columns indicating the number of ranks and threads that
926 executed the respective part of the run, wall-time and cycle
927 count aggregates (across all threads and ranks) averaged over the entire run.
928 The last column also shows what precentage of the total runtime each row represents.
929 Note that the :ref:`gmx mdrun` timer resetting functionalities (``-resethway`` and ``-resetstep``)
930 reset the performance counters and therefore are useful to avoid startup overhead and
931 performance instability (e.g. due to load balancing) at the beginning of the run.
933 The performance counters are:
935 * Particle-particle during Particle mesh Ewald
936 * Domain decomposition
937 * Domain decomposition communication load
938 * Domain decomposition communication bounds
939 * Virtual site constraints
940 * Send X to Particle mesh Ewald
942 * Launch GPU operations
943 * Communication of coordinates
945 * Waiting + Communication of force
946 * Particle mesh Ewald
951 * PME 3D-FFT Communication
952 * PME solve Lennard-Jones
955 * PME wait for particle-particle
956 * Wait + Receive PME force
959 * Wait PME GPU spread
960 * Wait PME GPU gather
961 * Reduce PME GPU Force
962 * Non-bonded position/force buffer operations
963 * Virtual site spread
965 * AWH (accelerated weight histogram method)
969 * Communication of energies
971 * Add rotational forces
975 As performance data is collected for every run, they are essential to assessing
976 and tuning the performance of :ref:`gmx mdrun` performance. Therefore, they benefit
977 both code developers as well as users of the program.
978 The counters are an average of the time/cycles different parts of the simulation take,
979 hence can not directly reveal fluctuations during a single run (although comparisons across
980 multiple runs are still very useful).
982 Counters will appear in an MD log file only if the related parts of the code were
983 executed during the :ref:`gmx mdrun` run. There is also a special counter called "Rest" which
984 indicates the amount of time not accounted for by any of the counters above. Therefore,
985 a significant amount "Rest" time (more than a few percent) will often be an indication of
986 parallelization inefficiency (e.g. serial code) and it is recommended to be reported to the
989 An additional set of subcounters can offer more fine-grained inspection of performance. They are:
991 * Domain decomposition redistribution
992 * DD neighbor search grid + sort
993 * DD setup communication
995 * DD make constraints
997 * Neighbor search grid local
1000 * NS search non-local
1004 * Listed buffer operations
1007 * Launch non-bonded GPU tasks
1008 * Launch PME GPU tasks
1009 * Ewald force correction
1010 * Non-bonded position buffer operations
1011 * Non-bonded force buffer operations
1013 Subcounters are geared toward developers and have to be enabled during compilation. See
1014 :doc:`/dev-manual/build-system` for more information.
1019 - red flags in log files, how to interpret wallcycle output
1020 - hints to devs how to extend wallcycles
1022 .. _gmx-mdrun-on-gpu:
1024 Running :ref:`mdrun <gmx mdrun>` with GPUs
1025 ------------------------------------------
1032 To better understand the later sections on different GPU use cases for
1033 calculation of :ref:`short range<gmx-gpu-pp>`, :ref:`PME<gmx-gpu-pme>`,
1034 :ref:`bonded interactions<gmx-gpu-bonded>` and
1035 :ref:`update and constraints <gmx-gpu-update>`
1036 we first introduce the concept of different GPU tasks. When thinking about
1037 running a simulation, several different kinds of interactions between the atoms
1038 have to be calculated (for more information please refer to the reference manual).
1039 The calculation can thus be split into several distinct parts that are largely independent
1040 of each other (hence can be calculated in any order, e.g. sequentially or concurrently),
1041 with the information from each of them combined at the end of
1042 time step to obtain the final forces on each atom and to propagate the system
1043 to the next time point. For a better understanding also please see the section
1044 on :ref:`domain decomposition <gmx-domain-decomp>`.
1046 Of all calculations required for an MD step,
1047 GROMACS aims to optimize performance bottom-up for each step
1048 from the lowest level (SIMD unit, cores, sockets, accelerators, etc.).
1049 Therefore many of the individual computation units are
1050 highly tuned for the lowest level of hardware parallelism: the SIMD units.
1051 Additionally, with GPU accelerators used as *co-processors*, some of the work
1052 can be *offloaded*, that is calculated simultaneously/concurrently with the CPU
1053 on the accelerator device, with the result being communicated to the CPU.
1054 Right now, |Gromacs| supports GPU accelerator offload of two tasks:
1055 the short-range :ref:`nonbonded interactions in real space <gmx-gpu-pp>`,
1056 and :ref:`PME <gmx-gpu-pme>`.
1058 **Please note that the solving of PME on GPU is still only the initial
1059 version supporting this behaviour, and comes with a set of limitations
1060 outlined further below.**
1062 Right now, we generally support short-range nonbonded offload with and
1063 without dynamic pruning on a wide range of GPU accelerators
1064 (both NVIDIA and AMD). This is compatible with the grand majority of
1065 the features and parallelization modes and can be used to scale to large machines.
1067 Simultaneously offloading both short-range nonbonded and long-range
1068 PME work to GPU accelerators is a new feature that that has some
1069 restrictions in terms of feature and parallelization
1070 compatibility (please see the :ref:`section below <gmx-pme-gpu-limitations>`).
1074 GPU computation of short range nonbonded interactions
1075 .....................................................
1077 .. todo:: make this more elaborate and include figures
1079 Using the GPU for the short-ranged nonbonded interactions provides
1080 the majority of the available speed-up compared to run using only the CPU.
1081 Here, the GPU acts as an accelerator that can effectively parallelize
1082 this problem and thus reduce the calculation time.
1086 GPU accelerated calculation of PME
1087 ..................................
1089 .. todo:: again, extend this and add some actual useful information concerning performance etc...
1091 |Gromacs| now allows the offloading of the PME calculation
1092 to the GPU, to further reduce the load on the CPU and improve usage overlap between
1093 CPU and GPU. Here, the solving of PME will be performed in addition to the calculation
1094 of the short range interactions on the same GPU as the short range interactions.
1096 .. _gmx-pme-gpu-limitations:
1101 **Please note again the limitations outlined below!**
1103 - Only a PME order of 4 is supported on GPUs.
1105 - PME will run on a GPU only when exactly one rank has a
1106 PME task, ie. decompositions with multiple ranks doing PME are not supported.
1108 - Only single precision is supported.
1110 - Only dynamical integrators are supported (ie. leap-frog, Velocity Verlet,
1111 stochastic dynamics)
1113 - LJ PME is not supported on GPUs.
1117 GPU accelerated calculation of bonded interactions (CUDA only)
1118 ..............................................................
1120 .. todo:: again, extend this and add some actual useful information concerning performance etc...
1122 |Gromacs| now allows the offloading of the bonded part of the PP
1123 workload to a CUDA-compatible GPU. This is treated as part of the PP
1124 work, and requires that the short-ranged non-bonded task also runs on
1125 a GPU. Typically, there is a performance advantage to offloading
1126 bonded interactions in particular when the amount of CPU resources per GPU
1127 is relatively little (either because the CPU is weak or there are few CPU
1128 cores assigned to a GPU in a run) or when there are other computations on the CPU.
1129 A typical case for the latter is free-energy calculations.
1133 GPU accelerated calculation of constraints and coordinate update (CUDA only)
1134 ............................................................................
1136 .. TODO again, extend this and add some actual useful information concerning performance etc...
1138 |Gromacs| makes it possible to also perform the coordinate update and (if requested)
1139 constraint calculation on a CUDA-compatible GPU. This allows executing all
1140 (supported) computation of a simulation step on the GPU.
1141 This feature is supported in single domain runs (unless using the experimental
1142 GPU domain decomposition feature), and needs to be explicitly requested by the user.
1143 This is a new parallelization mode where all force and coordinate
1144 data can be "GPU resident" for a number of steps, typically between neighbor searching steps.
1145 This has the benefit that there is less coupling between CPU host and GPU and
1146 on typical MD steps data does not need to be transferred between CPU and GPU.
1147 In this scheme it is however still possible for part of the computation to be
1148 executed on the CPU concurrently with GPU calculation.
1149 This helps supporting the broad range of |Gromacs| features not all of which are
1150 ported to GPUs. At the same time, it also allows improving performance by making
1151 use of the otherwise mostly idle CPU. It can often be advantageous to move the bonded
1152 or PME calculation back to the CPU, but the details of this will depending on the
1153 relative performance if the CPU cores paired in a simulation with a GPU.
1155 It is possible to change the default behaviour by setting the
1156 ``GMX_FORCE_UPDATE_DEFAULT_GPU`` environment variable to a non-zero value. In this
1157 case simulations will try to run all parts by default on the GPU, and will only fall
1158 back to the CPU based calculation if the simulation is not compatible.
1160 Using this parallelization mode is typically advantageous in cases where a fast GPU is
1161 used with a weak CPU, in particular if there is only single simulation assigned to a GPU.
1162 However, in typical throughput cases where multiple runs are assigned to each GPU,
1163 offloading everything, especially without moving back some of the work to the CPU
1164 can perform worse than the parallelization mode where only force computation is offloaded.
1167 Assigning tasks to GPUs
1168 .......................
1170 Depending on which tasks should be performed on which hardware, different kinds of
1171 calculations can be combined on the same or different GPUs, according to the information
1172 provided for running :ref:`mdrun <gmx mdrun>`.
1174 It is possible to assign the calculation of the different computational tasks to the same GPU, meaning
1175 that they will share the computational resources on the same device, or to different processing units
1176 that will each perform one task each.
1178 One overview over the possible task assignments is given below:
1180 |Gromacs| version 2018:
1182 Two different types of assignable GPU accelerated tasks are available, NB and PME.
1183 Each PP rank has a NB task that can be offloaded to a GPU.
1184 If there is only one rank with a PME task (including if that rank is a
1185 PME-only rank), then that task can be offloaded to a GPU. Such a PME
1186 task can run wholly on the GPU, or have its latter stages run only on the CPU.
1188 Limitations are that PME on GPU does not support PME domain decomposition,
1189 so that only one PME task can be offloaded to a single GPU
1190 assigned to a separate PME rank, while NB can be decomposed and offloaded to multiple GPUs.
1192 |Gromacs| version 2019:
1194 No new assignable GPU tasks are available, but any bonded interactions
1195 may run on the same GPU as the short-ranged interactions for a PP task.
1196 This can be influenced with the ``-bonded`` flag.
1198 Performance considerations for GPU tasks
1199 ........................................
1201 #) The performance balance depends on the speed and number of CPU cores you
1202 have vs the speed and number of GPUs you have.
1204 #) With slow/old GPUs and/or fast/modern CPUs with many
1205 cores, it might make more sense to let the CPU do PME calculation,
1206 with the GPUs focused on the calculation of the NB.
1208 #) With fast/modern GPUs and/or slow/old CPUs with few cores,
1209 it generally helps to have the GPU do PME.
1211 #) Offloading bonded work to a GPU will often not improve simulation performance
1212 as efficient CPU-based kernels can complete the bonded computation
1213 before the GPU is done with other offloaded work. Therefore,
1214 `gmx mdrun` will default to no bonded offload when PME is offloaded.
1215 Typical cases where performance can be improvement with bonded offload are:
1216 with significant bonded work (e.g. pure lipid or mostly polymer systems with little solvent),
1217 with very few and/or slow CPU cores per GPU, or when the CPU does
1218 other computation (e.g. PME, free energy).
1220 #) It *is* possible to use multiple GPUs with PME offload
1222 3 MPI ranks use one GPU each for short-range interactions,
1223 while a fourth rank does the PME on its GPU.
1225 #) The only way to know for sure what alternative is best for
1226 your machine is to test and check performance.
1228 .. todo:: we need to be more concrete here, i.e. what machine/software aspects to take into consideration, when will default run mode be using PME-GPU and when will it not, when/how should the user reason about testing different settings than the default.
1230 .. todo:: someone who knows about the mixed mode should comment further.
1232 Reducing overheads in GPU accelerated runs
1233 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1235 In order for CPU cores and GPU(s) to execute concurrently, tasks are
1236 launched and executed asynchronously on the GPU(s) while the CPU cores
1237 execute non-offloaded force computation (like long-range PME electrostatics).
1238 Asynchronous task launches are handled by GPU device driver and
1239 require CPU involvement. Therefore, the work of scheduling
1240 GPU tasks will incur an overhead that can in some cases significantly
1241 delay or interfere with the CPU execution.
1243 Delays in CPU execution are caused by the latency of launching GPU tasks,
1244 an overhead that can become significant as simulation ns/day increases
1245 (i.e. with shorter wall-time per step).
1246 The overhead is measured by :ref:`gmx mdrun` and reported in the performance
1247 summary section of the log file ("Launch GPU ops" row).
1248 A few percent of runtime spent in this category is normal,
1249 but in fast-iterating and multi-GPU parallel runs 10% or larger overheads can be observed.
1250 In general, a user can do little to avoid such overheads, but there
1251 are a few cases where tweaks can give performance benefits.
1252 In single-rank runs timing of GPU tasks is by default enabled and,
1253 while in most cases its impact is small, in fast runs performance can be affected.
1254 The performance impact will be most significant on NVIDIA GPUs with CUDA,
1255 less on AMD and Intel with OpenCL.
1256 In these cases, when more than a few percent of "Launch GPU ops" time is observed,
1257 it is recommended to turn off timing by setting the ``GMX_DISABLE_GPU_TIMING``
1258 environment variable.
1259 In parallel runs with many ranks sharing a GPU,
1260 launch overheads can also be reduced by starting fewer thread-MPI
1261 or MPI ranks per GPU; e.g. most often one rank per thread or core is not optimal.
1263 The second type of overhead, interference of the GPU driver with CPU computation,
1264 is caused by the scheduling and coordination of GPU tasks.
1265 A separate GPU driver thread can require CPU resources
1266 which may clash with the concurrently running non-offloaded tasks,
1267 potentially degrading the performance of PME or bonded force computation.
1268 This effect is most pronounced when using AMD GPUs with OpenCL with
1269 older driver releases (e.g. fglrx 12.15).
1270 To minimize the overhead it is recommended to
1271 leave a CPU hardware thread unused when launching :ref:`gmx mdrun`,
1272 especially on CPUs with high core counts and/or HyperThreading enabled.
1273 E.g. on a machine with a 4-core CPU and eight threads (via HyperThreading) and an AMD GPU,
1274 try ``gmx mdrun -ntomp 7 -pin on``.
1275 This will leave free CPU resources for the GPU task scheduling
1276 reducing interference with CPU computation.
1277 Note that assigning fewer resources to :ref:`gmx mdrun` CPU computation
1278 involves a tradeoff which may outweigh the benefits of reduced GPU driver overhead,
1279 in particular without HyperThreading and with few CPU cores.
1281 .. todo:: In future patch: any tips not covered above
1283 Running the OpenCL version of mdrun
1284 -----------------------------------
1286 Currently supported hardware architectures are:
1287 - GCN-based AMD GPUs;
1288 - NVIDIA GPUs (with at least OpenCL 1.2 support);
1290 Make sure that you have the latest drivers installed. For AMD GPUs,
1291 the compute-oriented `ROCm <https://rocm.github.io/>`_ stack is recommended;
1292 alternatively, the AMDGPU-PRO stack is also compatible; using the outdated
1293 and unsupported ``fglrx`` proprietary driver and runtime is not recommended (but
1294 for certain older hardware that may be the only way to obtain support).
1295 In addition Mesa version 17.0 or newer with LLVM 4.0 or newer is also supported.
1296 For NVIDIA GPUs, using the proprietary driver is
1297 required as the open source nouveau driver (available in Mesa) does not
1298 provide the OpenCL support.
1299 For Intel integrated GPUs, the `Neo driver <https://github.com/intel/compute-runtime/releases>`_ is
1301 .. seealso:: :issue:`3268` add more Intel driver recommendations
1303 The minimum OpenCL version required is |REQUIRED_OPENCL_MIN_VERSION|. See
1304 also the :ref:`known limitations <opencl-known-limitations>`.
1306 Devices from the AMD GCN architectures (all series) are compatible
1307 and regularly tested; NVIDIA Kepler and later (compute capability 3.0)
1308 are known to work, but before doing production runs always make sure that the |Gromacs| tests
1309 pass successfully on the hardware.
1311 The OpenCL GPU kernels are compiled at run time. Hence,
1312 building the OpenCL program can take a few seconds, introducing a slight
1313 delay in the :ref:`gmx mdrun` startup. This is not normally a
1314 problem for long production MD, but you might prefer to do some kinds
1315 of work, e.g. that runs very few steps, on just the CPU (e.g. see ``-nb`` above).
1317 The same ``-gpu_id`` option (or ``GMX_GPU_ID`` environment variable)
1318 used to select CUDA devices, or to define a mapping of GPUs to PP
1319 ranks, is used for OpenCL devices.
1321 Some other :ref:`OpenCL management <opencl-management>` environment
1322 variables may be of interest to developers.
1324 .. _opencl-known-limitations:
1326 Known limitations of the OpenCL support
1327 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1329 Limitations in the current OpenCL support of interest to |Gromacs| users:
1331 - Intel integrated GPUs are supported. Intel CPUs and Xeon Phi are not supported.
1332 - Due to blocking behavior of some asynchronous task enqueuing functions
1333 in the NVIDIA OpenCL runtime, with the affected driver versions there is
1334 almost no performance gain when using NVIDIA GPUs.
1335 The issue affects NVIDIA driver versions up to 349 series, but it
1336 known to be fixed 352 and later driver releases.
1337 - On NVIDIA GPUs the OpenCL kernels achieve much lower performance
1338 than the equivalent CUDA kernels due to limitations of the NVIDIA OpenCL
1340 - On the NVIDIA Volta an Turing architectures the OpenCL code is known to produce
1341 incorrect results with driver version up to 440.x (most likely due to compiler issues).
1342 Runs typically fail on these architectures.
1344 Limitations of interest to |Gromacs| developers:
1346 - The current implementation requires a minimum execution with of 16; kernels
1347 compiled for narrower execution width (be it due to hardware requirements or
1348 compiler choice) will not be suitable and will trigger a runtime error.
1350 Performance checklist
1351 ---------------------
1353 There are many different aspects that affect the performance of simulations in
1354 |Gromacs|. Most simulations require a lot of computational resources, therefore
1355 it can be worthwhile to optimize the use of those resources. Several issues
1356 mentioned in the list below could lead to a performance difference of a factor
1357 of 2. So it can be useful go through the checklist.
1359 |Gromacs| configuration
1360 ^^^^^^^^^^^^^^^^^^^^^^^
1362 * Don't use double precision unless you're absolute sure you need it.
1363 * Compile the FFTW library (yourself) with the correct flags on x86 (in most
1364 cases, the correct flags are automatically configured).
1365 * On x86, use gcc as the compiler (not icc, pgi or the Cray compiler).
1366 * On POWER, use gcc instead of IBM's xlc.
1367 * Use a new compiler version, especially for gcc (e.g. from version 5 to 6
1368 the performance of the compiled code improved a lot).
1369 * MPI library: OpenMPI usually has good performance and causes little trouble.
1370 * Make sure your compiler supports OpenMP (some versions of Clang don't).
1371 * If you have GPUs that support either CUDA or OpenCL, use them.
1373 * Configure with ``-DGMX_GPU=CUDA `` or ``-DGMX_GPU=OpenCL``.
1374 * For CUDA, use the newest CUDA available for your GPU to take advantage of the
1375 latest performance enhancements.
1376 * Use a recent GPU driver.
1377 * Make sure you use an :ref:`gmx mdrun` with ``GMX_SIMD`` appropriate for the CPU
1378 architecture; the log file will contain a warning note if suboptimal setting is used.
1379 However, prefer ``AVX2` over ``AVX512`` in GPU or highly parallel MPI runs (for more
1380 information see the :ref:`intra-core parallelization information <intra-core-parallelization>`).
1381 * If compiling on a cluster head node, make sure that ``GMX_SIMD``
1382 is appropriate for the compute nodes.
1387 * For an approximately spherical solute, use a rhombic dodecahedron unit cell.
1388 * When using a time-step of 2 fs, use :mdp-value:`constraints=h-bonds`
1389 (and not :mdp-value:`constraints=all-bonds`), since this is faster, especially with GPUs,
1390 and most force fields have been parametrized with only bonds involving
1391 hydrogens constrained.
1392 * You can increase the time-step to 4 or 5 fs when using virtual interaction
1393 sites (``gmx pdb2gmx -vsite h``).
1394 * For massively parallel runs with PME, you might need to try different numbers
1395 of PME ranks (``gmx mdrun -npme ???``) to achieve best performance;
1396 :ref:`gmx tune_pme` can help automate this search.
1397 * For massively parallel runs (also ``gmx mdrun -multidir``), or with a slow
1398 network, global communication can become a bottleneck and you can reduce it
1399 by choosing larger periods for algorithms such as temperature and
1402 Checking and improving performance
1403 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1405 * Look at the end of the ``md.log`` file to see the performance and the cycle
1406 counters and wall-clock time for different parts of the MD calculation. The
1407 PP/PME load ratio is also printed, with a warning when a lot of performance is
1408 lost due to imbalance.
1409 * Adjust the number of PME ranks and/or the cut-off and PME grid-spacing when
1410 there is a large PP/PME imbalance. Note that even with a small reported
1411 imbalance, the automated PME-tuning might have reduced the initial imbalance.
1412 You could still gain performance by changing the mdp parameters or increasing
1413 the number of PME ranks.
1414 * If the neighbor searching takes a lot of time, increase nstlist. If a Verlet
1415 buffer tolerance is used, this is done automatically by :ref:`gmx mdrun`
1416 and the pair-list buffer is increased to keep the energy drift constant.
1418 * If ``Comm. energies`` takes a lot of time (a note will be printed in the log
1419 file), increase nstcalcenergy.
1420 * If all communication takes a lot of time, you might be running on too many
1421 cores, or you could try running combined MPI/OpenMP parallelization with 2
1422 or 4 OpenMP threads per MPI process.