3 Getting good performance from mdrun
4 ===================================
5 The |Gromacs| build system and the :ref:`gmx mdrun` tool has a lot of built-in
6 and configurable intelligence to detect your hardware and make pretty
7 effective use of that hardware. For a lot of casual and serious use of
8 :ref:`gmx mdrun`, the automatic machinery works well enough. But to get the
9 most from your hardware to maximize your scientific quality, read on!
11 Hardware background information
12 -------------------------------
13 Modern computer hardware is complex and heterogeneous, so we need to
14 discuss a little bit of background information and set up some
15 definitions. Experienced HPC users can skip this section.
20 A hardware compute unit that actually executes
21 instructions. There is normally more than one core in a
22 processor, often many more.
25 A special kind of memory local to core(s) that is much faster
26 to access than main memory, kind of like the top of a human's
27 desk, compared to their filing cabinet. There are often
28 several layers of caches associated with a core.
31 A group of cores that share some kind of locality, such as a
32 shared cache. This makes it more efficient to spread
33 computational work over cores within a socket than over cores
34 in different sockets. Modern processors often have more than
38 A group of sockets that share coarser-level locality, such as
39 shared access to the same memory without requiring any network
40 hardware. A normal laptop or desktop computer is a node. A
41 node is often the smallest amount of a large compute cluster
42 that a user can request to use.
45 A stream of instructions for a core to execute. There are many
46 different programming abstractions that create and manage
47 spreading computation over multiple threads, such as OpenMP,
48 pthreads, winthreads, CUDA, OpenCL, and OpenACC. Some kinds of
49 hardware can map more than one software thread to a core; on
50 Intel x86 processors this is called "hyper-threading", while
51 the more general concept is often called SMT for
52 "simultaneous multi-threading". IBM Power8 can for instance use
53 up to 8 hardware threads per core.
54 This feature can usually be enabled or disabled either in
55 the hardware bios or through a setting in the Linux operating
56 system. |Gromacs| can typically make use of this, for a moderate
57 free performance boost. In most cases it will be
58 enabled by default e.g. on new x86 processors, but in some cases
59 the system administrators might have disabled it. If that is the
60 case, ask if they can re-enable it for you. If you are not sure
61 if it is enabled, check the output of the CPU information in
62 the log file and compare with CPU specifications you find online.
64 thread affinity (pinning)
65 By default, most operating systems allow software threads to migrate
66 between cores (or hardware threads) to help automatically balance
67 workload. However, the performance of :ref:`gmx mdrun` can deteriorate
68 if this is permitted and will degrade dramatically especially when
69 relying on multi-threading within a rank. To avoid this,
70 :ref:`gmx mdrun` will by default
71 set the affinity of its threads to individual cores/hardware threads,
72 unless the user or software environment has already done so
73 (or not the entire node is used for the run, i.e. there is potential
75 Setting thread affinity is sometimes called thread "pinning".
78 The dominant multi-node parallelization-scheme, which provides
79 a standardized language in which programs can be written that
80 work across more than one node.
83 In MPI, a rank is the smallest grouping of hardware used in
84 the multi-node parallelization scheme. That grouping can be
85 controlled by the user, and might correspond to a core, a
86 socket, a node, or a group of nodes. The best choice varies
87 with the hardware, software and compute task. Sometimes an MPI
88 rank is called an MPI process.
91 A graphics processing unit, which is often faster and more
92 efficient than conventional processors for particular kinds of
93 compute workloads. A GPU is always associated with a
94 particular node, and often a particular socket within that
98 A standardized technique supported by many compilers to share
99 a compute workload over multiple cores. Often combined with
100 MPI to achieve hybrid MPI/OpenMP parallelism.
103 A proprietary parallel computing framework and API developed by NVIDIA
104 that allows targeting their accelerator hardware.
105 |Gromacs| uses CUDA for GPU acceleration support with NVIDIA hardware.
108 An open standard-based parallel computing framework that consists
109 of a C99-based compiler and a programming API for targeting heterogeneous
110 and accelerator hardware. |Gromacs| uses OpenCL for GPU acceleration
111 on AMD devices (both GPUs and APUs); NVIDIA hardware is also supported.
114 Modern CPU cores have instructions that can execute large
115 numbers of floating-point instructions in a single cycle.
118 |Gromacs| background information
119 --------------------------------
120 The algorithms in :ref:`gmx mdrun` and their implementations are most relevant
121 when choosing how to make good use of the hardware. For details,
122 see the Reference Manual. The most important of these are
126 .. _gmx-domain-decomp:
129 The domain decomposition (DD) algorithm decomposes the
130 (short-ranged) component of the non-bonded interactions into
131 domains that share spatial locality, which permits the use of
132 efficient algorithms. Each domain handles all of the
133 particle-particle (PP) interactions for its members, and is
134 mapped to a single MPI rank. Within a PP rank, OpenMP threads
135 can share the workload, and some work can be off-loaded to a
136 GPU. The PP rank also handles any bonded interactions for the
137 members of its domain. A GPU may perform work for more than
138 one PP rank, but it is normally most efficient to use a single
139 PP rank per GPU and for that rank to have thousands of
140 particles. When the work of a PP rank is done on the CPU, mdrun
141 will make extensive use of the SIMD capabilities of the
142 core. There are various `command-line options
143 <controlling-the-domain-decomposition-algorithm` to control
144 the behaviour of the DD algorithm.
147 The particle-mesh Ewald (PME) algorithm treats the long-ranged
148 components of the non-bonded interactions (Coulomb and/or
149 Lennard-Jones). Either all, or just a subset of ranks may
150 participate in the work for computing long-ranged component
151 (often inaccurately called simple the "PME"
152 component). Because the algorithm uses a 3D FFT that requires
153 global communication, its performance gets worse as more ranks
154 participate, which can mean it is fastest to use just a subset
155 of ranks (e.g. one-quarter to one-half of the ranks). If
156 there are separate PME ranks, then the remaining ranks handle
157 the PP work. Otherwise, all ranks do both PP and PME work.
159 Running mdrun within a single node
160 ----------------------------------
162 :ref:`gmx mdrun` can be configured and compiled in several different ways that
163 are efficient to use within a single :term:`node`. The default configuration
164 using a suitable compiler will deploy a multi-level hybrid parallelism
165 that uses CUDA, OpenMP and the threading platform native to the
166 hardware. For programming convenience, in |Gromacs|, those native
167 threads are used to implement on a single node the same MPI scheme as
168 would be used between nodes, but much more efficient; this is called
169 thread-MPI. From a user's perspective, real MPI and thread-MPI look
170 almost the same, and |Gromacs| refers to MPI ranks to mean either kind,
171 except where noted. A real external MPI can be used for :ref:`gmx mdrun` within
172 a single node, but runs more slowly than the thread-MPI version.
174 By default, :ref:`gmx mdrun` will inspect the hardware available at run time
175 and do its best to make fairly efficient use of the whole node. The
176 log file, stdout and stderr are used to print diagnostics that
177 inform the user about the choices made and possible consequences.
179 A number of command-line parameters are available to modify the default
183 The total number of threads to use. The default, 0, will start as
184 many threads as available cores. Whether the threads are
185 thread-MPI ranks, and/or OpenMP threads within such ranks depends on
189 The total number of thread-MPI ranks to use. The default, 0,
190 will start one rank per GPU (if present), and otherwise one rank
194 The total number of OpenMP threads per rank to start. The
195 default, 0, will start one thread on each available core.
196 Alternatively, mdrun will honor the appropriate system
197 environment variable (e.g. ``OMP_NUM_THREADS``) if set.
200 The total number of ranks to dedicate to the long-ranged
201 component of PME, if used. The default, -1, will dedicate ranks
202 only if the total number of threads is at least 12, and will use
203 around a quarter of the ranks for the long-ranged component.
206 When using PME with separate PME ranks,
207 the total number of OpenMP threads per separate PME ranks.
208 The default, 0, copies the value from ``-ntomp``.
211 Can be set to "auto," "on" or "off" to control whether
212 mdrun will attempt to set the affinity of threads to cores.
213 Defaults to "auto," which means that if mdrun detects that all the
214 cores on the node are being used for mdrun, then it should behave
215 like "on," and attempt to set the affinities (unless they are
216 already set by something else).
219 If ``-pin on``, specifies the logical core number to
220 which mdrun should pin the first thread. When running more than
221 one instance of mdrun on a node, use this option to to avoid
222 pinning threads from different mdrun instances to the same core.
225 If ``-pin on``, specifies the stride in logical core
226 numbers for the cores to which mdrun should pin its threads. When
227 running more than one instance of mdrun on a node, use this option
228 to to avoid pinning threads from different mdrun instances to the
229 same core. Use the default, 0, to minimize the number of threads
230 per physical core - this lets mdrun manage the hardware-, OS- and
231 configuration-specific details of how to map logical cores to
235 Can be set to "interleave," "pp_pme" or "cartesian."
236 Defaults to "interleave," which means that any separate PME ranks
237 will be mapped to MPI ranks in an order like PP, PP, PME, PP, PP,
238 PME, ... etc. This generally makes the best use of the available
239 hardware. "pp_pme" maps all PP ranks first, then all PME
240 ranks. "cartesian" is a special-purpose mapping generally useful
241 only on special torus networks with accelerated global
242 communication for Cartesian communicators. Has no effect if there
243 are no separate PME ranks.
246 Used to set where to execute the non-bonded interactions.
247 Can be set to "auto", "cpu", "gpu."
248 Defaults to "auto," which uses a compatible GPU if available.
249 Setting "cpu" requires that no GPU is used. Setting "gpu" requires
250 that a compatible GPU be available and will be used.
253 A string that specifies the ID numbers of the GPUs that
254 are available to be used by ranks on this node. For example,
255 "12" specifies that the GPUs with IDs 1 and 2 (as reported
256 by the GPU runtime) can be used by mdrun. This is useful
257 when sharing a node with other computations, or if a GPU
258 is best used to support a display. If many GPUs are
259 present, a comma may be used to separate the IDs, so
260 "12,13" would make GPUs 12 and 13 available to mdrun.
261 It could be necessary to use different GPUs on different
262 nodes of a simulation, in which case the environment
263 variable ``GMX_GPU_ID`` can be set differently for the ranks
264 on different nodes to achieve that result.
267 A string that specifies the ID numbers of the GPUs to be
268 used by corresponding GPU tasks on this node. For example,
269 "0011" specifies that the first two GPU tasks will use GPU 0,
270 and the other two use GPU 1. When using this option, the
271 number of ranks must be known to mdrun, as well as where
272 tasks of different types should be run, such as by using
275 Examples for mdrun on one node
276 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
282 Starts mdrun using all the available resources. mdrun
283 will automatically choose a fairly efficient division
284 into thread-MPI ranks, OpenMP threads and assign work
285 to compatible GPUs. Details will vary with hardware
286 and the kind of simulation being run.
292 Starts mdrun using 8 threads, which might be thread-MPI
293 or OpenMP threads depending on hardware and the kind
294 of simulation being run.
298 gmx mdrun -ntmpi 2 -ntomp 4
300 Starts mdrun using eight total threads, with four thread-MPI
301 ranks and two OpenMP threads per core. You should only use
302 these options when seeking optimal performance, and
303 must take care that the ranks you create can have
304 all of their OpenMP threads run on the same socket.
305 The number of ranks must be a multiple of the number of
306 sockets, and the number of cores per node must be
307 a multiple of the number of threads per rank.
313 Starts mdrun using GPUs with IDs 1 and 2 (e.g. because
314 GPU 0 is dedicated to running a display). This requires
315 two thread-MPI ranks, and will split the available
316 CPU cores between them using OpenMP threads.
320 gmx mdrun -ntmpi 4 -nb gpu -gputasks 1122
322 Starts mdrun using four thread-MPI ranks, and maps them
323 to GPUs with IDs 1 and 2. The CPU cores available will
324 be split evenly between the ranks using OpenMP threads.
328 gmx mdrun -nt 6 -pin on -pinoffset 0
329 gmx mdrun -nt 6 -pin on -pinoffset 3
331 Starts two mdrun processes, each with six total threads.
332 Threads will have their affinities set to particular
333 logical cores, beginning from the logical core
334 with rank 0 or 3, respectively. The above would work
335 well on an Intel CPU with six physical cores and
336 hyper-threading enabled. Use this kind of setup only
337 if restricting mdrun to a subset of cores to share a
338 node with other processes.
342 mpirun -np 2 gmx_mpi mdrun
344 When using an :ref:`gmx mdrun` compiled with external MPI,
345 this will start two ranks and as many OpenMP threads
346 as the hardware and MPI setup will permit. If the
347 MPI setup is restricted to one node, then the resulting
348 :ref:`gmx mdrun` will be local to that node.
350 Running mdrun on more than one node
351 -----------------------------------
352 This requires configuring |Gromacs| to build with an external MPI
353 library. By default, this mdrun executable is run with
354 :ref:`mdrun_mpi`. All of the considerations for running single-node
355 mdrun still apply, except that ``-ntmpi`` and ``-nt`` cause a fatal
356 error, and instead the number of ranks is controlled by the
358 Settings such as ``-npme`` are much more important when
359 using multiple nodes. Configuring the MPI environment to
360 produce one rank per core is generally good until one
361 approaches the strong-scaling limit. At that point, using
362 OpenMP to spread the work of an MPI rank over more than one
363 core is needed to continue to improve absolute performance.
364 The location of the scaling limit depends on the processor,
365 presence of GPUs, network, and simulation algorithm, but
366 it is worth measuring at around ~200 particles/core if you
367 need maximum throughput.
369 There are further command-line parameters that are relevant in these
373 Defaults to "on." If "on," a Verlet-scheme simulation will
374 optimize various aspects of the PME and DD algorithms, shifting
375 load between ranks and/or GPUs to maximize throughput. Some
376 mdrun features are not compatible with this, and these ignore
380 Can be set to "auto," "no," or "yes."
381 Defaults to "auto." Doing Dynamic Load Balancing between MPI ranks
382 is needed to maximize performance. This is particularly important
383 for molecular systems with heterogeneous particle or interaction
384 density. When a certain threshold for performance loss is
385 exceeded, DLB activates and shifts particles between ranks to improve
389 During the simulation :ref:`gmx mdrun` must communicate between all ranks to
390 compute quantities such as kinetic energy. By default, this
391 happens whenever plausible, and is influenced by a lot of :ref:`[.mdp]
392 options. <mdp-general>` The period between communication phases
393 must be a multiple of :mdp:`nstlist`, and defaults to
394 the minimum of :mdp:`nstcalcenergy` and :mdp:`nstlist`.
395 ``mdrun -gcom`` sets the number of steps that must elapse between
396 such communication phases, which can improve performance when
397 running on a lot of ranks. Note that this means that _e.g._
398 temperature coupling algorithms will
399 effectively remain at constant energy until the next
400 communication phase. :ref:`gmx mdrun` will always honor the
401 setting of ``mdrun -gcom``, by changing :mdp:`nstcalcenergy`,
402 :mdp:`nstenergy`, :mdp:`nstlog`, :mdp:`nsttcouple` and/or
403 :mdp:`nstpcouple` if necessary.
405 Note that ``-tunepme`` has more effect when there is more than one
406 :term:`node`, because the cost of communication for the PP and PME
407 ranks differs. It still shifts load between PP and PME ranks, but does
408 not change the number of separate PME ranks in use.
410 Note also that ``-dlb`` and ``-tunepme`` can interfere with each other, so
411 if you experience performance variation that could result from this,
412 you may wish to tune PME separately, and run the result with ``mdrun
413 -notunepme -dlb yes``.
415 The :ref:`gmx tune_pme` utility is available to search a wider
416 range of parameter space, including making safe
417 modifications to the :ref:`tpr` file, and varying ``-npme``.
418 It is only aware of the number of ranks created by
419 the MPI environment, and does not explicitly manage
420 any aspect of OpenMP during the optimization.
422 Examples for mdrun on more than one node
423 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
424 The examples and explanations for for single-node mdrun are
425 still relevant, but ``-nt`` is no longer the way
426 to choose the number of MPI ranks.
430 mpirun -np 16 gmx_mpi mdrun
432 Starts :ref:`mdrun_mpi` with 16 ranks, which are mapped to
433 the hardware by the MPI library, e.g. as specified
434 in an MPI hostfile. The available cores will be
435 automatically split among ranks using OpenMP threads,
436 depending on the hardware and any environment settings
437 such as ``OMP_NUM_THREADS``.
441 mpirun -np 16 gmx_mpi mdrun -npme 5
443 Starts :ref:`mdrun_mpi` with 16 ranks, as above, and
444 require that 5 of them are dedicated to the PME
449 mpirun -np 11 gmx_mpi mdrun -ntomp 2 -npme 6 -ntomp_pme 1
451 Starts :ref:`mdrun_mpi` with 11 ranks, as above, and
452 require that six of them are dedicated to the PME
453 component with one OpenMP thread each. The remaining
454 five do the PP component, with two OpenMP threads
459 mpirun -np 4 gmx mdrun -ntomp 6 -nb gpu -gputasks 00
461 Starts :ref:`mdrun_mpi` on a machine with two nodes, using
462 four total ranks, each rank with six OpenMP threads,
463 and both ranks on a node sharing GPU with ID 0.
467 mpirun -np 8 gmx mdrun -ntomp 3 -gputasks 0000
469 Using a same/similar hardware as above,
470 starts :ref:`mdrun_mpi` on a machine with two nodes, using
471 eight total ranks, each rank with three OpenMP threads,
472 and all four ranks on a node sharing GPU with ID 0.
473 This may or may not be faster than the previous setup
474 on the same hardware.
478 mpirun -np 20 gmx_mpi mdrun -ntomp 4 -gputasks 00
480 Starts :ref:`mdrun_mpi` with 20 ranks, and assigns the CPU cores evenly
481 across ranks each to one OpenMP thread. This setup is likely to be
482 suitable when there are ten nodes, each with one GPU, and each node
483 has two sockets each of four cores.
487 mpirun -np 10 gmx_mpi mdrun -gpu_id 1
489 Starts :ref:`mdrun_mpi` with 20 ranks, and assigns the CPU cores evenly
490 across ranks each to one OpenMP thread. This setup is likely to be
491 suitable when there are ten nodes, each with two GPUs, but another
492 job on each node is using GPU 0. The job scheduler should set the
493 affinity of threads of both jobs to their allocated cores, or the
494 performance of mdrun will suffer greatly.
498 mpirun -np 20 gmx_mpi mdrun -gpu_id 01
500 Starts :ref:`mdrun_mpi` with 20 ranks. This setup is likely
501 to be suitable when there are ten nodes, each with two
502 GPUs, but there is no need to specify ``-gpu_id`` for the
503 normal case where all the GPUs on the node are available
506 Controlling the domain decomposition algorithm
507 ----------------------------------------------
508 This section lists all the options that affect how the domain
509 decomposition algorithm decomposes the workload to the available
513 Can be used to set the required maximum distance for inter
514 charge-group bonded interactions. Communication for two-body
515 bonded interactions below the non-bonded cut-off distance always
516 comes for free with the non-bonded communication. Particles beyond
517 the non-bonded cut-off are only communicated when they have
518 missing bonded interactions; this means that the extra cost is
519 minor and nearly independent of the value of ``-rdd``. With dynamic
520 load balancing, option ``-rdd`` also sets the lower limit for the
521 domain decomposition cell sizes. By default ``-rdd`` is determined
522 by :ref:`gmx mdrun` based on the initial coordinates. The chosen value will
523 be a balance between interaction range and communication cost.
526 On by default. When inter charge-group bonded interactions are
527 beyond the bonded cut-off distance, :ref:`gmx mdrun` terminates with an
528 error message. For pair interactions and tabulated bonds that do
529 not generate exclusions, this check can be turned off with the
530 option ``-noddcheck``.
533 When constraints are present, option ``-rcon`` influences
534 the cell size limit as well.
535 Particles connected by NC constraints, where NC is the LINCS order
536 plus 1, should not be beyond the smallest cell size. A error
537 message is generated when this happens, and the user should change
538 the decomposition or decrease the LINCS order and increase the
539 number of LINCS iterations. By default :ref:`gmx mdrun` estimates the
540 minimum cell size required for P-LINCS in a conservative
541 fashion. For high parallelization, it can be useful to set the
542 distance required for P-LINCS with ``-rcon``.
545 Sets the minimum allowed x, y and/or z scaling of the cells with
546 dynamic load balancing. :ref:`gmx mdrun` will ensure that the cells can
547 scale down by at least this factor. This option is used for the
548 automated spatial decomposition (when not using ``-dd``) as well as
549 for determining the number of grid pulses, which in turn sets the
550 minimum allowed cell size. Under certain circumstances the value
551 of ``-dds`` might need to be adjusted to account for high or low
552 spatial inhomogeneity of the system.
554 Finding out how to run mdrun better
555 -----------------------------------
557 The Wallcycle module is used for runtime performance measurement of :ref:`gmx mdrun`.
558 At the end of the log file of each run, the "Real cycle and time accounting" section
559 provides a table with runtime statistics for different parts of the :ref:`gmx mdrun` code
560 in rows of the table.
561 The table contains colums indicating the number of ranks and threads that
562 executed the respective part of the run, wall-time and cycle
563 count aggregates (across all threads and ranks) averaged over the entire run.
564 The last column also shows what precentage of the total runtime each row represents.
565 Note that the :ref:`gmx mdrun` timer resetting functionalities (`-resethway` and `-resetstep`)
566 reset the performance counters and therefore are useful to avoid startup overhead and
567 performance instability (e.g. due to load balancing) at the beginning of the run.
569 The performance counters are:
571 * Particle-particle during Particle mesh Ewald
572 * Domain decomposition
573 * Domain decomposition communication load
574 * Domain decomposition communication bounds
575 * Virtual site constraints
576 * Send X to Particle mesh Ewald
578 * Launch GPU operations
579 * Communication of coordinates
581 * Waiting + Communication of force
582 * Particle mesh Ewald
586 * PME 3D-FFT Communication
587 * PME solve Lennard-Jones
589 * PME wait for particle-particle
590 * Wait + Receive PME force
593 * Non-bonded position/force buffer operations
594 * Virtual site spread
599 * Communication of energies
601 * Add rotational forces
605 As performance data is collected for every run, they are essential to assessing
606 and tuning the performance of :ref:`gmx mdrun` performance. Therefore, they benefit
607 both code developers as well as users of the program.
608 The counters are an average of the time/cycles different parts of the simulation take,
609 hence can not directly reveal fluctuations during a single run (although comparisons across
610 multiple runs are still very useful).
612 Counters will appear in MD log file only if the related parts of the code were
613 executed during the :ref:`gmx mdrun` run. There is also a special counter called "Rest" which
614 indicated for the amount of time not accounted for by any of the counters above. Theerfore,
615 a significant amount "Rest" time (more than a few percent) will often be an indication of
616 parallelization inefficiency (e.g. serial code) and it is recommended to be reported to the
619 An additional set of subcounters can offer more fine-grained inspection of performance. They are:
621 * Domain decomposition redistribution
622 * DD neighbor search grid + sort
623 * DD setup communication
625 * DD make constraints
627 * Neighbor search grid local
630 * NS search non-local
634 * Listed buffer operations
636 * Ewald force correction
637 * Non-bonded position buffer operations
638 * Non-bonded force buffer operations
640 Subcounters are geared toward developers and have to be enabled during compilation. See
641 :doc:`/dev-manual/build-system` for more information.
643 TODO In future patch:
644 - red flags in log files, how to interpret wallcycle output
645 - hints to devs how to extend wallcycles
647 TODO In future patch: import wiki page stuff on performance checklist; maybe here,
650 .. _gmx-mdrun-on-gpu:
652 Running mdrun with GPUs
653 -----------------------
655 NVIDIA GPUs from the professional line (Tesla or Quadro) starting with
656 the Kepler generation (compute capability 3.5 and later) support changing the
657 processor and memory clock frequency with the help of the applications clocks feature.
658 With many workloads, using higher clock rates than the default provides significant
659 performance improvements.
660 For more information see the `NVIDIA blog article`_ on this topic.
661 For |Gromacs| the highest application clock rates are optimal on all hardware
662 available to date (up to and including Maxwell, compute capability 5.2).
664 Application clocks can be set using the NVIDIA system managemet tool
665 ``nvidia-smi``. If the system permissions allow, :ref:`gmx mdrun` has
666 built-in support to set application clocks if built with :ref:`NVML support<CUDA GPU acceleration>`.
667 Note that application clocks are a global setting, hence affect the
668 performance of all applications that use the respective GPU(s).
669 For this reason, :ref:`gmx mdrun` sets application clocks at initialization
670 to the values optimal for |Gromacs| and it restores them before exiting
671 to the values found at startup, unless it detects that they were altered
674 .. _NVIDIA blog article: https://devblogs.nvidia.com/parallelforall/increase-performance-gpu-boost-k80-autoboost/
681 To better understand the later sections on different GPU use cases for
682 calculation of :ref:`short range<gmx-gpu-pp>` and :ref:`PME <gmx-gpu-pme>`,
683 we first introduce the concept of different GPU tasks. When thinking about
684 running a simulation, several different kinds of interactions between the atoms
685 have to be calculated (for more information please refer to the reference manual).
686 The calculation can thus be split into several distinct parts that are largely independent
687 of each other (hence can be calculated in any order, e.g. sequentially or concurrently),
688 with the information from each of them combined at the end of
689 time step to obtain the final forces on each atom and to propagate the system
690 to the next time point. For a better understanding also please see the section
691 on :ref:`domain decomposition <gmx-domain-decomp>`.
693 Of all calculations required for an MD step,
694 GROMACS aims to optimize performance bottom-up for each step
695 from the lowest level (SIMD unit, cores, sockets, accelerators, etc.).
696 Therefore much of the indivdual computation units are
697 highly tuned for the lowest level of hardware parallelism: the SIMD units.
698 Additionally, with GPU accelerators used as *co-processors*, some of the work
699 can be *offloaded*, that is calculated simultaneously/concurrently with the CPU
700 on the accelerator device, with the result being communicated to the CPU.
701 Right now, |Gromacs| supports GPU accelerator offload of two tasks:
702 the short-range :ref:`nonbonded interactions in real space <gmx-gpu-pp>`,
703 and :ref:`PME <gmx-gpu-pme>`.
705 **Please note that the solving of PME on GPU is still only the initial
706 version supporting this behaviour, and comes with a set of limitations
707 outlined further below.**
709 Right now, we generally support short-range nonbonded offload with and
710 without dynamic pruning on a wide range of GPU accelerators
711 (both NVIDIA and AMD). This is compatible with the grand majority of
712 the features and parallelization modes and can be used to scale to large machines.
714 Simultaneously offloading both short-range nonbonded and long-range
715 PME work to GPU accelerators is a new feature that that has some
716 restrictions in terms of feature and parallelization
717 compatibility (please see the :ref:`section below <gmx-pme-gpu-limitations>`).
721 GPU computation of short range nonbonded interactions
722 .....................................................
724 .. TODO make this more elaborate and include figures
726 Using the GPU for the short-ranged nonbonded interactions provides
727 the majority of the available speed-up compared to run using only the CPU.
728 Here, the GPU acts as an accelerator that can effectively parallelize
729 this problem and thus reduce the calculation time.
733 GPU accelerated calculation of PME
734 ..................................
736 .. TODO again, extend this and add some actual useful information concerning performance etc...
738 Recent additions to |Gromacs| now also allow the off-loading of the PME calculation
739 to the GPU, to further reduce the load on the CPU and improve usage overlap between
740 CPU and GPU. Here, the solving of PME will be performed in addition to the calculation
741 of the short range interactions on the same GPU as the short range interactions.
743 .. _gmx-pme-gpu-limitations:
748 **Please note again the limitations outlined above!**
750 - Only compilation with CUDA is supported.
752 - Only a PME order of 4 is supported in GPU.
754 - PME will run on a GPU only when exactly one rank has a
755 PME task, ie. decompositions with multiple ranks doing PME are not supported.
757 - Only single precision is supported.
759 - Free energy calculations are not supported, because only single PME grids can be calculated.
761 - LJ PME is not supported on GPU.
763 Assigning tasks to GPUs
764 .......................
766 Depending on which tasks should be performed on which hardware, different kinds of
767 calculations can be combined on the same or different GPUs, according to the information
768 provided for running :ref:`mdrun <gmx mdrun>`.
770 .. Someone more knowledgeable than me should check the accuracy of this part, so that
771 I don't say something that is factually wrong :)
773 It is possible to assign the calculation of the different computational tasks to the same GPU, meaning
774 that they will share the computational resources on the same device, or to different processing units
775 that will each perform one task each.
777 One overview over the possible task assignments is given below:
779 |Gromacs| version 2018:
781 Two different types of GPU accelerated tasks are available, NB and PME.
782 Each PP rank has a NB task that can be offloaded to a GPU.
783 If there is only one rank with a PME task (including if that rank is a
784 PME-only rank), then that task can be offloaded to a GPU. Such a PME
785 task can run wholly on the GPU, or have its latter stages run only on the CPU.
787 Limitations are that PME on GPU does not support PME domain decomposition,
788 so that only one PME task can be offloaded to a single GPU
789 assigned to a separate PME rank, while NB can be decomposed and offloaded to multiple GPUs.
791 .. Future |Gromacs| versions past 2018:
793 .. Combinations of different number of NB and single PME ranks on different
794 GPUs are being planned to be implemented in the near future. In addition,
795 we plan to add support for using multiple GPUs for each rank (e.g. having one GPU
796 each to solve the NB and PME part for a single rank), and to
797 implement domain decomposition on GPUs to allow the separation of the PME
798 part to different GPU tasks.
801 Performance considerations for GPU tasks
802 ........................................
804 #) The performace balance depends on how many (and how fast) CPU
805 cores you have, vs. how many and how fast the GPUs are that you have.
807 #) With slow/old GPUs and/or fast/modern CPUs with many
808 cores, it might make more sense to let the CPU do PME calculation,
809 with the GPUs focused on the calculation of the NB.
811 #) With fast/modern GPUs and/or slow/old CPUs with few cores,
812 it generally helps to have the GPU do PME.
814 #) It *is* possible to use multiple GPUs with PME offload
816 3 MPI ranks use one GPU each for short-range interactions,
817 while a fourth rank does the PME on its GPU.
819 #) The only way to know for sure what alternative is best for
820 your machine is to test and check performance.
822 .. TODO: we need to be more concrete here, i.e. what machine/software aspects to take into consideration, when will default run mode be using PME-GPU and when will it not, when/how should the user reason about testing different settings than the default.
824 .. TODO someone who knows about the mixed mode should comment further.
826 Reducing overheads in GPU accelerated runs
827 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
829 In order for CPU cores and GPU(s) to execute concurrently, tasks are
830 launched and executed asynchronously on the GPU(s) while the CPU cores
831 execute non-offloaded force computation (like long-range PME electrostatics).
832 Asynchronous task launches are handled by GPU device driver and
833 require CPU involvement. Therefore, the work of scheduling
834 GPU tasks will incur an overhead that can in some cases significantly
835 delay or interfere with the CPU execution.
837 Delays in CPU execution are caused by the latency of launching GPU tasks,
838 an overhead that can become significant as simulation ns/day increases
839 (i.e. with shorter wall-time per step).
840 The overhead is measured by :ref:`gmx mdrun` and reported in the performance
841 summary section of the log file ("Launch GPU ops" row).
842 A few percent of runtime spent in this category is normal,
843 but in fast-iterating and multi-GPU parallel runs 10% or larger overheads can be observed.
844 In general, there a user can do little to avoid such overheads, but there
845 are a few cases where tweaks can give performance benefits.
846 In single-rank runs timing of GPU tasks is by default enabled and,
847 while in most cases its impact is small, in fast runs performance can be affected.
848 The performance impact will be most significant on NVIDIA GPUs with CUDA,
849 less on AMD with OpenCL.
850 In these cases, when more than a few percent of "Launch GPU ops" time is observed,
851 it is recommended turning off timing by setting the ``GMX_DISABLE_GPU_TIMING``
852 environment variable.
853 In parallel runs with with many ranks sharing a GPU
854 launch overheads can also be reduced by staring fewer thread-MPI
855 or MPI ranks per GPU; e.g. most often one rank per thread or core is not optimal.
857 The second type of overhead, interference of the GPU driver with CPU computation,
858 is caused by the scheduling and coordination of GPU tasks.
859 A separate GPU driver thread can require CPU resources
860 which may clash with the concurrently running non-offloaded tasks,
861 potentially degrading the performance of PME or bonded force computation.
862 This effect is most pronounced when using AMD GPUs with OpenCL with
863 older driver releases (e.g. fglrx 12.15).
864 To minimize the overhead it is recommended to
865 leave a CPU hardware thread unused when launching :ref:`gmx mdrun`,
866 especially on CPUs with high core count and/or HyperThreading enabled.
867 E.g. on a machine with a 4-core CPU and eight threads (via HyperThreading) and an AMD GPU,
868 try ``gmx mdrun -ntomp 7 -pin on``.
869 This will leave free CPU resources for the GPU task scheduling
870 reducing interference with CPU computation.
871 Note that assigning fewer resources to :ref:`gmx mdrun` CPU computation
872 involves a tradeoff which may outweigh the benefits of reduced GPU driver overhead,
873 in particular without HyperThreading and with few CPU cores.
875 TODO In future patch: any tips not covered above
877 Running the OpenCL version of mdrun
878 -----------------------------------
880 The current version works with GCN-based AMD GPUs, and NVIDIA CUDA
881 GPUs. Make sure that you have the latest drivers installed. For AMD GPUs,
882 the compute-oriented `ROCm <https://rocm.github.io/>`_ stack is recommended;
883 alternatively, the AMDGPU-PRO stack is also compatible; using the outdated
884 and unsupported `fglrx` proprietary driver and runtime is not recommended (but
885 for certain older hardware that may be the only way to obtain support).
886 In addition Mesa version 17.0 or newer with LLVM 4.0 or newer is also supported.
887 For NVIDIA GPUs, using the proprietary driver is
888 required as the open source nouveau driver (available in Mesa) does not
889 provide the OpenCL support.
890 The minimum OpenCL version required is |REQUIRED_OPENCL_MIN_VERSION|. See
891 also the :ref:`known limitations <opencl-known-limitations>`.
893 Devices from the AMD GCN architectures (all series) are compatible
894 and regularly tested; NVIDIA Fermi and later (compute capability 2.0)
895 are known to work, but before doing production runs always make sure that the |Gromacs| tests
896 pass successfully on the hardware.
898 The OpenCL GPU kernels are compiled at run time. Hence,
899 building the OpenCL program can take a few seconds introducing a slight
900 delay in the :ref:`gmx mdrun` startup. This is not normally a
901 problem for long production MD, but you might prefer to do some kinds
902 of work, e.g. that runs very few steps, on just the CPU (e.g. see ``-nb`` above).
904 The same ``-gpu_id`` option (or ``GMX_GPU_ID`` environment variable)
905 used to select CUDA devices, or to define a mapping of GPUs to PP
906 ranks, is used for OpenCL devices.
908 Some other :ref:`OpenCL management <opencl-management>` environment
909 variables may be of interest to developers.
911 .. _opencl-known-limitations:
913 Known limitations of the OpenCL support
914 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
916 Limitations in the current OpenCL support of interest to |Gromacs| users:
918 - PME GPU offload is not supported with OpenCL.
919 - No Intel devices (CPUs, GPUs or Xeon Phi) are supported
920 - Due to blocking behavior of some asynchronous task enqueuing functions
921 in the NVIDIA OpenCL runtime, with the affected driver versions there is
922 almost no performance gain when using NVIDIA GPUs.
923 The issue affects NVIDIA driver versions up to 349 series, but it
924 known to be fixed 352 and later driver releases.
925 - On NVIDIA GPUs the OpenCL kernels achieve much lower performance
926 than the equivalent CUDA kernels due to limitations of the NVIDIA OpenCL
929 Limitations of interest to |Gromacs| developers:
931 - The current implementation is not compatible with OpenCL devices that are
932 not using warp/wavefronts or for which the warp/wavefront size is not a
935 Performance checklist
936 ---------------------
938 There are many different aspects that affect the performance of simulations in
939 |Gromacs|. Most simulations require a lot of computational resources, therefore
940 it can be worthwhile to optimize the use of those resources. Several issues
941 mentioned in the list below could lead to a performance difference of a factor
942 of 2. So it can be useful go through the checklist.
944 |Gromacs| configuration
945 ^^^^^^^^^^^^^^^^^^^^^^^
947 * Don't use double precision unless you're absolute sure you need it.
948 * Compile the FFTW library (yourself) with the correct flags on x86 (in most
949 cases, the correct flags are automatically configured).
950 * On x86, use gcc or icc as the compiler (not pgi or the Cray compiler).
951 * On POWER, use gcc instead of IBM's xlc.
952 * Use a new compiler version, especially for gcc (e.g. from the version 5 to 6
953 the performance of the compiled code improved a lot).
954 * MPI library: OpenMPI usually has good performance and causes little trouble.
955 * Make sure your compiler supports OpenMP (some versions of Clang don't).
956 * If you have GPUs that support either CUDA or OpenCL, use them.
958 * Configure with ``-DGMX_GPU=ON`` (add ``-DGMX_USE_OPENCL=ON`` for OpenCL).
959 * For CUDA, use the newest CUDA availabe for your GPU to take advantage of the
960 latest performance enhancements.
961 * Use a recent GPU driver.
962 * If compiling on a cluster head node, make sure that ``GMX_SIMD``
963 is appropriate for the compute nodes.
968 * For an approximately spherical solute, use a rhombic dodecahedron unit cell.
969 * When using a time-step of 2 fs, use :mdp:`cutoff-scheme` = :mdp-value:`constraints=h-bonds`
970 (and not :mdp-value:`constraints=all-bonds`), since this is faster, especially with GPUs,
971 and most force fields have been parametrized with only bonds involving
972 hydrogens constrained.
973 * You can increase the time-step to 4 or 5 fs when using virtual interaction
974 sites (``gmx pdb2gmx -vsite h``).
975 * For massively parallel runs with PME, you might need to try different numbers
976 of PME ranks (``gmx mdrun -npme ???``) to achieve best performance;
977 :ref:`gmx tune_pme` can help automate this search.
978 * For massively parallel runs (also ``gmx mdrun -multidir``), or with a slow
979 network, global communication can become a bottleneck and you can reduce it
980 with ``gmx mdrun -gcom`` (note that this does affect the frequency of
981 temperature and pressure coupling).
983 Checking and improving performance
984 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
986 * Look at the end of the ``md.log`` file to see the performance and the cycle
987 counters and wall-clock time for different parts of the MD calculation. The
988 PP/PME load ratio is also printed, with a warning when a lot of performance is
989 lost due to imbalance.
990 * Adjust the number of PME ranks and/or the cut-off and PME grid-spacing when
991 there is a large PP/PME imbalance. Note that even with a small reported
992 imbalance, the automated PME-tuning might have reduced the initial imbalance.
993 You could still gain performance by changing the mdp parameters or increasing
994 the number of PME ranks.
995 * If the neighbor searching takes a lot of time, increase nstlist (with the
996 Verlet cut-off scheme, this automatically adjusts the size of the neighbour
997 list to do more non-bonded computation to keep energy drift constant).
999 * If ``Comm. energies`` takes a lot of time (a note will be printed in the log
1000 file), increase nstcalcenergy or use ``mdrun -gcom``.
1001 * If all communication takes a lot of time, you might be running on too many
1002 cores, or you could try running combined MPI/OpenMP parallelization with 2
1003 or 4 OpenMP threads per MPI process.
biod.pnpi.spb.ru / alexxy / gromacs.git / blob