3 Getting good performance from mdrun
4 ===================================
5 The |Gromacs| build system and the :ref:`gmx mdrun` tool has a lot of built-in
6 and configurable intelligence to detect your hardware and make pretty
7 effective use of that hardware. For a lot of casual and serious use of
8 :ref:`gmx mdrun`, the automatic machinery works well enough. But to get the
9 most from your hardware to maximize your scientific quality, read on!
11 Hardware background information
12 -------------------------------
13 Modern computer hardware is complex and heterogeneous, so we need to
14 discuss a little bit of background information and set up some
15 definitions. Experienced HPC users can skip this section.
20 A hardware compute unit that actually executes
21 instructions. There is normally more than one core in a
22 processor, often many more.
25 A special kind of memory local to core(s) that is much faster
26 to access than main memory, kind of like the top of a human's
27 desk, compared to their filing cabinet. There are often
28 several layers of caches associated with a core.
31 A group of cores that share some kind of locality, such as a
32 shared cache. This makes it more efficient to spread
33 computational work over cores within a socket than over cores
34 in different sockets. Modern processors often have more than
38 A group of sockets that share coarser-level locality, such as
39 shared access to the same memory without requiring any network
40 hardware. A normal laptop or desktop computer is a node. A
41 node is often the smallest amount of a large compute cluster
42 that a user can request to use.
45 A stream of instructions for a core to execute. There are many
46 different programming abstractions that create and manage
47 spreading computation over multiple threads, such as OpenMP,
48 pthreads, winthreads, CUDA, OpenCL, and OpenACC. Some kinds of
49 hardware can map more than one software thread to a core; on
50 Intel x86 processors this is called "hyper-threading", while
51 the more general concept is often called SMT for
52 "simultaneous multi-threading". IBM Power8 can for instance use
53 up to 8 hardware threads per core.
54 This feature can usually be enabled or disabled either in
55 the hardware bios or through a setting in the Linux operating
56 system. |Gromacs| can typically make use of this, for a moderate
57 free performance boost. In most cases it will be
58 enabled by default e.g. on new x86 processors, but in some cases
59 the system administrators might have disabled it. If that is the
60 case, ask if they can re-enable it for you. If you are not sure
61 if it is enabled, check the output of the CPU information in
62 the log file and compare with CPU specifications you find online.
64 thread affinity (pinning)
65 By default, most operating systems allow software threads to migrate
66 between cores (or hardware threads) to help automatically balance
67 workload. However, the performance of :ref:`gmx mdrun` can deteriorate
68 if this is permitted and will degrade dramatically especially when
69 relying on multi-threading within a rank. To avoid this,
70 :ref:`gmx mdrun` will by default
71 set the affinity of its threads to individual cores/hardware threads,
72 unless the user or software environment has already done so
73 (or not the entire node is used for the run, i.e. there is potential
75 Setting thread affinity is sometimes called thread "pinning".
78 The dominant multi-node parallelization-scheme, which provides
79 a standardized language in which programs can be written that
80 work across more than one node.
83 In MPI, a rank is the smallest grouping of hardware used in
84 the multi-node parallelization scheme. That grouping can be
85 controlled by the user, and might correspond to a core, a
86 socket, a node, or a group of nodes. The best choice varies
87 with the hardware, software and compute task. Sometimes an MPI
88 rank is called an MPI process.
91 A graphics processing unit, which is often faster and more
92 efficient than conventional processors for particular kinds of
93 compute workloads. A GPU is always associated with a
94 particular node, and often a particular socket within that
98 A standardized technique supported by many compilers to share
99 a compute workload over multiple cores. Often combined with
100 MPI to achieve hybrid MPI/OpenMP parallelism.
103 A proprietary parallel computing framework and API developed by NVIDIA
104 that allows targeting their accelerator hardware.
105 |Gromacs| uses CUDA for GPU acceleration support with NVIDIA hardware.
108 An open standard-based parallel computing framework that consists
109 of a C99-based compiler and a programming API for targeting heterogeneous
110 and accelerator hardware. |Gromacs| uses OpenCL for GPU acceleration
111 on AMD devices (both GPUs and APUs); NVIDIA hardware is also supported.
114 Modern CPU cores have instructions that can execute large
115 numbers of floating-point instructions in a single cycle.
118 |Gromacs| background information
119 --------------------------------
120 The algorithms in :ref:`gmx mdrun` and their implementations are most relevant
121 when choosing how to make good use of the hardware. For details,
122 see the Reference Manual. The most important of these are
127 The domain decomposition (DD) algorithm decomposes the
128 (short-ranged) component of the non-bonded interactions into
129 domains that share spatial locality, which permits the use of
130 efficient algorithms. Each domain handles all of the
131 particle-particle (PP) interactions for its members, and is
132 mapped to a single MPI rank. Within a PP rank, OpenMP threads
133 can share the workload, and some work can be off-loaded to a
134 GPU. The PP rank also handles any bonded interactions for the
135 members of its domain. A GPU may perform work for more than
136 one PP rank, but it is normally most efficient to use a single
137 PP rank per GPU and for that rank to have thousands of
138 particles. When the work of a PP rank is done on the CPU, mdrun
139 will make extensive use of the SIMD capabilities of the
140 core. There are various `command-line options
141 <controlling-the-domain-decomposition-algorithm` to control
142 the behaviour of the DD algorithm.
145 The particle-mesh Ewald (PME) algorithm treats the long-ranged
146 components of the non-bonded interactions (Coulomb and/or
147 Lennard-Jones). Either all, or just a subset of ranks may
148 participate in the work for computing long-ranged component
149 (often inaccurately called simple the "PME"
150 component). Because the algorithm uses a 3D FFT that requires
151 global communication, its performance gets worse as more ranks
152 participate, which can mean it is fastest to use just a subset
153 of ranks (e.g. one-quarter to one-half of the ranks). If
154 there are separate PME ranks, then the remaining ranks handle
155 the PP work. Otherwise, all ranks do both PP and PME work.
157 Running mdrun within a single node
158 ----------------------------------
160 :ref:`gmx mdrun` can be configured and compiled in several different ways that
161 are efficient to use within a single :term:`node`. The default configuration
162 using a suitable compiler will deploy a multi-level hybrid parallelism
163 that uses CUDA, OpenMP and the threading platform native to the
164 hardware. For programming convenience, in |Gromacs|, those native
165 threads are used to implement on a single node the same MPI scheme as
166 would be used between nodes, but much more efficient; this is called
167 thread-MPI. From a user's perspective, real MPI and thread-MPI look
168 almost the same, and |Gromacs| refers to MPI ranks to mean either kind,
169 except where noted. A real external MPI can be used for :ref:`gmx mdrun` within
170 a single node, but runs more slowly than the thread-MPI version.
172 By default, :ref:`gmx mdrun` will inspect the hardware available at run time
173 and do its best to make fairly efficient use of the whole node. The
174 log file, stdout and stderr are used to print diagnostics that
175 inform the user about the choices made and possible consequences.
177 A number of command-line parameters are available to modify the default
181 The total number of threads to use. The default, 0, will start as
182 many threads as available cores. Whether the threads are
183 thread-MPI ranks, and/or OpenMP threads within such ranks depends on
187 The total number of thread-MPI ranks to use. The default, 0,
188 will start one rank per GPU (if present), and otherwise one rank
192 The total number of OpenMP threads per rank to start. The
193 default, 0, will start one thread on each available core.
194 Alternatively, mdrun will honor the appropriate system
195 environment variable (e.g. ``OMP_NUM_THREADS``) if set.
198 The total number of ranks to dedicate to the long-ranged
199 component of PME, if used. The default, -1, will dedicate ranks
200 only if the total number of threads is at least 12, and will use
201 around a quarter of the ranks for the long-ranged component.
204 When using PME with separate PME ranks,
205 the total number of OpenMP threads per separate PME ranks.
206 The default, 0, copies the value from ``-ntomp``.
209 Can be set to "auto," "on" or "off" to control whether
210 mdrun will attempt to set the affinity of threads to cores.
211 Defaults to "auto," which means that if mdrun detects that all the
212 cores on the node are being used for mdrun, then it should behave
213 like "on," and attempt to set the affinities (unless they are
214 already set by something else).
217 If ``-pin on``, specifies the logical core number to
218 which mdrun should pin the first thread. When running more than
219 one instance of mdrun on a node, use this option to to avoid
220 pinning threads from different mdrun instances to the same core.
223 If ``-pin on``, specifies the stride in logical core
224 numbers for the cores to which mdrun should pin its threads. When
225 running more than one instance of mdrun on a node, use this option
226 to to avoid pinning threads from different mdrun instances to the
227 same core. Use the default, 0, to minimize the number of threads
228 per physical core - this lets mdrun manage the hardware-, OS- and
229 configuration-specific details of how to map logical cores to
233 Can be set to "interleave," "pp_pme" or "cartesian."
234 Defaults to "interleave," which means that any separate PME ranks
235 will be mapped to MPI ranks in an order like PP, PP, PME, PP, PP,
236 PME, ... etc. This generally makes the best use of the available
237 hardware. "pp_pme" maps all PP ranks first, then all PME
238 ranks. "cartesian" is a special-purpose mapping generally useful
239 only on special torus networks with accelerated global
240 communication for Cartesian communicators. Has no effect if there
241 are no separate PME ranks.
244 Used to set where to execute the non-bonded interactions.
245 Can be set to "auto", "cpu", "gpu."
246 Defaults to "auto," which uses a compatible GPU if available.
247 Setting "cpu" requires that no GPU is used. Setting "gpu" requires
248 that a compatible GPU be available and will be used.
251 A string that specifies the ID numbers of the GPUs that
252 are available to be used by ranks on this node. For example,
253 "12" specifies that the GPUs with IDs 1 and 2 (as reported
254 by the GPU runtime) can be used by mdrun. This is useful
255 when sharing a node with other computations, or if a GPU
256 is best used to support a display. If many GPUs are
257 present, a comma may be used to separate the IDs, so
258 "12,13" would make GPUs 12 and 13 available to mdrun.
259 It could be necessary to use different GPUs on different
260 nodes of a simulation, in which case the environment
261 variable ``GMX_GPU_ID`` can be set differently for the ranks
262 on different nodes to achieve that result.
265 A string that specifies the ID numbers of the GPUs to be
266 used by corresponding GPU tasks on this node. For example,
267 "0011" specifies that the first two GPU tasks will use GPU 0,
268 and the other two use GPU 1. When using this option, the
269 number of ranks must be known to mdrun, as well as where
270 tasks of different types should be run, such as by using
273 Examples for mdrun on one node
274 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
280 Starts mdrun using all the available resources. mdrun
281 will automatically choose a fairly efficient division
282 into thread-MPI ranks, OpenMP threads and assign work
283 to compatible GPUs. Details will vary with hardware
284 and the kind of simulation being run.
290 Starts mdrun using 8 threads, which might be thread-MPI
291 or OpenMP threads depending on hardware and the kind
292 of simulation being run.
296 gmx mdrun -ntmpi 2 -ntomp 4
298 Starts mdrun using eight total threads, with four thread-MPI
299 ranks and two OpenMP threads per core. You should only use
300 these options when seeking optimal performance, and
301 must take care that the ranks you create can have
302 all of their OpenMP threads run on the same socket.
303 The number of ranks must be a multiple of the number of
304 sockets, and the number of cores per node must be
305 a multiple of the number of threads per rank.
311 Starts mdrun using GPUs with IDs 1 and 2 (e.g. because
312 GPU 0 is dedicated to running a display). This requires
313 two thread-MPI ranks, and will split the available
314 CPU cores between them using OpenMP threads.
318 gmx mdrun -ntmpi 4 -nb gpu -gputasks 1122
320 Starts mdrun using four thread-MPI ranks, and maps them
321 to GPUs with IDs 1 and 2. The CPU cores available will
322 be split evenly between the ranks using OpenMP threads.
326 gmx mdrun -nt 6 -pin on -pinoffset 0
327 gmx mdrun -nt 6 -pin on -pinoffset 3
329 Starts two mdrun processes, each with six total threads.
330 Threads will have their affinities set to particular
331 logical cores, beginning from the logical core
332 with rank 0 or 3, respectively. The above would work
333 well on an Intel CPU with six physical cores and
334 hyper-threading enabled. Use this kind of setup only
335 if restricting mdrun to a subset of cores to share a
336 node with other processes.
340 mpirun -np 2 gmx_mpi mdrun
342 When using an :ref:`gmx mdrun` compiled with external MPI,
343 this will start two ranks and as many OpenMP threads
344 as the hardware and MPI setup will permit. If the
345 MPI setup is restricted to one node, then the resulting
346 :ref:`gmx mdrun` will be local to that node.
348 Running mdrun on more than one node
349 -----------------------------------
350 This requires configuring |Gromacs| to build with an external MPI
351 library. By default, this mdrun executable is run with
352 :ref:`mdrun_mpi`. All of the considerations for running single-node
353 mdrun still apply, except that ``-ntmpi`` and ``-nt`` cause a fatal
354 error, and instead the number of ranks is controlled by the
356 Settings such as ``-npme`` are much more important when
357 using multiple nodes. Configuring the MPI environment to
358 produce one rank per core is generally good until one
359 approaches the strong-scaling limit. At that point, using
360 OpenMP to spread the work of an MPI rank over more than one
361 core is needed to continue to improve absolute performance.
362 The location of the scaling limit depends on the processor,
363 presence of GPUs, network, and simulation algorithm, but
364 it is worth measuring at around ~200 particles/core if you
365 need maximum throughput.
367 There are further command-line parameters that are relevant in these
371 Defaults to "on." If "on," a Verlet-scheme simulation will
372 optimize various aspects of the PME and DD algorithms, shifting
373 load between ranks and/or GPUs to maximize throughput. Some
374 mdrun features are not compatible with this, and these ignore
378 Can be set to "auto," "no," or "yes."
379 Defaults to "auto." Doing Dynamic Load Balancing between MPI ranks
380 is needed to maximize performance. This is particularly important
381 for molecular systems with heterogeneous particle or interaction
382 density. When a certain threshold for performance loss is
383 exceeded, DLB activates and shifts particles between ranks to improve
387 During the simulation :ref:`gmx mdrun` must communicate between all ranks to
388 compute quantities such as kinetic energy. By default, this
389 happens whenever plausible, and is influenced by a lot of :ref:`[.mdp]
390 options. <mdp-general>` The period between communication phases
391 must be a multiple of :mdp:`nstlist`, and defaults to
392 the minimum of :mdp:`nstcalcenergy` and :mdp:`nstlist`.
393 ``mdrun -gcom`` sets the number of steps that must elapse between
394 such communication phases, which can improve performance when
395 running on a lot of ranks. Note that this means that _e.g._
396 temperature coupling algorithms will
397 effectively remain at constant energy until the next
398 communication phase. :ref:`gmx mdrun` will always honor the
399 setting of ``mdrun -gcom``, by changing :mdp:`nstcalcenergy`,
400 :mdp:`nstenergy`, :mdp:`nstlog`, :mdp:`nsttcouple` and/or
401 :mdp:`nstpcouple` if necessary.
403 Note that ``-tunepme`` has more effect when there is more than one
404 :term:`node`, because the cost of communication for the PP and PME
405 ranks differs. It still shifts load between PP and PME ranks, but does
406 not change the number of separate PME ranks in use.
408 Note also that ``-dlb`` and ``-tunepme`` can interfere with each other, so
409 if you experience performance variation that could result from this,
410 you may wish to tune PME separately, and run the result with ``mdrun
411 -notunepme -dlb yes``.
413 The :ref:`gmx tune_pme` utility is available to search a wider
414 range of parameter space, including making safe
415 modifications to the :ref:`tpr` file, and varying ``-npme``.
416 It is only aware of the number of ranks created by
417 the MPI environment, and does not explicitly manage
418 any aspect of OpenMP during the optimization.
420 Examples for mdrun on more than one node
421 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
422 The examples and explanations for for single-node mdrun are
423 still relevant, but ``-nt`` is no longer the way
424 to choose the number of MPI ranks.
428 mpirun -np 16 gmx_mpi mdrun
430 Starts :ref:`mdrun_mpi` with 16 ranks, which are mapped to
431 the hardware by the MPI library, e.g. as specified
432 in an MPI hostfile. The available cores will be
433 automatically split among ranks using OpenMP threads,
434 depending on the hardware and any environment settings
435 such as ``OMP_NUM_THREADS``.
439 mpirun -np 16 gmx_mpi mdrun -npme 5
441 Starts :ref:`mdrun_mpi` with 16 ranks, as above, and
442 require that 5 of them are dedicated to the PME
447 mpirun -np 11 gmx_mpi mdrun -ntomp 2 -npme 6 -ntomp_pme 1
449 Starts :ref:`mdrun_mpi` with 11 ranks, as above, and
450 require that six of them are dedicated to the PME
451 component with one OpenMP thread each. The remaining
452 five do the PP component, with two OpenMP threads
457 mpirun -np 4 gmx mdrun -ntomp 6 -nb gpu -gputasks 00
459 Starts :ref:`mdrun_mpi` on a machine with two nodes, using
460 four total ranks, each rank with six OpenMP threads,
461 and both ranks on a node sharing GPU with ID 0.
465 mpirun -np 8 gmx mdrun -ntomp 3 -gputasks 0000
467 Using a same/similar hardware as above,
468 starts :ref:`mdrun_mpi` on a machine with two nodes, using
469 eight total ranks, each rank with three OpenMP threads,
470 and all four ranks on a node sharing GPU with ID 0.
471 This may or may not be faster than the previous setup
472 on the same hardware.
476 mpirun -np 20 gmx_mpi mdrun -ntomp 4 -gputasks 00
478 Starts :ref:`mdrun_mpi` with 20 ranks, and assigns the CPU cores evenly
479 across ranks each to one OpenMP thread. This setup is likely to be
480 suitable when there are ten nodes, each with one GPU, and each node
481 has two sockets each of four cores.
485 mpirun -np 10 gmx_mpi mdrun -gpu_id 1
487 Starts :ref:`mdrun_mpi` with 20 ranks, and assigns the CPU cores evenly
488 across ranks each to one OpenMP thread. This setup is likely to be
489 suitable when there are ten nodes, each with two GPUs, but another
490 job on each node is using GPU 0. The job scheduler should set the
491 affinity of threads of both jobs to their allocated cores, or the
492 performance of mdrun will suffer greatly.
496 mpirun -np 20 gmx_mpi mdrun -gpu_id 01
498 Starts :ref:`mdrun_mpi` with 20 ranks. This setup is likely
499 to be suitable when there are ten nodes, each with two
500 GPUs, but there is no need to specify ``-gpu_id`` for the
501 normal case where all the GPUs on the node are available
504 Controlling the domain decomposition algorithm
505 ----------------------------------------------
506 This section lists all the options that affect how the domain
507 decomposition algorithm decomposes the workload to the available
511 Can be used to set the required maximum distance for inter
512 charge-group bonded interactions. Communication for two-body
513 bonded interactions below the non-bonded cut-off distance always
514 comes for free with the non-bonded communication. Particles beyond
515 the non-bonded cut-off are only communicated when they have
516 missing bonded interactions; this means that the extra cost is
517 minor and nearly independent of the value of ``-rdd``. With dynamic
518 load balancing, option ``-rdd`` also sets the lower limit for the
519 domain decomposition cell sizes. By default ``-rdd`` is determined
520 by :ref:`gmx mdrun` based on the initial coordinates. The chosen value will
521 be a balance between interaction range and communication cost.
524 On by default. When inter charge-group bonded interactions are
525 beyond the bonded cut-off distance, :ref:`gmx mdrun` terminates with an
526 error message. For pair interactions and tabulated bonds that do
527 not generate exclusions, this check can be turned off with the
528 option ``-noddcheck``.
531 When constraints are present, option ``-rcon`` influences
532 the cell size limit as well.
533 Particles connected by NC constraints, where NC is the LINCS order
534 plus 1, should not be beyond the smallest cell size. A error
535 message is generated when this happens, and the user should change
536 the decomposition or decrease the LINCS order and increase the
537 number of LINCS iterations. By default :ref:`gmx mdrun` estimates the
538 minimum cell size required for P-LINCS in a conservative
539 fashion. For high parallelization, it can be useful to set the
540 distance required for P-LINCS with ``-rcon``.
543 Sets the minimum allowed x, y and/or z scaling of the cells with
544 dynamic load balancing. :ref:`gmx mdrun` will ensure that the cells can
545 scale down by at least this factor. This option is used for the
546 automated spatial decomposition (when not using ``-dd``) as well as
547 for determining the number of grid pulses, which in turn sets the
548 minimum allowed cell size. Under certain circumstances the value
549 of ``-dds`` might need to be adjusted to account for high or low
550 spatial inhomogeneity of the system.
552 Finding out how to run mdrun better
553 -----------------------------------
555 The Wallcycle module is used for runtime performance measurement of :ref:`gmx mdrun`.
556 At the end of the log file of each run, the "Real cycle and time accounting" section
557 provides a table with runtime statistics for different parts of the :ref:`gmx mdrun` code
558 in rows of the table.
559 The table contains colums indicating the number of ranks and threads that
560 executed the respective part of the run, wall-time and cycle
561 count aggregates (across all threads and ranks) averaged over the entire run.
562 The last column also shows what precentage of the total runtime each row represents.
563 Note that the :ref:`gmx mdrun` timer resetting functionalities (`-resethway` and `-resetstep`)
564 reset the performance counters and therefore are useful to avoid startup overhead and
565 performance instability (e.g. due to load balancing) at the beginning of the run.
567 The performance counters are:
569 * Particle-particle during Particle mesh Ewald
570 * Domain decomposition
571 * Domain decomposition communication load
572 * Domain decomposition communication bounds
573 * Virtual site constraints
574 * Send X to Particle mesh Ewald
576 * Launch GPU operations
577 * Communication of coordinates
580 * Waiting + Communication of force
581 * Particle mesh Ewald
585 * PME 3D-FFT Communication
586 * PME solve Lennard-Jones
588 * PME wait for particle-particle
589 * Wait + Receive PME force
592 * Non-bonded position/force buffer operations
593 * Virtual site spread
598 * Communication of energies
600 * Add rotational forces
604 As performance data is collected for every run, they are essential to assessing
605 and tuning the performance of :ref:`gmx mdrun` performance. Therefore, they benefit
606 both code developers as well as users of the program.
607 The counters are an average of the time/cycles different parts of the simulation take,
608 hence can not directly reveal fluctuations during a single run (although comparisons across
609 multiple runs are still very useful).
611 Counters will appear in MD log file only if the related parts of the code were
612 executed during the :ref:`gmx mdrun` run. There is also a special counter called "Rest" which
613 indicated for the amount of time not accounted for by any of the counters above. Theerfore,
614 a significant amount "Rest" time (more than a few percent) will often be an indication of
615 parallelization inefficiency (e.g. serial code) and it is recommended to be reported to the
618 An additional set of subcounters can offer more fine-grained inspection of performance. They are:
620 * Domain decomposition redistribution
621 * DD neighbor search grid + sort
622 * DD setup communication
624 * DD make constraints
626 * Neighbor search grid local
629 * NS search non-local
633 * Listed buffer operations
635 * Ewald force correction
636 * Non-bonded position buffer operations
637 * Non-bonded force buffer operations
639 Subcounters are geared toward developers and have to be enabled during compilation. See
640 :doc:`/dev-manual/build-system` for more information.
642 TODO In future patch:
643 - red flags in log files, how to interpret wallcycle output
644 - hints to devs how to extend wallcycles
646 TODO In future patch: import wiki page stuff on performance checklist; maybe here,
649 .. _gmx-mdrun-on-gpu:
651 Running mdrun with GPUs
652 -----------------------
654 NVIDIA GPUs from the professional line (Tesla or Quadro) starting with
655 the Kepler generation (compute capability 3.5 and later) support changing the
656 processor and memory clock frequency with the help of the applications clocks feature.
657 With many workloads, using higher clock rates than the default provides significant
658 performance improvements.
659 For more information see the `NVIDIA blog article`_ on this topic.
660 For |Gromacs| the highest application clock rates are optimal on all hardware
661 available to date (up to and including Maxwell, compute capability 5.2).
663 Application clocks can be set using the NVIDIA system managemet tool
664 ``nvidia-smi``. If the system permissions allow, :ref:`gmx mdrun` has
665 built-in support to set application clocks if built with :ref:`NVML support<CUDA GPU acceleration>`.
666 Note that application clocks are a global setting, hence affect the
667 performance of all applications that use the respective GPU(s).
668 For this reason, :ref:`gmx mdrun` sets application clocks at initialization
669 to the values optimal for |Gromacs| and it restores them before exiting
670 to the values found at startup, unless it detects that they were altered
673 .. _NVIDIA blog article: https://devblogs.nvidia.com/parallelforall/increase-performance-gpu-boost-k80-autoboost/
675 Reducing overheads in GPU accelerated runs
676 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
678 In order for CPU cores and GPU(s) to execute concurrently, tasks are
679 launched and executed asynchronously on the GPU(s) while the CPU cores
680 execute non-offloaded force computation (like long-range PME electrostatics).
681 Asynchronous task launches are handled by GPU device driver and
682 require CPU involvement. Therefore, the work of scheduling
683 GPU tasks will incur an overhead that can in some cases significantly
684 delay or interfere with the CPU execution.
686 Delays in CPU execution are caused by the latency of launching GPU tasks,
687 an overhead that can become significant as simulation ns/day increases
688 (i.e. with shorter wall-time per step).
689 The overhead is measured by :ref:`gmx mdrun` and reported in the performance
690 summary section of the log file ("Launch GPU ops" row).
691 A few percent of runtime spent in this category is normal,
692 but in fast-iterating and multi-GPU parallel runs 10% or larger overheads can be observed.
693 In general, there a user can do little to avoid such overheads, but there
694 are a few cases where tweaks can give performance benefits.
695 In single-rank runs timing of GPU tasks is by default enabled and,
696 while in most cases its impact is small, in fast runs performance can be affected.
697 The performance impact will be most significant on NVIDIA GPUs with CUDA,
698 less on AMD with OpenCL.
699 In these cases, when more than a few percent of "Launch GPU ops" time is observed,
700 it is recommended turning off timing by setting the ``GMX_DISABLE_GPU_TIMING``
701 environment variable.
702 In parallel runs with with many ranks sharing a GPU
703 launch overheads can also be reduced by staring fewer thread-MPI
704 or MPI ranks per GPU; e.g. most often one rank per thread or core is not optimal.
706 The second type of overhead, interference of the GPU driver with CPU computation,
707 is caused by the scheduling and coordination of GPU tasks.
708 A separate GPU driver thread can require CPU resources
709 which may clash with the concurrently running non-offloaded tasks,
710 potentially degrading the performance of PME or bonded force computation.
711 This effect is most pronounced when using AMD GPUs with OpenCL with
712 all stable driver releases to date (up to and including fglrx 12.15).
713 To minimize the overhead it is recommended to
714 leave a CPU hardware thread unused when launching :ref:`gmx mdrun`,
715 especially on CPUs with high core count and/or HyperThreading enabled.
716 E.g. on a machine with a 4-core CPU and eight threads (via HyperThreading) and an AMD GPU,
717 try ``gmx mdrun -ntomp 7 -pin on``.
718 This will leave free CPU resources for the GPU task scheduling
719 reducing interference with CPU computation.
720 Note that assigning fewer resources to :ref:`gmx mdrun` CPU computation
721 involves a tradeoff which may outweigh the benefits of reduced GPU driver overhead,
722 in particular without HyperThreading and with few CPU cores.
724 TODO In future patch: any tips not covered above
726 Running the OpenCL version of mdrun
727 -----------------------------------
729 The current version works with GCN-based AMD GPUs, and NVIDIA CUDA
730 GPUs. Make sure that you have the latest drivers installed. For AMD GPUs,
731 Mesa version 17.0 or newer with LLVM 4.0 or newer is supported in addition
732 to the proprietary driver. For NVIDIA GPUs, using the proprietary driver is
733 required as the open source nouveau driver (available in Mesa) does not
734 provide the OpenCL support.
735 The minimum OpenCL version required is |REQUIRED_OPENCL_MIN_VERSION|. See
736 also the :ref:`known limitations <opencl-known-limitations>`.
738 Devices from the AMD GCN architectures (all series) and NVIDIA Fermi
739 and later (compute capability 2.0) are known to work, but before
740 doing production runs always make sure that the |Gromacs| tests
741 pass successfully on the hardware.
743 The OpenCL GPU kernels are compiled at run time. Hence,
744 building the OpenCL program can take a few seconds introducing a slight
745 delay in the :ref:`gmx mdrun` startup. This is not normally a
746 problem for long production MD, but you might prefer to do some kinds
747 of work, e.g. that runs very few steps, on just the CPU (e.g. see ``-nb`` above).
749 The same ``-gpu_id`` option (or ``GMX_GPU_ID`` environment variable)
750 used to select CUDA devices, or to define a mapping of GPUs to PP
751 ranks, is used for OpenCL devices.
753 Some other :ref:`OpenCL management <opencl-management>` environment
754 variables may be of interest to developers.
756 .. _opencl-known-limitations:
758 Known limitations of the OpenCL support
759 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
761 Limitations in the current OpenCL support of interest to |Gromacs| users:
763 - No Intel devices (CPUs, GPUs or Xeon Phi) are supported
764 - Due to blocking behavior of some asynchronous task enqueuing functions
765 in the NVIDIA OpenCL runtime, with the affected driver versions there is
766 almost no performance gain when using NVIDIA GPUs.
767 The issue affects NVIDIA driver versions up to 349 series, but it
768 known to be fixed 352 and later driver releases.
769 - On NVIDIA GPUs the OpenCL kernels achieve much lower performance
770 than the equivalent CUDA kernels due to limitations of the NVIDIA OpenCL
772 - The AMD APPSDK version 3.0 ships with OpenCL compiler/runtime components,
773 libamdocl12cl64.so and libamdocl64.so (only in earlier releases),
774 that conflict with newer fglrx GPU drivers which provide the same libraries.
775 This conflict manifests in kernel launch failures as, due to the library path
776 setup, the OpenCL runtime loads the APPSDK version of the aforementioned
777 libraries instead of the ones provided by the driver installer.
778 The recommended workaround is to remove or rename the APPSDK versions of the
781 Limitations of interest to |Gromacs| developers:
783 - The current implementation is not compatible with OpenCL devices that are
784 not using warp/wavefronts or for which the warp/wavefront size is not a
786 - Some Ewald tabulated kernels are known to produce incorrect results, so
787 (correct) analytical kernels are used instead.
789 Performance checklist
790 ---------------------
792 There are many different aspects that affect the performance of simulations in
793 |Gromacs|. Most simulations require a lot of computational resources, therefore
794 it can be worthwhile to optimize the use of those resources. Several issues
795 mentioned in the list below could lead to a performance difference of a factor
796 of 2. So it can be useful go through the checklist.
798 |Gromacs| configuration
799 ^^^^^^^^^^^^^^^^^^^^^^^
801 * Don't use double precision unless you're absolute sure you need it.
802 * Compile the FFTW library (yourself) with the correct flags on x86 (in most
803 cases, the correct flags are automatically configured).
804 * On x86, use gcc or icc as the compiler (not pgi or the Cray compiler).
805 * On POWER, use gcc instead of IBM's xlc.
806 * Use a new compiler version, especially for gcc (e.g. from the version 5 to 6
807 the performance of the compiled code improved a lot).
808 * MPI library: OpenMPI usually has good performance and causes little trouble.
809 * Make sure your compiler supports OpenMP (some versions of Clang don't).
810 * If you have GPUs that support either CUDA or OpenCL, use them.
812 * Configure with ``-DGMX_GPU=ON`` (add ``-DGMX_USE_OPENCL=ON`` for OpenCL).
813 * For CUDA, use the newest CUDA availabe for your GPU to take advantage of the
814 latest performance enhancements.
815 * Use a recent GPU driver.
816 * If compiling on a cluster head node, make sure that ``GMX_SIMD``
817 is appropriate for the compute nodes.
822 * For an approximately spherical solute, use a rhombic dodecahedron unit cell.
823 * When using a time-step of 2 fs, use :mdp:`cutoff-scheme` = :mdp-value:`constraints=h-bonds`
824 (and not :mdp-value:`constraints=all-bonds`), since this is faster, especially with GPUs,
825 and most force fields have been parametrized with only bonds involving
826 hydrogens constrained.
827 * You can increase the time-step to 4 or 5 fs when using virtual interaction
828 sites (``gmx pdb2gmx -vsite h``).
829 * For massively parallel runs with PME, you might need to try different numbers
830 of PME ranks (``gmx mdrun -npme ???``) to achieve best performance;
831 :ref:`gmx tune_pme` can help automate this search.
832 * For massively parallel runs (also ``gmx mdrun -multidir``), or with a slow
833 network, global communication can become a bottleneck and you can reduce it
834 with ``gmx mdrun -gcom`` (note that this does affect the frequency of
835 temperature and pressure coupling).
837 Checking and improving performance
838 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
840 * Look at the end of the ``md.log`` file to see the performance and the cycle
841 counters and wall-clock time for different parts of the MD calculation. The
842 PP/PME load ratio is also printed, with a warning when a lot of performance is
843 lost due to imbalance.
844 * Adjust the number of PME ranks and/or the cut-off and PME grid-spacing when
845 there is a large PP/PME imbalance. Note that even with a small reported
846 imbalance, the automated PME-tuning might have reduced the initial imbalance.
847 You could still gain performance by changing the mdp parameters or increasing
848 the number of PME ranks.
849 * If the neighbor searching takes a lot of time, increase nstlist (with the
850 Verlet cut-off scheme, this automatically adjusts the size of the neighbour
851 list to do more non-bonded computation to keep energy drift constant).
853 * If ``Comm. energies`` takes a lot of time (a note will be printed in the log
854 file), increase nstcalcenergy or use ``mdrun -gcom``.
855 * If all communication takes a lot of time, you might be running on too many
856 cores, or you could try running combined MPI/OpenMP parallelization with 2
857 or 4 OpenMP threads per MPI process.
biod.pnpi.spb.ru / alexxy / gromacs.git / blob