1 Getting good performance from mdrun
2 ===================================
3 The |Gromacs| build system and the :ref:`gmx mdrun` tool has a lot of built-in
4 and configurable intelligence to detect your hardware and make pretty
5 effective use of that hardware. For a lot of casual and serious use of
6 :ref:`gmx mdrun`, the automatic machinery works well enough. But to get the
7 most from your hardware to maximize your scientific quality, read on!
9 Hardware background information
10 -------------------------------
11 Modern computer hardware is complex and heterogeneous, so we need to
12 discuss a little bit of background information and set up some
13 definitions. Experienced HPC users can skip this section.
18 A hardware compute unit that actually executes
19 instructions. There is normally more than one core in a
20 processor, often many more.
23 A special kind of memory local to core(s) that is much faster
24 to access than main memory, kind of like the top of a human's
25 desk, compared to their filing cabinet. There are often
26 several layers of caches associated with a core.
29 A group of cores that share some kind of locality, such as a
30 shared cache. This makes it more efficient to spread
31 computational work over cores within a socket than over cores
32 in different sockets. Modern processors often have more than
36 A group of sockets that share coarser-level locality, such as
37 shared access to the same memory without requiring any network
38 hardware. A normal laptop or desktop computer is a node. A
39 node is often the smallest amount of a large compute cluster
40 that a user can request to use.
43 A stream of instructions for a core to execute. There are many
44 different programming abstractions that create and manage
45 spreading computation over multiple threads, such as OpenMP,
46 pthreads, winthreads, CUDA, OpenCL, and OpenACC. Some kinds of
47 hardware can map more than one software thread to a core; on
48 Intel x86 processors this is called "hyper-threading", while
49 the more general concept is often called SMT for
50 "simultaneous multi-threading". IBM Power8 can for instance use
51 up to 8 hardware threads per core.
52 This feature can usually be enabled or disabled either in
53 the hardware bios or through a setting in the Linux operating
54 system. |Gromacs| can typically make use of this, for a moderate
55 free performance boost. In most cases it will be
56 enabled by default e.g. on new x86 processors, but in some cases
57 the system administrators might have disabled it. If that is the
58 case, ask if they can re-enable it for you. If you are not sure
59 if it is enabled, check the output of the CPU information in
60 the log file and compare with CPU specifications you find online.
62 thread affinity (pinning)
63 By default, most operating systems allow software threads to migrate
64 between cores (or hardware threads) to help automatically balance
65 workload. However, the performance of :ref:`gmx mdrun` can deteriorate
66 if this is permitted and will degrade dramatically especially when
67 relying on multi-threading within a rank. To avoid this,
68 :ref:`gmx mdrun` will by default
69 set the affinity of its threads to individual cores/hardware threads,
70 unless the user or software environment has already done so
71 (or not the entire node is used for the run, i.e. there is potential
73 Setting thread affinity is sometimes called thread "pinning".
76 The dominant multi-node parallelization-scheme, which provides
77 a standardized language in which programs can be written that
78 work across more than one node.
81 In MPI, a rank is the smallest grouping of hardware used in
82 the multi-node parallelization scheme. That grouping can be
83 controlled by the user, and might correspond to a core, a
84 socket, a node, or a group of nodes. The best choice varies
85 with the hardware, software and compute task. Sometimes an MPI
86 rank is called an MPI process.
89 A graphics processing unit, which is often faster and more
90 efficient than conventional processors for particular kinds of
91 compute workloads. A GPU is always associated with a
92 particular node, and often a particular socket within that
96 A standardized technique supported by many compilers to share
97 a compute workload over multiple cores. Often combined with
98 MPI to achieve hybrid MPI/OpenMP parallelism.
101 A proprietary parallel computing framework and API developed by NVIDIA
102 that allows targeting their accelerator hardware.
103 |Gromacs| uses CUDA for GPU acceleration support with NVIDIA hardware.
106 An open standard-based parallel computing framework that consists
107 of a C99-based compiler and a programming API for targeting heterogeneous
108 and accelerator hardware. |Gromacs| uses OpenCL for GPU acceleration
109 on AMD devices (both GPUs and APUs); NVIDIA hardware is also supported.
112 Modern CPU cores have instructions that can execute large
113 numbers of floating-point instructions in a single cycle.
116 |Gromacs| background information
117 --------------------------------
118 The algorithms in :ref:`gmx mdrun` and their implementations are most relevant
119 when choosing how to make good use of the hardware. For details,
120 see the Reference Manual. The most important of these are
125 The domain decomposition (DD) algorithm decomposes the
126 (short-ranged) component of the non-bonded interactions into
127 domains that share spatial locality, which permits the use of
128 efficient algorithms. Each domain handles all of the
129 particle-particle (PP) interactions for its members, and is
130 mapped to a single MPI rank. Within a PP rank, OpenMP threads
131 can share the workload, and some work can be off-loaded to a
132 GPU. The PP rank also handles any bonded interactions for the
133 members of its domain. A GPU may perform work for more than
134 one PP rank, but it is normally most efficient to use a single
135 PP rank per GPU and for that rank to have thousands of
136 particles. When the work of a PP rank is done on the CPU, mdrun
137 will make extensive use of the SIMD capabilities of the
138 core. There are various `command-line options
139 <controlling-the-domain-decomposition-algorithm` to control
140 the behaviour of the DD algorithm.
143 The particle-mesh Ewald (PME) algorithm treats the long-ranged
144 components of the non-bonded interactions (Coulomb and/or
145 Lennard-Jones). Either all, or just a subset of ranks may
146 participate in the work for computing long-ranged component
147 (often inaccurately called simple the "PME"
148 component). Because the algorithm uses a 3D FFT that requires
149 global communication, its performance gets worse as more ranks
150 participate, which can mean it is fastest to use just a subset
151 of ranks (e.g. one-quarter to one-half of the ranks). If
152 there are separate PME ranks, then the remaining ranks handle
153 the PP work. Otherwise, all ranks do both PP and PME work.
155 Running mdrun within a single node
156 ----------------------------------
158 :ref:`gmx mdrun` can be configured and compiled in several different ways that
159 are efficient to use within a single :term:`node`. The default configuration
160 using a suitable compiler will deploy a multi-level hybrid parallelism
161 that uses CUDA, OpenMP and the threading platform native to the
162 hardware. For programming convenience, in |Gromacs|, those native
163 threads are used to implement on a single node the same MPI scheme as
164 would be used between nodes, but much more efficient; this is called
165 thread-MPI. From a user's perspective, real MPI and thread-MPI look
166 almost the same, and |Gromacs| refers to MPI ranks to mean either kind,
167 except where noted. A real external MPI can be used for :ref:`gmx mdrun` within
168 a single node, but runs more slowly than the thread-MPI version.
170 By default, :ref:`gmx mdrun` will inspect the hardware available at run time
171 and do its best to make fairly efficient use of the whole node. The
172 log file, stdout and stderr are used to print diagnostics that
173 inform the user about the choices made and possible consequences.
175 A number of command-line parameters are available to modify the default
179 The total number of threads to use. The default, 0, will start as
180 many threads as available cores. Whether the threads are
181 thread-MPI ranks, and/or OpenMP threads within such ranks depends on
185 The total number of thread-MPI ranks to use. The default, 0,
186 will start one rank per GPU (if present), and otherwise one rank
190 The total number of OpenMP threads per rank to start. The
191 default, 0, will start one thread on each available core.
192 Alternatively, mdrun will honor the appropriate system
193 environment variable (e.g. ``OMP_NUM_THREADS``) if set.
196 The total number of ranks to dedicate to the long-ranged
197 component of PME, if used. The default, -1, will dedicate ranks
198 only if the total number of threads is at least 12, and will use
199 around a quarter of the ranks for the long-ranged component.
202 When using PME with separate PME ranks,
203 the total number of OpenMP threads per separate PME ranks.
204 The default, 0, copies the value from ``-ntomp``.
207 A string that specifies the ID numbers of the GPUs to be
208 used by corresponding PP ranks on this node. For example,
209 "0011" specifies that the lowest two PP ranks use GPU 0,
210 and the other two use GPU 1.
213 Can be set to "auto," "on" or "off" to control whether
214 mdrun will attempt to set the affinity of threads to cores.
215 Defaults to "auto," which means that if mdrun detects that all the
216 cores on the node are being used for mdrun, then it should behave
217 like "on," and attempt to set the affinities (unless they are
218 already set by something else).
221 If ``-pin on``, specifies the logical core number to
222 which mdrun should pin the first thread. When running more than
223 one instance of mdrun on a node, use this option to to avoid
224 pinning threads from different mdrun instances to the same core.
227 If ``-pin on``, specifies the stride in logical core
228 numbers for the cores to which mdrun should pin its threads. When
229 running more than one instance of mdrun on a node, use this option
230 to to avoid pinning threads from different mdrun instances to the
231 same core. Use the default, 0, to minimize the number of threads
232 per physical core - this lets mdrun manage the hardware-, OS- and
233 configuration-specific details of how to map logical cores to
237 Can be set to "interleave," "pp_pme" or "cartesian."
238 Defaults to "interleave," which means that any separate PME ranks
239 will be mapped to MPI ranks in an order like PP, PP, PME, PP, PP,
240 PME, ... etc. This generally makes the best use of the available
241 hardware. "pp_pme" maps all PP ranks first, then all PME
242 ranks. "cartesian" is a special-purpose mapping generally useful
243 only on special torus networks with accelerated global
244 communication for Cartesian communicators. Has no effect if there
245 are no separate PME ranks.
248 Used to set where to execute the non-bonded interactions.
249 Can be set to "auto", "cpu", "gpu", "gpu_cpu."
250 Defaults to "auto," which uses a compatible GPU if available.
251 Setting "cpu" requires that no GPU is used. Setting "gpu" requires
252 that a compatible GPU be available and will be used. Setting
253 "gpu_cpu" lets the GPU compute the local and the CPU the non-local
254 non-bonded interactions. Is only faster under a narrow range of
257 Examples for mdrun on one node
258 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
264 Starts mdrun using all the available resources. mdrun
265 will automatically choose a fairly efficient division
266 into thread-MPI ranks, OpenMP threads and assign work
267 to compatible GPUs. Details will vary with hardware
268 and the kind of simulation being run.
274 Starts mdrun using 8 threads, which might be thread-MPI
275 or OpenMP threads depending on hardware and the kind
276 of simulation being run.
280 gmx mdrun -ntmpi 2 -ntomp 4
282 Starts mdrun using eight total threads, with four thread-MPI
283 ranks and two OpenMP threads per core. You should only use
284 these options when seeking optimal performance, and
285 must take care that the ranks you create can have
286 all of their OpenMP threads run on the same socket.
287 The number of ranks must be a multiple of the number of
288 sockets, and the number of cores per node must be
289 a multiple of the number of threads per rank.
295 Starts mdrun using GPUs with IDs 1 and 2 (e.g. because
296 GPU 0 is dedicated to running a display). This requires
297 two thread-MPI ranks, and will split the available
298 CPU cores between them using OpenMP threads.
302 gmx mdrun -ntmpi 4 -gpu_id "1122"
304 Starts mdrun using four thread-MPI ranks, and maps them
305 to GPUs with IDs 1 and 2. The CPU cores available will
306 be split evenly between the ranks using OpenMP threads.
310 gmx mdrun -nt 6 -pin on -pinoffset 0
311 gmx mdrun -nt 6 -pin on -pinoffset 3
313 Starts two mdrun processes, each with six total threads.
314 Threads will have their affinities set to particular
315 logical cores, beginning from the logical core
316 with rank 0 or 3, respectively. The above would work
317 well on an Intel CPU with six physical cores and
318 hyper-threading enabled. Use this kind of setup only
319 if restricting mdrun to a subset of cores to share a
320 node with other processes.
324 mpirun -np 2 gmx_mpi mdrun
326 When using an :ref:`gmx mdrun` compiled with external MPI,
327 this will start two ranks and as many OpenMP threads
328 as the hardware and MPI setup will permit. If the
329 MPI setup is restricted to one node, then the resulting
330 :ref:`gmx mdrun` will be local to that node.
332 Running mdrun on more than one node
333 -----------------------------------
334 This requires configuring |Gromacs| to build with an external MPI
335 library. By default, this mdrun executable is run with
336 :ref:`mdrun_mpi`. All of the considerations for running single-node
337 mdrun still apply, except that ``-ntmpi`` and ``-nt`` cause a fatal
338 error, and instead the number of ranks is controlled by the
340 Settings such as ``-npme`` are much more important when
341 using multiple nodes. Configuring the MPI environment to
342 produce one rank per core is generally good until one
343 approaches the strong-scaling limit. At that point, using
344 OpenMP to spread the work of an MPI rank over more than one
345 core is needed to continue to improve absolute performance.
346 The location of the scaling limit depends on the processor,
347 presence of GPUs, network, and simulation algorithm, but
348 it is worth measuring at around ~200 particles/core if you
349 need maximum throughput.
351 There are further command-line parameters that are relevant in these
355 Defaults to "on." If "on," a Verlet-scheme simulation will
356 optimize various aspects of the PME and DD algorithms, shifting
357 load between ranks and/or GPUs to maximize throughput. Some
358 mdrun features are not compatible with this, and these ignore
362 Can be set to "auto," "no," or "yes."
363 Defaults to "auto." Doing Dynamic Load Balancing between MPI ranks
364 is needed to maximize performance. This is particularly important
365 for molecular systems with heterogeneous particle or interaction
366 density. When a certain threshold for performance loss is
367 exceeded, DLB activates and shifts particles between ranks to improve
371 During the simulation :ref:`gmx mdrun` must communicate between all ranks to
372 compute quantities such as kinetic energy. By default, this
373 happens whenever plausible, and is influenced by a lot of [.mdp
374 options](#mdp-options). The period between communication phases
375 must be a multiple of :mdp:`nstlist`, and defaults to
376 the minimum of :mdp:`nstcalcenergy` and :mdp:`nstlist`.
377 ``mdrun -gcom`` sets the number of steps that must elapse between
378 such communication phases, which can improve performance when
379 running on a lot of ranks. Note that this means that _e.g._
380 temperature coupling algorithms will
381 effectively remain at constant energy until the next
382 communication phase. :ref:`gmx mdrun` will always honor the
383 setting of ``mdrun -gcom``, by changing :mdp:`nstcalcenergy`,
384 :mdp:`nstenergy`, :mdp:`nstlog`, :mdp:`nsttcouple` and/or
385 :mdp:`nstpcouple` if necessary.
387 Note that ``-tunepme`` has more effect when there is more than one
388 :term:`node`, because the cost of communication for the PP and PME
389 ranks differs. It still shifts load between PP and PME ranks, but does
390 not change the number of separate PME ranks in use.
392 Note also that ``-dlb`` and ``-tunepme`` can interfere with each other, so
393 if you experience performance variation that could result from this,
394 you may wish to tune PME separately, and run the result with ``mdrun
395 -notunepme -dlb yes``.
397 The :ref:`gmx tune_pme` utility is available to search a wider
398 range of parameter space, including making safe
399 modifications to the :ref:`tpr` file, and varying ``-npme``.
400 It is only aware of the number of ranks created by
401 the MPI environment, and does not explicitly manage
402 any aspect of OpenMP during the optimization.
404 Examples for mdrun on more than one node
405 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
406 The examples and explanations for for single-node mdrun are
407 still relevant, but ``-nt`` is no longer the way
408 to choose the number of MPI ranks.
412 mpirun -np 16 gmx_mpi mdrun
414 Starts :ref:`mdrun_mpi` with 16 ranks, which are mapped to
415 the hardware by the MPI library, e.g. as specified
416 in an MPI hostfile. The available cores will be
417 automatically split among ranks using OpenMP threads,
418 depending on the hardware and any environment settings
419 such as ``OMP_NUM_THREADS``.
423 mpirun -np 16 gmx_mpi mdrun -npme 5
425 Starts :ref:`mdrun_mpi` with 16 ranks, as above, and
426 require that 5 of them are dedicated to the PME
431 mpirun -np 11 gmx_mpi mdrun -ntomp 2 -npme 6 -ntomp_pme 1
433 Starts :ref:`mdrun_mpi` with 11 ranks, as above, and
434 require that six of them are dedicated to the PME
435 component with one OpenMP thread each. The remaining
436 five do the PP component, with two OpenMP threads
441 mpirun -np 4 gmx mdrun -ntomp 6 -gpu_id 00
443 Starts :ref:`mdrun_mpi` on a machine with two nodes, using
444 four total ranks, each rank with six OpenMP threads,
445 and both ranks on a node sharing GPU with ID 0.
449 mpirun -np 8 gmx mdrun -ntomp 3 -gpu_id 0000
451 Using a same/similar hardware as above,
452 starts :ref:`mdrun_mpi` on a machine with two nodes, using
453 eight total ranks, each rank with three OpenMP threads,
454 and all four ranks on a node sharing GPU with ID 0.
455 This may or may not be faster than the previous setup
456 on the same hardware.
460 mpirun -np 20 gmx_mpi mdrun -ntomp 4 -gpu_id 0
462 Starts :ref:`mdrun_mpi` with 20 ranks, and assigns the CPU cores evenly
463 across ranks each to one OpenMP thread. This setup is likely to be
464 suitable when there are ten nodes, each with one GPU, and each node
469 mpirun -np 20 gmx_mpi mdrun -gpu_id 00
471 Starts :ref:`mdrun_mpi` with 20 ranks, and assigns the CPU cores evenly
472 across ranks each to one OpenMP thread. This setup is likely to be
473 suitable when there are ten nodes, each with one GPU, and each node
478 mpirun -np 20 gmx_mpi mdrun -gpu_id 01
480 Starts :ref:`mdrun_mpi` with 20 ranks. This setup is likely
481 to be suitable when there are ten nodes, each with two
486 mpirun -np 40 gmx_mpi mdrun -gpu_id 0011
488 Starts :ref:`mdrun_mpi` with 40 ranks. This setup is likely
489 to be suitable when there are ten nodes, each with two
490 GPUs, and OpenMP performs poorly on the hardware.
492 Controlling the domain decomposition algorithm
493 ----------------------------------------------
494 This section lists all the options that affect how the domain
495 decomposition algorithm decomposes the workload to the available
499 Can be used to set the required maximum distance for inter
500 charge-group bonded interactions. Communication for two-body
501 bonded interactions below the non-bonded cut-off distance always
502 comes for free with the non-bonded communication. Particles beyond
503 the non-bonded cut-off are only communicated when they have
504 missing bonded interactions; this means that the extra cost is
505 minor and nearly independent of the value of ``-rdd``. With dynamic
506 load balancing, option ``-rdd`` also sets the lower limit for the
507 domain decomposition cell sizes. By default ``-rdd`` is determined
508 by :ref:`gmx mdrun` based on the initial coordinates. The chosen value will
509 be a balance between interaction range and communication cost.
512 On by default. When inter charge-group bonded interactions are
513 beyond the bonded cut-off distance, :ref:`gmx mdrun` terminates with an
514 error message. For pair interactions and tabulated bonds that do
515 not generate exclusions, this check can be turned off with the
516 option ``-noddcheck``.
519 When constraints are present, option ``-rcon`` influences
520 the cell size limit as well.
521 Particles connected by NC constraints, where NC is the LINCS order
522 plus 1, should not be beyond the smallest cell size. A error
523 message is generated when this happens, and the user should change
524 the decomposition or decrease the LINCS order and increase the
525 number of LINCS iterations. By default :ref:`gmx mdrun` estimates the
526 minimum cell size required for P-LINCS in a conservative
527 fashion. For high parallelization, it can be useful to set the
528 distance required for P-LINCS with ``-rcon``.
531 Sets the minimum allowed x, y and/or z scaling of the cells with
532 dynamic load balancing. :ref:`gmx mdrun` will ensure that the cells can
533 scale down by at least this factor. This option is used for the
534 automated spatial decomposition (when not using ``-dd``) as well as
535 for determining the number of grid pulses, which in turn sets the
536 minimum allowed cell size. Under certain circumstances the value
537 of ``-dds`` might need to be adjusted to account for high or low
538 spatial inhomogeneity of the system.
540 Finding out how to run mdrun better
541 -----------------------------------
543 The Wallcycle module is used for runtime performance measurement of :ref:`gmx mdrun`.
544 At the end of the log file of each run, the "Real cycle and time accounting" section
545 provides a table with runtime statistics for different parts of the :ref:`gmx mdrun` code
546 in rows of the table.
547 The table contains colums indicating the number of ranks and threads that
548 executed the respective part of the run, wall-time and cycle
549 count aggregates (across all threads and ranks) averaged over the entire run.
550 The last column also shows what precentage of the total runtime each row represents.
551 Note that the :ref:`gmx mdrun` timer resetting functionalities (`-resethway` and `-resetstep`)
552 reset the performance counters and therefore are useful to avoid startup overhead and
553 performance instability (e.g. due to load balancing) at the beginning of the run.
555 The performance counters are:
557 * Particle-particle during Particle mesh Ewald
558 * Domain decomposition
559 * Domain decomposition communication load
560 * Domain decomposition communication bounds
561 * Virtual site constraints
562 * Send X to Particle mesh Ewald
564 * Launch GPU operations
565 * Communication of coordinates
568 * Waiting + Communication of force
569 * Particle mesh Ewald
573 * PME 3D-FFT Communication
574 * PME solve Lennard-Jones
576 * PME wait for particle-particle
577 * Wait + Receive PME force
580 * Non-bonded position/force buffer operations
581 * Virtual site spread
586 * Communication of energies
588 * Add rotational forces
592 As performance data is collected for every run, they are essential to assessing
593 and tuning the performance of :ref:`gmx mdrun` performance. Therefore, they benefit
594 both code developers as well as users of the program.
595 The counters are an average of the time/cycles different parts of the simulation take,
596 hence can not directly reveal fluctuations during a single run (although comparisons across
597 multiple runs are still very useful).
599 Counters will appear in MD log file only if the related parts of the code were
600 executed during the :ref:`gmx mdrun` run. There is also a special counter called "Rest" which
601 indicated for the amount of time not accounted for by any of the counters above. Theerfore,
602 a significant amount "Rest" time (more than a few percent) will often be an indication of
603 parallelization inefficiency (e.g. serial code) and it is recommended to be reported to the
606 An additional set of subcounters can offer more fine-grained inspection of performance. They are:
608 * Domain decomposition redistribution
609 * DD neighbor search grid + sort
610 * DD setup communication
612 * DD make constraints
614 * Neighbor search grid local
617 * NS search non-local
621 * Listed buffer operations
623 * Ewald force correction
624 * Non-bonded position buffer operations
625 * Non-bonded force buffer operations
627 Subcounters are geared toward developers and have to be enabled during compilation. See
628 :doc:`/dev-manual/build-system` for more information.
630 TODO In future patch:
631 - red flags in log files, how to interpret wallcycle output
632 - hints to devs how to extend wallcycles
634 TODO In future patch: import wiki page stuff on performance checklist; maybe here,
637 Running mdrun with GPUs
638 -----------------------
640 NVIDIA GPUs from the professional line (Tesla or Quadro) starting with
641 the Kepler generation (compute capability 3.5 and later) support changing the
642 processor and memory clock frequency with the help of the applications clocks feature.
643 With many workloads, using higher clock rates than the default provides significant
644 performance improvements.
645 For more information see the `NVIDIA blog article`_ on this topic.
646 For |Gromacs| the highest application clock rates are optimal on all hardware
647 available to date (up to and including Maxwell, compute capability 5.2).
649 Application clocks can be set using the NVIDIA system managemet tool
650 ``nvidia-smi``. If the system permissions allow, :ref:`gmx mdrun` has
651 built-in support to set application clocks if built with NVML support. # TODO add ref to relevant section
652 Note that application clocks are a global setting, hence affect the
653 performance of all applications that use the respective GPU(s).
654 For this reason, :ref:`gmx mdrun` sets application clocks at initialization
655 to the values optimal for |Gromacs| and it restores them before exiting
656 to the values found at startup, unless it detects that they were altered
659 .. _NVIDIA blog article: https://devblogs.nvidia.com/parallelforall/increase-performance-gpu-boost-k80-autoboost/
661 Reducing overheads in GPU accelerated runs
662 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
664 In order for CPU cores and GPU(s) to execute concurrently, tasks are
665 launched and executed asynchronously on the GPU(s) while the CPU cores
666 execute non-offloaded force computation (like long-range PME electrostatics).
667 Asynchronous task launches are handled by GPU device driver and
668 require CPU involvement. Therefore, the work of scheduling
669 GPU tasks will incur an overhead that can in some cases significantly
670 delay or interfere with the CPU execution.
672 Delays in CPU execution are caused by the latency of launching GPU tasks,
673 an overhead that can become significant as simulation ns/day increases
674 (i.e. with shorter wall-time per step).
675 The overhead is measured by :ref:`gmx mdrun` and reported in the performance
676 summary section of the log file ("Launch GPU ops" row).
677 A few percent of runtime spent in this category is normal,
678 but in fast-iterating and multi-GPU parallel runs 10% or larger overheads can be observed.
679 In general, there a user can do little to avoid such overheads, but there
680 are a few cases where tweaks can give performance benefits.
681 In single-rank runs timing of GPU tasks is by default enabled and,
682 while in most cases its impact is small, in fast runs performance can be affected.
683 The performance impact will be most significant on NVIDIA GPUs with CUDA,
684 less on AMD with OpenCL.
685 In these cases, when more than a few percent of "Launch GPU ops" time is observed,
686 it is recommended turning off timing by setting the ``GMX_DISABLE_GPU_TIMING``
687 environment variable.
688 In parallel runs with with many ranks sharing a GPU
689 launch overheads can also be reduced by staring fewer thread-MPI
690 or MPI ranks per GPU; e.g. most often one rank per thread or core is not optimal.
692 The second type of overhead, interference of the GPU driver with CPU computation,
693 is caused by the scheduling and coordination of GPU tasks.
694 A separate GPU driver thread can require CPU resources
695 which may clash with the concurrently running non-offloaded tasks,
696 potentially degrading the performance of PME or bonded force computation.
697 This effect is most pronounced when using AMD GPUs with OpenCL with
698 all stable driver releases to date (up to and including fglrx 12.15).
699 To minimize the overhead it is recommended to
700 leave a CPU hardware thread unused when launching :ref:`gmx mdrun`,
701 especially on CPUs with high core count and/or HyperThreading enabled.
702 E.g. on a machine with a 4-core CPU and eight threads (via HyperThreading) and an AMD GPU,
703 try ``gmx mdrun -ntomp 7 -pin on``.
704 This will leave free CPU resources for the GPU task scheduling
705 reducing interference with CPU computation.
706 Note that assigning fewer resources to :ref:`gmx mdrun` CPU computation
707 involves a tradeoff which may outweigh the benefits of reduced GPU driver overhead,
708 in particular without HyperThreading and with few CPU cores.
710 TODO In future patch: any tips not covered above
712 Running the OpenCL version of mdrun
713 -----------------------------------
715 The current version works with GCN-based AMD GPUs, and NVIDIA CUDA
716 GPUs. Make sure that you have the latest drivers installed. For AMD GPUs,
717 Mesa version 17.0 or newer with LLVM 4.0 or newer is supported in addition
718 to the proprietary driver. For NVIDIA GPUs, using the proprietary driver is
719 required as the open source nouveau driver (available in Mesa) does not
720 provide the OpenCL support.
721 The minimum OpenCL version required is |REQUIRED_OPENCL_MIN_VERSION|. See
722 also the :ref:`known limitations <opencl-known-limitations>`.
724 Devices from the AMD GCN architectures (all series) and NVIDIA Fermi
725 and later (compute capability 2.0) are known to work, but before
726 doing production runs always make sure that the |Gromacs| tests
727 pass successfully on the hardware.
729 The OpenCL GPU kernels are compiled at run time. Hence,
730 building the OpenCL program can take a few seconds introducing a slight
731 delay in the :ref:`gmx mdrun` startup. This is not normally a
732 problem for long production MD, but you might prefer to do some kinds
733 of work, e.g. that runs very few steps, on just the CPU (e.g. see ``-nb`` above).
735 The same ``-gpu_id`` option (or ``GMX_GPU_ID`` environment variable)
736 used to select CUDA devices, or to define a mapping of GPUs to PP
737 ranks, is used for OpenCL devices.
739 Some other :ref:`OpenCL management <opencl-management>` environment
740 variables may be of interest to developers.
742 .. _opencl-known-limitations:
744 Known limitations of the OpenCL support
745 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
747 Limitations in the current OpenCL support of interest to |Gromacs| users:
749 - No Intel devices (CPUs, GPUs or Xeon Phi) are supported
750 - Due to blocking behavior of some asynchronous task enqueuing functions
751 in the NVIDIA OpenCL runtime, with the affected driver versions there is
752 almost no performance gain when using NVIDIA GPUs.
753 The issue affects NVIDIA driver versions up to 349 series, but it
754 known to be fixed 352 and later driver releases.
755 - On NVIDIA GPUs the OpenCL kernels achieve much lower performance
756 than the equivalent CUDA kernels due to limitations of the NVIDIA OpenCL
758 - The AMD APPSDK version 3.0 ships with OpenCL compiler/runtime components,
759 libamdocl12cl64.so and libamdocl64.so (only in earlier releases),
760 that conflict with newer fglrx GPU drivers which provide the same libraries.
761 This conflict manifests in kernel launch failures as, due to the library path
762 setup, the OpenCL runtime loads the APPSDK version of the aforementioned
763 libraries instead of the ones provided by the driver installer.
764 The recommended workaround is to remove or rename the APPSDK versions of the
767 Limitations of interest to |Gromacs| developers:
769 - The current implementation is not compatible with OpenCL devices that are
770 not using warp/wavefronts or for which the warp/wavefront size is not a
772 - Some Ewald tabulated kernels are known to produce incorrect results, so
773 (correct) analytical kernels are used instead.
775 Performance checklist
776 ---------------------
778 There are many different aspects that affect the performance of simulations in
779 |Gromacs|. Most simulations require a lot of computational resources, therefore
780 it can be worthwhile to optimize the use of those resources. Several issues
781 mentioned in the list below could lead to a performance difference of a factor
782 of 2. So it can be useful go through the checklist.
784 |Gromacs| configuration
785 ^^^^^^^^^^^^^^^^^^^^^^^
787 * Don't use double precision unless you're absolute sure you need it.
788 * Compile the FFTW library (yourself) with the correct flags on x86 (in most
789 cases, the correct flags are automatically configured).
790 * On x86, use gcc or icc as the compiler (not pgi or the Cray compiler).
791 * On POWER, use gcc instead of IBM's xlc.
792 * Use a new compiler version, especially for gcc (e.g. from the version 5 to 6
793 the performance of the compiled code improved a lot).
794 * MPI library: OpenMPI usually has good performance and causes little trouble.
795 * Make sure your compiler supports OpenMP (some versions of Clang don't).
796 * If you have GPUs that support either CUDA or OpenCL, use them.
798 * Configure with ``-DGMX_GPU=ON`` (add ``-DGMX_USE_OPENCL=ON`` for OpenCL).
799 * For CUDA, use the newest CUDA availabe for your GPU to take advantage of the
800 latest performance enhancements.
801 * Use a recent GPU driver.
802 * If compiling on a cluster head node, make sure that ``GMX_CPU_ACCELERATION``
803 is appropriate for the compute nodes.
808 * For an approximately spherical solute, use a rhombic dodecahedron unit cell.
809 * When using a time-step of 2 fs, use :mdp:`cutoff-scheme` = :mdp:`h-bonds`
810 (and not :mdp:`all-bonds`), since this is faster, especially with GPUs,
811 and most force fields have been parametrized with only bonds involving
812 hydrogens constrained.
813 * You can increase the time-step to 4 or 5 fs when using virtual interaction
814 sites (``gmx pdb2gmx -vsite h``).
815 * For massively parallel runs with PME, you might need to try different numbers
816 of PME ranks (``gmx mdrun -npme ???``) to achieve best performance;
817 ``gmx tune_pme`` can help automate this search.
818 * For massively parallel runs (also ``gmx mdrun -multidir``), or with a slow
819 network, global communication can become a bottleneck and you can reduce it
820 with ``gmx mdrun -gcom`` (note that this does affect the frequency of
821 temperature and pressure coupling).
823 Checking and improving performance
824 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
826 * Look at the end of the ``md.log`` file to see the performance and the cycle
827 counters and wall-clock time for different parts of the MD calculation. The
828 PP/PME load ratio is also printed, with a warning when a lot of performance is
829 lost due to imbalance.
830 * Adjust the number of PME ranks and/or the cut-off and PME grid-spacing when
831 there is a large PP/PME imbalance. Note that even with a small reported
832 imbalance, the automated PME-tuning might have reduced the initial imbalance.
833 You could still gain performance by changing the mdp parameters or increasing
834 the number of PME ranks.
835 * If the neighbor searching takes a lot of time, increase nstlist (with the
836 Verlet cut-off scheme, this automatically adjusts the size of the neighbour
837 list to do more non-bonded computation to keep energy drift constant).
839 * If ``Comm. energies`` takes a lot of time (a note will be printed in the log
840 file), increase nstcalcenergy or use ``mdrun -gcom``.
841 * If all communication takes a lot of time, you might be running on too many
842 cores, or you could try running combined MPI/OpenMP parallelization with 2
843 or 4 OpenMP threads per MPI process.
biod.pnpi.spb.ru / alexxy / gromacs.git / blob