on AMD devices (both GPUs and APUs); NVIDIA hardware is also supported.
SIMD
- Modern CPU cores have instructions that can execute large
+ A type of CPU instruction by which modern CPU cores can execute large
numbers of floating-point instructions in a single cycle.
when choosing how to make good use of the hardware. For details,
see the Reference Manual. The most important of these are
-.. glossary::
-
.. _gmx-domain-decomp:
+.. glossary::
+
Domain Decomposition
The domain decomposition (DD) algorithm decomposes the
(short-ranged) component of the non-bonded interactions into
members of its domain. A GPU may perform work for more than
one PP rank, but it is normally most efficient to use a single
PP rank per GPU and for that rank to have thousands of
- particles. When the work of a PP rank is done on the CPU, :ref:`mdrun <gmx mdrun>`
- will make extensive use of the SIMD capabilities of the
- core. There are various `command-line options
- <controlling-the-domain-decomposition-algorithm` to control
- the behaviour of the DD algorithm.
+ particles. When the work of a PP rank is done on the CPU,
+ :ref:`mdrun <gmx mdrun>` will make extensive use of the SIMD
+ capabilities of the core. There are various
+ :ref:`command-line options <controlling-the-domain-decomposition-algorithm>`
+ to control the behaviour of the DD algorithm.
Particle-mesh Ewald
The particle-mesh Ewald (PME) algorithm treats the long-ranged
- components of the non-bonded interactions (Coulomb and/or
+ component of the non-bonded interactions (Coulomb and/or
Lennard-Jones). Either all, or just a subset of ranks may
- participate in the work for computing long-ranged component
- (often inaccurately called simple the "PME"
+ participate in the work for computing the long-ranged component
+ (often inaccurately called simply the "PME"
component). Because the algorithm uses a 3D FFT that requires
global communication, its performance gets worse as more ranks
participate, which can mean it is fastest to use just a subset
are no separate PME ranks.
``-nb``
- Used to set where to execute the non-bonded interactions.
+ Used to set where to execute the short-range non-bonded interactions.
Can be set to "auto", "cpu", "gpu."
Defaults to "auto," which uses a compatible GPU if available.
Setting "cpu" requires that no GPU is used. Setting "gpu" requires
that a compatible GPU be available and will be used.
+``-pme``
+ Used to set where to execute the long-range non-bonded interactions.
+ Can be set to "auto", "cpu", "gpu."
+ Defaults to "auto," which uses a compatible GPU if available.
+ Setting "gpu" requires that a compatible GPU be available and will be used.
+ Multiple PME ranks are not supported with PME on GPU, so if a GPU is used
+ for the PME calculation -npme must be set to 1.
+
``-gpu_id``
A string that specifies the ID numbers of the GPUs that
are available to be used by ranks on this node. For example,
gmx mdrun -ntmpi 2 -ntomp 4
-Starts :ref:`mdrun <gmx mdrun>` using eight total threads, with four thread-MPI
-ranks and two OpenMP threads per core. You should only use
+Starts :ref:`mdrun <gmx mdrun>` using eight total threads, with two thread-MPI
+ranks and four OpenMP threads per rank. You should only use
these options when seeking optimal performance, and
must take care that the ranks you create can have
all of their OpenMP threads run on the same socket.
sockets, and the number of cores per node must be
a multiple of the number of threads per rank.
+::
+
+ gmx mdrun -ntmpi 4 -nb gpu -pme cpu -gputasks 0011
+
+Starts :ref:`mdrun <gmx mdrun>` using four thread-MPI ranks, and maps them
+to GPUs with IDs 0 and 1. The CPU cores available will be split evenly between
+the ranks using OpenMP threads, with the first two ranks offloading short-range
+nonbonded force calculations to GPU 0, and the last two ranks offloading to GPU 1.
+The long-range component of the forces are calculated on CPUs. This may be optimal
+on hardware where the CPUs are relatively powerful compared to the GPUs.
+
+::
+
+ gmx mdrun -ntmpi 4 -nb gpu -pme gpu -npme 1 -gputasks 0001
+
+Starts :ref:`mdrun <gmx mdrun>` using four thread-MPI ranks, one of which is
+dedicated to the long-range PME calculation. The first 3 threads offload their
+short-range non-bonded calculations to the GPU with ID 0, the 4th (PME) thread
+offloads its calculations to the GPU with ID 1.
+
+::
+
+ gmx mdrun -ntmpi 4 -nb gpu -pme gpu -npme 1 -gputasks 0011
+
+Similar to the above example, with 3 ranks assigned to calculating short-range
+non-bonded forces, and one rank assigned to calculate the long-range forces.
+In this case, 2 of the 3 short-range ranks offload their nonbonded force
+calculations to GPU 0. The GPU with ID 1 calculates the short-ranged forces of
+the 3rd short-range rank, as well as the long-range forces of the PME-dedicated
+rank. Whether this or the above example is optimal will depend on the capabilities
+of the individual GPUs and the system composition.
+
::
gmx mdrun -gpu_id 12
::
- gmx mdrun -ntmpi 4 -nb gpu -gputasks 1122
+ gmx mdrun -nt 6 -pin on -pinoffset 0 -pinstride 1
+ gmx mdrun -nt 6 -pin on -pinoffset 6 -pinstride 1
-Starts :ref:`mdrun <gmx mdrun>` using four thread-MPI ranks, and maps them
-to GPUs with IDs 1 and 2. The CPU cores available will
-be split evenly between the ranks using OpenMP threads.
-
-::
-
- gmx mdrun -nt 6 -pin on -pinoffset 0
- gmx mdrun -nt 6 -pin on -pinoffset 3
-
-Starts two :ref:`mdrun <gmx mdrun>` processes, each with six total threads.
+Starts two :ref:`mdrun <gmx mdrun>` processes, each with six total threads
+arranged so that the processes affect each other as little as possible by
+being assigned to disjoint sets of physical cores.
Threads will have their affinities set to particular
-logical cores, beginning from the logical core
-with rank 0 or 3, respectively. The above would work
-well on an Intel CPU with six physical cores and
+logical cores, beginning from the first and 7th logical cores, respectively. The
+above would work well on an Intel CPU with six physical cores and
hyper-threading enabled. Use this kind of setup only
if restricting :ref:`mdrun <gmx mdrun>` to a subset of cores to share a
node with other processes.
``-gcom``
During the simulation :ref:`gmx mdrun` must communicate between all ranks to
compute quantities such as kinetic energy. By default, this
- happens whenever plausible, and is influenced by a lot of :ref:`[.mdp]
- options. <mdp-general>` The period between communication phases
+ happens whenever plausible, and is influenced by a lot of
+ :ref:`mdp options. <mdp-general>` The period between communication phases
must be a multiple of :mdp:`nstlist`, and defaults to
the minimum of :mdp:`nstcalcenergy` and :mdp:`nstlist`.
``mdrun -gcom`` sets the number of steps that must elapse between
normal case where all the GPUs on the node are available
for use.
+.. _controlling-the-domain-decomposition-algorithm:
+
Controlling the domain decomposition algorithm
----------------------------------------------
This section lists all the options that affect how the domain
``-rcon``
When constraints are present, option ``-rcon`` influences
- the cell size limit as well.
+ the cell size limit as well.
Particles connected by NC constraints, where NC is the LINCS order
plus 1, should not be beyond the smallest cell size. A error
message is generated when this happens, and the user should change
* Neighbor search
* Launch GPU operations
* Communication of coordinates
-* Born radii
* Force
* Waiting + Communication of force
* Particle mesh Ewald
hence can not directly reveal fluctuations during a single run (although comparisons across
multiple runs are still very useful).
-Counters will appear in MD log file only if the related parts of the code were
+Counters will appear in an MD log file only if the related parts of the code were
executed during the :ref:`gmx mdrun` run. There is also a special counter called "Rest" which
-indicated for the amount of time not accounted for by any of the counters above. Theerfore,
+indicates the amount of time not accounted for by any of the counters above. Therefore,
a significant amount "Rest" time (more than a few percent) will often be an indication of
parallelization inefficiency (e.g. serial code) and it is recommended to be reported to the
developers.
- red flags in log files, how to interpret wallcycle output
- hints to devs how to extend wallcycles
-TODO In future patch: import wiki page stuff on performance checklist; maybe here,
-maybe elsewhere
-
.. _gmx-mdrun-on-gpu:
Running :ref:`mdrun <gmx mdrun>` with GPUs
Of all calculations required for an MD step,
GROMACS aims to optimize performance bottom-up for each step
from the lowest level (SIMD unit, cores, sockets, accelerators, etc.).
-Therefore much of the indivdual computation units are
+Therefore many of the individual computation units are
highly tuned for the lowest level of hardware parallelism: the SIMD units.
Additionally, with GPU accelerators used as *co-processors*, some of the work
can be *offloaded*, that is calculated simultaneously/concurrently with the CPU
task can run wholly on the GPU, or have its latter stages run only on the CPU.
Limitations are that PME on GPU does not support PME domain decomposition,
- so that only one PME task can be offloaded to a single GPU
+ so that only one PME task can be offloaded to a single GPU
assigned to a separate PME rank, while NB can be decomposed and offloaded to multiple GPUs.
.. Future |Gromacs| versions past 2018:
Performance considerations for GPU tasks
........................................
-#) The performace balance depends on how many (and how fast) CPU
- cores you have, vs. how many and how fast the GPUs are that you have.
+#) The performance balance depends on the speed and number of CPU cores you
+ have vs the speed and number of GPUs you have.
#) With slow/old GPUs and/or fast/modern CPUs with many
cores, it might make more sense to let the CPU do PME calculation,
an overhead that can become significant as simulation ns/day increases
(i.e. with shorter wall-time per step).
The overhead is measured by :ref:`gmx mdrun` and reported in the performance
-summary section of the log file ("Launch GPU ops" row).
-A few percent of runtime spent in this category is normal,
+summary section of the log file ("Launch GPU ops" row).
+A few percent of runtime spent in this category is normal,
but in fast-iterating and multi-GPU parallel runs 10% or larger overheads can be observed.
-In general, there a user can do little to avoid such overheads, but there
+In general, a user can do little to avoid such overheads, but there
are a few cases where tweaks can give performance benefits.
In single-rank runs timing of GPU tasks is by default enabled and,
while in most cases its impact is small, in fast runs performance can be affected.
The performance impact will be most significant on NVIDIA GPUs with CUDA,
less on AMD with OpenCL.
In these cases, when more than a few percent of "Launch GPU ops" time is observed,
-it is recommended turning off timing by setting the ``GMX_DISABLE_GPU_TIMING``
+it is recommended to turn off timing by setting the ``GMX_DISABLE_GPU_TIMING``
environment variable.
-In parallel runs with with many ranks sharing a GPU
-launch overheads can also be reduced by staring fewer thread-MPI
+In parallel runs with many ranks sharing a GPU,
+launch overheads can also be reduced by starting fewer thread-MPI
or MPI ranks per GPU; e.g. most often one rank per thread or core is not optimal.
The second type of overhead, interference of the GPU driver with CPU computation,
older driver releases (e.g. fglrx 12.15).
To minimize the overhead it is recommended to
leave a CPU hardware thread unused when launching :ref:`gmx mdrun`,
-especially on CPUs with high core count and/or HyperThreading enabled.
+especially on CPUs with high core counts and/or HyperThreading enabled.
E.g. on a machine with a 4-core CPU and eight threads (via HyperThreading) and an AMD GPU,
try ``gmx mdrun -ntomp 7 -pin on``.
This will leave free CPU resources for the GPU task scheduling
pass successfully on the hardware.
The OpenCL GPU kernels are compiled at run time. Hence,
-building the OpenCL program can take a few seconds introducing a slight
+building the OpenCL program can take a few seconds, introducing a slight
delay in the :ref:`gmx mdrun` startup. This is not normally a
problem for long production MD, but you might prefer to do some kinds
of work, e.g. that runs very few steps, on just the CPU (e.g. see ``-nb`` above).
cases, the correct flags are automatically configured).
* On x86, use gcc or icc as the compiler (not pgi or the Cray compiler).
* On POWER, use gcc instead of IBM's xlc.
-* Use a new compiler version, especially for gcc (e.g. from the version 5 to 6
+* Use a new compiler version, especially for gcc (e.g. from version 5 to 6
the performance of the compiled code improved a lot).
* MPI library: OpenMPI usually has good performance and causes little trouble.
* Make sure your compiler supports OpenMP (some versions of Clang don't).
biod.pnpi.spb.ru / alexxy / gromacs.git / blobdiff