2 * This file is part of the GROMACS molecular simulation package.
4 * Copyright (c) 2017,2018,2019, by the GROMACS development team, led by
5 * Mark Abraham, David van der Spoel, Berk Hess, and Erik Lindahl,
6 * and including many others, as listed in the AUTHORS file in the
7 * top-level source directory and at http://www.gromacs.org.
9 * GROMACS is free software; you can redistribute it and/or
10 * modify it under the terms of the GNU Lesser General Public License
11 * as published by the Free Software Foundation; either version 2.1
12 * of the License, or (at your option) any later version.
14 * GROMACS is distributed in the hope that it will be useful,
15 * but WITHOUT ANY WARRANTY; without even the implied warranty of
16 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
17 * Lesser General Public License for more details.
19 * You should have received a copy of the GNU Lesser General Public
20 * License along with GROMACS; if not, see
21 * http://www.gnu.org/licenses, or write to the Free Software Foundation,
22 * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
24 * If you want to redistribute modifications to GROMACS, please
25 * consider that scientific software is very special. Version
26 * control is crucial - bugs must be traceable. We will be happy to
27 * consider code for inclusion in the official distribution, but
28 * derived work must not be called official GROMACS. Details are found
29 * in the README & COPYING files - if they are missing, get the
30 * official version at http://www.gromacs.org.
32 * To help us fund GROMACS development, we humbly ask that you cite
33 * the research papers on the package. Check out http://www.gromacs.org.
35 /*! \libinternal \file
36 * \brief Declares functionality for deciding whether tasks will run on GPUs.
38 * \author Mark Abraham <mark.j.abraham@gmail.com>
39 * \ingroup module_taskassignment
43 #ifndef GMX_TASKASSIGNMENT_DECIDEGPUUSAGE_H
44 #define GMX_TASKASSIGNMENT_DECIDEGPUUSAGE_H
51 enum class PmeRunMode;
56 //! Record where a compute task is targetted.
57 enum class TaskTarget : int
64 //! Help pass GPU-emulation parameters with type safety.
65 enum class EmulateGpuNonbonded : bool
67 //! Do not emulate GPUs.
75 /*! \brief Decide whether this thread-MPI simulation will run
76 * nonbonded tasks on GPUs.
78 * The number of GPU tasks and devices influences both the choice of
79 * the number of ranks, and checks upon any such choice made by the
80 * user. So we need to consider this before any automated choice of
81 * the number of thread-MPI ranks.
83 * \param[in] nonbondedTarget The user's choice for mdrun -nb for where to assign short-ranged nonbonded interaction tasks.
84 * \param[in] gpuIdsToUse The compatible GPUs that the user permitted us to use.
85 * \param[in] userGpuTaskAssignment The user-specified assignment of GPU tasks to device IDs.
86 * \param[in] emulateGpuNonbonded Whether we will emulate GPU calculation of nonbonded interactions.
87 * \param[in] buildSupportsNonbondedOnGpu Whether GROMACS was built with GPU support.
88 * \param[in] nonbondedOnGpuIsUseful Whether computing nonbonded interactions on a GPU is useful for this calculation.
89 * \param[in] numRanksPerSimulation The number of ranks in each simulation.
91 * \returns Whether the simulation will run nonbonded tasks on GPUs.
93 * \throws std::bad_alloc If out of memory
94 * InconsistentInputError If the user requirements are inconsistent. */
95 bool decideWhetherToUseGpusForNonbondedWithThreadMpi(TaskTarget nonbondedTarget,
96 const std::vector<int>& gpuIdsToUse,
97 const std::vector<int>& userGpuTaskAssignment,
98 EmulateGpuNonbonded emulateGpuNonbonded,
99 bool buildSupportsNonbondedOnGpu,
100 bool nonbondedOnGpuIsUseful,
101 int numRanksPerSimulation);
103 /*! \brief Decide whether this thread-MPI simulation will run
106 * The number of GPU tasks and devices influences both the choice of
107 * the number of ranks, and checks upon any such choice made by the
108 * user. So we need to consider this before any automated choice of
109 * the number of thread-MPI ranks.
111 * \param[in] useGpuForNonbonded Whether GPUs will be used for nonbonded interactions.
112 * \param[in] pmeTarget The user's choice for mdrun -pme for where to assign
113 * long-ranged PME nonbonded interaction tasks.
114 * \param[in] gpuIdsToUse The compatible GPUs that the user permitted us to use.
115 * \param[in] userGpuTaskAssignment The user-specified assignment of GPU tasks to device IDs.
116 * \param[in] hardwareInfo Hardware information
117 * \param[in] inputrec The user input
118 * \param[in] mtop Global system topology
119 * \param[in] numRanksPerSimulation The number of ranks in each simulation.
120 * \param[in] numPmeRanksPerSimulation The number of PME ranks in each simulation.
122 * \returns Whether the simulation will run PME tasks on GPUs.
124 * \throws std::bad_alloc If out of memory
125 * InconsistentInputError If the user requirements are inconsistent. */
126 bool decideWhetherToUseGpusForPmeWithThreadMpi(bool useGpuForNonbonded,
127 TaskTarget pmeTarget,
128 const std::vector<int>& gpuIdsToUse,
129 const std::vector<int>& userGpuTaskAssignment,
130 const gmx_hw_info_t& hardwareInfo,
131 const t_inputrec& inputrec,
132 const gmx_mtop_t& mtop,
133 int numRanksPerSimulation,
134 int numPmeRanksPerSimulation);
136 /*! \brief Decide whether the simulation will try to run nonbonded
139 * The final decision cannot be made until after the duty of the rank
140 * is known. But we need to know if nonbonded will run on GPUs for
141 * setting up DD (particularly rlist) and determining duty. If the
142 * user requires GPUs for the tasks of that duty, then it will be an
143 * error when none are found.
145 * With thread-MPI, calls have been made to
146 * decideWhetherToUseGpusForNonbondedWithThreadMpi() and
147 * decideWhetherToUseGpusForPmeWithThreadMpi() to help determine
148 * the number of ranks and run some checks, but the final
149 * decision is made in this routine, along with many more
150 * consistency checks.
152 * \param[in] nonbondedTarget The user's choice for mdrun -nb for where to assign short-ranged nonbonded interaction tasks.
153 * \param[in] userGpuTaskAssignment The user-specified assignment of GPU tasks to device IDs.
154 * \param[in] emulateGpuNonbonded Whether we will emulate GPU calculation of nonbonded interactions.
155 * \param[in] buildSupportsNonbondedOnGpu Whether GROMACS was build with GPU support.
156 * \param[in] nonbondedOnGpuIsUseful Whether computing nonbonded interactions on a GPU is useful for this calculation.
157 * \param[in] gpusWereDetected Whether compatible GPUs were detected on any node.
159 * \returns Whether the simulation will run nonbonded and PME tasks, respectively, on GPUs.
161 * \throws std::bad_alloc If out of memory
162 * InconsistentInputError If the user requirements are inconsistent. */
163 bool decideWhetherToUseGpusForNonbonded(TaskTarget nonbondedTarget,
164 const std::vector<int>& userGpuTaskAssignment,
165 EmulateGpuNonbonded emulateGpuNonbonded,
166 bool buildSupportsNonbondedOnGpu,
167 bool nonbondedOnGpuIsUseful,
168 bool gpusWereDetected);
170 /*! \brief Decide whether the simulation will try to run tasks of
171 * different types on GPUs.
173 * The final decision cannot be made until after the duty of the rank
174 * is known. But we need to know if nonbonded will run on GPUs for
175 * setting up DD (particularly rlist) and determining duty. If the
176 * user requires GPUs for the tasks of that duty, then it will be an
177 * error when none are found.
179 * With thread-MPI, calls have been made to
180 * decideWhetherToUseGpusForNonbondedWithThreadMpi() and
181 * decideWhetherToUseGpusForPmeWithThreadMpi() to help determine
182 * the number of ranks and run some checks, but the final
183 * decision is made in this routine, along with many more
184 * consistency checks.
186 * \param[in] useGpuForNonbonded Whether GPUs will be used for nonbonded interactions.
187 * \param[in] pmeTarget The user's choice for mdrun -pme for where to assign long-ranged PME nonbonded interaction tasks.
188 * \param[in] userGpuTaskAssignment The user-specified assignment of GPU tasks to device IDs.
189 * \param[in] hardwareInfo Hardware information
190 * \param[in] inputrec The user input
191 * \param[in] mtop Global system topology
192 * \param[in] numRanksPerSimulation The number of ranks in each simulation.
193 * \param[in] numPmeRanksPerSimulation The number of PME ranks in each simulation.
194 * \param[in] gpusWereDetected Whether compatible GPUs were detected on any node.
196 * \returns Whether the simulation will run nonbonded and PME tasks, respectively, on GPUs.
198 * \throws std::bad_alloc If out of memory
199 * InconsistentInputError If the user requirements are inconsistent. */
200 bool decideWhetherToUseGpusForPme(bool useGpuForNonbonded,
201 TaskTarget pmeTarget,
202 const std::vector<int>& userGpuTaskAssignment,
203 const gmx_hw_info_t& hardwareInfo,
204 const t_inputrec& inputrec,
205 const gmx_mtop_t& mtop,
206 int numRanksPerSimulation,
207 int numPmeRanksPerSimulation,
208 bool gpusWereDetected);
210 /*! \brief Decide whether the simulation will try to run bonded tasks on GPUs.
212 * \param[in] useGpuForNonbonded Whether GPUs will be used for nonbonded interactions.
213 * \param[in] useGpuForPme Whether GPUs will be used for PME interactions.
214 * \param[in] bondedTarget The user's choice for mdrun -bonded for where to assign tasks.
215 * \param[in] canUseGpuForBonded Whether the bonded interactions can run on a GPU
216 * \param[in] usingLJPme Whether Vdw interactions use LJ-PME.
217 * \param[in] usingElecPmeOrEwald Whether a PME or Ewald type method is used for electrostatics.
218 * \param[in] numPmeRanksPerSimulation The number of PME ranks in each simulation, can be -1 for auto.
219 * \param[in] gpusWereDetected Whether compatible GPUs were detected on any node.
221 * \returns Whether the simulation will run bondeded tasks on GPUs.
223 * \throws std::bad_alloc If out of memory
224 * InconsistentInputError If the user requirements are inconsistent. */
225 bool decideWhetherToUseGpusForBonded(bool useGpuForNonbonded,
227 TaskTarget bondedTarget,
228 bool canUseGpuForBonded,
230 bool usingElecPmeOrEwald,
231 int numPmeRanksPerSimulation,
232 bool gpusWereDetected);
234 /*! \brief Decide whether to use GPU for update.
236 * \param[in] forceGpuUpdateDefault If update should run on GPU by default.
237 * \param[in] isDomainDecomposition Whether there more than one domain.
238 * \param[in] useUpdateGroups If the constraints can be split across domains.
239 * \param[in] pmeRunMode PME running mode: CPU, GPU or mixed.
240 * \param[in] havePmeOnlyRank If there is a PME-only rank in the simulation.
241 * \param[in] useGpuForNonbonded Whether GPUs will be used for nonbonded interactions.
242 * \param[in] updateTarget User choice for running simulation on GPU.
243 * \param[in] gpusWereDetected Whether compatible GPUs were detected on any node.
244 * \param[in] inputrec The user input.
245 * \param[in] mtop The global topology.
246 * \param[in] useEssentialDynamics If essential dynamics is active.
247 * \param[in] doOrientationRestraints If orientation restraints are enabled.
248 * \param[in] useReplicaExchange If this is a REMD simulation.
249 * \param[in] doRerun It this is a rerun.
251 * \returns Whether complete simulation can be run on GPU.
252 * \throws std::bad_alloc If out of memory
253 * InconsistentInputError If the user requirements are inconsistent.
255 bool decideWhetherToUseGpuForUpdate(bool forceGpuUpdateDefault,
256 bool isDomainDecomposition,
257 bool useUpdateGroups,
258 PmeRunMode pmeRunMode,
259 bool havePmeOnlyRank,
260 bool useGpuForNonbonded,
261 TaskTarget updateTarget,
262 bool gpusWereDetected,
263 const t_inputrec& inputrec,
264 const gmx_mtop_t& mtop,
265 bool useEssentialDynamics,
266 bool doOrientationRestraints,
267 bool useReplicaExchange,