From: Artem Zhmurov <zhmurov@gmail.com>
Date: Mon, 16 Sep 2019 14:23:48 +0000 (+0200)
Subject: Disable GPU update/constraints when neither PME nor buffer ops are offloaded.
X-Git-Url: http://biod.pnpi.spb.ru/gitweb/?a=commitdiff_plain;h=9b6824791cb7422f16b1b9f0129336febda0e6c0;p=alexxy%2Fgromacs.git

Disable GPU update/constraints when neither PME nor buffer ops are offloaded.

Using the GPU-version of update makes sense if forces are already on the GPU,
i.e. if at least:
1. PME is on the GPU (there should be a copy of coordinates on a GPU in rvec
   format for PME spread).
2. Non-bonded interactions and buffer ops are on the GPU.

This is temporary solution, needed because the buffer ops offload switch is
operated by the environment variable. More favorable behavior would be to
switch on the GPU buffer ops in the second case rather then disabling the
GPU update.

Refs. #2816.

Change-Id: I37a9969dd6c74dcfa41a95da13ae54d014c9ea60
---

diff --git a/src/gromacs/mdrun/runner.cpp b/src/gromacs/mdrun/runner.cpp
index 3347dd39dd..a1928eaa1b 100644
--- a/src/gromacs/mdrun/runner.cpp
+++ b/src/gromacs/mdrun/runner.cpp
@@ -1485,7 +1485,9 @@ int Mdrunner::mdrunner()
 
         // Before we start the actual simulator, try if we can run the update task on the GPU.
         useGpuForUpdate = decideWhetherToUseGpuForUpdate(DOMAINDECOMP(cr),
+                                                         useGpuForPme,
                                                          useGpuForNonbonded,
+                                                         c_enableGpuBufOps,
                                                          updateTarget,
                                                          gpusWereDetected,
                                                          *inputrec,
diff --git a/src/gromacs/taskassignment/decidegpuusage.cpp b/src/gromacs/taskassignment/decidegpuusage.cpp
index b9f0d7aff8..1487659935 100644
--- a/src/gromacs/taskassignment/decidegpuusage.cpp
+++ b/src/gromacs/taskassignment/decidegpuusage.cpp
@@ -492,7 +492,9 @@ bool decideWhetherToUseGpusForBonded(const bool       useGpuForNonbonded,
 }
 
 bool decideWhetherToUseGpuForUpdate(bool              isDomainDecomposition,
+                                    bool              useGpuForPme,
                                     bool              useGpuForNonbonded,
+                                    bool              useGpuForBufferOps,
                                     TaskTarget        updateTarget,
                                     bool              gpusWereDetected,
                                     const t_inputrec &inputrec,
@@ -512,9 +514,12 @@ bool decideWhetherToUseGpuForUpdate(bool              isDomainDecomposition,
     {
         errorMessage += "Domain decomposition is not supported.\n";
     }
-    if (!useGpuForNonbonded)
+    // Using the GPU-version of update makes sense if forces are already on the GPU, i.e. if at least:
+    // 1. PME is on the GPU (there should be a copy of coordinates on a GPU in rvec format for PME spread).
+    // 2. Non-bonded interactions and buffer ops are on the GPU.
+    if (!(useGpuForPme || (useGpuForNonbonded && useGpuForBufferOps)))
     {
-        errorMessage += "Short-ranged non-bonded interaction tasks must run on the GPU.\n";
+        errorMessage += "Either PME or short-ranged non-bonded interaction tasks must run on the GPU.\n";
     }
     if (!gpusWereDetected)
     {
diff --git a/src/gromacs/taskassignment/decidegpuusage.h b/src/gromacs/taskassignment/decidegpuusage.h
index 707b1e49b3..48ecf2e88f 100644
--- a/src/gromacs/taskassignment/decidegpuusage.h
+++ b/src/gromacs/taskassignment/decidegpuusage.h
@@ -232,7 +232,9 @@ bool decideWhetherToUseGpusForBonded(bool       useGpuForNonbonded,
 /*! \brief Decide whether to use GPU for update.
  *
  * \param[in]  isDomainDecomposition     Whether there more than one domain.
+ * \param[in]  useGpuForPme              Whether GPUs will be used for PME interactions.
  * \param[in]  useGpuForNonbonded        Whether GPUs will be used for nonbonded interactions.
+ * \param[in]  useGpuForBufferOps        Whether GPUs will be used for buffer operations.
  * \param[in]  updateTarget              User choice for running simulation on GPU.
  * \param[in]  gpusWereDetected          Whether compatible GPUs were detected on any node.
  * \param[in]  inputrec                  The user input.
@@ -246,7 +248,9 @@ bool decideWhetherToUseGpusForBonded(bool       useGpuForNonbonded,
  *             InconsistentInputError    If the user requirements are inconsistent.
  */
 bool decideWhetherToUseGpuForUpdate(bool              isDomainDecomposition,
+                                    bool              useGpuForPme,
                                     bool              useGpuForNonbonded,
+                                    bool              useGpuForBufferOps,
                                     TaskTarget        updateTarget,
                                     bool              gpusWereDetected,
                                     const t_inputrec &inputrec,