Divide default communicator from DD communicators
The communicators mpi_comm_mysim and mpi_comm_mygroup inside
t_commrec got initialized in init_commrec (to MPI_COMM_WORLD
if no multisim, to a subset otherwise). These communicators
were then used in subsequent setup work, before they got
reassigned during the construction of the DDBuilder object
and the construction of the actual domain decomposition object.
Effectively, this means that the same communicators (and, hence,
identical function calls) do very different things depending on
whether they get used before or after the setup of domain
decomposition. It also means that before DD set up, mpi_comm_mysim
and mpi_comm_mygroup are *identical*.
This change introduces an additional communicator within
t_commrec, mpiDefaulCommunicator, which helps to make these
implicit assumptions explicit. Consequently, this also redefines
PAR(cr), MASTER(cr), and SIMMASTER(cr).
This change will allow to move the sim and group communicators,
which are now only created at DD time, into the DD object,
logically separating the DD object from t_commrec.
Refs #2395
23 files changed: