Add PP decomposition simluationWorkload flags

Add PP decomposition simluationWorkload flags

Add the complete set of flags that indicate whether there is PP
decomposition using CPU/GPU communication.

These flags allow simplifications in the force schedule.

The additional benefit is a reduced reliance on passing around commrec
for checks related to parallelization.

Refs #3913