Revert "Wrap more device pointers in DeviceBuffer" (!1244)