TL;DR yes you have to map memory by location due to limits of FP and size of the problem for simulation. It’s not just a problem for path tracing anymore.
BTW games that have voxel layers overlaid the warped space map to 3d textures that are also warped. So they’re not actually neat little evenly distributed boxes at all.