How is global data "All" synchronized across all processes?

From: Mark Baumann <markb_at_physics.utexas.edu>
Date: Fri, 1 Apr 2011 16:33:13 -0500 (CDT)

Dear GADGET users,

I have implemented sink particles across multiple processes in GADGET-2.
In doing so, I am decrementing the variable All.TotNumPart as particles
are deleted from the simulation.

I am experiencing a problem in that different processes end up with
different values of All.TotNumPart. Initial debugging seems to show that
each process only registers the decrements that the process itself has
made. This leads to my question: how is the struct "All" synchronized, if
at all?

By looking through the code I've found a few other places where All is
modified, for instance All.Time is modified within run.c. However I do
not see any blocking or communication code around these lines, leading me
to wonder if changes to All.Time are registered on the other processes or
not. Is there any place in the code that synchronizes the struct across
all processes?

If not, then I am guessing I will need to broadcast new values of
All.TotNumPart whenever it is decremented by any process. Or, perhaps
better, to do an Allreduce on particle number everytime a particle is
deleted so that all processes will have an up-to-date value for
All.TotNumPart. Before I do that, I wanted to ask you in case I am
overlooking some synchronization code that already exists somewhere.

Thank you for any insights,

Mark
Received on 2011-04-01 23:43:26