negative particle sum

From: Robert Thompson <rthompsonj_at_gmail.com>
Date: Thu, 4 Mar 2010 00:29:24 -0800

I'm having an issue running a large N-Body simulation on the supercomputer
Ranger _at_ TACC. I was able to successfully run a 2Gpc^3 N1000^3 simulation
on Lonestar (also _at_ TACC) using Gadget-3. I wanted to run a larger sim to
examine box size effects but due to memory limitations on Lonestar I needed
to move to Ranger. I recompiled gadget on Ranger, took the same parameter
files from the previous 2Gpc run, and simply changed the box size to 3Gpc
along with the particle count to roughly N=1500^3. The IC file was
generated in the exact same fashion as the 2Gpc run, but again the parameter
file was altered to change the box size and particle count. Gadget compiles
and runs fine, except that after reading in all of the input files and
displaying the correct total number of particles it then says that their sum
is a negative number. The code continues to run but outputs very tiny
snapshot files when they should be on the order of 89Gb.

I have compiled the code using Ranger's pgi and intel compilers, both had
the same result. I have also tried anywhere from 2048-4096 processors and
it all ends the same. Has anyone run into this type of issue before and how
was it solved? I can re-run the smaller successful run on Ranger to see if
it is system dependent but I do not want to waste SU's if at all possible.
Thank you in advance for your time, there is a snippet of the log file below
showing just after gadget finishes reading in the snapshot files. You can
see that sum2 is a negative number even though the line before it has the
correct total particle count.

-Robert Thompson
University of Nevada Las Vegas


reading file `/scratch/01026/tg802801/3GpcN1500_ngen/ics.127' on task=2032
(contains 24882176 particles.)
distributing this file to tasks 2032-2047
Type 0 (gas): 0 (tot= 0000000000) masstab=0
Type 1 (halo): 24882176 (tot= 3402072064) masstab=57.7304
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading block 0 (Coordinates)...
reading block 1 (Velocities)...
reading block 2 (ParticleIDs)...
reading block 3 (Masses)...
reading block 4 (InternalEnergy)...
reading done.
Total number of particles : 3402072064

sum1=0 sum2=-892895232
Testing ID uniqueness...
success.
Received on 2010-03-04 09:29:25