Hi all,
I've been running some 3 & 30 million particle galaxy simulations in
GADGET2 using most of the defaults in the provided galaxy.Makefile, but
they've been going very slow (think ~0.015Gyr for ~60hr of wallclock time
for 30 million particles with 0.15kpc disk & bulge softening lengths). My
ultimate aim is to run 3Gyr simulations on the order of ~10 million
particles with varying halo concentrations but high enough resolution to
look at structure (<0.15kpc disk soft length is my current guess).
Apologies for the long e-mail (not sure if some small details may be
relevant).
Here is a quick run down of the various software/settings I've been using:
--> Initial conditions: GalactICS 2008's MW_original model of the Milky Way
& most defaults. (Widrow, Pym, Dubinski 2008).
30 million particles: 26 500 000 halo; 500 000 bulge; 3 000 000 disk.
Main output: disk/bulge/halo/galaxy files with particle mass, x y z
positions & velocities.
--> Conversion of galaxy (disk + bulge + halo files) to Tipsy to Gadget
format: In-house scripts.
Classify bulge & disk as "stellar" particles (so Gadget soft. lengths set
under "stellar").
--> Gadget2:
Used most of the defaults in galaxy.Makefile, tried TreePM on & off &
different mesh sizes (not much difference).
Other settings: Open MPI 1.6; GSL 1.15; FFTW 2.15; gcc 4.5; Coupled with
linux-based supercomputer.
Parameters: Most of defaults in galaxy.param; tried different snapshot &
rewrite file write frequencies (not much difference); 1kpc halo soft length
for all runs; 0.01-0.15kpc "stellar" (bulge & disk) soft length for test
runs; 0.5 & 1kpc "stellar" soft lengths for varying halo concentrations,
all runs 30-million particles (1kpc runs finished in 16hrs-9days, 0.5kpc
only ~0.03-0.06Gyr in ~60hrs wallclock).
--> Cluster settings: I don't know much about the hardware itself, but I've
been using the Razor queues (this link:
http://hpc.uark.edu/hpc/about/hardware.html) on my university's cluster; 12
procs/node, but a parallel extension with 8 procs/node has been recently
added & I will try that.
Tried 36-180 procs; 96 & 180 much too slow (even with TreeAllocFac
increased from 0.8 to 1.5), 48 faster than 96+ & 36-.
At this point, the one thing I haven't yet tried changing that may help is
the memory buffer (25 MB by default in galaxy.param) and so far,
simultaneously increasing the "stellar" soft length/TreeAllocFac doesn't
seem like it will help. I'm still trying to figure out why varying the
halo concentration results in such a wide range of finished sim times, but
in general, any "stellar" softening length below 0.5kpc runs much too slow
with any combination of IC's & parameters.
Thanks, and apologies for long e-mail once more!
Jazmin Berlanga
--
Jazmin Berlanga M.
M.S. Student
Department of Physics
University of Arkansas - Fayetteville
AGES Office: (479) 575-2479
Received on 2012-09-06 01:21:48