Hello,
probably a stupid question, but did you check if you ran out of memory
and started swapping??
Every processor you add increases the amount of memory you need!
12 processors sharing 24GB with 33 million particles could be to much.
At home I have a 6 processor pc with 4GB of memory. 4.5 million particles
with 5 processors need around 3.8 GB of memory. With 6 processors it
started
swapping.
with regards Enders
> Hi all,
>
> I've been running some 3 & 30 million particle galaxy simulations in
> GADGET2 using most of the defaults in the provided galaxy.Makefile, but
> they've been going very slow (think ~0.015Gyr for ~60hr of wallclock time
> for 30 million particles with 0.15kpc disk & bulge softening lengths).
> My
> ultimate aim is to run 3Gyr simulations on the order of ~10 million
> particles with varying halo concentrations but high enough resolution to
> look at structure (<0.15kpc disk soft length is my current guess).
> Apologies for the long e-mail (not sure if some small details may be
> relevant).
>
> Here is a quick run down of the various software/settings I've been
> using:
>
> --> Initial conditions: GalactICS 2008's MW_original model of the Milky
> Way
> & most defaults. (Widrow, Pym, Dubinski 2008).
> 30 million particles: 26 500 000 halo; 500 000 bulge; 3 000 000 disk.
> Main output: disk/bulge/halo/galaxy files with particle mass, x y z
> positions & velocities.
>
> --> Conversion of galaxy (disk + bulge + halo files) to Tipsy to Gadget
> format: In-house scripts.
> Classify bulge & disk as "stellar" particles (so Gadget soft. lengths set
> under "stellar").
>
> --> Gadget2:
> Used most of the defaults in galaxy.Makefile, tried TreePM on & off &
> different mesh sizes (not much difference).
> Other settings: Open MPI 1.6; GSL 1.15; FFTW 2.15; gcc 4.5; Coupled with
> linux-based supercomputer.
>
> Parameters: Most of defaults in galaxy.param; tried different snapshot &
> rewrite file write frequencies (not much difference); 1kpc halo soft
> length
> for all runs; 0.01-0.15kpc "stellar" (bulge & disk) soft length for test
> runs; 0.5 & 1kpc "stellar" soft lengths for varying halo concentrations,
> all runs 30-million particles (1kpc runs finished in 16hrs-9days, 0.5kpc
> only ~0.03-0.06Gyr in ~60hrs wallclock).
>
> --> Cluster settings: I don't know much about the hardware itself, but
> I've
> been using the Razor queues (this link:
> http://hpc.uark.edu/hpc/about/hardware.html) on my university's cluster;
> 12
> procs/node, but a parallel extension with 8 procs/node has been recently
> added & I will try that.
> Tried 36-180 procs; 96 & 180 much too slow (even with TreeAllocFac
> increased from 0.8 to 1.5), 48 faster than 96+ & 36-.
>
> At this point, the one thing I haven't yet tried changing that may help
> is
> the memory buffer (25 MB by default in galaxy.param) and so far,
> simultaneously increasing the "stellar" soft length/TreeAllocFac doesn't
> seem like it will help. I'm still trying to figure out why varying the
> halo concentration results in such a wide range of finished sim times,
> but
> in general, any "stellar" softening length below 0.5kpc runs much too
> slow
> with any combination of IC's & parameters.
>
>
> Thanks, and apologies for long e-mail once more!
> Jazmin Berlanga
>
--
Erstellt mit Operas revolutionärem E-Mail-Modul: http://www.opera.com/mail/
Received on 2012-09-06 05:38:01