Hi Johan,
Yes, the scalability of the domain decomposition in gadget2 is quite
limited, especially for small simulations... (note that your test
problem is actually quite small, and you are trying to carry out a
"strong scaling" test)
You can probably improve the speed of the domain decomposition
substantially by changing one line in the function
domain_topslit_local(), line ~1034, which reads:
if(TopNodes[sub].Count > All.TotNumPart / (TOPNODEFACTOR * NTask * NTask))
This should actually better be
if(TopNodes[sub].Count > All.TotNumPart / (TOPNODEFACTOR * NTask))
This is probably responsible for a substantial fraction of the slowdown
you observed for large CPU numbers and small particle load.
Cheers,
Volker
Johan Maes wrote:
> Dear Gadget list,
>
> We are currently testing the scalability of our new computer cluster
> (with dual-core AMD Opteron processors) by simulating the adiabatic
> collapse of a 1/r gas sphere. The same simulation was repeated for
> 1,8,16,24,32 and 64 CPU's. However, as from 24 processors, the
> computation time starts to increase. Looking at the cpu.txt files, it
> seems that the domain decomposition is taking much more time than it's
> supposed to.
>
> #CPU's DD time fraction (%)
> 1 2.5
> 8 5
> 16 21
> 24 27
> 32 50
>
> For 64 CPU's, it even takes up ~90% of the computation time:
>
> Step 1010, Time: 0.809326, CPUs: 64
> 12829.22 359.31 546.15 11790.72
>
> What could be causing this problem?
>
> Also, I used 3e5 particles for these (dimensionless) simulations.
> Strangely enough, when I increase the number to 3e6, the DD time
> fraction for 64 CPU's decreases to 63%. Or is this supposed to happen? I
> don't know a whole lot about the domain decomposition so this could just
> be me doing something wrong...
>
> Thanks for any suggestions/help!
>
> Best regards,
>
> Johan
>
>
>
>
> -----------------------------------------------------------
>
> If you wish to unsubscribe from this mailing, send mail to
> minimalist_at_MPA-Garching.MPG.de with a subject of: unsubscribe gadget-list
> A web-archive of this mailing list is available here:
> http://www.mpa-garching.mpg.de/gadget/gadget-list
Received on 2008-03-01 19:42:09