No domain decomposition that stays within memory bounds is possible from Arlindo Trindade _at_ gmail on 2014-04-16 (GADGET General Discussion Mailing List)

From: Arlindo Trindade _at_ gmail <arlindo.trindade_at_gmail.com>
Date: Wed, 16 Apr 2014 17:45:16 +0100

Hello all,

I'm trying to run a Gadget2 simulation with N=2*(512)^3 particles (DM+
gas) and Boxsize=200Mpc/h on a cluster. However, I get the error shown
below.

The same error occurs for simulations with much smaller number of
particles (2*16^3, 2*64^3, 2*128^4 for example).

I've tried to change the values of PartAllocFactor, TreeAllocFactor and
BufferSize, as suggested in some other posts in the gadget-list, but
still I was not able to overcome me problem.

Can someone has an idea of what is happening?

Best regards,
Arlindo Trindade

----------------------------------------------------------------------------------------------------------------------------------------------
This is Gadget, version `2.0'.

Running on 60 processors.

found 5 times in output-list.

Allocated 10000 MByte communication buffer per processor.

Communication buffer has room for 43086940 particles in gravity computation
Communication buffer has room for 14811136 particles in density computation
Communication buffer has room for 11848908 particles in hydro computation
Communication buffer has room for 11022240 particles in domain decomposition

Hubble (internal units) = 100
G (internal units) = 43.0071
UnitMass_in_g = 1.989e+43
UnitTime_in_s = 3.08568e+19
UnitVelocity_in_cm_per_s = 100000
UnitDensity_in_cgs = 6.76991e-31
UnitEnergy_in_cgs = 1.989e+53

Task=0 FFT-Slabs=18
Task=1 FFT-Slabs=18
Task=2 FFT-Slabs=18
Task=3 FFT-Slabs=18
Task=4 FFT-Slabs=18
Task=5 FFT-Slabs=18
Task=6 FFT-Slabs=18
Task=7 FFT-Slabs=18
Task=8 FFT-Slabs=18
Task=9 FFT-Slabs=18
Task=10 FFT-Slabs=18
Task=11 FFT-Slabs=18
Task=12 FFT-Slabs=18
Task=13 FFT-Slabs=18
Task=14 FFT-Slabs=18
Task=15 FFT-Slabs=18
Task=16 FFT-Slabs=18
Task=17 FFT-Slabs=18
Task=18 FFT-Slabs=18
Task=19 FFT-Slabs=18
Task=20 FFT-Slabs=18
Task=21 FFT-Slabs=18
Task=22 FFT-Slabs=18
Task=23 FFT-Slabs=18
Task=24 FFT-Slabs=18
Task=25 FFT-Slabs=18
Task=26 FFT-Slabs=18
Task=27 FFT-Slabs=18
Task=28 FFT-Slabs=18
Task=29 FFT-Slabs=18
Task=30 FFT-Slabs=18
Task=31 FFT-Slabs=18
Task=32 FFT-Slabs=18
Task=33 FFT-Slabs=18
Task=34 FFT-Slabs=18
Task=35 FFT-Slabs=18
Task=36 FFT-Slabs=18
Task=37 FFT-Slabs=18
Task=38 FFT-Slabs=18
Task=39 FFT-Slabs=18
Task=40 FFT-Slabs=18
Task=41 FFT-Slabs=18
Task=42 FFT-Slabs=18
Task=43 FFT-Slabs=18
Task=44 FFT-Slabs=18
Task=45 FFT-Slabs=18
Task=46 FFT-Slabs=18
Task=47 FFT-Slabs=18
Task=48 FFT-Slabs=18
Task=49 FFT-Slabs=18
Task=50 FFT-Slabs=18
Task=51 FFT-Slabs=18
Task=52 FFT-Slabs=18
Task=53 FFT-Slabs=18
Task=54 FFT-Slabs=18
Task=55 FFT-Slabs=18
Task=56 FFT-Slabs=16
Task=57 FFT-Slabs=0
Task=58 FFT-Slabs=0
Task=59 FFT-Slabs=0

Allocated 1365.33 MByte for particle storage. 80

Allocated 716.8 MByte for storage of SPH data. 84

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.0' on task=0
(contains 8388608 particles.)
distributing this file to tasks 0-2
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.1' on task=3
(contains 8388608 particles.)
distributing this file to tasks 3-4
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.2' on task=5
(contains 8388608 particles.)
distributing this file to tasks 5-6
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.3' on task=7
(contains 8388608 particles.)
distributing this file to tasks 7-8
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.4' on task=9
(contains 8388608 particles.)
distributing this file to tasks 9-10
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.5' on task=11
(contains 8388608 particles.)
distributing this file to tasks 11-12
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.6' on task=13
(contains 8388608 particles.)
distributing this file to tasks 13-13
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.7' on task=14
(contains 8388608 particles.)
distributing this file to tasks 14-14
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.8' on task=15
(contains 8388608 particles.)
distributing this file to tasks 15-17
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.9' on task=18
(contains 8388608 particles.)
distributing this file to tasks 18-19
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.10' on task=20
(contains 8388608 particles.)
distributing this file to tasks 20-21
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.11' on task=22
(contains 8388608 particles.)
distributing this file to tasks 22-23
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.12' on task=24
(contains 8388608 particles.)
distributing this file to tasks 24-25
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.13' on task=26
(contains 8388608 particles.)
distributing this file to tasks 26-27
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.14' on task=28
(contains 8388608 particles.)
distributing this file to tasks 28-28
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.15' on task=29
(contains 8388608 particles.)
distributing this file to tasks 29-29
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.16' on task=30
(contains 8388608 particles.)
distributing this file to tasks 30-32
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.17' on task=33
(contains 8388608 particles.)
distributing this file to tasks 33-34
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.18' on task=35
(contains 8388608 particles.)
distributing this file to tasks 35-36
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.19' on task=37
(contains 8388608 particles.)
distributing this file to tasks 37-38
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.20' on task=39
(contains 8388608 particles.)
distributing this file to tasks 39-40
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.21' on task=41
(contains 8388608 particles.)
distributing this file to tasks 41-42
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.22' on task=43
(contains 8388608 particles.)
distributing this file to tasks 43-43
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.23' on task=44
(contains 8388608 particles.)
distributing this file to tasks 44-44
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.24' on task=45
(contains 8388608 particles.)
distributing this file to tasks 45-47
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.25' on task=48
(contains 8388608 particles.)
distributing this file to tasks 48-49
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.26' on task=50
(contains 8388608 particles.)
distributing this file to tasks 50-51
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.27' on task=52
(contains 8388608 particles.)
distributing this file to tasks 52-53
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.28' on task=54
(contains 8388608 particles.)
distributing this file to tasks 54-55
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.29' on task=56
(contains 8388608 particles.)
distributing this file to tasks 56-57
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.30' on task=58
(contains 8388608 particles.)
distributing this file to tasks 58-58
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading file `/mnt/lustre/atrindade/2LPT/IC_test/ics.31' on task=59
(contains 8388608 particles.)
distributing this file to tasks 59-59
Type 0 (gas): 4194304 (tot= 0134217728) masstab=0.0661731
Type 1 (halo): 4194304 (tot= 0134217728) masstab=0.430125
Type 2 (disk): 0 (tot= 0000000000) masstab=0
Type 3 (bulge): 0 (tot= 0000000000) masstab=0
Type 4 (stars): 0 (tot= 0000000000) masstab=0
Type 5 (bndry): 0 (tot= 0000000000) masstab=0

reading done.
Total number of particles : 0268435456

allocated 0.0762939 Mbyte for ngb search.

Allocated 4506.36 MByte for BH-tree. 64

domain decomposition...
NTopleaves= 127

No domain decomposition that stays within memory bounds is possible.
---------------------------------------------------------------------------------------------------
Received on 2014-04-16 18:45:19