The Simulation suddenly crashed at redshift 1.49612

From: Hassan Fathivavsari <>
Date: Fri, 9 Jul 2021 07:21:27 +0200

Dear all,

I have run a DM + Gas simulation. Everything seemed to be working fine
but when it reached at redshift 1.49612 it crashed with following report:
(This redshift corresponds to time = 0.4 at which an output was requested
to be generated):

Sync-Point 6985, Time: 0.400621, Redshift: 1.49612, Systemstep:
7.87001e-05, Dloga: 0.000196465, Nsync-grv: 524288, Nsync-hyd:
DOMAIN: Begin domain decomposition (sync-point 6985).
DOMAIN: New shift vector determined (-0.57978 -1.13576 -1.36276)
DOMAIN: Sum=4 TotalCost=4 NumTimeBinsToBeBalanced=1 MultipleDomains=4
DOMAIN: NTopleaves=316, determination of top-level tree involved 8
iterations and took 0.0430765 sec
DOMAIN: we are going to try at most 135 different settings for combining
the domains on tasks=5, nnodes=1
DOMAIN: total_cost=4 total_load=2
DOMAIN: total_cost=4 total_load=2
DOMAIN: total_cost=4 total_load=2
DOMAIN: total_cost=4 total_load=2
DOMAIN: total_cost=4 total_load=2
DOMAIN: total_cost=4 total_load=2
DOMAIN: best solution found after 6 iterations by task=3 for nextra=0,
reaching maximum imbalance of 1.14521|1.14552
DOMAIN: combining multiple-domains took 0.00363554 sec
DOMAIN: exchange of 262813 particles
DOMAIN: particle exchange done. (took 0.031764 sec)
DOMAIN: domain decomposition done. (took in total 0.0888818 sec)
PEANO: Begin Peano-Hilbert order...
PEANO: done, took 0.0318906 sec.
NGBTREE: Ngb-tree construction done. took 0.0324755 sec <numnodes>=8117.4
 NTopnodes=361 NTopleaves=316
SPH-DENSITY: Begin density calculation. (presently allocated=69.4507 MB)
SPH-DENSITY: Ndensities=261475 (task zero has: NumGas=59889,
SPH-DENSITY: ngb iteration 1: took 0.914 , need to repeat for
000000000902 local particles.
SPH-DENSITY: ngb iteration 2: took 0.021
SPH-DENSITY: density computation done. took 1.186
SPH-HYDRO: Begin hydro-force calculation. (presently allocated=70.3647 MB)
SPH-HYDRO: global Nhydro=261475 (task zero: NumGas=59889, Nhydro=59889)
SPH-HYDRO: hydro-force computation done. took 3.293
ACCEL: Start tree gravity force computation... (524288 particles)
TREEPM: Starting PM part of force calculation. (timebin=19)
PM-PERIODIC: Starting periodic PM calculation. (Rcut=0.109375) presently
allocated=69.4507 MB
PM-PERIODIC: done. (took 2.1174 seconds)
TREEPM: Finished PM part of force calculation.
TREE: Full tree construction for all particles. (presently
allocated=71.0386 MB)
GRAVTREE: Tree construction done. took 0.067678 sec <numnodes>=17780.8
 NTopnodes=361 NTopleaves=316 tree-build-scalability=0.983758
GRAVTREE: Begin tree force. timebin=19 (presently allocated=71.9547 MB)
GRAVTREE: tree-forces are calculated, with 2 cycles took 6.47065 sec
GRAVTREE: tree-force is done.
ACCEL: tree force computation done.
KICKS: 2nd gravity for highest active timebin=19: particles 524288
GRAVTREE/FMM: Setting OldAcc!
SFR: spawned 0 stars, converted 1 gas particles into stars

FOF: We shall first compute a group catalog for this snapshot file
FOF: Begin to compute FoF group catalogue... (presently allocated=71.4042
FOF: Comoving linking length: 0.009375
FOFTREE: Ngb-tree construction done. took 0.0579319 sec <numnodes>=9876.4
 NTopnodes=361 NTopleaves=316
FOF: Start linking particles (presently allocated=76.3867 MB)
FOF: linking of small cells took 0.000385365 sec
FOF: local links done (took 0.10742 sec, avg-work=0.088653,
FOF: Marked=42436 out of the 262144 primaries which are linked
FOF: begin linking across processors (presently allocated=77.0021 MB)
FOF: have done 804 cross links (processed 42436, took
0.0437611 sec)
FOF: have done 269 cross links (processed 25989, took
0.0270564 sec)
FOF: have done 104 cross links (processed 4470, took
0.00817215 sec)
FOF: have done 26 cross links (processed 537, took
0.00249025 sec)
FOF: have done 5 cross links (processed 104, took
0.00189013 sec)
FOF: have done 0 cross links (processed 20, took
0.00173687 sec)
FOF: Local groups found.
FOF: primary group finding took = 0.200323 sec
FOF: Start finding nearest dm-particle (presently allocated=76.3867 MB)
FOF: done finding nearest dm-particle
FOF: attaching gas and star particles to nearest dm particles took =
0.000317938 sec
FOF: compiling local group data and catalogue took = 0.037896 sec
FOF: Total number of FOF groups with at least 32 particles: 241
FOF: Largest FOF group has 29876 particles.
FOF: Total number of particles in FOF groups: 103518
FOF: group properties are now allocated.. (presently allocated=72.797 MB)
FOF: computation of group properties took = 0.000784035 sec
FOF: start assigning group numbers
FOF: Assigning of group numbers took = 0.00289581 sec
FOF: Finished computing FoF groups. Complete work took 0.308229 sec
 (presently allocated=71.4239 MB)
FOF/SUBFIND: writing group catalogue file: './output/fof_tab_004' (file 1
of 1)
FOF/SUBFIND: writing group catalogue block 0 (GroupLen)...
FOF/SUBFIND: writing group catalogue block 1 (GroupMass)...
FOF/SUBFIND: writing group catalogue block 2 (GroupPos)...
FOF/SUBFIND: writing group catalogue block 3 (GroupVel)...
FOF/SUBFIND: writing group catalogue block 4 (GroupLenType)...
FOF/SUBFIND: writing group catalogue block 5 (GroupOffsetType)...
FOF/SUBFIND: writing group catalogue block 6 (GroupMassType)...
FOF/SUBFIND: writing group catalogue block 7 (GroupAscale)...
FOF/SUBFIND: writing group catalogue block 8 (GroupSFR)...
FOF/SUBFIND: Group catalogues saved. took = 0.00427187 sec, total size
0.0312576 MB, corresponds to effective I/O rate of 7.31709 MB/sec
Code termination on task=4, function particle_exchange_based_on_PS(), file
src/domain/, line 546: Bummer2
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 4

I would be grateful for any help with this issue,

Best regards,
Received on 2021-07-09 07:21:40

This archive was generated by hypermail 2.3.0 : 2022-09-01 14:03:43 CEST