Re: The Simulation suddenly crashed at redshift 1.49612

From: Leonard Romano <leonard.romano_at_tum.de>
Date: Fri, 9 Jul 2021 10:47:45 +0200

Dear Hassan,

This looks very similar to this issue:
https://wwwmpa.mpa-garching.mpg.de/gadget/gadget-list/0846.html
Have you pulled from the main repository since April 15th?
The issue here is that there is still a star in the gas block.
In the fix included on April 15th a domain decomposition is enforced
before FOF is run, making sure that this does not happen.
Probably pulling from the main repository will fix your problem.

Best,
Leonard

On 09.07.21 07:21, Hassan Fathivavsari wrote:
> Dear all,
>
> I have run a DM + Gas simulation. Everything seemed to be working fine
> but when it reached at redshift 1.49612 it crashed with following report:
> (This redshift corresponds to time = 0.4 at which an output was requested
> to be generated):
>
> Sync-Point 6985, Time: 0.400621, Redshift: 1.49612, Systemstep:
> 7.87001e-05, Dloga: 0.000196465, Nsync-grv: 524288, Nsync-hyd:     261475
> DOMAIN: Begin domain decomposition (sync-point 6985).
> DOMAIN: New shift vector determined (-0.57978 -1.13576 -1.36276)
> DOMAIN: Sum=4  TotalCost=4  NumTimeBinsToBeBalanced=1  MultipleDomains=4
> DOMAIN: NTopleaves=316, determination of top-level tree involved 8
> iterations and took 0.0430765 sec
> DOMAIN: we are going to try at most 135 different settings for
> combining the domains on tasks=5, nnodes=1
> DOMAIN: total_cost=4  total_load=2
> DOMAIN: total_cost=4  total_load=2
> DOMAIN: total_cost=4  total_load=2
> DOMAIN: total_cost=4  total_load=2
> DOMAIN: total_cost=4  total_load=2
> DOMAIN: total_cost=4  total_load=2
> DOMAIN: best solution found after 6 iterations by task=3 for nextra=0,
> reaching maximum imbalance of 1.14521|1.14552
> DOMAIN: combining multiple-domains took 0.00363554 sec
> DOMAIN: exchange of 262813 particles
> DOMAIN: particle exchange done. (took 0.031764 sec)
> DOMAIN: domain decomposition done. (took in total 0.0888818 sec)
> PEANO: Begin Peano-Hilbert order...
> PEANO: done, took 0.0318906 sec.
> NGBTREE: Ngb-tree construction done. took 0.0324755 sec
>  <numnodes>=8117.4  NTopnodes=361 NTopleaves=316
> SPH-DENSITY: Begin density calculation. (presently allocated=69.4507 MB)
> SPH-DENSITY: Ndensities=261475 (task zero has: NumGas=59889,
> Ndensities=59889)
> SPH-DENSITY: ngb iteration    1: took    0.914  , need to repeat for
> 000000000902 local particles.
> SPH-DENSITY: ngb iteration    2: took    0.021
> SPH-DENSITY: density computation done. took    1.186
> SPH-HYDRO: Begin hydro-force calculation.  (presently
> allocated=70.3647 MB)
> SPH-HYDRO: global Nhydro=261475 (task zero: NumGas=59889, Nhydro=59889)
> SPH-HYDRO: hydro-force computation done. took    3.293
> ACCEL: Start tree gravity force computation... (524288 particles)
> TREEPM: Starting PM part of force calculation. (timebin=19)
> PM-PERIODIC: Starting periodic PM calculation. (Rcut=0.109375)
>  presently allocated=69.4507 MB
> PM-PERIODIC: done.  (took 2.1174 seconds)
> TREEPM: Finished PM part of force calculation.
> TREE: Full tree construction for all particles. (presently
> allocated=71.0386 MB)
> GRAVTREE: Tree construction done. took 0.067678 sec
>  <numnodes>=17780.8  NTopnodes=361 NTopleaves=316
> tree-build-scalability=0.983758
> GRAVTREE: Begin tree force. timebin=19 (presently allocated=71.9547 MB)
> GRAVTREE: tree-forces are calculated, with 2 cycles took 6.47065 sec
> GRAVTREE: tree-force is done.
> ACCEL: tree force computation done.
> KICKS: 2nd gravity for highest active timebin=19:  particles 524288
> GRAVTREE/FMM: Setting OldAcc!
> SFR: spawned 0 stars, converted 1 gas particles into stars
>
> FOF: We shall first compute a group catalog for this snapshot file
> FOF: Begin to compute FoF group catalogue...  (presently
> allocated=71.4042 MB)
> FOF: Comoving linking length: 0.009375
> FOFTREE: Ngb-tree construction done. took 0.0579319 sec
>  <numnodes>=9876.4  NTopnodes=361 NTopleaves=316
> FOF: Start linking particles (presently allocated=76.3867 MB)
> FOF: linking of small cells took 0.000385365 sec
> FOF: local links done (took 0.10742 sec, avg-work=0.088653,
> imbalance=1.20979).
> FOF: Marked=42436 out of the 262144 primaries which are linked
> FOF: begin linking across processors (presently allocated=77.0021 MB)
> FOF: have done             804 cross links (processed  42436, took
> 0.0437611 sec)
> FOF: have done             269 cross links (processed  25989, took
> 0.0270564 sec)
> FOF: have done             104 cross links (processed   4470, took
> 0.00817215 sec)
> FOF: have done              26 cross links (processed    537, took
> 0.00249025 sec)
> FOF: have done               5 cross links (processed    104, took
> 0.00189013 sec)
> FOF: have done               0 cross links (processed     20, took
> 0.00173687 sec)
> FOF: Local groups found.
> FOF: primary group finding took = 0.200323 sec
> FOF: Start finding nearest dm-particle (presently allocated=76.3867 MB)
> FOF: done finding nearest dm-particle
> FOF: attaching gas and star particles to nearest dm particles took =
> 0.000317938 sec
> FOF: compiling local group data and catalogue took = 0.037896 sec
> FOF: Total number of FOF groups with at least 32 particles: 241
> FOF: Largest FOF group has 29876 particles.
> FOF: Total number of particles in FOF groups: 103518
> FOF: group properties are now allocated.. (presently allocated=72.797 MB)
> FOF: computation of group properties took = 0.000784035 sec
> FOF: start assigning group numbers
> FOF: Assigning of group numbers took = 0.00289581 sec
> FOF: Finished computing FoF groups.  Complete work took 0.308229 sec
>  (presently allocated=71.4239 MB)
> FOF/SUBFIND: writing group catalogue file: './output/fof_tab_004'
> (file 1 of 1)
> FOF/SUBFIND: writing group catalogue block 0 (GroupLen)...
> FOF/SUBFIND: writing group catalogue block 1 (GroupMass)...
> FOF/SUBFIND: writing group catalogue block 2 (GroupPos)...
> FOF/SUBFIND: writing group catalogue block 3 (GroupVel)...
> FOF/SUBFIND: writing group catalogue block 4 (GroupLenType)...
> FOF/SUBFIND: writing group catalogue block 5 (GroupOffsetType)...
> FOF/SUBFIND: writing group catalogue block 6 (GroupMassType)...
> FOF/SUBFIND: writing group catalogue block 7 (GroupAscale)...
> FOF/SUBFIND: writing group catalogue block 8 (GroupSFR)..
> FOF/SUBFIND: Group catalogues saved. took = 0.00427187 sec, total size
> 0.0312576 MB, corresponds to effective I/O rate of 7.31709 MB/sec
> Code termination on task=4, function particle_exchange_based_on_PS(),
> file src/domain/domain_exchange.cc, line 546: Bummer2
> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 4
> pc_at_ubuntu:~/Desktop/SPH/gadget4-master/examples/DM-L50-N128/Including_Gas$
>
> I would be grateful for any help with this issue,
>
> Best regards,
> Hassan
>
>
-- 
===================================================
Leonard Romano, B.Sc.(レオナルド・ロマノ)
Physics Department
Technical University of Munich (TUM), Germany
Theoretical Astrophysics Group
Department of Earth and Space Science
Graduate School of Science, Osaka University, Japan
he / him / his
===================================================
Received on 2021-07-09 10:47:50

This archive was generated by hypermail 2.3.0 : 2022-09-01 14:03:43 CEST