Dear Hassan,
This looks very similar to this issue:
https://wwwmpa.mpa-garching.mpg.de/gadget/gadget-list/0846.html
Have you pulled from the main repository since April 15th?
The issue here is that there is still a star in the gas block.
In the fix included on April 15th a domain decomposition is enforced
before FOF is run, making sure that this does not happen.
Probably pulling from the main repository will fix your problem.
Best,
Leonard
On 09.07.21 07:21, Hassan Fathivavsari wrote:
> Dear all,
>
> I have run a DM + Gas simulation. Everything seemed to be working fine
> but when it reached at redshift 1.49612 it crashed with following report:
> (This redshift corresponds to time = 0.4 at which an output was requested
> to be generated):
>
> Sync-Point 6985, Time: 0.400621, Redshift: 1.49612, Systemstep:
> 7.87001e-05, Dloga: 0.000196465, Nsync-grv: 524288, Nsync-hyd: 261475
> DOMAIN: Begin domain decomposition (sync-point 6985).
> DOMAIN: New shift vector determined (-0.57978 -1.13576 -1.36276)
> DOMAIN: Sum=4 TotalCost=4 NumTimeBinsToBeBalanced=1 MultipleDomains=4
> DOMAIN: NTopleaves=316, determination of top-level tree involved 8
> iterations and took 0.0430765 sec
> DOMAIN: we are going to try at most 135 different settings for
> combining the domains on tasks=5, nnodes=1
> DOMAIN: total_cost=4 total_load=2
> DOMAIN: total_cost=4 total_load=2
> DOMAIN: total_cost=4 total_load=2
> DOMAIN: total_cost=4 total_load=2
> DOMAIN: total_cost=4 total_load=2
> DOMAIN: total_cost=4 total_load=2
> DOMAIN: best solution found after 6 iterations by task=3 for nextra=0,
> reaching maximum imbalance of 1.14521|1.14552
> DOMAIN: combining multiple-domains took 0.00363554 sec
> DOMAIN: exchange of 262813 particles
> DOMAIN: particle exchange done. (took 0.031764 sec)
> DOMAIN: domain decomposition done. (took in total 0.0888818 sec)
> PEANO: Begin Peano-Hilbert order...
> PEANO: done, took 0.0318906 sec.
> NGBTREE: Ngb-tree construction done. took 0.0324755 sec
> <numnodes>=8117.4 NTopnodes=361 NTopleaves=316
> SPH-DENSITY: Begin density calculation. (presently allocated=69.4507 MB)
> SPH-DENSITY: Ndensities=261475 (task zero has: NumGas=59889,
> Ndensities=59889)
> SPH-DENSITY: ngb iteration 1: took 0.914 , need to repeat for
> 000000000902 local particles.
> SPH-DENSITY: ngb iteration 2: took 0.021
> SPH-DENSITY: density computation done. took 1.186
> SPH-HYDRO: Begin hydro-force calculation. (presently
> allocated=70.3647 MB)
> SPH-HYDRO: global Nhydro=261475 (task zero: NumGas=59889, Nhydro=59889)
> SPH-HYDRO: hydro-force computation done. took 3.293
> ACCEL: Start tree gravity force computation... (524288 particles)
> TREEPM: Starting PM part of force calculation. (timebin=19)
> PM-PERIODIC: Starting periodic PM calculation. (Rcut=0.109375)
> presently allocated=69.4507 MB
> PM-PERIODIC: done. (took 2.1174 seconds)
> TREEPM: Finished PM part of force calculation.
> TREE: Full tree construction for all particles. (presently
> allocated=71.0386 MB)
> GRAVTREE: Tree construction done. took 0.067678 sec
> <numnodes>=17780.8 NTopnodes=361 NTopleaves=316
> tree-build-scalability=0.983758
> GRAVTREE: Begin tree force. timebin=19 (presently allocated=71.9547 MB)
> GRAVTREE: tree-forces are calculated, with 2 cycles took 6.47065 sec
> GRAVTREE: tree-force is done.
> ACCEL: tree force computation done.
> KICKS: 2nd gravity for highest active timebin=19: particles 524288
> GRAVTREE/FMM: Setting OldAcc!
> SFR: spawned 0 stars, converted 1 gas particles into stars
>
> FOF: We shall first compute a group catalog for this snapshot file
> FOF: Begin to compute FoF group catalogue... (presently
> allocated=71.4042 MB)
> FOF: Comoving linking length: 0.009375
> FOFTREE: Ngb-tree construction done. took 0.0579319 sec
> <numnodes>=9876.4 NTopnodes=361 NTopleaves=316
> FOF: Start linking particles (presently allocated=76.3867 MB)
> FOF: linking of small cells took 0.000385365 sec
> FOF: local links done (took 0.10742 sec, avg-work=0.088653,
> imbalance=1.20979).
> FOF: Marked=42436 out of the 262144 primaries which are linked
> FOF: begin linking across processors (presently allocated=77.0021 MB)
> FOF: have done 804 cross links (processed 42436, took
> 0.0437611 sec)
> FOF: have done 269 cross links (processed 25989, took
> 0.0270564 sec)
> FOF: have done 104 cross links (processed 4470, took
> 0.00817215 sec)
> FOF: have done 26 cross links (processed 537, took
> 0.00249025 sec)
> FOF: have done 5 cross links (processed 104, took
> 0.00189013 sec)
> FOF: have done 0 cross links (processed 20, took
> 0.00173687 sec)
> FOF: Local groups found.
> FOF: primary group finding took = 0.200323 sec
> FOF: Start finding nearest dm-particle (presently allocated=76.3867 MB)
> FOF: done finding nearest dm-particle
> FOF: attaching gas and star particles to nearest dm particles took =
> 0.000317938 sec
> FOF: compiling local group data and catalogue took = 0.037896 sec
> FOF: Total number of FOF groups with at least 32 particles: 241
> FOF: Largest FOF group has 29876 particles.
> FOF: Total number of particles in FOF groups: 103518
> FOF: group properties are now allocated.. (presently allocated=72.797 MB)
> FOF: computation of group properties took = 0.000784035 sec
> FOF: start assigning group numbers
> FOF: Assigning of group numbers took = 0.00289581 sec
> FOF: Finished computing FoF groups. Complete work took 0.308229 sec
> (presently allocated=71.4239 MB)
> FOF/SUBFIND: writing group catalogue file: './output/fof_tab_004'
> (file 1 of 1)
> FOF/SUBFIND: writing group catalogue block 0 (GroupLen)...
> FOF/SUBFIND: writing group catalogue block 1 (GroupMass)...
> FOF/SUBFIND: writing group catalogue block 2 (GroupPos)...
> FOF/SUBFIND: writing group catalogue block 3 (GroupVel)...
> FOF/SUBFIND: writing group catalogue block 4 (GroupLenType)...
> FOF/SUBFIND: writing group catalogue block 5 (GroupOffsetType)...
> FOF/SUBFIND: writing group catalogue block 6 (GroupMassType)...
> FOF/SUBFIND: writing group catalogue block 7 (GroupAscale)...
> FOF/SUBFIND: writing group catalogue block 8 (GroupSFR)..
> FOF/SUBFIND: Group catalogues saved. took = 0.00427187 sec, total size
> 0.0312576 MB, corresponds to effective I/O rate of 7.31709 MB/sec
> Code termination on task=4, function particle_exchange_based_on_PS(),
> file src/domain/domain_exchange.cc, line 546: Bummer2
> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 4
> pc_at_ubuntu:~/Desktop/SPH/gadget4-master/examples/DM-L50-N128/Including_Gas$
>
> I would be grateful for any help with this issue,
>
> Best regards,
> Hassan
>
>
--
===================================================
Leonard Romano, B.Sc.（レオナルド・ロマノ）
Physics Department
Technical University of Munich (TUM), Germany
Theoretical Astrophysics Group
Department of Earth and Space Science
Graduate School of Science, Osaka University, Japan
he / him / his
===================================================
Received on 2021-07-09 10:47:50