- Mail actions: [ respond to this message ] [ mail a new topic ]
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]

From: Hassan Fathivavsari <h.fathie_at_gmail.com>

Date: Fri, 9 Jul 2021 07:21:27 +0200

Dear all,

I have run a DM + Gas simulation. Everything seemed to be working fine

but when it reached at redshift 1.49612 it crashed with following report:

(This redshift corresponds to time = 0.4 at which an output was requested

to be generated):

Sync-Point 6985, Time: 0.400621, Redshift: 1.49612, Systemstep:

7.87001e-05, Dloga: 0.000196465, Nsync-grv: 524288, Nsync-hyd:

261475

DOMAIN: Begin domain decomposition (sync-point 6985).

DOMAIN: New shift vector determined (-0.57978 -1.13576 -1.36276)

DOMAIN: Sum=4 TotalCost=4 NumTimeBinsToBeBalanced=1 MultipleDomains=4

DOMAIN: NTopleaves=316, determination of top-level tree involved 8

iterations and took 0.0430765 sec

DOMAIN: we are going to try at most 135 different settings for combining

the domains on tasks=5, nnodes=1

DOMAIN: total_cost=4 total_load=2

DOMAIN: total_cost=4 total_load=2

DOMAIN: total_cost=4 total_load=2

DOMAIN: total_cost=4 total_load=2

DOMAIN: total_cost=4 total_load=2

DOMAIN: total_cost=4 total_load=2

DOMAIN: best solution found after 6 iterations by task=3 for nextra=0,

reaching maximum imbalance of 1.14521|1.14552

DOMAIN: combining multiple-domains took 0.00363554 sec

DOMAIN: exchange of 262813 particles

DOMAIN: particle exchange done. (took 0.031764 sec)

DOMAIN: domain decomposition done. (took in total 0.0888818 sec)

PEANO: Begin Peano-Hilbert order...

PEANO: done, took 0.0318906 sec.

NGBTREE: Ngb-tree construction done. took 0.0324755 sec <numnodes>=8117.4

NTopnodes=361 NTopleaves=316

SPH-DENSITY: Begin density calculation. (presently allocated=69.4507 MB)

SPH-DENSITY: Ndensities=261475 (task zero has: NumGas=59889,

Ndensities=59889)

SPH-DENSITY: ngb iteration 1: took 0.914 , need to repeat for

000000000902 local particles.

SPH-DENSITY: ngb iteration 2: took 0.021

SPH-DENSITY: density computation done. took 1.186

SPH-HYDRO: Begin hydro-force calculation. (presently allocated=70.3647 MB)

SPH-HYDRO: global Nhydro=261475 (task zero: NumGas=59889, Nhydro=59889)

SPH-HYDRO: hydro-force computation done. took 3.293

ACCEL: Start tree gravity force computation... (524288 particles)

TREEPM: Starting PM part of force calculation. (timebin=19)

PM-PERIODIC: Starting periodic PM calculation. (Rcut=0.109375) presently

allocated=69.4507 MB

PM-PERIODIC: done. (took 2.1174 seconds)

TREEPM: Finished PM part of force calculation.

TREE: Full tree construction for all particles. (presently

allocated=71.0386 MB)

GRAVTREE: Tree construction done. took 0.067678 sec <numnodes>=17780.8

NTopnodes=361 NTopleaves=316 tree-build-scalability=0.983758

GRAVTREE: Begin tree force. timebin=19 (presently allocated=71.9547 MB)

GRAVTREE: tree-forces are calculated, with 2 cycles took 6.47065 sec

GRAVTREE: tree-force is done.

ACCEL: tree force computation done.

KICKS: 2nd gravity for highest active timebin=19: particles 524288

GRAVTREE/FMM: Setting OldAcc!

SFR: spawned 0 stars, converted 1 gas particles into stars

FOF: We shall first compute a group catalog for this snapshot file

FOF: Begin to compute FoF group catalogue... (presently allocated=71.4042

MB)

FOF: Comoving linking length: 0.009375

FOFTREE: Ngb-tree construction done. took 0.0579319 sec <numnodes>=9876.4

NTopnodes=361 NTopleaves=316

FOF: Start linking particles (presently allocated=76.3867 MB)

FOF: linking of small cells took 0.000385365 sec

FOF: local links done (took 0.10742 sec, avg-work=0.088653,

imbalance=1.20979).

FOF: Marked=42436 out of the 262144 primaries which are linked

FOF: begin linking across processors (presently allocated=77.0021 MB)

FOF: have done 804 cross links (processed 42436, took

0.0437611 sec)

FOF: have done 269 cross links (processed 25989, took

0.0270564 sec)

FOF: have done 104 cross links (processed 4470, took

0.00817215 sec)

FOF: have done 26 cross links (processed 537, took

0.00249025 sec)

FOF: have done 5 cross links (processed 104, took

0.00189013 sec)

FOF: have done 0 cross links (processed 20, took

0.00173687 sec)

FOF: Local groups found.

FOF: primary group finding took = 0.200323 sec

FOF: Start finding nearest dm-particle (presently allocated=76.3867 MB)

FOF: done finding nearest dm-particle

FOF: attaching gas and star particles to nearest dm particles took =

0.000317938 sec

FOF: compiling local group data and catalogue took = 0.037896 sec

FOF: Total number of FOF groups with at least 32 particles: 241

FOF: Largest FOF group has 29876 particles.

FOF: Total number of particles in FOF groups: 103518

FOF: group properties are now allocated.. (presently allocated=72.797 MB)

FOF: computation of group properties took = 0.000784035 sec

FOF: start assigning group numbers

FOF: Assigning of group numbers took = 0.00289581 sec

FOF: Finished computing FoF groups. Complete work took 0.308229 sec

(presently allocated=71.4239 MB)

FOF/SUBFIND: writing group catalogue file: './output/fof_tab_004' (file 1

of 1)

FOF/SUBFIND: writing group catalogue block 0 (GroupLen)...

FOF/SUBFIND: writing group catalogue block 1 (GroupMass)...

FOF/SUBFIND: writing group catalogue block 2 (GroupPos)...

FOF/SUBFIND: writing group catalogue block 3 (GroupVel)...

FOF/SUBFIND: writing group catalogue block 4 (GroupLenType)...

FOF/SUBFIND: writing group catalogue block 5 (GroupOffsetType)...

FOF/SUBFIND: writing group catalogue block 6 (GroupMassType)...

FOF/SUBFIND: writing group catalogue block 7 (GroupAscale)...

FOF/SUBFIND: writing group catalogue block 8 (GroupSFR)...

FOF/SUBFIND: Group catalogues saved. took = 0.00427187 sec, total size

0.0312576 MB, corresponds to effective I/O rate of 7.31709 MB/sec

Code termination on task=4, function particle_exchange_based_on_PS(), file

src/domain/domain_exchange.cc, line 546: Bummer2

application called MPI_Abort(MPI_COMM_WORLD, 1) - process 4

pc_at_ubuntu:~/Desktop/SPH/gadget4-master/examples/DM-L50-N128/Including_Gas$

I would be grateful for any help with this issue,

Best regards,

Hassan

Received on 2021-07-09 07:21:40

Date: Fri, 9 Jul 2021 07:21:27 +0200

Dear all,

I have run a DM + Gas simulation. Everything seemed to be working fine

but when it reached at redshift 1.49612 it crashed with following report:

(This redshift corresponds to time = 0.4 at which an output was requested

to be generated):

Sync-Point 6985, Time: 0.400621, Redshift: 1.49612, Systemstep:

7.87001e-05, Dloga: 0.000196465, Nsync-grv: 524288, Nsync-hyd:

261475

DOMAIN: Begin domain decomposition (sync-point 6985).

DOMAIN: New shift vector determined (-0.57978 -1.13576 -1.36276)

DOMAIN: Sum=4 TotalCost=4 NumTimeBinsToBeBalanced=1 MultipleDomains=4

DOMAIN: NTopleaves=316, determination of top-level tree involved 8

iterations and took 0.0430765 sec

DOMAIN: we are going to try at most 135 different settings for combining

the domains on tasks=5, nnodes=1

DOMAIN: total_cost=4 total_load=2

DOMAIN: total_cost=4 total_load=2

DOMAIN: total_cost=4 total_load=2

DOMAIN: total_cost=4 total_load=2

DOMAIN: total_cost=4 total_load=2

DOMAIN: total_cost=4 total_load=2

DOMAIN: best solution found after 6 iterations by task=3 for nextra=0,

reaching maximum imbalance of 1.14521|1.14552

DOMAIN: combining multiple-domains took 0.00363554 sec

DOMAIN: exchange of 262813 particles

DOMAIN: particle exchange done. (took 0.031764 sec)

DOMAIN: domain decomposition done. (took in total 0.0888818 sec)

PEANO: Begin Peano-Hilbert order...

PEANO: done, took 0.0318906 sec.

NGBTREE: Ngb-tree construction done. took 0.0324755 sec <numnodes>=8117.4

NTopnodes=361 NTopleaves=316

SPH-DENSITY: Begin density calculation. (presently allocated=69.4507 MB)

SPH-DENSITY: Ndensities=261475 (task zero has: NumGas=59889,

Ndensities=59889)

SPH-DENSITY: ngb iteration 1: took 0.914 , need to repeat for

000000000902 local particles.

SPH-DENSITY: ngb iteration 2: took 0.021

SPH-DENSITY: density computation done. took 1.186

SPH-HYDRO: Begin hydro-force calculation. (presently allocated=70.3647 MB)

SPH-HYDRO: global Nhydro=261475 (task zero: NumGas=59889, Nhydro=59889)

SPH-HYDRO: hydro-force computation done. took 3.293

ACCEL: Start tree gravity force computation... (524288 particles)

TREEPM: Starting PM part of force calculation. (timebin=19)

PM-PERIODIC: Starting periodic PM calculation. (Rcut=0.109375) presently

allocated=69.4507 MB

PM-PERIODIC: done. (took 2.1174 seconds)

TREEPM: Finished PM part of force calculation.

TREE: Full tree construction for all particles. (presently

allocated=71.0386 MB)

GRAVTREE: Tree construction done. took 0.067678 sec <numnodes>=17780.8

NTopnodes=361 NTopleaves=316 tree-build-scalability=0.983758

GRAVTREE: Begin tree force. timebin=19 (presently allocated=71.9547 MB)

GRAVTREE: tree-forces are calculated, with 2 cycles took 6.47065 sec

GRAVTREE: tree-force is done.

ACCEL: tree force computation done.

KICKS: 2nd gravity for highest active timebin=19: particles 524288

GRAVTREE/FMM: Setting OldAcc!

SFR: spawned 0 stars, converted 1 gas particles into stars

FOF: We shall first compute a group catalog for this snapshot file

FOF: Begin to compute FoF group catalogue... (presently allocated=71.4042

MB)

FOF: Comoving linking length: 0.009375

FOFTREE: Ngb-tree construction done. took 0.0579319 sec <numnodes>=9876.4

NTopnodes=361 NTopleaves=316

FOF: Start linking particles (presently allocated=76.3867 MB)

FOF: linking of small cells took 0.000385365 sec

FOF: local links done (took 0.10742 sec, avg-work=0.088653,

imbalance=1.20979).

FOF: Marked=42436 out of the 262144 primaries which are linked

FOF: begin linking across processors (presently allocated=77.0021 MB)

FOF: have done 804 cross links (processed 42436, took

0.0437611 sec)

FOF: have done 269 cross links (processed 25989, took

0.0270564 sec)

FOF: have done 104 cross links (processed 4470, took

0.00817215 sec)

FOF: have done 26 cross links (processed 537, took

0.00249025 sec)

FOF: have done 5 cross links (processed 104, took

0.00189013 sec)

FOF: have done 0 cross links (processed 20, took

0.00173687 sec)

FOF: Local groups found.

FOF: primary group finding took = 0.200323 sec

FOF: Start finding nearest dm-particle (presently allocated=76.3867 MB)

FOF: done finding nearest dm-particle

FOF: attaching gas and star particles to nearest dm particles took =

0.000317938 sec

FOF: compiling local group data and catalogue took = 0.037896 sec

FOF: Total number of FOF groups with at least 32 particles: 241

FOF: Largest FOF group has 29876 particles.

FOF: Total number of particles in FOF groups: 103518

FOF: group properties are now allocated.. (presently allocated=72.797 MB)

FOF: computation of group properties took = 0.000784035 sec

FOF: start assigning group numbers

FOF: Assigning of group numbers took = 0.00289581 sec

FOF: Finished computing FoF groups. Complete work took 0.308229 sec

(presently allocated=71.4239 MB)

FOF/SUBFIND: writing group catalogue file: './output/fof_tab_004' (file 1

of 1)

FOF/SUBFIND: writing group catalogue block 0 (GroupLen)...

FOF/SUBFIND: writing group catalogue block 1 (GroupMass)...

FOF/SUBFIND: writing group catalogue block 2 (GroupPos)...

FOF/SUBFIND: writing group catalogue block 3 (GroupVel)...

FOF/SUBFIND: writing group catalogue block 4 (GroupLenType)...

FOF/SUBFIND: writing group catalogue block 5 (GroupOffsetType)...

FOF/SUBFIND: writing group catalogue block 6 (GroupMassType)...

FOF/SUBFIND: writing group catalogue block 7 (GroupAscale)...

FOF/SUBFIND: writing group catalogue block 8 (GroupSFR)...

FOF/SUBFIND: Group catalogues saved. took = 0.00427187 sec, total size

0.0312576 MB, corresponds to effective I/O rate of 7.31709 MB/sec

Code termination on task=4, function particle_exchange_based_on_PS(), file

src/domain/domain_exchange.cc, line 546: Bummer2

application called MPI_Abort(MPI_COMM_WORLD, 1) - process 4

pc_at_ubuntu:~/Desktop/SPH/gadget4-master/examples/DM-L50-N128/Including_Gas$

I would be grateful for any help with this issue,

Best regards,

Hassan

Received on 2021-07-09 07:21:40

*
This archive was generated by hypermail 2.3.0
: 2022-09-01 14:03:43 CEST
*