- Mail actions: [ respond to this message ] [ mail a new topic ]
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]

From: Weiguang Cui <cuiweiguang_at_gmail.com>

Date: Sat, 21 Aug 2021 09:10:39 +0100

Dear all,

I recently met another problem with the 2048^3, 200 mpc/h run.

treebuild in SUBFIND requires a higher value for TREE_NUM_BEFORE_NODESPLIT:

==========================================================

SUBFIND: We now execute a parallel version of SUBFIND.

SUBFIND: Previous subhalo catalogue had approximately a size 2.42768e+09,

and the summed squared subhalo size was 8.42698e+16

SUBFIND: Number of FOF halos treated with collective SubFind algorithm = 1

SUBFIND: Number of processors used in different partitions for the

collective SubFind code = 2

SUBFIND: (The adopted size-limit for the collective algorithm was 9631634

particles, for threshold size factor 0.6)

SUBFIND: The other 10021349 FOF halos are treated in parallel with serial

code

SUBFIND: subfind_distribute_groups() took 0.044379 sec

SUBFIND: particle balance=1.10537

SUBFIND: subfind_exchange() took 30.2562 sec

SUBFIND: particle balance for processing=1

SUBFIND: root-task=0: Collectively doing halo 0 of length 10426033 on 2

processors.

SUBFIND: subdomain decomposition took 8.54527 sec

SUBFIND: serial subfind subdomain decomposition took 6.0162 sec

SUBFIND: root-task=0: total number of subhalo coll_candidates=1454

SUBFIND: root-task=0: number of subhalo candidates small enough to be done

with one cpu: 1453. (Largest size 81455)

Code termination on task=0, function treebuild_insert_group_of_points(),

file src/tree/tree.cc, line 489: It appears we have reached the bottom of

the tree because there are more than TREE_NUM_BEFORE_NODESPLIT=16 particles

in the smallest tree node representable for BITS_FOR_POSITIONS=64.

Either eliminate the particles at (nearly) indentical coordinates, increase

the setting for TREE_NUM_BEFORE_NODESPLIT, or possibly enlarge

BITS_FOR_POSITIONS if you have really not enough dynamic range

==============================================

But, if I increase the TREE_NUM_BEFORE_NODESPLIT to 64, FMM seems not

working:

=============================================================

Sync-Point 19835, Time: 0.750591, Redshift: 0.332284, Systemstep:

5.27389e-05, Dloga: 7.02657e-05, Nsync-grv: 31415, Nsync-hyd:

0

ACCEL: Start tree gravity force computation... (31415 particles)

TREE: Full tree construction for all particles. (presently

allocated=7626.51 MB)

GRAVTREE: Tree construction done. took 13.4471 sec <numnodes>=206492

NTopnodes=115433 NTopleaves=101004 tree-build-scalability=0.441627

FMM: Begin tree force. timebin=13 (presently allocated=0.5 MB)

Code termination on task=0, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=887, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=40, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=888, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=889, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=3, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=890, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=6, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=891, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=9, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=892, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=893, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=894, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=20, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

======================================

I don't think fine-tuning the value for TREE_NUM_BEFORE_NODESPLIT is a

solution.

I can try to use BITS_FOR_POSITIONS=128 by setting POSITIONS_IN_128BIT, but

I am afraid that the code may not be able to run from restart files.

Any suggestions?

Many thanks.

Best,

Weiguang

-------------------------------------------

https://weiguangcui.github.io/

Received on 2021-08-21 10:11:22

Date: Sat, 21 Aug 2021 09:10:39 +0100

Dear all,

I recently met another problem with the 2048^3, 200 mpc/h run.

treebuild in SUBFIND requires a higher value for TREE_NUM_BEFORE_NODESPLIT:

==========================================================

SUBFIND: We now execute a parallel version of SUBFIND.

SUBFIND: Previous subhalo catalogue had approximately a size 2.42768e+09,

and the summed squared subhalo size was 8.42698e+16

SUBFIND: Number of FOF halos treated with collective SubFind algorithm = 1

SUBFIND: Number of processors used in different partitions for the

collective SubFind code = 2

SUBFIND: (The adopted size-limit for the collective algorithm was 9631634

particles, for threshold size factor 0.6)

SUBFIND: The other 10021349 FOF halos are treated in parallel with serial

code

SUBFIND: subfind_distribute_groups() took 0.044379 sec

SUBFIND: particle balance=1.10537

SUBFIND: subfind_exchange() took 30.2562 sec

SUBFIND: particle balance for processing=1

SUBFIND: root-task=0: Collectively doing halo 0 of length 10426033 on 2

processors.

SUBFIND: subdomain decomposition took 8.54527 sec

SUBFIND: serial subfind subdomain decomposition took 6.0162 sec

SUBFIND: root-task=0: total number of subhalo coll_candidates=1454

SUBFIND: root-task=0: number of subhalo candidates small enough to be done

with one cpu: 1453. (Largest size 81455)

Code termination on task=0, function treebuild_insert_group_of_points(),

file src/tree/tree.cc, line 489: It appears we have reached the bottom of

the tree because there are more than TREE_NUM_BEFORE_NODESPLIT=16 particles

in the smallest tree node representable for BITS_FOR_POSITIONS=64.

Either eliminate the particles at (nearly) indentical coordinates, increase

the setting for TREE_NUM_BEFORE_NODESPLIT, or possibly enlarge

BITS_FOR_POSITIONS if you have really not enough dynamic range

==============================================

But, if I increase the TREE_NUM_BEFORE_NODESPLIT to 64, FMM seems not

working:

=============================================================

Sync-Point 19835, Time: 0.750591, Redshift: 0.332284, Systemstep:

5.27389e-05, Dloga: 7.02657e-05, Nsync-grv: 31415, Nsync-hyd:

0

ACCEL: Start tree gravity force computation... (31415 particles)

TREE: Full tree construction for all particles. (presently

allocated=7626.51 MB)

GRAVTREE: Tree construction done. took 13.4471 sec <numnodes>=206492

NTopnodes=115433 NTopleaves=101004 tree-build-scalability=0.441627

FMM: Begin tree force. timebin=13 (presently allocated=0.5 MB)

Code termination on task=0, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=887, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=40, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=888, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=889, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=3, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=890, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=6, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=891, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=9, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=892, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=893, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=894, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

Code termination on task=20, function gravity_fmm(), file src/fmm/fmm.cc,

line 1879: Can't even process a single particle

======================================

I don't think fine-tuning the value for TREE_NUM_BEFORE_NODESPLIT is a

solution.

I can try to use BITS_FOR_POSITIONS=128 by setting POSITIONS_IN_128BIT, but

I am afraid that the code may not be able to run from restart files.

Any suggestions?

Many thanks.

Best,

Weiguang

-------------------------------------------

https://weiguangcui.github.io/

Received on 2021-08-21 10:11:22

*
This archive was generated by hypermail 2.3.0
: 2022-09-01 14:03:43 CEST
*