- Mail actions: [ respond to this message ] [ mail a new topic ]
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]

From: Weiguang Cui <cuiweiguang_at_gmail.com>

Date: Wed, 11 Aug 2021 09:46:05 +0100

Hi Volker,

Yes, `ErrTolTheta` is set to 0.5 which is just slightly small than the

proposed value in the document. But I really doubt that is the cause.

I can understand that the two massive halos, even treated with separated

tasks, can be costly. I suppose this is seen from

"SUBFIND: Processing overall took (total time=1482.26 sec)"

But there is about 2.5ks time difference to this line:

"SUBFIND: Finished with SUBFIND. (total time=4092.76 sec)"

Which should be reposeful for this cost? I can not see any significant cost

listed between the two lines. Maybe I am wrong in interpreting this and

they are not in sequence.

Thanks.

Best,

Weiguang

-------------------------------------------

https://weiguangcui.github.io/

On Tue, Aug 10, 2021 at 6:50 PM Volker Springel <

vspringel_at_mpa-garching.mpg.de> wrote:

*>
*

*> Hi Weiguang,
*

*>
*

*> No, the code has decided to treat the two biggest halos with 2 tasks each
*

*> (because they don't fit on a single processor with their ~6 million
*

*> particles), all the other tasks are used to process the 8913146 halos that
*

*> are smaller. Only for halos that are treated with multiple tasks you see
*

*> the "root-task" messages.
*

*>
*

*> The gravitational unbinding of highly resolved halos can take quite a bit
*

*> of time... it might be that you have set a quite small value of
*

*> ErrTolTheta, which would slow it down unnecessarily.
*

*>
*

*> Regards,
*

*> Volker
*

*>
*

*>
*

*> > On 7. Aug 2021, at 21:42, Weiguang Cui <cuiweiguang_at_gmail.com> wrote:
*

*> >
*

*> > Dear all,
*

*> >
*

*> > I understand that this simulation (2048^3 particles) is large to run, is
*

*> it normal that the SUBFIND takes about 4ks to complete? It seems that there
*

*> are only two tasks to run subfind. Are there some options in SUBFIND that I
*

*> did set not correctly?
*

*> >
*

*> > ```
*

*> > SUBFIND: We now execute a parallel version of SUBFIND.
*

*> > SUBFIND: Previous subhalo catalogue had approximately a size
*

*> 2.76992e+09, and the summed squared subhalo size was 1.16758e+14
*

*> > SUBFIND: Number of FOF halos treated with collective SubFind algorithm =
*

*> 2
*

*> > SUBFIND: Number of processors used in different partitions for the
*

*> collective SubFind code = 4
*

*> > SUBFIND: (The adopted size-limit for the collective algorithm was
*

*> 5965232 particles, for threshold size factor 0.6)
*

*> > SUBFIND: The other 8913146 FOF halos are treated in parallel with serial
*

*> code
*

*> > SUBFIND: subfind_distribute_groups() took 0.044987 sec
*

*> > SUBFIND: particle balance=1.23335
*

*> > SUBFIND: subfind_exchange() took 20.5998 sec
*

*> > SUBFIND: particle balance for processing=1
*

*> > SUBFIND: root-task=2: Collectively doing halo 1 of length 6001658 on
*

*> 2 processors.
*

*> > SUBFIND: root-task=0: Collectively doing halo 0 of length 6318694 on
*

*> 2 processors.
*

*> > SUBFIND: subdomain decomposition took 5.28067 sec
*

*> > SUBFIND: serial subfind subdomain decomposition took 4.50761 sec
*

*> > SUBFIND: root-task=2: total number of subhalo coll_candidates=2858
*

*> > SUBFIND: root-task=2: number of subhalo candidates small enough to be
*

*> done with one cpu: 2858. (Largest size 234093)
*

*> > SUBFIND: root-task=0: total number of subhalo coll_candidates=3148
*

*> > SUBFIND: root-task=0: number of subhalo candidates small enough to be
*

*> done with one cpu: 3148. (Largest size 1116065)
*

*> > SUBFIND: root-task=2: bringing the independent ones back took 1.43275 sec
*

*> > SUBFIND: root-task=2: the collective unbinding of remaining halos took
*

*> 728.107 sec
*

*> > SUBFIND: root-task=2: found 2312 bound substructures in FoF group of
*

*> length 6001658
*

*> > SUBFIND: root-task=2: Collective processing of halo 1 took 1212.43
*

*> > SUBFIND: root-task=0: bringing the independent ones back took 1.69057 sec
*

*> > SUBFIND: root-task=0: the collective unbinding of remaining halos took
*

*> 893.829 sec
*

*> > SUBFIND: root-task=0: found 2642 bound substructures in FoF group of
*

*> length 6318694
*

*> > SUBFIND: subfind_hbt_single_group() processing for Ngroups=1 took
*

*> 1459.14 sec
*

*> > SUBFIND: root-task=0: Collective processing of halo 0 took 1473.2
*

*> > SUBFIND: Processing overall took (total time=1482.26 sec)
*

*> > ALLOCATE: Changing to MaxPart = 17425843
*

*> > ALLOCATE: Changing to MaxPartSph = 0
*

*> > SUBFIND: subfind_exchange() (for return to original CPU) took 26.0112
*

*> sec
*

*> > TREE: Full tree construction for all particles. (presently
*

*> allocated=3948.2 MB)
*

*> > GRAVTREE: Tree construction done. took 71.5354 sec
*

*> <numnodes>=1.90281e+06 NTopnodes=70545 NTopleaves=61727
*

*> tree-build-scalability=0.962969
*

*> > SUBFIND: SO iteration 1: need to repeat for 7848033 halo centers.
*

*> (took 0.309375 sec)
*

*> > SUBFIND: SO iteration 2: need to repeat for 7848033 halo centers.
*

*> (took 0.2519 sec)
*

*> > SUBFIND: SO iteration 3: need to repeat for 7848033 halo centers.
*

*> (took 0.268218 sec)
*

*> > SUBFIND: SO iteration 4: need to repeat for 7848033 halo centers.
*

*> (took 0.266999 sec)
*

*> > SUBFIND: SO iteration 5: need to repeat for 7848033 halo centers.
*

*> (took 0.288053 sec)
*

*> > SUBFIND: SO iteration 6: need to repeat for 7848033 halo centers.
*

*> (took 0.266823 sec)
*

*> > SUBFIND: SO iteration 7: need to repeat for 7848033 halo centers.
*

*> (took 0.264547 sec)
*

*> > SUBFIND: SO iteration 8: need to repeat for 7848033 halo centers.
*

*> (took 0.263144 sec)
*

*> > SUBFIND: SO iteration 9: need to repeat for 7848033 halo centers.
*

*> (took 0.256196 sec)
*

*> > SUBFIND: SO iteration 10: need to repeat for 7848033 halo centers.
*

*> (took 0.264536 sec)
*

*> > SUBFIND: SO iteration 11: need to repeat for 7848033 halo centers.
*

*> (took 0.257483 sec)
*

*> > SUBFIND: SO iteration 12: need to repeat for 7848033 halo centers.
*

*> (took 0.25124 sec)
*

*> > SUBFIND: SO iteration 13: need to repeat for 7848033 halo centers.
*

*> (took 0.261861 sec)
*

*> > SUBFIND: SO iteration 14: need to repeat for 7848033 halo centers.
*

*> (took 0.257919 sec)
*

*> > SUBFIND: SO iteration 15: need to repeat for 7848030 halo centers.
*

*> (took 0.268111 sec)
*

*> > SUBFIND: SO iteration 16: need to repeat for 38913 halo centers.
*

*> (took 0.259465 sec)
*

*> > SUBFIND: SO iteration 17: need to repeat for 37 halo centers.
*

*> (took 0.0242081 sec)
*

*> > SUBFIND: SO iteration 1: need to repeat for 7848033 halo centers.
*

*> (took 0.264944 sec)
*

*> > SUBFIND: SO iteration 2: need to repeat for 7848033 halo centers.
*

*> (took 0.237437 sec)
*

*> > SUBFIND: SO iteration 3: need to repeat for 7848033 halo centers.
*

*> (took 0.257997 sec)
*

*> > SUBFIND: SO iteration 4: need to repeat for 7848033 halo centers.
*

*> (took 0.263542 sec)
*

*> > SUBFIND: SO iteration 5: need to repeat for 7848033 halo centers.
*

*> (took 0.268112 sec)
*

*> > SUBFIND: SO iteration 6: need to repeat for 7848033 halo centers.
*

*> (took 0.254724 sec)
*

*> > SUBFIND: SO iteration 7: need to repeat for 7848033 halo centers.
*

*> (took 0.268252 sec)
*

*> > SUBFIND: SO iteration 8: need to repeat for 7848033 halo centers.
*

*> (took 0.25666 sec)
*

*> > SUBFIND: SO iteration 9: need to repeat for 7848033 halo centers.
*

*> (took 0.272972 sec)
*

*> > SUBFIND: SO iteration 10: need to repeat for 7848033 halo centers.
*

*> (took 0.251183 sec)
*

*> > SUBFIND: SO iteration 11: need to repeat for 7848033 halo centers.
*

*> (took 0.265267 sec)
*

*> > SUBFIND: SO iteration 12: need to repeat for 7848033 halo centers.
*

*> (took 0.257801 sec)
*

*> > SUBFIND: SO iteration 13: need to repeat for 7848033 halo centers.
*

*> (took 0.269823 sec)
*

*> > SUBFIND: SO iteration 14: need to repeat for 7848033 halo centers.
*

*> (took 0.265494 sec)
*

*> > SUBFIND: SO iteration 15: need to repeat for 7848030 halo centers.
*

*> (took 0.247782 sec)
*

*> > SUBFIND: SO iteration 16: need to repeat for 36428 halo centers.
*

*> (took 0.259644 sec)
*

*> > SUBFIND: SO iteration 17: need to repeat for 35 halo centers.
*

*> (took 0.019228 sec)
*

*> > SUBFIND: SO iteration 1: need to repeat for 7848033 halo centers.
*

*> (took 0.265704 sec)
*

*> > SUBFIND: SO iteration 2: need to repeat for 7848033 halo centers.
*

*> (took 0.229607 sec)
*

*> > SUBFIND: SO iteration 3: need to repeat for 7848033 halo centers.
*

*> (took 0.254473 sec)
*

*> > SUBFIND: SO iteration 4: need to repeat for 7848033 halo centers.
*

*> (took 0.25696 sec)
*

*> > SUBFIND: SO iteration 5: need to repeat for 7848033 halo centers.
*

*> (took 0.26164 sec)
*

*> > SUBFIND: SO iteration 6: need to repeat for 7848033 halo centers.
*

*> (took 0.25506 sec)
*

*> > SUBFIND: SO iteration 7: need to repeat for 7848033 halo centers.
*

*> (took 0.240955 sec)
*

*> > SUBFIND: SO iteration 8: need to repeat for 7848033 halo centers.
*

*> (took 0.260188 sec)
*

*> > SUBFIND: SO iteration 9: need to repeat for 7848033 halo centers.
*

*> (took 0.248877 sec)
*

*> > SUBFIND: SO iteration 10: need to repeat for 7848033 halo centers.
*

*> (took 0.256545 sec)
*

*> > SUBFIND: SO iteration 11: need to repeat for 7848033 halo centers.
*

*> (took 0.25118 sec)
*

*> > SUBFIND: SO iteration 12: need to repeat for 7848033 halo centers.
*

*> (took 0.25854 sec)
*

*> > SUBFIND: SO iteration 13: need to repeat for 7848033 halo centers.
*

*> (took 0.246834 sec)
*

*> > SUBFIND: SO iteration 14: need to repeat for 7848033 halo centers.
*

*> (took 0.27007 sec)
*

*> > SUBFIND: SO iteration 15: need to repeat for 7848033 halo centers.
*

*> (took 0.258597 sec)
*

*> > SUBFIND: SO iteration 16: need to repeat for 122350 halo centers.
*

*> (took 0.248988 sec)
*

*> > SUBFIND: SO iteration 17: need to repeat for 181 halo centers.
*

*> (took 0.0358 sec)
*

*> > SUBFIND: SO iteration 1: need to repeat for 7848033 halo centers.
*

*> (took 0.269485 sec)
*

*> > SUBFIND: SO iteration 2: need to repeat for 7848033 halo centers.
*

*> (took 0.228188 sec)
*

*> > SUBFIND: SO iteration 3: need to repeat for 7848033 halo centers.
*

*> (took 0.234185 sec)
*

*> > SUBFIND: SO iteration 4: need to repeat for 7848033 halo centers.
*

*> (took 0.229443 sec)
*

*> > SUBFIND: SO iteration 5: need to repeat for 7848033 halo centers.
*

*> (took 0.224041 sec)
*

*> > SUBFIND: SO iteration 6: need to repeat for 7848033 halo centers.
*

*> (took 0.232641 sec)
*

*> > SUBFIND: SO iteration 7: need to repeat for 7848033 halo centers.
*

*> (took 0.230443 sec)
*

*> > SUBFIND: SO iteration 8: need to repeat for 7848033 halo centers.
*

*> (took 0.219413 sec)
*

*> > SUBFIND: SO iteration 9: need to repeat for 7848033 halo centers.
*

*> (took 0.220803 sec)
*

*> > SUBFIND: SO iteration 10: need to repeat for 7848033 halo centers.
*

*> (took 0.237108 sec)
*

*> > SUBFIND: SO iteration 11: need to repeat for 7848033 halo centers.
*

*> (took 0.232947 sec)
*

*> > SUBFIND: SO iteration 12: need to repeat for 7848033 halo centers.
*

*> (took 0.240728 sec)
*

*> > SUBFIND: SO iteration 13: need to repeat for 7848033 halo centers.
*

*> (took 0.228163 sec)
*

*> > SUBFIND: SO iteration 14: need to repeat for 7848033 halo centers.
*

*> (took 0.221182 sec)
*

*> > SUBFIND: SO iteration 15: need to repeat for 7848033 halo centers.
*

*> (took 0.224345 sec)
*

*> > SUBFIND: SO iteration 16: need to repeat for 6272558 halo centers.
*

*> (took 0.226659 sec)
*

*> > SUBFIND: SO iteration 17: need to repeat for 46051 halo centers.
*

*> (took 0.188434 sec)
*

*> > SUBFIND: SO iteration 18: need to repeat for 52 halo centers.
*

*> (took 0.01652 sec)
*

*> > SUBFIND: determining spherical overdensity masses took 16.475 sec
*

*> > SUBFIND: assembled and ordered groups and subhalos (took 5.52871 sec)
*

*> > FOF/SUBFIND: writing group catalogue file:
*

*> './snaps//groups_026/fof_subhalo_tab_026.0' (file 1 of 32)
*

*> > FOF/SUBFIND: writing group catalogue block 0 (GroupLen)...
*

*> > FOF/SUBFIND: writing group catalogue block 1 (GroupMass)..
*

*> > FOF/SUBFIND: writing group catalogue block 2 (GroupPos)...
*

*> > FOF/SUBFIND: writing group catalogue block 3 (GroupVel)...
*

*> > FOF/SUBFIND: writing group catalogue block 4 (GroupLenType)...
*

*> > FOF/SUBFIND: writing group catalogue block 5 (GroupOffsetType)...
*

*> > FOF/SUBFIND: writing group catalogue block 6 (GroupMassType)...
*

*> > FOF/SUBFIND: writing group catalogue block 7 (GroupAscale)...
*

*> > FOF/SUBFIND: writing group catalogue block 9 (Group_M_Mean200)...
*

*> > FOF/SUBFIND: writing group catalogue block 10 (Group_R_Mean200)...
*

*> > FOF/SUBFIND: writing group catalogue block 11 (Group_M_Crit200)..
*

*> > FOF/SUBFIND: writing group catalogue block 12 (Group_R_Crit200)...
*

*> > FOF/SUBFIND: writing group catalogue block 13 (Group_M_Crit500)...
*

*> > FOF/SUBFIND: writing group catalogue block 14 (Group_R_Crit500)...
*

*> > FOF/SUBFIND: writing group catalogue block 15 (Group_M_TopHat200)...
*

*> > FOF/SUBFIND: writing group catalogue block 16 (Group_R_TopHat200)...
*

*> > FOF/SUBFIND: writing group catalogue block 17 (GroupNsubs)...
*

*> > FOF/SUBFIND: writing group catalogue block 18 (GroupFirstSub)...
*

*> > FOF/SUBFIND: writing group catalogue block 19 (SubhaloGroupNr)...
*

*> > FOF/SUBFIND: writing group catalogue block 20 (SubhaloRankInGr)...
*

*> > FOF/SUBFIND: writing group catalogue block 21 (SubhaloLen)...
*

*> > FOF/SUBFIND: writing group catalogue block 22 (SubhaloMass)...
*

*> > FOF/SUBFIND: writing group catalogue block 23 (SubhaloPos)...
*

*> > FOF/SUBFIND: writing group catalogue block 24 (SubhaloVel)...
*

*> > FOF/SUBFIND: writing group catalogue block 25 (SubhaloLenType)...
*

*> > FOF/SUBFIND: writing group catalogue block 26 (SubhaloOffsetType)...
*

*> > FOF/SUBFIND: writing group catalogue block 27 (SubhaloMassType)...
*

*> > FOF/SUBFIND: writing group catalogue block 28 (SubhaloCM)...
*

*> > FOF/SUBFIND: writing group catalogue block 29 (SubhaloSpin)...
*

*> > FOF/SUBFIND: writing group catalogue block 30 (SubhaloVelDisp)...
*

*> > FOF/SUBFIND: writing group catalogue block 31 (SubhaloVmax)...
*

*> > FOF/SUBFIND: writing group catalogue block 32 (SubhaloVmaxRad)...
*

*> > FOF/SUBFIND: writing group catalogue block 33 (SubhaloHalfmassRad)...
*

*> > FOF/SUBFIND: writing group catalogue block 34 (SubhaloHalfmassRadType)...
*

*> > FOF/SUBFIND: writing group catalogue block 35 (SubhaloIDMostbound)...
*

*> > FOF/SUBFIND: writing group catalogue block 36 (SubhaloParentRank)...
*

*> > FOF/SUBFIND: Group catalogues saved. took = 8.89541 sec, total size
*

*> 3525.03 MB, corresponds to effective I/O rate of 396.275 MB/sec
*

*> > SUBFIND: Subgroup catalogues saved. took = 9.7367 sec
*

*> > SUBFIND: Finished with SUBFIND. (total time=4092.76 sec)
*

*> > SUBFIND: Total number of subhalos with at least 32 particles: 9849779
*

*> > SUBFIND: Largest subhalo has 4419279 particles/cells.
*

*> > SUBFIND: Total number of particles/cells in subhalos: 2897277494
*

*> > ```
*

*> >
*

*> > Many thanks for any suggestions. Please let me know if you need any more
*

*> information.
*

*> >
*

*> > Best,
*

*> > Weiguang
*

*> >
*

*> > -------------------------------------------
*

*> > https://weiguangcui.github.io/
*

*> >
*

*> > -----------------------------------------------------------
*

*> >
*

*> > If you wish to unsubscribe from this mailing, send mail to
*

*> > minimalist_at_MPA-Garching.MPG.de with a subject of: unsubscribe
*

*> gadget-list
*

*> > A web-archive of this mailing list is available here:
*

*> > http://www.mpa-garching.mpg.de/gadget/gadget-list
*

*>
*

*>
*

*>
*

*>
*

*> -----------------------------------------------------------
*

*>
*

*> If you wish to unsubscribe from this mailing, send mail to
*

*> minimalist_at_MPA-Garching.MPG.de with a subject of: unsubscribe gadget-list
*

*> A web-archive of this mailing list is available here:
*

*> http://www.mpa-garching.mpg.de/gadget/gadget-list
*

*>
*

Received on 2021-08-11 10:46:53

Date: Wed, 11 Aug 2021 09:46:05 +0100

Hi Volker,

Yes, `ErrTolTheta` is set to 0.5 which is just slightly small than the

proposed value in the document. But I really doubt that is the cause.

I can understand that the two massive halos, even treated with separated

tasks, can be costly. I suppose this is seen from

"SUBFIND: Processing overall took (total time=1482.26 sec)"

But there is about 2.5ks time difference to this line:

"SUBFIND: Finished with SUBFIND. (total time=4092.76 sec)"

Which should be reposeful for this cost? I can not see any significant cost

listed between the two lines. Maybe I am wrong in interpreting this and

they are not in sequence.

Thanks.

Best,

Weiguang

-------------------------------------------

https://weiguangcui.github.io/

On Tue, Aug 10, 2021 at 6:50 PM Volker Springel <

vspringel_at_mpa-garching.mpg.de> wrote:

Received on 2021-08-11 10:46:53

*
This archive was generated by hypermail 2.3.0
: 2023-01-10 10:01:33 CET
*