Hi Weiguang,
No, the code has decided to treat the two biggest halos with 2 tasks each (because they don't fit on a single processor with their ~6 million particles), all the other tasks are used to process the 8913146 halos that are smaller. Only for halos that are treated with multiple tasks you see the "root-task" messages.
The gravitational unbinding of highly resolved halos can take quite a bit of time... it might be that you have set a quite small value of ErrTolTheta, which would slow it down unnecessarily.
Regards,
Volker
> On 7. Aug 2021, at 21:42, Weiguang Cui <cuiweiguang_at_gmail.com> wrote:
>
> Dear all,
>
> I understand that this simulation (2048^3 particles) is large to run, is it normal that the SUBFIND takes about 4ks to complete? It seems that there are only two tasks to run subfind. Are there some options in SUBFIND that I did set not correctly?
>
> ```
> SUBFIND: We now execute a parallel version of SUBFIND.
> SUBFIND: Previous subhalo catalogue had approximately a size 2.76992e+09, and the summed squared subhalo size was 1.16758e+14
> SUBFIND: Number of FOF halos treated with collective SubFind algorithm = 2
> SUBFIND: Number of processors used in different partitions for the collective SubFind code = 4
> SUBFIND: (The adopted size-limit for the collective algorithm was 5965232 particles, for threshold size factor 0.6)
> SUBFIND: The other 8913146 FOF halos are treated in parallel with serial code
> SUBFIND: subfind_distribute_groups() took 0.044987 sec
> SUBFIND: particle balance=1.23335
> SUBFIND: subfind_exchange() took 20.5998 sec
> SUBFIND: particle balance for processing=1
> SUBFIND: root-task=2: Collectively doing halo 1 of length 6001658 on 2 processors.
> SUBFIND: root-task=0: Collectively doing halo 0 of length 6318694 on 2 processors.
> SUBFIND: subdomain decomposition took 5.28067 sec
> SUBFIND: serial subfind subdomain decomposition took 4.50761 sec
> SUBFIND: root-task=2: total number of subhalo coll_candidates=2858
> SUBFIND: root-task=2: number of subhalo candidates small enough to be done with one cpu: 2858. (Largest size 234093)
> SUBFIND: root-task=0: total number of subhalo coll_candidates=3148
> SUBFIND: root-task=0: number of subhalo candidates small enough to be done with one cpu: 3148. (Largest size 1116065)
> SUBFIND: root-task=2: bringing the independent ones back took 1.43275 sec
> SUBFIND: root-task=2: the collective unbinding of remaining halos took 728.107 sec
> SUBFIND: root-task=2: found 2312 bound substructures in FoF group of length 6001658
> SUBFIND: root-task=2: Collective processing of halo 1 took 1212.43
> SUBFIND: root-task=0: bringing the independent ones back took 1.69057 sec
> SUBFIND: root-task=0: the collective unbinding of remaining halos took 893.829 sec
> SUBFIND: root-task=0: found 2642 bound substructures in FoF group of length 6318694
> SUBFIND: subfind_hbt_single_group() processing for Ngroups=1 took 1459.14 sec
> SUBFIND: root-task=0: Collective processing of halo 0 took 1473.2
> SUBFIND: Processing overall took (total time=1482.26 sec)
> ALLOCATE: Changing to MaxPart = 17425843
> ALLOCATE: Changing to MaxPartSph = 0
> SUBFIND: subfind_exchange() (for return to original CPU) took 26.0112 sec
> TREE: Full tree construction for all particles. (presently allocated=3948.2 MB)
> GRAVTREE: Tree construction done. took 71.5354 sec <numnodes>=1.90281e+06 NTopnodes=70545 NTopleaves=61727 tree-build-scalability=0.962969
> SUBFIND: SO iteration 1: need to repeat for 7848033 halo centers. (took 0.309375 sec)
> SUBFIND: SO iteration 2: need to repeat for 7848033 halo centers. (took 0.2519 sec)
> SUBFIND: SO iteration 3: need to repeat for 7848033 halo centers. (took 0.268218 sec)
> SUBFIND: SO iteration 4: need to repeat for 7848033 halo centers. (took 0.266999 sec)
> SUBFIND: SO iteration 5: need to repeat for 7848033 halo centers. (took 0.288053 sec)
> SUBFIND: SO iteration 6: need to repeat for 7848033 halo centers. (took 0.266823 sec)
> SUBFIND: SO iteration 7: need to repeat for 7848033 halo centers. (took 0.264547 sec)
> SUBFIND: SO iteration 8: need to repeat for 7848033 halo centers. (took 0.263144 sec)
> SUBFIND: SO iteration 9: need to repeat for 7848033 halo centers. (took 0.256196 sec)
> SUBFIND: SO iteration 10: need to repeat for 7848033 halo centers. (took 0.264536 sec)
> SUBFIND: SO iteration 11: need to repeat for 7848033 halo centers. (took 0.257483 sec)
> SUBFIND: SO iteration 12: need to repeat for 7848033 halo centers. (took 0.25124 sec)
> SUBFIND: SO iteration 13: need to repeat for 7848033 halo centers. (took 0.261861 sec)
> SUBFIND: SO iteration 14: need to repeat for 7848033 halo centers. (took 0.257919 sec)
> SUBFIND: SO iteration 15: need to repeat for 7848030 halo centers. (took 0.268111 sec)
> SUBFIND: SO iteration 16: need to repeat for 38913 halo centers. (took 0.259465 sec)
> SUBFIND: SO iteration 17: need to repeat for 37 halo centers. (took 0.0242081 sec)
> SUBFIND: SO iteration 1: need to repeat for 7848033 halo centers. (took 0.264944 sec)
> SUBFIND: SO iteration 2: need to repeat for 7848033 halo centers. (took 0.237437 sec)
> SUBFIND: SO iteration 3: need to repeat for 7848033 halo centers. (took 0.257997 sec)
> SUBFIND: SO iteration 4: need to repeat for 7848033 halo centers. (took 0.263542 sec)
> SUBFIND: SO iteration 5: need to repeat for 7848033 halo centers. (took 0.268112 sec)
> SUBFIND: SO iteration 6: need to repeat for 7848033 halo centers. (took 0.254724 sec)
> SUBFIND: SO iteration 7: need to repeat for 7848033 halo centers. (took 0.268252 sec)
> SUBFIND: SO iteration 8: need to repeat for 7848033 halo centers. (took 0.25666 sec)
> SUBFIND: SO iteration 9: need to repeat for 7848033 halo centers. (took 0.272972 sec)
> SUBFIND: SO iteration 10: need to repeat for 7848033 halo centers. (took 0.251183 sec)
> SUBFIND: SO iteration 11: need to repeat for 7848033 halo centers. (took 0.265267 sec)
> SUBFIND: SO iteration 12: need to repeat for 7848033 halo centers. (took 0.257801 sec)
> SUBFIND: SO iteration 13: need to repeat for 7848033 halo centers. (took 0.269823 sec)
> SUBFIND: SO iteration 14: need to repeat for 7848033 halo centers. (took 0.265494 sec)
> SUBFIND: SO iteration 15: need to repeat for 7848030 halo centers. (took 0.247782 sec)
> SUBFIND: SO iteration 16: need to repeat for 36428 halo centers. (took 0.259644 sec)
> SUBFIND: SO iteration 17: need to repeat for 35 halo centers. (took 0.019228 sec)
> SUBFIND: SO iteration 1: need to repeat for 7848033 halo centers. (took 0.265704 sec)
> SUBFIND: SO iteration 2: need to repeat for 7848033 halo centers. (took 0.229607 sec)
> SUBFIND: SO iteration 3: need to repeat for 7848033 halo centers. (took 0.254473 sec)
> SUBFIND: SO iteration 4: need to repeat for 7848033 halo centers. (took 0.25696 sec)
> SUBFIND: SO iteration 5: need to repeat for 7848033 halo centers. (took 0.26164 sec)
> SUBFIND: SO iteration 6: need to repeat for 7848033 halo centers. (took 0.25506 sec)
> SUBFIND: SO iteration 7: need to repeat for 7848033 halo centers. (took 0.240955 sec)
> SUBFIND: SO iteration 8: need to repeat for 7848033 halo centers. (took 0.260188 sec)
> SUBFIND: SO iteration 9: need to repeat for 7848033 halo centers. (took 0.248877 sec)
> SUBFIND: SO iteration 10: need to repeat for 7848033 halo centers. (took 0.256545 sec)
> SUBFIND: SO iteration 11: need to repeat for 7848033 halo centers. (took 0.25118 sec)
> SUBFIND: SO iteration 12: need to repeat for 7848033 halo centers. (took 0.25854 sec)
> SUBFIND: SO iteration 13: need to repeat for 7848033 halo centers. (took 0.246834 sec)
> SUBFIND: SO iteration 14: need to repeat for 7848033 halo centers. (took 0.27007 sec)
> SUBFIND: SO iteration 15: need to repeat for 7848033 halo centers. (took 0.258597 sec)
> SUBFIND: SO iteration 16: need to repeat for 122350 halo centers. (took 0.248988 sec)
> SUBFIND: SO iteration 17: need to repeat for 181 halo centers. (took 0.0358 sec)
> SUBFIND: SO iteration 1: need to repeat for 7848033 halo centers. (took 0.269485 sec)
> SUBFIND: SO iteration 2: need to repeat for 7848033 halo centers. (took 0.228188 sec)
> SUBFIND: SO iteration 3: need to repeat for 7848033 halo centers. (took 0.234185 sec)
> SUBFIND: SO iteration 4: need to repeat for 7848033 halo centers. (took 0.229443 sec)
> SUBFIND: SO iteration 5: need to repeat for 7848033 halo centers. (took 0.224041 sec)
> SUBFIND: SO iteration 6: need to repeat for 7848033 halo centers. (took 0.232641 sec)
> SUBFIND: SO iteration 7: need to repeat for 7848033 halo centers. (took 0.230443 sec)
> SUBFIND: SO iteration 8: need to repeat for 7848033 halo centers. (took 0.219413 sec)
> SUBFIND: SO iteration 9: need to repeat for 7848033 halo centers. (took 0.220803 sec)
> SUBFIND: SO iteration 10: need to repeat for 7848033 halo centers. (took 0.237108 sec)
> SUBFIND: SO iteration 11: need to repeat for 7848033 halo centers. (took 0.232947 sec)
> SUBFIND: SO iteration 12: need to repeat for 7848033 halo centers. (took 0.240728 sec)
> SUBFIND: SO iteration 13: need to repeat for 7848033 halo centers. (took 0.228163 sec)
> SUBFIND: SO iteration 14: need to repeat for 7848033 halo centers. (took 0.221182 sec)
> SUBFIND: SO iteration 15: need to repeat for 7848033 halo centers. (took 0.224345 sec)
> SUBFIND: SO iteration 16: need to repeat for 6272558 halo centers. (took 0.226659 sec)
> SUBFIND: SO iteration 17: need to repeat for 46051 halo centers. (took 0.188434 sec)
> SUBFIND: SO iteration 18: need to repeat for 52 halo centers. (took 0.01652 sec)
> SUBFIND: determining spherical overdensity masses took 16.475 sec
> SUBFIND: assembled and ordered groups and subhalos (took 5.52871 sec)
> FOF/SUBFIND: writing group catalogue file: './snaps//groups_026/fof_subhalo_tab_026.0' (file 1 of 32)
> FOF/SUBFIND: writing group catalogue block 0 (GroupLen)...
> FOF/SUBFIND: writing group catalogue block 1 (GroupMass)..
> FOF/SUBFIND: writing group catalogue block 2 (GroupPos)...
> FOF/SUBFIND: writing group catalogue block 3 (GroupVel)...
> FOF/SUBFIND: writing group catalogue block 4 (GroupLenType)...
> FOF/SUBFIND: writing group catalogue block 5 (GroupOffsetType)...
> FOF/SUBFIND: writing group catalogue block 6 (GroupMassType)...
> FOF/SUBFIND: writing group catalogue block 7 (GroupAscale)...
> FOF/SUBFIND: writing group catalogue block 9 (Group_M_Mean200)...
> FOF/SUBFIND: writing group catalogue block 10 (Group_R_Mean200)...
> FOF/SUBFIND: writing group catalogue block 11 (Group_M_Crit200)..
> FOF/SUBFIND: writing group catalogue block 12 (Group_R_Crit200)...
> FOF/SUBFIND: writing group catalogue block 13 (Group_M_Crit500)...
> FOF/SUBFIND: writing group catalogue block 14 (Group_R_Crit500)...
> FOF/SUBFIND: writing group catalogue block 15 (Group_M_TopHat200)...
> FOF/SUBFIND: writing group catalogue block 16 (Group_R_TopHat200)...
> FOF/SUBFIND: writing group catalogue block 17 (GroupNsubs)...
> FOF/SUBFIND: writing group catalogue block 18 (GroupFirstSub)...
> FOF/SUBFIND: writing group catalogue block 19 (SubhaloGroupNr)...
> FOF/SUBFIND: writing group catalogue block 20 (SubhaloRankInGr)...
> FOF/SUBFIND: writing group catalogue block 21 (SubhaloLen)...
> FOF/SUBFIND: writing group catalogue block 22 (SubhaloMass)...
> FOF/SUBFIND: writing group catalogue block 23 (SubhaloPos)...
> FOF/SUBFIND: writing group catalogue block 24 (SubhaloVel)...
> FOF/SUBFIND: writing group catalogue block 25 (SubhaloLenType)...
> FOF/SUBFIND: writing group catalogue block 26 (SubhaloOffsetType)...
> FOF/SUBFIND: writing group catalogue block 27 (SubhaloMassType)...
> FOF/SUBFIND: writing group catalogue block 28 (SubhaloCM)...
> FOF/SUBFIND: writing group catalogue block 29 (SubhaloSpin)...
> FOF/SUBFIND: writing group catalogue block 30 (SubhaloVelDisp)...
> FOF/SUBFIND: writing group catalogue block 31 (SubhaloVmax)...
> FOF/SUBFIND: writing group catalogue block 32 (SubhaloVmaxRad)...
> FOF/SUBFIND: writing group catalogue block 33 (SubhaloHalfmassRad)...
> FOF/SUBFIND: writing group catalogue block 34 (SubhaloHalfmassRadType)...
> FOF/SUBFIND: writing group catalogue block 35 (SubhaloIDMostbound)...
> FOF/SUBFIND: writing group catalogue block 36 (SubhaloParentRank)...
> FOF/SUBFIND: Group catalogues saved. took = 8.89541 sec, total size 3525.03 MB, corresponds to effective I/O rate of 396.275 MB/sec
> SUBFIND: Subgroup catalogues saved. took = 9.7367 sec
> SUBFIND: Finished with SUBFIND. (total time=4092.76 sec)
> SUBFIND: Total number of subhalos with at least 32 particles: 9849779
> SUBFIND: Largest subhalo has 4419279 particles/cells.
> SUBFIND: Total number of particles/cells in subhalos: 2897277494
> ```
>
> Many thanks for any suggestions. Please let me know if you need any more information.
>
> Best,
> Weiguang
>
> -------------------------------------------
> https://weiguangcui.github.io/
>
> -----------------------------------------------------------
>
> If you wish to unsubscribe from this mailing, send mail to
> minimalist_at_MPA-Garching.MPG.de with a subject of: unsubscribe gadget-list
> A web-archive of this mailing list is available here:
> http://www.mpa-garching.mpg.de/gadget/gadget-list
Received on 2021-08-10 19:49:14