Shared memory islands host a minimum of 28 and a maximum of 28 MPI ranks. We shall use 48 MPI ranks in total for assisting one-sided communication (2 per shared memory node). ___ __ ____ ___ ____ ____ __ / __) /__\ ( _ \ / __)( ___)(_ _)___ /. | ( (_-. /(__)\ )(_) )( (_-. )__) )( (___)(_ _) \___/(__)(__)(____/ \___/(____) (__) (_) This is Gadget, version 4.0. Git commit b4bb065ce3dec478d2a2d7101cefc5f5faade084, Wed Dec 23 17:05:02 2020 +0100 Code was compiled with the following compiler and flags: mpiicpc -std=c++11 -ggdb -O0 -march=native -Wall -Wno-format-security -I/sc/home/ken.osato/usr/include -I/sc/home/ken.osato/usr/include -I/sc/home/ken.osato/usr/include -Ibuild -Isrc Code was compiled with the following settings: CREATE_GRID DEBUG DOUBLEPRECISION=2 FOF IDS_64BIT LEAN NGENIC=2048 NGENIC_2LPT NSOFTCLASSES=1 NTYPES=2 NUMBER_OF_MPI_LISTENERS_PER_NODE=2 PERIODIC PMGRID=2048 POSITIONS_IN_32BIT POWERSPEC_ON_OUTPUT PRESERVE_SHMEM_BINARY_INVARIANCE RANDOMIZE_DOMAINCENTER SELFGRAVITY SUBFIND SUBFIND_STORE_LOCAL_DENSITY TREEPM_NOTIMESPLIT Running on 1296 MPI tasks. BEGRUN: Size of particle structure 64 [bytes] BEGRUN: Size of sph particle structure 96 [bytes] BEGRUN: Size of gravity tree node 72 [bytes] BEGRUN: Size of neighbour tree node 112 [bytes] BEGRUN: Size of subfind auxiliary data 88 [bytes] ------------------------------------------------------------------------------------------------------------------------- AvailMem: Largest = 1539710.20 Mb (on task= 378), Smallest = 1538600.80 Mb (on task= 0), Average = 1539031.53 Mb Total Mem: Largest = 1546107.02 Mb (on task= 486), Smallest = 1546106.68 Mb (on task= 0), Average = 1546106.69 Mb Committed_AS: Largest = 7505.87 Mb (on task= 0), Smallest = 6396.48 Mb (on task= 378), Average = 7075.16 Mb SwapTotal: Largest = 8191.00 Mb (on task= 0), Smallest = 8191.00 Mb (on task= 0), Average = 8191.00 Mb SwapFree: Largest = 8190.75 Mb (on task= 378), Smallest = 7848.32 Mb (on task=1080), Average = 8148.01 Mb AllocMem: Largest = 7505.87 Mb (on task= 0), Smallest = 6396.48 Mb (on task= 378), Average = 7075.16 Mb avail /dev/shm: Largest = 773048.10 Mb (on task= 810), Smallest = 772977.14 Mb (on task= 191), Average = 773026.41 Mb ------------------------------------------------------------------------------------------------------------------------- Task=0 has the maximum commited memory and is host: cn0080 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Obtaining parameters from file 'param.txt': InitCondFile ./dummy.dat OutputDir ./output SnapshotFileBase snapshot OutputListFilename outputs.txt ICFormat 1 SnapFormat 3 TimeLimitCPU 86400 CpuTimeBetRestartFile 7200 MaxMemSize 13000 TimeBegin 0.015625 TimeMax 1 ComovingIntegrationOn 1 Omega0 0.3089 OmegaLambda 0.6911 OmegaBaryon 0.0486 HubbleParam 0.6774 Hubble 100 BoxSize 2000 OutputListOn 1 TimeBetSnapshot 0 TimeOfFirstSnapshot 0 TimeBetStatistics 0.01 NumFilesPerSnapshot 256 MaxFilesWithConcurrentIO 64 ErrTolIntAccuracy 0.01 CourantFac 0.3 MaxSizeTimestep 0.03 MinSizeTimestep 0 TypeOfOpeningCriterion 1 ErrTolTheta 0.75 ErrTolThetaMax 1 ErrTolForceAcc 0.002 TopNodeFactor 3 ActivePartFracForNewDomainDecomp 0.01 ActivePartFracForPMinsteadOfEwald 0.05 DesNumNgb 64 MaxNumNgbDeviation 1 DesLinkNgb 20 UnitLength_in_cm 3.08568e+24 UnitMass_in_g 1.989e+43 UnitVelocity_in_cm_per_s 100000 GravityConstantInternal 0 SofteningComovingClass0 0.05 SofteningMaxPhysClass0 0.05 SofteningClassOfPartType0 0 SofteningClassOfPartType1 0 ArtBulkViscConst 1 MinEgySpec 0 InitGasTemp 0 NSample 2048 GridSize 2048 Seed 181170 SphereMode 1 PowerSpectrumType 2 ReNormalizeInputSpectrum 1 PrimordialIndex 0.9667 ShapeGamma 0.21 Sigma8 0.8159 PowerSpectrumFile TNG_powerspec.dat InputSpectrum_UnitLength_in_cm 3.08568e+24 MALLOC: Allocation of shared memory took 94.0149 sec found 6 times in output-list. BEGRUN: Hubble (internal units) = 100 BEGRUN: h = 0.6774 BEGRUN: G (internal units) = 43.0187 BEGRUN: UnitMass_in_g = 1.989e+43 BEGRUN: UnitLenth_in_cm = 3.08568e+24 BEGRUN: UnitTime_in_s = 3.08568e+19 BEGRUN: UnitVelocity_in_cm_per_s = 100000 BEGRUN: UnitDensity_in_cgs = 6.76991e-31 BEGRUN: UnitEnergy_in_cgs = 1.989e+53 EWALD: initialize Ewald correction... EWALD: reading Ewald tables from file `ewald_table_1-1-1_64-64-64_precision8-order3.dat' EWALD: Initialization of periodic boundaries finished. NGENIC: generated grid of size 2048 NGENIC: computing displacement fields... NGENIC: vel_prefac1= 3557.04 hubble_a=28456.5 fom1=0.999995 NGENIC: vel_prefac2= 7114.08 hubble_a=28456.5 fom2=1.99999 found 139 rows in input spectrum table Normalization of spectrum in file: Sigma8 = 13.251 Normalization adjusted to Sigma8=0.8159 (Normfac=0.00379118) NGENIC: Dplus=50.1933 NGENIC_2LPT: Computing secondary source term, derivatices 0 0 NGENIC: setting up modes in kspace... NGENIC_2LPT: Computing secondary source term, derivatices 1 1 NGENIC: setting up modes in kspace... NGENIC_2LPT: Computing secondary source term, derivatices 2 2 NGENIC: setting up modes in kspace... NGENIC_2LPT: Computing secondary source term, derivatices 0 1 NGENIC: setting up modes in kspace... NGENIC_2LPT: Computing secondary source term, derivatices 0 2 NGENIC: setting up modes in kspace... NGENIC_2LPT: Computing secondary source term, derivatices 1 2 NGENIC: setting up modes in kspace... NGENIC_2LPT: Secondary source term computed in real space NGENIC_2LPT: Done transforming it to k-space NGENIC_2LPT: Obtaining second order displacements for axes=0 NGENIC_2LPT: Obtaining second order displacements for axes=1 NGENIC_2LPT: Obtaining second order displacements for axes=2 NGENIC_2LPT: Obtaining Zeldovich displacements for axes=0 NGENIC: setting up modes in kspace... NGENIC_2LPT: Obtaining Zeldovich displacements for axes=1 NGENIC: setting up modes in kspace... NGENIC_2LPT: Obtaining Zeldovich displacements for axes=2 NGENIC: setting up modes in kspace... NGENIC: Maximum displacement: 0.716217, in units of the part-spacing= 0.733407 NGENIC: Maximum velocity component: 2546.11 INIT: Testing ID uniqueness... INIT: success. took=4.6695 sec DOMAIN: Begin domain decomposition (sync-point 0). DOMAIN: New shift vector determined (-660.76 536.279 189.617) DOMAIN: Sum=2 TotalCost=2 NumTimeBinsToBeBalanced=1 MultipleDomains=2 DOMAIN: Increasing TopNodeAllocFactor=0.08 new value=0.104 DOMAIN: Increasing TopNodeAllocFactor=0.104 new value=0.1352 DOMAIN: Increasing TopNodeAllocFactor=0.1352 new value=0.17576 DOMAIN: Increasing TopNodeAllocFactor=0.17576 new value=0.228488 DOMAIN: Increasing TopNodeAllocFactor=0.228488 new value=0.297034 DOMAIN: Increasing TopNodeAllocFactor=0.297034 new value=0.386145 DOMAIN: Increasing TopNodeAllocFactor=0.386145 new value=0.501988 DOMAIN: Increasing TopNodeAllocFactor=0.501988 new value=0.652585 DOMAIN: Increasing TopNodeAllocFactor=0.652585 new value=0.84836 DOMAIN: Increasing TopNodeAllocFactor=0.84836 new value=1.10287 DOMAIN: Increasing TopNodeAllocFactor=1.10287 new value=1.43373 DOMAIN: Increasing TopNodeAllocFactor=1.43373 new value=1.86385 DOMAIN: Increasing TopNodeAllocFactor=1.86385 new value=2.423 DOMAIN: Increasing TopNodeAllocFactor=2.423 new value=3.1499 DOMAIN: Increasing TopNodeAllocFactor=3.1499 new value=4.09487 DOMAIN: Increasing TopNodeAllocFactor=4.09487 new value=5.32333 DOMAIN: NTopleaves=32768, determination of top-level tree involved 5 iterations and took 9.4247 sec DOMAIN: we are going to try at most 497 different settings for combining the domains on tasks=1296, nnodes=24 DOMAIN: total_cost=2 total_load=1 DOMAIN: best solution found after 1 iterations by task=34 for nextra=61, reaching maximum imbalance of 1.03397|1.0341 DOMAIN: combining multiple-domains took 0.485401 sec DOMAIN: exchange of 8589934592 particles DOMAIN: particle exchange done. (took 5.71959 sec) DOMAIN: domain decomposition done. (took in total 16.0766 sec) PEANO: Begin Peano-Hilbert order... PEANO: done, took 1.89233 sec. SNAPSHOT: Setting next time for snapshot file to Time_next= 0.251357 (DumpFlag=1) Sync-Point 0, Time: 0.015625, Redshift: 63, Systemstep: 0, Dloga: 0, Nsync-grv: 8589934592, Nsync-hyd: 0 DOMAIN: Begin domain decomposition (sync-point 0). DOMAIN: New shift vector determined (625.243 685.845 997.192) DOMAIN: Sum=2 TotalCost=2 NumTimeBinsToBeBalanced=1 MultipleDomains=2 DOMAIN: NTopleaves=32768, determination of top-level tree involved 5 iterations and took 6.45136 sec DOMAIN: we are going to try at most 497 different settings for combining the domains on tasks=1296, nnodes=24 DOMAIN: total_cost=2 total_load=1 DOMAIN: best solution found after 1 iterations by task=32 for nextra=49, reaching maximum imbalance of 1.03176|1.03177 DOMAIN: combining multiple-domains took 0.435799 sec DOMAIN: exchange of 8589934592 particles DOMAIN: particle exchange done. (took 5.55658 sec) DOMAIN: domain decomposition done. (took in total 12.908 sec) PEANO: Begin Peano-Hilbert order... PEANO: done, took 1.83768 sec. ACCEL: Start tree gravity force computation... (8589934592 particles) TREEPM: Starting PM part of force calculation. (timebin=0) PM-PERIODIC: Starting periodic PM calculation. (Rcut=8.54492) presently allocated=801.396 MB PM-PERIODIC: done. (took 7.8307 seconds) TREEPM: Finished PM part of force calculation. TREE: Full tree construction for all particles. (presently allocated=942.289 MB) GRAVTREE: Tree construction done. took 21.2147 sec =984281 NTopnodes=37449 NTopleaves=32768 tree-build-scalability=0.961982 GRAVTREE: Begin tree force. timebin=0 (presently allocated=927.275 MB) GRAVTREE: tree-forces are calculated, with 22 cycles took 562.677 sec GRAVTREE: tree-force is done. GRAVTREE/FMM: Setting OldAcc! TREEPM: Starting PM part of force calculation. (timebin=0) PM-PERIODIC: Starting periodic PM calculation. (Rcut=8.54492) presently allocated=801.396 MB PM-PERIODIC: done. (took 7.93629 seconds) TREEPM: Finished PM part of force calculation. TREE: Full tree construction for all particles. (presently allocated=874.976 MB) GRAVTREE: Tree construction done. took 10.8666 sec =984281 NTopnodes=37449 NTopleaves=32768 tree-build-scalability=0.961982 GRAVTREE: Begin tree force. timebin=0 (presently allocated=927.275 MB) GRAVTREE: tree-forces are calculated, with 126 cycles took 4678.53 sec GRAVTREE: tree-force is done. ACCEL: tree force computation done. KICKS: 2nd gravity for highest active timebin=0: particles 8589934592 GRAVTREE/FMM: Setting OldAcc! KICKS: 1st gravity for highest active timebin=0: particles 8589934592 DOMAIN: Begin domain decomposition (sync-point 0). DOMAIN: New shift vector determined (-564.688 -670.297 917.116) DOMAIN: Sum=2 TotalCost=2 NumTimeBinsToBeBalanced=1 MultipleDomains=2 DOMAIN: NTopleaves=32768, determination of top-level tree involved 5 iterations and took 6.63602 sec DOMAIN: we are going to try at most 497 different settings for combining the domains on tasks=1296, nnodes=24 DOMAIN: total_cost=2 total_load=1 DOMAIN: best solution found after 1 iterations by task=85 for nextra=12, reaching maximum imbalance of 1.06318|1.06426 DOMAIN: combining multiple-domains took 0.525916 sec DOMAIN: exchange of 8589934592 particles DOMAIN: particle exchange done. (took 5.71438 sec) DOMAIN: domain decomposition done. (took in total 13.2942 sec) PEANO: Begin Peano-Hilbert order... PEANO: done, took 1.79417 sec. /* omitted */ Sync-Point 1478, Time: 0.382304, Redshift: 1.61572, Systemstep: 0.000194037, Dloga: 0.000507676, Nsync-grv: 11830954, Nsync-hyd: 0 ACCEL: Start tree gravity force computation... (11830954 particles) TREE: Full tree construction for all particles. (presently allocated=899.754 MB) GRAVTREE: Tree construction done. took 13.5731 sec =1.25328e+06 NTopnodes=47425 NTopleaves=41497 tree-build-scalability=0.962188 GRAVTREE: Begin tree force. timebin=17 (presently allocated=958.225 MB) GRAVTREE: tree-forces are calculated, with 14 cycles took 11.6152 sec GRAVTREE: tree-force is done. ACCEL: tree force computation done. KICKS: 2nd gravity for highest active timebin=17: particles 11830954 GRAVTREE/FMM: Setting OldAcc! KICKS: 1st gravity for highest active timebin=17: particles 11830954 Sync-Point 1479, Time: 0.382498, Redshift: 1.61439, Systemstep: 0.000194136, Dloga: 0.000507676, Nsync-grv: 968, Nsync-hyd: 0 ACCEL: Start tree gravity force computation... (968 particles) TREE: Full tree construction for all particles. (presently allocated=899.754 MB) GRAVTREE: Tree construction done. took 13.6165 sec =1.25339e+06 NTopnodes=47425 NTopleaves=41497 tree-build-scalability=0.962192 GRAVTREE: Begin tree force. timebin=16 (presently allocated=958.233 MB)