Hi Julianne,
Getting extremely small timesteps is typically a physical problem of a setup, indepedent of machine/compiler/parallelization issues. In general, it is important to analyse why the code thinks such a short timestep is needed. In other words, which timestep criterion is giving you this short step? Why are the physical quantitites that enter this criterion take on extreme values? I'd recommend to set MinSizeTimestep to a large enough value such that the code stops well before it runs out of timebins at the lower end, then you can get access to some of this information.
In your concrete setup, several things look problematic to me. Your gravitational softening lengths are vanishingly small for your mass resolution. If you have 70 million particles in a 50 Mpc box, the mean particle separation is about 120 kpc. If all the DM particles have the same mass, you could at most go to a softening of ~1/50 of that, which is ~2.5 kpc. Yet, they all get assigned 0.08 kpc... If you have a zoom setup, you probably want to assign a softening based on the adaptive mass of the particles, but you currently don't. Note that assigning an inappropriately small softening to a heavy particle is easily producing huge accelerations, and thus tiny timesteps.
Another thing I noticed is that you set CritPhysDensity to a non-zero value. This not only overrides the computation of the star formation threshold of the subgrid model, the specific value you use is also extremely large. This will effectively prevent star formation, I think. As cooling is active however, you'll then end up with massive, cold clumps of gas that will become extremely dense. They will also be associated with tiny timesteps.
Best regards,
Volker
> On 25. Jul 2021, at 23:06, Goddard, Julianne K. <Julianne.Goddard_at_uky.edu> wrote:
>
> Hello all,
>
> I am running a cosmological zoom simulation with a 50Mpc periodic box including gas and dark matter from z=99 with about 70 million total particles. I have included the details of the latest version of this run from the log at the end of this email. I have tried several different configurations and settings within the code and with the cluster I am running on (number of nodes/processors), however I keep encountering the same error at some redshift between z=30 and z=9 depending on the configuration:
>
> "Code termination on task=1, function timebins_get_bin_and_do_validity_checks(), file src/time_integration/timestep.cc, line 508: time-step of integer size 1 not allowed"
>
> I have not been able to identify the source of this error, any guidance or help would be greatly appreciated.
>
> Thank you,
>
> Julianne
>
> Code was compiled with the following compiler and flags:
> mpicxx -std=c++11 -ggdb -O3 -march=native -Wall -Wno-format-security -I/home/jkgo232/hdf5/include -I/opt/ohpc/pub/libs/intel/ccs/gsl/2.5/include -I/home/jkgo232/fftw3/include -Igalaxy_zoom4/build -Isrc
>
>
> Code was compiled with the following settings:
> ADAPTIVE_HYDRO_SOFTENING
> ASMTH=1.25
> COOLING
> DOUBLEPRECISION=1
> DOUBLEPRECISION_FFTW
> EVALPOTENTIAL
> FMM
> FOF
> GADGET2_HEADER
> HIERARCHICAL_GRAVITY
> NSOFTCLASSES=6
> NTYPES=6
> OUTPUT_COOLHEAT
> OUTPUT_ENTROPY
> OUTPUT_POTENTIAL
> OUTPUT_PRESSURE
> PERIODIC
> PMGRID=128
> PRESSURE_ENTROPY_SPH
> SELFGRAVITY
> STARFORMATION
> SUBFIND
> TREEPM_NOTIMESPLIT
> TREE_NUM_BEFORE_NODESPLIT=8
> WENDLAND_BIAS_CORRECTION
> WENDLAND_C4_KERNEL
>
>
> Running on 46 MPI tasks.
>
>
> BEGRUN: Size of particle structure 144 [bytes]
> BEGRUN: Size of sph particle structure 288 [bytes]
> BEGRUN: Size of gravity tree node 104 [bytes]
> BEGRUN: Size of neighbour tree node 192 [bytes]
> BEGRUN: Size of subfind auxiliary data 104 [bytes]
>
>
> -------------------------------------------------------------------------------------------------------------------------
> AvailMem: Largest = 183846.24 Mb (on task= 0), Smallest = 183846.24 Mb (on task= 0), Average = 183846.24 Mb
> Total Mem: Largest = 191839.78 Mb (on task= 0), Smallest = 191839.78 Mb (on task= 0), Average = 191839.78 Mb
> Committed_AS: Largest = 7993.54 Mb (on task= 0), Smallest = 7993.54 Mb (on task= 0), Average = 7993.54 Mb
> SwapFree: Largest = 22792.30 Mb (on task= 0), Smallest = 22792.30 Mb (on task= 0), Average = 22792.30 Mb
> AllocMem: Largest = 7993.54 Mb (on task= 0), Smallest = 7993.54 Mb (on task= 0), Average = 7993.54 Mb
> avail /dev/shm: Largest = 95871.05 Mb (on task= 0), Smallest = 95871.05 Mb (on task= 0), Average = 95871.05 Mb
> -------------------------------------------------------------------------------------------------------------------------
> Task=0 has the maximum commited memory and is host: cascadeb002
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Obtaining parameters from file 'param.txt':
>
> InitCondFile ./ICs/ics_zoom_gadget_12.dat
> OutputDir ./gz2
> SnapshotFileBase snapshot
> OutputListFilename output_list1.txt
> ICFormat 1
> SnapFormat 3
> TimeLimitCPU 1.2096e+06
> CpuTimeBetRestartFile 86400
> MaxMemSize 2000
> TimeBegin 0.01
> TimeMax 1
> ComovingIntegrationOn 1
> Omega0 0.308
> OmegaLambda 0.692
> OmegaBaryon 0.0481
> HubbleParam 0.678
> BoxSize 50000
> Hubble 0.1
> OutputListOn 1
> TimeBetSnapshot 1.05
> TimeOfFirstSnapshot 0.01
> TimeBetStatistics 1.05
> InitCondFile ./ICs/ics_zoom_gadget_12.dat
> OutputDir ./gz2
> SnapshotFileBase snapshot
> OutputListFilename output_list1.txt
> ICFormat 1
> SnapFormat 3
> TimeLimitCPU 1.2096e+06
> CpuTimeBetRestartFile 86400
> MaxMemSize 2000
> TimeBegin 0.01
> TimeMax 1
> ComovingIntegrationOn 1
> Omega0 0.308
> OmegaLambda 0.692
> OmegaBaryon 0.0481
> HubbleParam 0.678
> BoxSize 50000
> Hubble 0.1
> OutputListOn 1
> TimeBetSnapshot 1.05
> TimeOfFirstSnapshot 0.01
> TimeBetStatistics 1.05
> NumFilesPerSnapshot 1
> MaxFilesWithConcurrentIO 1
> ErrTolIntAccuracy 0.012
> CourantFac 0.2
> MaxSizeTimestep 0.01
> MinSizeTimestep 1e-12
> TypeOfOpeningCriterion 1
> ErrTolTheta 0.4
> ErrTolThetaMax 1
> ErrTolForceAcc 0.02
> TopNodeFactor 3
> ActivePartFracForNewDomainDecomp 0.01
> ActivePartFracForPMinsteadOfEwald 0.1
> DesNumNgb 32
> MaxNumNgbDeviation 2
> UnitLength_in_cm 3.08568e+21
> UnitMass_in_g 1.989e+43
> UnitVelocity_in_cm_per_s 100000
> GravityConstantInternal 0
> GasSoftFactor 1.5
> MinimumComovingHydroSoftening 0.05
> AdaptiveHydroSofteningSpacing 1.05
> SofteningComovingClass0 0.05
> SofteningComovingClass1 0.08
> SofteningComovingClass2 0.08
> SofteningComovingClass3 0.08
> SofteningComovingClass4 0.05
> SofteningComovingClass5 1.2
> SofteningMaxPhysClass0 0.05
> SofteningMaxPhysClass1 0.08
>
> SofteningMaxPhysClass2 0.08
> SofteningMaxPhysClass3 0.08
> SofteningMaxPhysClass4 0.05
> SofteningMaxPhysClass5 1.2
> SofteningClassOfPartType0 0
> SofteningClassOfPartType1 1
> SofteningClassOfPartType2 2
> SofteningClassOfPartType3 3
> SofteningClassOfPartType4 4
> SofteningClassOfPartType5 5
> ArtBulkViscConst 1
> MinEgySpec 0.1
> InitGasTemp 275.5
> MaxSfrTimescale 4
> FactorSN 0.1
> FactorEVP 3000
> TempSupernova 3e+08
> TempClouds 1000
> CritOverDensity 57.7
> CritPhysDensity 4
> TreecoolFile TREECOOL
> DesLinkNgb 32
>
> MALLOC: Allocation of shared memory took 0.00267887 sec
>
> found 89 times in output-list.
> BEGRUN: Hubble (internal units) = 0.1
> BEGRUN: h = 0.678
> BEGRUN: G (internal units) = 43018.7
> BEGRUN: UnitMass_in_g = 1.989e+43
> BEGRUN: UnitLenth_in_cm = 3.08568e+21
> BEGRUN: UnitTime_in_s = 3.08568e+16
> BEGRUN: UnitVelocity_in_cm_per_s = 100000
> BEGRUN: UnitDensity_in_cgs = 6.76991e-22
> BEGRUN: UnitEnergy_in_cgs = 1.989e+53
>
> COOLING: read ionization table with 171 entries in file `TREECOOL'.
> COOLING: using 170 ionization table entries from file `TREECOOL'.
> EWALD: initialize Ewald correction...
>
> EWALD: reading Ewald tables from file `ewald_table_1-1-1_64-64-64_precision8-order3.dat'
> EWALD: Initialization of periodic boundaries finished.
>
> READIC: filenr=0, './ICs/ics_zoom_gadget_12.dat' contains:
> READIC: Type 0 (gas): 26556504 (tot= 26556504) masstab= 2.42811e-06
> READIC: Type 1: 26556504 (tot= 26556504) masstab= 1.31199e-05
> READIC: Type 2: 0 (tot= 0) masstab= 0
> READIC: Type 3: 0 (tot= 0) masstab= 0
> READIC: Type 4: 0 (tot= 0) masstab= 0
> READIC: Type 5: 18115397 (tot= 18115397) masstab= 0
>
> READIC: Reading file `./ICs/ics_zoom_gadget_12.dat' on task=0 and distribute it to 0 to 45.
> READIC: reading block 0 (Coordinates)...
> READIC: reading block 1 (Velocities)...
> READIC: reading block 2 (ParticleIDs)...
> READIC: reading block 3 (Masses)...
> READIC: reading block 4 (InternalEnergy)...
> READIC: reading done. Took 11.3321 sec, total size 2072.41 MB, corresponds to effective I/O rate of 182.88 MB/sec
>
> READIC: Total number of particles : 71228405
>
> INIT: Testing ID uniqueness...
> INIT: success. took=0.236911 sec
>
>
>
>
>
> -----------------------------------------------------------
>
> If you wish to unsubscribe from this mailing, send mail to
> minimalist_at_MPA-Garching.MPG.de with a subject of: unsubscribe gadget-list
> A web-archive of this mailing list is available here:
> http://www.mpa-garching.mpg.de/gadget/gadget-list
Received on 2021-07-27 10:34:08