Re: Not enough memory

From: Volker Springel <vspringel_at_MPA-Garching.MPG.DE>
Date: Fri, 12 Feb 2021 10:29:30 +0100

Hi Hassan,

Further to Tiago's message, your setting of

MaxMemSize 100

in the parameterfile allows the code to use at most 100 MB. Without increasing this sufficiently, the code cannot allocate ~685 MB... (independent of what physical memory your machine has)

Regards,
Volker


> On 12. Feb 2021, at 10:07, Tiago Castro <tiagobscastro_at_gmail.com> wrote:
>
> Dear Hassan,
>
> I believe you face the same problem that I did some months ago, your laptop shmem is apparently set to half the total memory of your machine. Check if this thread might help you: https://wwwmpa.mpa-garching.mpg.de/gadget/gadget-list/0804.html
>
> Regards,
> Tiago Castro Post Doc, Department of Physics / UNITS / OATS
> Phone: (+39 040 3199 120)
> Mobile: (+39 388 794 1562)
> Email: tiagobscastro_at_gmail.com
> Website: tiagobscastro.com
> Skype: tiagobscastro
> Address: Osservatorio Astronomico di Trieste / Villa Bazzoni
> Via Bazzoni, 2, 34143 Trieste TS
>
>
>
>
> Em qui., 11 de fev. de 2021 às 20:00, Hassan Fathivavsari <h.fathie_at_gmail.com> escreveu:
> Dear all,
>
> I am a new user of GADGET.
> I have created an initial condition file using MUSIC. But when I execute the GADGET
> code (mpirun -np 6 ./Gadget4 param.txt), I got the following error:
>
> ------------------------
> Code termination on task=5, function mymalloc_movable_fullinfo(), file src/data/mymalloc.cc, line 326:
> Not enough memory in mymalloc_fullinfo() to allocate 685.336 MB for variable 'rhogrid' at pmforce_zoom_optimized_prepare_density()/src/pm/pm_periodic.cc/line 378 (FreeBytes=97.1207 MB).
> -------------------------
>
> But I have enough memory on my Laptop. I have also pasted below the full message
> printed out in the Terminal:
> I would be very grateful if some one could help me in this issue.
>
> With best regards,
> Hassan
>
> ===================== Full Terminal Message =========================
>
> pc_at_ubuntu:~/Desktop/SPH/gadget4-master/IC_from_MUSIC$ mpirun -np 6 ./Gadget4 param.txt
> Shared memory islands host a minimum of 6 and a maximum of 6 MPI ranks.
>
> ___ __ ____ ___ ____ ____ __
> / __) /__\ ( _ \ / __)( ___)(_ _)___ /. |
> ( (_-. /(__)\ )(_) )( (_-. )__) )( (___)(_ _)
> \___/(__)(__)(____/ \___/(____) (__) (_)
>
> This is Gadget, version 4.0.
> Git commit unknown, unknown
>
> Code was compiled with the following compiler and flags:
> mpicxx -std=c++11 -O3 -Wall -I/hdf5/build/include -I/usr/local/include -I/usr/local/include -I./IC_from_MUSIC//build -Isrc
>
>
> Code was compiled with the following settings:
> EVALPOTENTIAL
> FOF
> FOF_GROUP_MIN_LEN=32
> FOF_PRIMARY_LINK_TYPES=2
> GADGET2_HEADER
> INDIVIDUAL_GRAVITY_SOFTENING=4+8+16+32
> NSOFTCLASSES=4
> PERIODIC
> PMGRID=1024
> PM_ZOOM_OPTIMIZED
> POWERSPEC_ON_OUTPUT
> SELFGRAVITY
> SUBFIND
>
>
> Running on 6 MPI tasks.
>
>
> BEGRUN: Size of particle structure 84 [bytes]
> BEGRUN: Size of sph particle structure 96 [bytes]
> BEGRUN: Size of gravity tree node 72 [bytes]
> BEGRUN: Size of neighbour tree node 112 [bytes]
> BEGRUN: Size of subfind auxiliary data 80 [bytes]
>
>
> -------------------------------------------------------------------------------------------------------------------------
> AvailMem: Largest = 2795.93 Mb (on task= 0), Smallest = 2795.93 Mb (on task= 0), Average = 2795.93 Mb
> Total Mem: Largest = 6267.95 Mb (on task= 0), Smallest = 6267.95 Mb (on task= 0), Average = 6267.95 Mb
> Committed_AS: Largest = 3472.02 Mb (on task= 0), Smallest = 3472.02 Mb (on task= 0), Average = 3472.02 Mb
> SwapTotal: Largest = 2048.00 Mb (on task= 0), Smallest = 2048.00 Mb (on task= 0), Average = 2048.00 Mb
> SwapFree: Largest = 2048.00 Mb (on task= 0), Smallest = 2048.00 Mb (on task= 0), Average = 2048.00 Mb
> AllocMem: Largest = 3472.02 Mb (on task= 0), Smallest = 3472.02 Mb (on task= 0), Average = 3472.02 Mb
> avail /dev/shm: Largest = 3132.36 Mb (on task= 0), Smallest = 3132.36 Mb (on task= 0), Average = 3132.36 Mb
> -------------------------------------------------------------------------------------------------------------------------
> Task=0 has the maximum commited memory and is host: ubuntu
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Obtaining parameters from file 'param.txt':
>
> InitCondFile ./ics_gadget.dat
> OutputDir ./output
> SnapshotFileBase snapshot
> OutputListFilename ./ExpansionList_16
> ICFormat 1
> SnapFormat 3
> TimeLimitCPU 86400
> CpuTimeBetRestartFile 7200
> MaxMemSize 100
> TimeBegin 0.0078125
> TimeMax 1
> ComovingIntegrationOn 1
> Omega0 2.20836e-06
> OmegaLambda 0.75
> OmegaBaryon 0.04
> HubbleParam 0.73
> BoxSize 100
> Hubble 100
> OutputListOn 1
> TimeBetSnapshot 0
> TimeOfFirstSnapshot 0
> TimeBetStatistics 0.01
> NumFilesPerSnapshot 1
> MaxFilesWithConcurrentIO 1
> ErrTolIntAccuracy 0.012
> CourantFac 0.3
> MaxSizeTimestep 0.005
> MinSizeTimestep 0
> TypeOfOpeningCriterion 1
> ErrTolTheta 0.7
> ErrTolThetaMax 1
> ErrTolForceAcc 0.0025
> TopNodeFactor 5
> ActivePartFracForNewDomainDecomp 0.01
> DesNumNgb 64
> MaxNumNgbDeviation 1
> UnitLength_in_cm 3.08568e+24
> UnitMass_in_g 1.989e+43
> UnitVelocity_in_cm_per_s 100000
> GravityConstantInternal 0
> SofteningComovingClass0 0.001
> SofteningComovingClass1 0.007
> SofteningComovingClass2 0.025
> SofteningComovingClass3 0.081
> SofteningMaxPhysClass0 0.0005
> SofteningMaxPhysClass1 0.007
> SofteningMaxPhysClass2 0.025
> SofteningMaxPhysClass3 0.081
> SofteningClassOfPartType0 0
> SofteningClassOfPartType1 0
> SofteningClassOfPartType2 1
> SofteningClassOfPartType3 2
> SofteningClassOfPartType4 3
> SofteningClassOfPartType5 3
> DesLinkNgb 20
> ArtBulkViscConst 1
> MinEgySpec 0
> InitGasTemp 0
>
> MALLOC: Allocation of shared memory took 0.000141637 sec
>
> found 16 times in output-list.
> BEGRUN: Hubble (internal units) = 100
> BEGRUN: h = 0.73
> BEGRUN: G (internal units) = 43.0187
> BEGRUN: UnitMass_in_g = 1.989e+43
> BEGRUN: UnitLenth_in_cm = 3.08568e+24
> BEGRUN: UnitTime_in_s = 3.08568e+19
> BEGRUN: UnitVelocity_in_cm_per_s = 100000
> BEGRUN: UnitDensity_in_cgs = 6.76991e-31
> BEGRUN: UnitEnergy_in_cgs = 1.989e+53
>
>
> READIC: filenr=0, './ics_gadget.dat' contains:
> READIC: Type 0 (gas): 0 (tot= 0) masstab= 0
> READIC: Type 1: 17576 (tot= 17576) masstab= 3.65236e-09
> READIC: Type 2: 0 (tot= 0) masstab= 0
> READIC: Type 3: 0 (tot= 0) masstab= 0
> READIC: Type 4: 0 (tot= 0) masstab= 0
> READIC: Type 5: 54196 (tot= 54196) masstab= 0
>
> READIC: Reading file `./ics_gadget.dat' on task=0 and distribute it to 0 to 5.
> READIC: reading block 0 (Coordinates)...
> READIC: reading block 1 (Velocities)...
> READIC: reading block 2 (ParticleIDs)...
> READIC: reading block 3 (Masses)...
> READIC: reading done. Took 0.00612891 sec, total size 2.12379 MB, corresponds to effective I/O rate of 346.521 MB/sec
>
> READIC: Total number of particles : 71772
>
> INIT: Testing ID uniqueness...
> INIT: success. took=0.00309694 sec
>
> DOMAIN: Begin domain decomposition (sync-point 0).
> DOMAIN: Sum=2 TotalCost=2 NumTimeBinsToBeBalanced=1 MultipleDomains=2
> DOMAIN: Increasing TopNodeAllocFactor=0.08 new value=0.104
> DOMAIN: Increasing TopNodeAllocFactor=0.104 new value=0.1352
> DOMAIN: Increasing TopNodeAllocFactor=0.1352 new value=0.17576
> DOMAIN: Increasing TopNodeAllocFactor=0.17576 new value=0.228488
> DOMAIN: Increasing TopNodeAllocFactor=0.228488 new value=0.297034
> DOMAIN: Increasing TopNodeAllocFactor=0.297034 new value=0.386145
> DOMAIN: Increasing TopNodeAllocFactor=0.386145 new value=0.501988
> DOMAIN: Increasing TopNodeAllocFactor=0.501988 new value=0.652585
> DOMAIN: Increasing TopNodeAllocFactor=0.652585 new value=0.84836
> DOMAIN: Increasing TopNodeAllocFactor=0.84836 new value=1.10287
> DOMAIN: Increasing TopNodeAllocFactor=1.10287 new value=1.43373
> DOMAIN: Increasing TopNodeAllocFactor=1.43373 new value=1.86385
> DOMAIN: Increasing TopNodeAllocFactor=1.86385 new value=2.423
> DOMAIN: Increasing TopNodeAllocFactor=2.423 new value=3.1499
> DOMAIN: NTopleaves=246, determination of top-level tree involved 14 iterations and took 0.00647044 sec
> DOMAIN: we are going to try at most 149 different settings for combining the domains on tasks=6, nnodes=1
> DOMAIN: total_cost=2 total_load=1
> DOMAIN: best solution found after 1 iterations by task=5 for nextra=0, reaching maximum imbalance of 1.02449|1.03052
> DOMAIN: combining multiple-domains took 0.000829803 sec
> DOMAIN: exchange of 71772 particles
> DOMAIN: particle exchange done. (took 0.00680363 sec)
> DOMAIN: domain decomposition done. (took in total 0.0150899 sec)
> PEANO: Begin Peano-Hilbert order...
> PEANO: done, took 0.00189733 sec.
> INIT: AvgType1Mass = 3.65236e-09 (min=3.65236e-09 max=3.65236e-09) Ndm1tot=17576
> INIT: For this AvgType1Mass, the mean particle spacing is -nan and the assigned softening is 0.001
>
> SNAPSHOT: Setting next time for snapshot file to Time_next= 0.0288901 (DumpFlag=1)
>
>
>
> Sync-Point 0, Time: 0.0078125, Redshift: 127, Systemstep: 0, Dloga: 0, Nsync-grv: 71772, Nsync-hyd: 0
> DOMAIN: Begin domain decomposition (sync-point 0).
> DOMAIN: Sum=2 TotalCost=2 NumTimeBinsToBeBalanced=1 MultipleDomains=2
> DOMAIN: NTopleaves=246, determination of top-level tree involved 14 iterations and took 0.00247401 sec
> DOMAIN: we are going to try at most 149 different settings for combining the domains on tasks=6, nnodes=1
> DOMAIN: total_cost=2 total_load=1
> DOMAIN: best solution found after 1 iterations by task=5 for nextra=0, reaching maximum imbalance of 1.02449|1.03052
> DOMAIN: combining multiple-domains took 0.00157532 sec
> DOMAIN: exchange of 71772 particles
> DOMAIN: particle exchange done. (took 0.00363552 sec)
> DOMAIN: domain decomposition done. (took in total 0.00819422 sec)
> PEANO: Begin Peano-Hilbert order...
> PEANO: done, took 0.000835626 sec.
> ACCEL: Start tree gravity force computation... (71772 particles)
> PM-PERIODIC: Starting periodic PM calculation. (Rcut=0.854492) presently allocated=1.83447 MB
> -------------------------- Allocated Memory Blocks---- ( Step 0 )------------------
> Task Nr F Variable MBytes Cumulative Function|File|Linenumber
> ------------------------------------------------------------------------------------------
> 4 0 0 GetGhostRankForSimulCommRank 0.0001 0.0001 mymalloc_init()|src/data/mymalloc.cc|137
> 4 1 0 GetShmRankForSimulCommRank 0.0001 0.0001 mymalloc_init()|src/data/mymalloc.cc|138
> 4 2 0 GetNodeIDForSimulCommRank 0.0001 0.0002 mymalloc_init()|src/data/mymalloc.cc|139
> 4 3 0 SharedMemBaseAddr 0.0001 0.0002 mymalloc_init()|src/data/mymalloc.cc|153
> 4 4 1 slab_to_task 0.0039 0.0042 my_slab_based_fft_init()|src/pm/pm_mpi_fft.cc|45
> 4 5 1 slabs_x_per_task 0.0001 0.0042 my_slab_based_fft_init()|src/pm/pm_mpi_fft.cc|60
> 4 6 1 first_slab_x_of_task 0.0001 0.0043 my_slab_based_fft_init()|src/pm/pm_mpi_fft.cc|63
> 4 7 1 slabs_y_per_task 0.0001 0.0043 my_slab_based_fft_init()|src/pm/pm_mpi_fft.cc|66
> 4 8 1 first_slab_y_of_task 0.0001 0.0044 my_slab_based_fft_init()|src/pm/pm_mpi_fft.cc|69
> 4 9 1 P 1.5972 1.6016 allocate_memory()|src/io/../fof/../data/simparticles.h|273
> 4 10 1 SphP 0.0001 1.6017 allocate_memory()|src/io/../fof/../data/simparticles.h|274
> 4 11 1 NextActiveParticleHydro 0.0001 1.6017 timebins_allocate()|src/time_integration/timestep.cc|463
> 4 12 1 NextInTimeBinHydro 0.0001 1.6018 timebins_allocate()|src/time_integration/timestep.cc|466
> 4 13 1 PrevInTimeBinHydro 0.0001 1.6019 timebins_allocate()|src/time_integration/timestep.cc|469
> 4 14 1 NextActiveParticleGravity 0.0761 1.6780 timebins_allocate()|src/time_integration/timestep.cc|463
> 4 15 1 NextInTimeBinGravity 0.0761 1.7541 timebins_allocate()|src/time_integration/timestep.cc|466
> 4 16 1 PrevInTimeBinGravity 0.0761 1.8302 timebins_allocate()|src/time_integration/timestep.cc|469
> 4 17 1 FirstTopleafOfTask 0.0001 1.8303 domain_allocate()|src/domain/domain.cc|163
> 4 18 1 NumTopleafOfTask 0.0001 1.8303 domain_allocate()|src/domain/domain.cc|164
> 4 19 1 TopNodes 0.0022 1.8325 domain_allocate()|src/domain/domain.cc|165
> 4 20 1 TaskOfLeaf 0.0010 1.8335 domain_allocate()|src/domain/domain.cc|166
> 4 21 1 ListOfTopleaves 0.0010 1.8345 domain_decomposition()|src/domain/domain.cc|118
> 4 22 0 part 1.0726 2.9070 pmforce_zoom_optimized_prepare_density()|src/pm/pm_periodic.cc|184
> 4 23 1 localfield_globalindex 0.0001 2.9071 pmforce_zoom_optimized_prepare_density()|src/pm/pm_periodic.cc|272
> 4 24 1 localfield_data 0.0001 2.9072 pmforce_zoom_optimized_prepare_density()|src/pm/pm_periodic.cc|274
> 4 25 1 localfield_first 0.0001 2.9072 pmforce_zoom_optimized_prepare_density()|src/pm/pm_periodic.cc|275
> 4 26 1 localfield_sendcount 0.0001 2.9073 pmforce_zoom_optimized_prepare_density()|src/pm/pm_periodic.cc|276
> 4 27 1 localfield_offset 0.0001 2.9073 pmforce_zoom_optimized_prepare_density()|src/pm/pm_periodic.cc|277
> 4 28 1 localfield_recvcount 0.0001 2.9074 pmforce_zoom_optimized_prepare_density()|src/pm/pm_periodic.cc|278
> ------------------------------------------------------------------------------------------
> Code termination on task=4, function mymalloc_movable_fullinfo(), file src/data/mymalloc.cc, line 326:
> Not enough memory in mymalloc_fullinfo() to allocate 685.336 MB for variable 'rhogrid' at pmforce_zoom_optimized_prepare_density()/src/pm/pm_periodic.cc/line 378 (FreeBytes=97.0926 MB).
>
> application called MPI_Abort(MPI_COMM_WORLD, 1) - process 4
> pc_at_ubuntu:~/Desktop/SPH/gadget4-master/IC_from_MUSIC$
>
>
> -----------------------------------------------------------
>
> If you wish to unsubscribe from this mailing, send mail to
> minimalist_at_MPA-Garching.MPG.de with a subject of: unsubscribe gadget-list
> A web-archive of this mailing list is available here:
> http://www.mpa-garching.mpg.de/gadget/gadget-list
Received on 2021-02-12 10:29:30

This archive was generated by hypermail 2.3.0 : 2022-09-01 14:03:43 CEST