Currently Loaded Modulefiles: 1) intel_compiler/.2020.4.304(default) 3) intel/2020.4.304(default) 2) intel_mpi/.2020.4.304(default) slurmstepd: error: Detected 1 oom-kill event(s) in StepId=151171.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler. srun: error: cn0111: task 1008: Out Of Memory slurmstepd: error: Detected 2 oom-kill event(s) in StepId=151171.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler. slurmstepd: error: Detected 1 oom-kill event(s) in StepId=151171.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler. slurmstepd: error: Detected 2 oom-kill event(s) in StepId=151171.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler. slurmstepd: error: Detected 1 oom-kill event(s) in StepId=151171.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler. slurmstepd: error: Detected 1 oom-kill event(s) in StepId=151171.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler. slurmstepd: error: Detected 1 oom-kill event(s) in StepId=151171.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler. slurmstepd: error: Detected 1 oom-kill event(s) in StepId=151171.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler. [cn0085:467223:0:467223] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e00072117) slurmstepd: error: Detected 1 oom-kill event(s) in StepId=151171.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler. ==== backtrace (tid: 467223) ==== 0 0x0000000000012b20 .annobin_sigaction.c() sigaction.c:0 1 0x0000000000006e1c rdma_getaddrinfo() ???:0 2 0x000000000000a82e vrb_msg_ep_reject() verbs_cm.c:0 3 0x000000000000a1ae rxm_conn_handle_event() rxm_conn.c:0 4 0x000000000000b4d4 rxm_msg_eq_progress() osd.c:0 5 0x000000000000b665 rxm_cmap_connect() osd.c:0 6 0x0000000000011be3 rxm_ep_tsend() rxm_ep.c:0 7 0x00000000003396fc fi_tsend() /usr/include/rdma/fi_tagged.h:116 8 0x00000000003396fc MPIDI_OFI_send_handler_vci() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/ofi_impl.h:645 9 0x00000000003396fc MPIDI_OFI_send_normal() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/ofi_send.h:393 10 0x00000000003396fc MPIDI_OFI_send() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/ofi_send.h:610 11 0x00000000003396fc MPIDI_NM_mpi_isend() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/ofi_send.h:804 12 0x00000000003396fc MPIDI_isend_unsafe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_send.h:322 13 0x00000000003396fc MPIDI_isend_safe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_send.h:609 14 0x00000000003396fc MPID_Isend() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_send.h:828 15 0x00000000003396fc MPID_Isend_coll() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_send.h:847 16 0x00000000003396fc MPIC_Isend() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/helper_fns.c:504 17 0x000000000011aa34 MPIR_Alltoallv_intra_scattered_impl() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/intel/alltoallv/alltoallv_intra_scattered.c:205 18 0x00000000001781ba MPIDI_NM_mpi_alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/intel/ofi_coll.h:769 19 0x00000000001781ba MPIDI_Alltoallv_intra_composition_alpha() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_impl.h:1499 20 0x00000000001781ba MPID_Alltoallv_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:2265 21 0x00000000001781ba MPIDI_coll_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:3319 22 0x0000000000154f1e MPIDI_coll_select() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_globals_default.c:129 23 0x000000000021cf9f MPID_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll.h:238 24 0x000000000011b9fd PMPI_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/alltoallv/alltoallv.c:350 25 0x0000000000450f1c myMPI_Alltoallv() /home/uchu/ken.osato/Gadget-4/src/mpi_utils/myalltoall.cc:201 26 0x000000000045d14c pm_mpi_fft::my_slab_transpose() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:287 27 0x000000000045c68b pm_mpi_fft::my_slab_based_fft() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:384 28 0x00000000005017ad ngenic::ngenic_get_derivate_from_fourier_field() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:670 29 0x00000000004ff6c8 ngenic::ngenic_displace_particles() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:166 30 0x0000000000406baa sim::init() /home/uchu/ken.osato/Gadget-4/src/main/init.cc:56 31 0x000000000040f467 main() /home/uchu/ken.osato/Gadget-4/src/main/main.cc:323 32 0x0000000000023493 __libc_start_main() ???:0 33 0x0000000000404d2e _start() ???:0 ================================= slurmstepd: error: Detected 1 oom-kill event(s) in StepId=151171.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler. slurmstepd: error: Detected 2 oom-kill event(s) in StepId=151171.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler. slurmstepd: error: Detected 1 oom-kill event(s) in StepId=151171.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler. slurmstepd: error: Detected 1 oom-kill event(s) in StepId=151171.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler. [cn0128:370393:0:370393] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e0005a6d9) ==== backtrace (tid: 370393) ==== 0 0x0000000000012b20 .annobin_sigaction.c() sigaction.c:0 1 0x000000000001486c rxm_cq_handle_seg_data() osd.c:0 2 0x000000000000c45f rxm_handle_unexp_sar() rxm_ep.c:0 3 0x000000000000c726 rxm_ep_trecv_common() rxm_ep.c:0 4 0x000000000000c7c6 rxm_ep_trecv() rxm_ep.c:0 5 0x000000000033d954 fi_trecv() /usr/include/rdma/fi_tagged.h:93 6 0x000000000033d954 MPIDI_OFI_do_irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/intel/ofi_recv.h:129 7 0x000000000033d954 MPIDI_NM_mpi_irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/ofi_recv.h:371 8 0x000000000033d954 MPIDI_irecv_unsafe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:312 9 0x000000000033d954 MPIDI_irecv_safe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:501 10 0x000000000033d954 MPID_Irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:714 11 0x000000000033d954 MPIC_Irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/helper_fns.c:588 12 0x000000000011a963 MPIR_Alltoallv_intra_scattered_impl() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/intel/alltoallv/alltoallv_intra_scattered.c:184 13 0x00000000001781ba MPIDI_NM_mpi_alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/intel/ofi_coll.h:769 14 0x00000000001781ba MPIDI_Alltoallv_intra_composition_alpha() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_impl.h:1499 15 0x00000000001781ba MPID_Alltoallv_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:2265 16 0x00000000001781ba MPIDI_coll_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:3319 17 0x0000000000154f1e MPIDI_coll_select() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_globals_default.c:129 18 0x000000000021cf9f MPID_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll.h:238 19 0x000000000011b9fd PMPI_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/alltoallv/alltoallv.c:350 20 0x0000000000450f1c myMPI_Alltoallv() /home/uchu/ken.osato/Gadget-4/src/mpi_utils/myalltoall.cc:201 21 0x000000000045d14c pm_mpi_fft::my_slab_transpose() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:287 22 0x000000000045c68b pm_mpi_fft::my_slab_based_fft() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:384 23 0x00000000005017ad ngenic::ngenic_get_derivate_from_fourier_field() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:670 24 0x00000000004ff6c8 ngenic::ngenic_displace_particles() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:166 25 0x0000000000406baa sim::init() /home/uchu/ken.osato/Gadget-4/src/main/init.cc:56 26 0x000000000040f467 main() /home/uchu/ken.osato/Gadget-4/src/main/main.cc:323 27 0x0000000000023493 __libc_start_main() ???:0 28 0x0000000000404d2e _start() ???:0 ================================= [cn0063:284322:0:284322] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456a2) ==== backtrace (tid: 284322) ==== 0 0x0000000000012b20 .annobin_sigaction.c() sigaction.c:0 1 0x000000000001486c rxm_cq_handle_seg_data() osd.c:0 2 0x000000000000c45f rxm_handle_unexp_sar() rxm_ep.c:0 3 0x000000000000c726 rxm_ep_trecv_common() rxm_ep.c:0 4 0x000000000000c7c6 rxm_ep_trecv() rxm_ep.c:0 5 0x000000000033d954 fi_trecv() /usr/include/rdma/fi_tagged.h:93 6 0x000000000033d954 MPIDI_OFI_do_irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/intel/ofi_recv.h:129 7 0x000000000033d954 MPIDI_NM_mpi_irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/ofi_recv.h:371 8 0x000000000033d954 MPIDI_irecv_unsafe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:312 9 0x000000000033d954 MPIDI_irecv_safe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:501 10 0x000000000033d954 MPID_Irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:714 11 0x000000000033d954 MPIC_Irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/helper_fns.c:588 12 0x000000000011a963 MPIR_Alltoallv_intra_scattered_impl() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/intel/alltoallv/alltoallv_intra_scattered.c:184 13 0x00000000001781ba MPIDI_NM_mpi_alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/intel/ofi_coll.h:769 14 0x00000000001781ba MPIDI_Alltoallv_intra_composition_alpha() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_impl.h:1499 15 0x00000000001781ba MPID_Alltoallv_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:2265 16 0x00000000001781ba MPIDI_coll_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:3319 17 0x0000000000154f1e MPIDI_coll_select() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_globals_default.c:129 18 0x000000000021cf9f MPID_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll.h:238 19 0x000000000011b9fd PMPI_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/alltoallv/alltoallv.c:350 20 0x0000000000450f1c myMPI_Alltoallv() /home/uchu/ken.osato/Gadget-4/src/mpi_utils/myalltoall.cc:201 21 0x000000000045d14c pm_mpi_fft::my_slab_transpose() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:287 22 0x000000000045c68b pm_mpi_fft::my_slab_based_fft() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:384 23 0x00000000005017ad ngenic::ngenic_get_derivate_from_fourier_field() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:670 24 0x00000000004ff6c8 ngenic::ngenic_displace_particles() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:166 25 0x0000000000406baa sim::init() /home/uchu/ken.osato/Gadget-4/src/main/init.cc:56 26 0x000000000040f467 main() /home/uchu/ken.osato/Gadget-4/src/main/main.cc:323 27 0x0000000000023493 __libc_start_main() ???:0 28 0x0000000000404d2e _start() ???:0 ================================= [cn0127:385471:0:385471] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e0005e1bf) slurmstepd: error: Detected 1 oom-kill event(s) in StepId=151171.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler. ==== backtrace (tid: 385471) ==== 0 0x0000000000012b20 .annobin_sigaction.c() sigaction.c:0 1 0x0000000000006e1c rdma_getaddrinfo() ???:0 2 0x000000000000a82e vrb_msg_ep_reject() verbs_cm.c:0 3 0x000000000000a1ae rxm_conn_handle_event() rxm_conn.c:0 4 0x000000000000b4d4 rxm_msg_eq_progress() osd.c:0 5 0x000000000000b665 rxm_cmap_connect() osd.c:0 6 0x0000000000011be3 rxm_ep_tsend() rxm_ep.c:0 7 0x00000000003396fc fi_tsend() /usr/include/rdma/fi_tagged.h:116 8 0x00000000003396fc MPIDI_OFI_send_handler_vci() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/ofi_impl.h:645 9 0x00000000003396fc MPIDI_OFI_send_normal() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/ofi_send.h:393 10 0x00000000003396fc MPIDI_OFI_send() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/ofi_send.h:610 11 0x00000000003396fc MPIDI_NM_mpi_isend() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/ofi_send.h:804 12 0x00000000003396fc MPIDI_isend_unsafe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_send.h:322 13 0x00000000003396fc MPIDI_isend_safe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_send.h:609 14 0x00000000003396fc MPID_Isend() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_send.h:828 15 0x00000000003396fc MPID_Isend_coll() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_send.h:847 16 0x00000000003396fc MPIC_Isend() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/helper_fns.c:504 17 0x000000000011aa34 MPIR_Alltoallv_intra_scattered_impl() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/intel/alltoallv/alltoallv_intra_scattered.c:205 18 0x00000000001781ba MPIDI_NM_mpi_alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/intel/ofi_coll.h:769 19 0x00000000001781ba MPIDI_Alltoallv_intra_composition_alpha() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_impl.h:1499 20 0x00000000001781ba MPID_Alltoallv_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:2265 21 0x00000000001781ba MPIDI_coll_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:3319 22 0x0000000000154f1e MPIDI_coll_select() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_globals_default.c:129 23 0x000000000021cf9f MPID_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll.h:238 24 0x000000000011b9fd PMPI_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/alltoallv/alltoallv.c:350 25 0x0000000000450f1c myMPI_Alltoallv() /home/uchu/ken.osato/Gadget-4/src/mpi_utils/myalltoall.cc:201 26 0x000000000045d14c pm_mpi_fft::my_slab_transpose() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:287 27 0x000000000045c68b pm_mpi_fft::my_slab_based_fft() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:384 28 0x00000000005017ad ngenic::ngenic_get_derivate_from_fourier_field() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:670 29 0x00000000004ff6c8 ngenic::ngenic_displace_particles() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:166 30 0x0000000000406baa sim::init() /home/uchu/ken.osato/Gadget-4/src/main/init.cc:56 31 0x000000000040f467 main() /home/uchu/ken.osato/Gadget-4/src/main/main.cc:323 32 0x0000000000023493 __libc_start_main() ???:0 33 0x0000000000404d2e _start() ???:0 ================================= slurmstepd: error: Detected 1 oom-kill event(s) in StepId=151171.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler. [cn0081:424074:0:424074] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e0006788a) ==== backtrace (tid: 424074) ==== 0 0x0000000000012b20 .annobin_sigaction.c() sigaction.c:0 1 0x000000000001486c rxm_cq_handle_seg_data() osd.c:0 2 0x000000000000c45f rxm_handle_unexp_sar() rxm_ep.c:0 3 0x000000000000c726 rxm_ep_trecv_common() rxm_ep.c:0 4 0x000000000000c7c6 rxm_ep_trecv() rxm_ep.c:0 5 0x000000000033d954 fi_trecv() /usr/include/rdma/fi_tagged.h:93 6 0x000000000033d954 MPIDI_OFI_do_irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/intel/ofi_recv.h:129 7 0x000000000033d954 MPIDI_NM_mpi_irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/ofi_recv.h:371 8 0x000000000033d954 MPIDI_irecv_unsafe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:312 9 0x000000000033d954 MPIDI_irecv_safe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:501 10 0x000000000033d954 MPID_Irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:714 11 0x000000000033d954 MPIC_Irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/helper_fns.c:588 12 0x000000000011a963 MPIR_Alltoallv_intra_scattered_impl() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/intel/alltoallv/alltoallv_intra_scattered.c:184 13 0x00000000001781ba MPIDI_NM_mpi_alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/intel/ofi_coll.h:769 14 0x00000000001781ba MPIDI_Alltoallv_intra_composition_alpha() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_impl.h:1499 15 0x00000000001781ba MPID_Alltoallv_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:2265 16 0x00000000001781ba MPIDI_coll_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:3319 17 0x0000000000154f1e MPIDI_coll_select() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_globals_default.c:129 18 0x000000000021cf9f MPID_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll.h:238 19 0x000000000011b9fd PMPI_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/alltoallv/alltoallv.c:350 20 0x0000000000450f1c myMPI_Alltoallv() /home/uchu/ken.osato/Gadget-4/src/mpi_utils/myalltoall.cc:201 21 0x000000000045d14c pm_mpi_fft::my_slab_transpose() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:287 22 0x000000000045c68b pm_mpi_fft::my_slab_based_fft() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:384 23 0x00000000005017ad ngenic::ngenic_get_derivate_from_fourier_field() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:670 24 0x00000000004ff6c8 ngenic::ngenic_displace_particles() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:166 25 0x0000000000406baa sim::init() /home/uchu/ken.osato/Gadget-4/src/main/init.cc:56 26 0x000000000040f467 main() /home/uchu/ken.osato/Gadget-4/src/main/main.cc:323 27 0x0000000000023493 __libc_start_main() ???:0 28 0x0000000000404d2e _start() ???:0 ================================= slurmstepd: error: Detected 1 oom-kill event(s) in StepId=151171.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler. [cn0063:284332:0:284332] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456ac) [cn0063:284330:0:284330] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456aa) ==== backtrace (tid: 284332) ==== 0 0x0000000000012b20 .annobin_sigaction.c() sigaction.c:0 1 0x000000000001486c rxm_cq_handle_seg_data() osd.c:0 2 0x000000000000c45f rxm_handle_unexp_sar() rxm_ep.c:0 3 0x000000000000c726 rxm_ep_trecv_common() rxm_ep.c:0 4 0x000000000000c7c6 rxm_ep_trecv() rxm_ep.c:0 5 0x000000000033d954 fi_trecv() /usr/include/rdma/fi_tagged.h:93 6 0x000000000033d954 MPIDI_OFI_do_irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/intel/ofi_recv.h:129 7 0x000000000033d954 MPIDI_NM_mpi_irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/ofi_recv.h:371 8 0x000000000033d954 MPIDI_irecv_unsafe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:312 9 0x000000000033d954 MPIDI_irecv_safe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:501 10 0x000000000033d954 MPID_Irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:714 11 0x000000000033d954 MPIC_Irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/helper_fns.c:588 12 0x000000000011a963 MPIR_Alltoallv_intra_scattered_impl() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/intel/alltoallv/alltoallv_intra_scattered.c:184 13 0x00000000001781ba MPIDI_NM_mpi_alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/intel/ofi_coll.h:769 14 0x00000000001781ba MPIDI_Alltoallv_intra_composition_alpha() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_impl.h:1499 15 0x00000000001781ba MPID_Alltoallv_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:2265 16 0x00000000001781ba MPIDI_coll_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:3319 17 0x0000000000154f1e MPIDI_coll_select() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_globals_default.c:129 18 0x000000000021cf9f MPID_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll.h:238 19 0x000000000011b9fd PMPI_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/alltoallv/alltoallv.c:350 20 0x0000000000450f1c myMPI_Alltoallv() /home/uchu/ken.osato/Gadget-4/src/mpi_utils/myalltoall.cc:201 21 0x000000000045d14c pm_mpi_fft::my_slab_transpose() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:287 22 0x000000000045c68b pm_mpi_fft::my_slab_based_fft() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:384 23 0x00000000005017ad ngenic::ngenic_get_derivate_from_fourier_field() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:670 24 0x00000000004ff6c8 ngenic::ngenic_displace_particles() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:166 25 0x0000000000406baa sim::init() /home/uchu/ken.osato/Gadget-4/src/main/init.cc:56 26 0x000000000040f467 main() /home/uchu/ken.osato/Gadget-4/src/main/main.cc:323 27 0x0000000000023493 __libc_start_main() ???:0 28 0x0000000000404d2e _start() ???:0 ================================= ==== backtrace (tid: 284330) ==== 0 0x0000000000012b20 .annobin_sigaction.c() sigaction.c:0 1 0x000000000001486c rxm_cq_handle_seg_data() osd.c:0 2 0x000000000000c45f rxm_handle_unexp_sar() rxm_ep.c:0 3 0x000000000000c726 rxm_ep_trecv_common() rxm_ep.c:0 4 0x000000000000c7c6 rxm_ep_trecv() rxm_ep.c:0 5 0x000000000033d954 fi_trecv() /usr/include/rdma/fi_tagged.h:93 6 0x000000000033d954 MPIDI_OFI_do_irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/intel/ofi_recv.h:129 7 0x000000000033d954 MPIDI_NM_mpi_irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/ofi_recv.h:371 8 0x000000000033d954 MPIDI_irecv_unsafe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:312 9 0x000000000033d954 MPIDI_irecv_safe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:501 10 0x000000000033d954 MPID_Irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:714 11 0x000000000033d954 MPIC_Irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/helper_fns.c:588 12 0x000000000011a963 MPIR_Alltoallv_intra_scattered_impl() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/intel/alltoallv/alltoallv_intra_scattered.c:184 13 0x00000000001781ba MPIDI_NM_mpi_alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/intel/ofi_coll.h:769 14 0x00000000001781ba MPIDI_Alltoallv_intra_composition_alpha() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_impl.h:1499 15 0x00000000001781ba MPID_Alltoallv_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:2265 16 0x00000000001781ba MPIDI_coll_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:3319 17 0x0000000000154f1e MPIDI_coll_select() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_globals_default.c:129 18 0x000000000021cf9f MPID_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll.h:238 19 0x000000000011b9fd PMPI_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/alltoallv/alltoallv.c:350 20 0x0000000000450f1c myMPI_Alltoallv() /home/uchu/ken.osato/Gadget-4/src/mpi_utils/myalltoall.cc:201 21 0x000000000045d14c pm_mpi_fft::my_slab_transpose() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:287 22 0x000000000045c68b pm_mpi_fft::my_slab_based_fft() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:384 23 0x00000000005017ad ngenic::ngenic_get_derivate_from_fourier_field() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:670 24 0x00000000004ff6c8 ngenic::ngenic_displace_particles() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:166 25 0x0000000000406baa sim::init() /home/uchu/ken.osato/Gadget-4/src/main/init.cc:56 26 0x000000000040f467 main() /home/uchu/ken.osato/Gadget-4/src/main/main.cc:323 27 0x0000000000023493 __libc_start_main() ???:0 28 0x0000000000404d2e _start() ???:0 ================================= [cn0063:284349:0:284349] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456bd) [cn0063:284348:0:284348] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456bc) [cn0063:284352:0:284352] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456c0) [cn0063:284351:0:284351] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456bf) [cn0063:284346:0:284346] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456ba) [cn0063:284344:0:284344] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456b8) [cn0063:284350:0:284350] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456be) [cn0063:284343:0:284343] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456b7) [cn0063:284339:0:284339] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456b3) [cn0063:284345:0:284345] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456b9) [cn0063:284340:0:284340] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456b4) [cn0063:284337:0:284337] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456b1) [cn0063:284336:0:284336] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456b0) [cn0063:284338:0:284338] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456b2) [cn0063:284335:0:284335] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456af) [cn0063:284342:0:284342] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456b6) [cn0063:284333:0:284333] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456ad) [cn0063:284334:0:284334] Caught signal 11 (Segmentation fault: tkill(2) or tgkill(2) at address 0xc8e000456ae) ==== backtrace (tid: 284350) ==== 0 0x0000000000012b20 .annobin_sigaction.c() sigaction.c:0 1 0x000000000001486c rxm_cq_handle_seg_data() osd.c:0 2 0x000000000000c45f rxm_handle_unexp_sar() rxm_ep.c:0 3 0x000000000000c726 rxm_ep_trecv_common() rxm_ep.c:0 4 0x000000000000c7c6 rxm_ep_trecv() rxm_ep.c:0 5 0x000000000033d954 fi_trecv() /usr/include/rdma/fi_tagged.h:93 6 0x000000000033d954 MPIDI_OFI_do_irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/intel/ofi_recv.h:129 7 0x000000000033d954 MPIDI_NM_mpi_irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/ofi_recv.h:371 8 0x000000000033d954 MPIDI_irecv_unsafe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:312 9 0x000000000033d954 MPIDI_irecv_safe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:501 10 0x000000000033d954 MPID_Irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:714 11 0x000000000033d954 MPIC_Irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/helper_fns.c:588 12 0x000000000011a963 MPIR_Alltoallv_intra_scattered_impl() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/intel/alltoallv/alltoallv_intra_scattered.c:184 13 0x00000000001781ba MPIDI_NM_mpi_alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/intel/ofi_coll.h:769 14 0x00000000001781ba MPIDI_Alltoallv_intra_composition_alpha() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_impl.h:1499 15 0x00000000001781ba MPID_Alltoallv_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:2265 16 0x00000000001781ba MPIDI_coll_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:3319 17 0x0000000000154f1e MPIDI_coll_select() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_globals_default.c:129 18 0x000000000021cf9f MPID_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll.h:238 ==== backtrace (tid: 284348) ==== 0 0x0000000000012b20 .annobin_sigaction.c() sigaction.c:0 1 0x000000000001486c rxm_cq_handle_seg_data() osd.c:0 2 0x000000000000c45f rxm_handle_unexp_sar() rxm_ep.c:0 3 0x000000000000c726 rxm_ep_trecv_common() rxm_ep.c:0 4 0x000000000000c7c6 rxm_ep_trecv() rxm_ep.c:0 5 0x000000000033d954 fi_trecv() /usr/include/rdma/fi_tagged.h:93 6 0x000000000033d954 MPIDI_OFI_do_irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/intel/ofi_recv.h:129 7 0x000000000033d954 MPIDI_NM_mpi_irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/ofi_recv.h:371 8 0x000000000033d954 MPIDI_irecv_unsafe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:312 9 0x000000000033d954 MPIDI_irecv_safe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:501 10 0x000000000033d954 MPID_Irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:714 11 0x000000000033d954 MPIC_Irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/helper_fns.c:588 12 0x000000000011a963 MPIR_Alltoallv_intra_scattered_impl() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/intel/alltoallv/alltoallv_intra_scattered.c:184 13 0x00000000001781ba MPIDI_NM_mpi_alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/intel/ofi_coll.h:769 14 0x00000000001781ba MPIDI_Alltoallv_intra_composition_alpha() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_impl.h:1499 15 0x00000000001781ba MPID_Alltoallv_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:2265 16 0x00000000001781ba MPIDI_coll_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:3319 17 0x0000000000154f1e MPIDI_coll_select() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_globals_default.c:129 18 0x000000000021cf9f MPID_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll.h:238 19 0x000000000011b9fd PMPI_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/alltoallv/alltoallv.c:350 20 0x0000000000450f1c myMPI_Alltoallv() /home/uchu/ken.osato/Gadget-4/src/mpi_utils/myalltoall.cc:201 21 0x000000000045d14c pm_mpi_fft::my_slab_transpose() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:287 22 0x000000000045c68b pm_mpi_fft::my_slab_based_fft() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:384 23 0x00000000005017ad ngenic::ngenic_get_derivate_from_fourier_field() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:670 24 0x00000000004ff6c8 ngenic::ngenic_displace_particles() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:166 25 0x0000000000406baa sim::init() /home/uchu/ken.osato/Gadget-4/src/main/init.cc:56 26 0x000000000040f467 main() /home/uchu/ken.osato/Gadget-4/src/main/main.cc:323 27 0x0000000000023493 __libc_start_main() ???:0 28 0x0000000000404d2e _start() ???:0 ================================= /* abridged */ ==== backtrace (tid: 284333) ==== 0 0x0000000000012b20 .annobin_sigaction.c() sigaction.c:0 1 0x0000000000014883 rxm_cq_handle_seg_data() osd.c:0 2 0x000000000000c45f rxm_handle_unexp_sar() rxm_ep.c:0 3 0x000000000000c726 rxm_ep_trecv_common() rxm_ep.c:0 4 0x000000000000c7c6 rxm_ep_trecv() rxm_ep.c:0 5 0x000000000033d954 fi_trecv() /usr/include/rdma/fi_tagged.h:93 6 0x000000000033d954 MPIDI_OFI_do_irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/intel/ofi_recv.h:129 7 0x000000000033d954 MPIDI_NM_mpi_irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/ofi_recv.h:371 8 0x000000000033d954 MPIDI_irecv_unsafe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:312 9 0x000000000033d954 MPIDI_irecv_safe() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:501 10 0x000000000033d954 MPID_Irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/ch4_recv.h:714 11 0x000000000033d954 MPIC_Irecv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/helper_fns.c:588 12 0x000000000011a963 MPIR_Alltoallv_intra_scattered_impl() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/intel/alltoallv/alltoallv_intra_scattered.c:184 13 0x00000000001781ba MPIDI_NM_mpi_alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/netmod/include/../ofi/intel/ofi_coll.h:769 14 0x00000000001781ba MPIDI_Alltoallv_intra_composition_alpha() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_impl.h:1499 15 0x00000000001781ba MPID_Alltoallv_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:2265 16 0x00000000001781ba MPIDI_coll_invoke() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_select_utils.c:3319 17 0x0000000000154f1e MPIDI_coll_select() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll_globals_default.c:129 18 0x000000000021cf9f MPID_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpid/ch4/src/intel/ch4_coll.h:238 19 0x000000000011b9fd PMPI_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/alltoallv/alltoallv.c:350 20 0x0000000000450f1c myMPI_Alltoallv() /home/uchu/ken.osato/Gadget-4/src/mpi_utils/myalltoall.cc:201 21 0x000000000045d14c pm_mpi_fft::my_slab_transpose() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:287 22 0x000000000045c68b pm_mpi_fft::my_slab_based_fft() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:384 23 0x00000000005017ad ngenic::ngenic_get_derivate_from_fourier_field() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:670 24 0x00000000004ff6c8 ngenic::ngenic_displace_particles() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:166 25 0x0000000000406baa sim::init() /home/uchu/ken.osato/Gadget-4/src/main/init.cc:56 26 0x000000000040f467 main() /home/uchu/ken.osato/Gadget-4/src/main/main.cc:323 27 0x0000000000023493 __libc_start_main() ???:0 28 0x0000000000404d2e _start() ???:0 ================================= 19 0x000000000011b9fd PMPI_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/alltoallv/alltoallv.c:350 20 0x0000000000450f1c myMPI_Alltoallv() /home/uchu/ken.osato/Gadget-4/src/mpi_utils/myalltoall.cc:201 21 0x000000000045d14c pm_mpi_fft::my_slab_transpose() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:287 22 0x000000000045c68b pm_mpi_fft::my_slab_based_fft() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:384 23 0x00000000005017ad ngenic::ngenic_get_derivate_from_fourier_field() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:670 24 0x00000000004ff6c8 ngenic::ngenic_displace_particles() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:166 25 0x0000000000406baa sim::init() /home/uchu/ken.osato/Gadget-4/src/main/init.cc:56 26 0x000000000040f467 main() /home/uchu/ken.osato/Gadget-4/src/main/main.cc:323 27 0x0000000000023493 __libc_start_main() ???:0 28 0x0000000000404d2e _start() ???:0 ================================= 19 0x000000000011b9fd PMPI_Alltoallv() /localdisk/jenkins/workspace/workspace/ch4-build-linux-2019/impi-ch4-build-linux_build/CONF/impi-ch4-build-linux-release/label/impi-ch4-build-linux-intel64/_buildspace/release/../../src/mpi/coll/alltoallv/alltoallv.c:350 20 0x0000000000450f1c myMPI_Alltoallv() /home/uchu/ken.osato/Gadget-4/src/mpi_utils/myalltoall.cc:201 21 0x000000000045d14c pm_mpi_fft::my_slab_transpose() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:287 22 0x000000000045c68b pm_mpi_fft::my_slab_based_fft() /home/uchu/ken.osato/Gadget-4/src/pm/pm_mpi_fft.cc:384 23 0x00000000005017ad ngenic::ngenic_get_derivate_from_fourier_field() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:670 24 0x00000000004ff6c8 ngenic::ngenic_displace_particles() /home/uchu/ken.osato/Gadget-4/src/ngenic/ngenic.cc:166 25 0x0000000000406baa sim::init() /home/uchu/ken.osato/Gadget-4/src/main/init.cc:56 26 0x000000000040f467 main() /home/uchu/ken.osato/Gadget-4/src/main/main.cc:323 27 0x0000000000023493 __libc_start_main() ???:0 28 0x0000000000404d2e _start() ???:0 ================================= slurmstepd: error: *** JOB 151171 ON cn0063 CANCELLED AT 2022-01-27T17:49:00 *** srun: Job step aborted: Waiting up to 32 seconds for job step to finish. slurmstepd: error: *** STEP 151171.0 ON cn0063 CANCELLED AT 2022-01-27T17:49:00 *** slurmstepd: error: Detected 6 oom-kill event(s) in StepId=151171.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.