Problems with GADGET2 execution on 32 cores.

From: Imran Aziz Ahmed <imranaziz_at_gmail.com>
Date: Wed, 29 Sep 2010 12:53:38 +0530

Hi folks,

I am sorry if this is a repeat.
If I try to execute GADGET2 on 32 cores or more, I can see the errors which
I have pasted at the end of this e-mail.

The program fails after step 1339. However, it works fine for 4 cores and 24
cores.

Kindly help.

Thank you!

Regards,
-Imran Aziz Ahmed


====================================================================================

Begin Step 1339, Time: 0.679503, Redshift: 0.471665, Systemstep:
0.000263559, Dloga: 0.000387946

domain decomposition...

NTopleaves= 2710

work-load balance=1.8138 memory-balance=1.56344

exchange of 0000006891 particles

domain decomposition done.

begin Peano-Hilbert order...

Peano-Hilbert done.

Start force computation...

Tree construction.

task 12: maximum number 11059 of tree-nodes reached.

for particle 13462

task 12: endrun called with an error level of 1





application called MPI_Abort(MPI_COMM_WORLD, 1) - process 12

Fatal error in MPI_Sendrecv: Other MPI error, error stack:

MPI_Sendrecv(218).................: MPI_Sendrecv(sbuf=0x7fffedd26e90,
scount=5088, MPI_BYTE, dest=0, stag=15, rbuf=0x7fffedd1c010, rcount=3008,
MPI_BYTE, src=0, rtag=15, MPI_COMM_WORLD, status=0x7fffffffdd10) failed

MPIDI_CH3I_Progress(150)..........:

MPID_nem_mpich2_blocking_recv(948):

MPID_nem_tcp_connpoll(1720).......:

state_commrdy_handler(1556).......:

MPID_nem_tcp_recv_handler(1446)...: socket closed

Fatal error in MPI_Sendrecv: Other MPI error, error stack:

MPI_Sendrecv(218).................: MPI_Sendrecv(sbuf=0x7fffedd29130,
scount=3232, MPI_BYTE, dest=4, stag=15, rbuf=0x7fffedd1f0f0, rcount=2272,
MPI_BYTE, src=4, rtag=15, MPI_COMM_WORLD, status=0x7fffffffdd10) failed

MPIDI_CH3I_Progress(150)..........:

MPID_nem_mpich2_blocking_recv(948):

MPID_nem_tcp_connpoll(1720).......:

state_commrdy_handler(1556).......:

MPID_nem_tcp_recv_handler(1446)...: socket closed

Fatal error in MPI_Sendrecv: Other MPI error, error stack:

MPI_Sendrecv(218).................: MPI_Sendrecv(sbuf=0x7fffedd2b5d0,
scount=3520, MPI_BYTE, dest=8, stag=15, rbuf=0x7fffedd21cb0, rcount=1408,
MPI_BYTE, src=8, rtag=15, MPI_COMM_WORLD, status=0x7fffffffdd10) failed

MPIDI_CH3I_Progress(150)..........:

MPID_nem_mpich2_blocking_recv(948):

MPID_nem_tcp_connpoll(1720).......:

state_commrdy_handler(1556).......:

MPID_nem_tcp_recv_handler(1446)...: socket closed

Fatal error in MPI_Sendrecv: Other MPI error, error stack:

MPI_Sendrecv(218).................: MPI_Sendrecv(sbuf=0x7fffedd2c390,
scount=3008, MPI_BYTE, dest=9, stag=15, rbuf=0x7fffedd22230, rcount=1152,
MPI_BYTE, src=9, rtag=15, MPI_COMM_WORLD, status=0x7fffffffdd10) failed

MPIDI_CH3I_Progress(150)..........:

MPID_nem_mpich2_blocking_recv(948):

MPID_nem_tcp_connpoll(1720).......:

state_commrdy_handler(1556).......:

MPID_nem_tcp_recv_handler(1446)...: socket closed

Fatal error in MPI_Sendrecv: Other MPI error, error stack:

MPI_Sendrecv(218)..Fatal error in MPI_Sendrecv: Other MPI error, error
stack:

MPI_Sendrecv(218).................: MPI_Sendrecv(sbuf=0x7fffedd2e870,
scount=2400, MPI_BYTE, dest=12, stag=15, rbuf=0x7fffedd235d0, rcount=4448,
MPI_BYTE, src=12, rtag=15, MPI_COMM_WORLD, status=0x7fffffffdd10) failed

MPIDI_CH3I_Progress(150)..........:

MPID_nem_mpich2_blocking_recv(948):

MPID_nem_tcp_connpoll(1709).......: Communication error

Fatal error in MPI_Sendrecv: Other MPI error, error stack:
MPI_Sendrecv(218).................: MPI_Sendrecv(sbuf=0x7fffedd2db90,
scount=3296, MPI_BYTE, dest=11, stag=15, rbuf=0x7fffedd22f30, rcount=1696,
MPI_BYTE, src=11, rtag=15, MPI_COMM_WORLD, sta
Received on 2010-09-29 09:23:39