BUG and solution for ALTIX 4000 series(particularly at LRZ). On intel compiler for ALTIX causes wrong results with PM module in P-Gadget2.

From: Arman Khalatyan <arm2arm_at_googlemail.com>
Date: Thu, 22 Feb 2007 11:45:55 +0100

Hello everyone who uses Gadget2 on Altix 4700 machines. I would like to
report the bug in Intel compiler 9.0> at LRZ (Leibniz Reichnung Zentrum).
I was fighting with question: Why Haloes produced on Altix machine at LRZ
and haloes on other machined are differ in properties form 10-20%.
Finally I found that the PM module (pm_periodic.c/pm_nonperiodic.c) does not
produces correct "P[i].GravPM[dim] = acc_dim;" it seems is truncated in
memory from double to float or something else. In single PM step the values
for P[i].GravPM[dim] are differ in order of 1e-11 up to 1e-5. If we add
exponential growth of errors in cosmological simulations we will end up with
unacceptable results.

I was testing Gadget-2 over several supercomputers/clusters: at AIP(
P4HT3.6Gh, AMD), JUMP(IBM-p690+), Marenostrum(IBM,p970dualcore), UAM(SGI
ALTIX 3000), LRZ(SGI ALTIX 4700,CPU type is ITANIUM2 Madison 9M).

There results where exactly same except on ITANIUM-2.
Important thing to point that FFTW alone produces correct results.
Gadget-2 without PM also correct.

To solve the problem .
Only combination of this 4 options can produce correct code:
icc -no-gcc -ansi -ansi_alias -ipo
NOTE:
-mp1 or -mp does not effect anything,these options force compiler do not do
any precision optimizations.

I was testing also all possible compilers (intel8.1 to intel10.0b)at LRZ
here is tests:

module load gsl
module load fftw/mpi

icc 8.1 + gcc 3.3.4 running ok (probably there is no aggressive
optimizations for ITANIUM-2 cpus)

icc 9.0 and above + gcc 4.01 or 4.1.0 does not running correctly with PM
module in Gagdet2 in file pm_periodic.c/pm_nonperiodic.c the P[i].GravPM
wrong

solution: -O3 -no-gcc -ansi -ansi_alias -ipo
or for debugging:
-O0 -g -traceback (-O0 is important to switch off all possible
optimizations)
I found good to know.

Arman.

+++++++++++++++++++++++++++++++++
Arman Khalatyan
Address:
Astrophysical Institute Potsdam,
An der Sternwarte 16,
D-14482 Potsdam, GERMANY
Phones: (+049) 0331 7499517(work),
        (+049) 030 26558524(priv).
+++++++++++++++++++++++++++++++++
Received on 2007-02-22 11:45:57