<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(0,0,102)">Hi all<br><br>In the case for example when I'm using m1.tiny with two cores total which is a power of 2, when P and Q are equal to 1 and 2 (and np 2) it stops with the following error, when I set P and Q equal to 2 and 2 (and np 4 or higher) it hangs at the beginning of MPIRandomAccess, which is beginning of the test.<br>
In worst case it takes couple of minutes for two medium nodes to pass this stage.<br>I waited seven hours. Still I get four hpcc process with 100 percent usage of CPU, but nothing's going on.<br><br>For MPICH I receive this error:<br>
<br>=====================================================================================<br>= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br>= EXIT CODE: 139<br>= CLEANING UP REMAINING PROCESSES<br>= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES<br>
=====================================================================================<br>[proxy:0:1@ubuntu-benchmark02] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:928): assert (!closed) failed<br>[proxy:0:1@ubuntu-benchmark02] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status<br>
[proxy:0:1@ubuntu-benchmark02] main (./pm/pmiserv/pmip.c:226): demux engine error waiting for event<br>[mpiexec@ubuntu-benchmark01] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:70): one of the processes terminated badly; aborting<br>
[mpiexec@ubuntu-benchmark01] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion<br>[mpiexec@ubuntu-benchmark01] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:191): launcher returned error waiting for completion<br>
[mpiexec@ubuntu-benchmark01] main (./ui/mpich/mpiexec.c:405): process manager error waiting for completion<br><br>and their developer answer:<br>>BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br>Here's the key part -- it sounds like your application process died<br>
badly. Sounds like a problem with your benchmark.<br>The rest is just cleanup of the remaining processes.<br><br>and in further discussion on of them said:<br>If cpi works then I suspect MPICH is not your problem.<br><br>
and here is Open MPI error:<br><br>localadmin@ubuntu-benchmark:~/hpcc-1.4.2$ mpirun -np 2 --hostfile hosts2 hpcc<br><a href="mailto:localadmin@192.168.100.3">localadmin@192.168.100.3</a>'s password:<br>[ubuntu-benchmark:01828] *** Process received signal ***<br>
[ubuntu-benchmark:02164] *** Process received signal ***<br>[ubuntu-benchmark:02164] Signal: Segmentation fault (11)<br>[ubuntu-benchmark:02164] Signal code: Address not mapped (1)<br>[ubuntu-benchmark:02164] Failing at address: 0xda3000<br>
[ubuntu-benchmark:01828] Signal: Segmentation fault (11)<br>[ubuntu-benchmark:01828] Signal code: Address not mapped (1)<br>[ubuntu-benchmark:01828] Failing at address: 0x2039000<br>[ubuntu-benchmark:02164] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x2b791e2d74a0]<br>
[ubuntu-benchmark:02164] [ 1] hpcc(HPCC_Power2NodesMPIRandomAccessCheck+0xa31) [0x423961]<br>[ubuntu-benchmark:02164] [ 2] hpcc(HPCC_MPIRandomAccess+0x87a) [0x41e53a]<br>[ubuntu-benchmark:02164] [ 3] hpcc(main+0xfbf) [0x40a2bf]<br>
[ubuntu-benchmark:02164] [ 4] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x2b791e2c276d]<br>[ubuntu-benchmark:02164] [ 5] hpcc() [0x40abfd]<br>[ubuntu-benchmark:02164] *** End of error message ***<br>[ubuntu-benchmark:01828] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x2acd639c34a0]<br>
[ubuntu-benchmark:01828] [ 1] hpcc(HPCC_Power2NodesMPIRandomAccessCheck+0x9c8) [0x4238f8]<br>[ubuntu-benchmark:01828] [ 2] hpcc(HPCC_MPIRandomAccess+0x87a) [0x41e53a]<br>[ubuntu-benchmark:01828] [ 3] hpcc(main+0xfbf) [0x40a2bf]<br>
[ubuntu-benchmark:01828] [ 4] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x2acd639ae76d]<br>[ubuntu-benchmark:01828] [ 5] hpcc() [0x40abfd]<br>[ubuntu-benchmark:01828] *** End of error message ***<br>--------------------------------------------------------------------------<br>
mpirun noticed that process rank 1 with PID 2164 on node 192.168.100.3 exited on signal 11 (Segmentation fault).<br>--------------------------------------------------------------------------<br><br>@Lorin<br>On more thing to add, I asked Open MPI developers and they said you don't need distributed file system when Open MPI installed in the same location on every machine.<br>
<br>@Dave<br>I have used Open MPI in a virtualized environment this way. I didn't understand what incompatibility you mean? By the way, It's not important to me using Open MPI, MPICH or anything else. my goal is to run the test.<br>
<br>@Brian<br>Unfortunately --mca btl_openib_free_list_max didn't help and I received the same error or hanging situation.<br><br>@Jacob<br><br>I set memory overcommitment ratio to one both on server and instances in<br>
/proc/sys/vm/overcommitment_ratio<br>and disabled hyper threads in<br>/sys/devices/system/node/node0/cpu{1,3,5,7}/online<br>but unfortunately they didn't help and I received the same error or hanging situation.<br>I could do the test on one instance.<br>
<br><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 16 May 2013 20:56, Jacob Liberman <span dir="ltr"><<a href="mailto:jliberma@redhat.com" target="_blank">jliberma@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div><div class="h5">
<div>On 05/15/2013 08:08 AM, Reza
Bakhshayeshi wrote:<br>
</div>
</div></div><blockquote type="cite"><div><div class="h5">
<div dir="ltr">
<div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(0,0,102)">Hi <br>
<br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(0,0,102)">I
want to perform a MPI program across the instances. I've
already done it on a traditional and virtual cluster, so I'm
pretty sure about the healthiness of my installation.<br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(0,0,102)">Unfortunately
I can't perform it on a cluster of OpenStack instances.<br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(0,0,102)">
My MPI program is HPCC, it stops at the begging of
MPIRandomAccess.<br>
<br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(0,0,102)">I
would be so grateful if anyone had a similar experience or can
guess some possibilities and solutions.<br>
<br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(0,0,102)">Regards,<br>
</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif;color:rgb(0,0,102)">Reza<br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div></div><div class="im"><pre>_______________________________________________
OpenStack-operators mailing list
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a>
</pre>
</div></blockquote>
Does it hang or does it fail with an error? Please send along any
errors.<br>
<br>
The HPCC random access test will size the problem to half the
available RAM in the whole system.<br>
<br>
I would make sure your memory over commitment ratio is set to 1.<br>
<br>
I would also disable hyperthreading and make sure you are running on
a power of 2 processor count.<br>
<br>
You can start by running the MPI test within a single instance on a
single host.<br>
</div>
<br>_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br></blockquote></div><br></div>