<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On 25 November 2015 at 08:42, Sean M. Collins <span dir="ltr"><<a href="mailto:sean@coreitpro.com" target="_blank">sean@coreitpro.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">The first run for the multinode grenade job completed.<br>
<br>
<a href="http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/" rel="noreferrer" target="_blank">http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/</a><br>
<br>
I'm still getting my bearings in the grenade log output, but if I<br>
understand it correctly, after upgrading Neutron, when spawning a new<br>
instance we do not get successful connectivity. The odd part is we see<br>
the failure twice. Once when upgrading Nova[1], then once when upgrading<br>
Cinder[2]. I would have thought Grenade would have exited after just<br>
failing the first time. It did do a call to worlddump.<br>
<br>
The odd part is that the first failure is a little strange - it pings<br>
then sleeps until it successfully gets a packet back. Which it does -<br>
but then it does a call to worlddump. Odd?<br>
<br>
Anyway, then it goes on to upgrade cinder, then tries to create an<br>
instance and ssh in, then it fails and grenade exits completely.<br>
<br>
I'll continue digging through the logs to see exactly why we don't get<br>
connectivity. I see some interesting warnings in the L3 agent log about<br>
cleaning up a non-existent router[3], which may be the culprit.<br>
<br>
[1]: <a href="http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/grenade.sh.txt.gz#_2015-11-23_20_34_06_742" rel="noreferrer" target="_blank">http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/grenade.sh.txt.gz#_2015-11-23_20_34_06_742</a><br>
<br>
[2]: <a href="http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/grenade.sh.txt.gz#_2015-11-23_21_45_15_133" rel="noreferrer" target="_blank">http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/grenade.sh.txt.gz#_2015-11-23_21_45_15_133</a><br>
<br>
[3]: <a href="http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/old/screen-q-l3.txt.gz?level=WARNING" rel="noreferrer" target="_blank">http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/old/screen-q-l3.txt.gz?level=WARNING</a></blockquote><div><br></div><div>Good stuff Sean.</div><div><br></div><div>A question:</div><div><br></div><div>if the workflow is: upgrade server first, run some connectivity tests to then proceed to upgrading the rest to then re-run tempest, where's the log of the new server?</div><div><br></div><div>All I can see is this one:</div><div><br></div><div><a href="http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/old/screen-q-svc.txt.gz">http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/old/screen-q-svc.txt.gz</a><br></div><div><br></div><div>And I see no restart in it.</div><div><br></div><div>On the subnode (compute), I only see 'old' logs.</div><div><br></div><div><a href="http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/subnode-2/old/">http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/subnode-2/old/</a></div><div><br></div><div>Thoughts?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
<div class=""><div class="h5">--<br>
Sean M. Collins<br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>