Open Stack

On 11/25/2015 11:42 AM, Sean M. Collins wrote:
> The first run for the multinode grenade job completed.
> 
> http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/
> 
> I'm still getting my bearings in the grenade log output, but if I
> understand it correctly, after upgrading Neutron, when spawning a new
> instance we do not get successful connectivity. The odd part is we see
> the failure twice. Once when upgrading Nova[1], then once when upgrading
> Cinder[2]. I would have thought Grenade would have exited after just
> failing the first time. It did do a call to worlddump.

We're calling worlddump a bunch of times in grenade during success
operations to try to help track down why connectivity sometimes goes
away when we don't do any actions which we think should affect it. The
Nova operation succeeded, there is a successful ping at the end of it.

Cinder pinged, but also does ssh verification. That failed.

> 
> The odd part is that the first failure is a little strange - it pings
> then sleeps until it successfully gets a packet back. Which it does -
> but then it does a call to worlddump. Odd?
> 
> Anyway, then it goes on to upgrade cinder, then tries to create an
> instance and ssh in, then it fails and grenade exits completely.
> 
> I'll continue digging through the logs to see exactly why we don't get
> connectivity. I see some interesting warnings in the L3 agent log about
> cleaning up a non-existent router[3], which may be the culprit.
> 
> [1]: http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/grenade.sh.txt.gz#_2015-11-23_20_34_06_742
> 
> [2]: http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/grenade.sh.txt.gz#_2015-11-23_21_45_15_133
> 
> [3]: http://logs.openstack.org/35/187235/11/experimental/gate-grenade-dsvm-neutron-multinode/011124b/logs/old/screen-q-l3.txt.gz?level=WARNING
> 


-- 
Sean Dague
http://dague.net