Open Stack

On Tue, Jan 6, 2015 at 5:12 PM, Sam Morrison <sorrison at gmail.com> wrote:

> I’m working on upgrading our nova to Juno.
>
> First I upgrade the control infrastructure, conductor,api,scheduler etc.
> and migrate the DB
> I also set the following on the control hosts
> [upgrade_levels]
> compute=icehouse
>
> I don’t touch the nodes running nova-compute.
>
> Things seem to work however when I start a nova-compute I get the
> following when it registers its service:
>

Are you restarting existing icehouse nova-compute and not juno?

Can you share sanitized copies of your config file as not sure what the
issue is based on the stacktrace alone.


>
> Line in question:
> nova/virt/libvirt/driver.py
> def _set_host_enabled()
> service = service_obj.Service.get_by_compute_host(ctx, CONF.host)
>
>
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py",
> line 155, in call
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     return
> self.serializer.deserialize_entity(ctxt, result)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/nova/nova/rpc.py", line 111, in deserialize_entity
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     return
> self._base.deserialize_entity(context, entity)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/nova/nova/objects/base.py", line 575, in deserialize_entity
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     entity =
> self._process_object(context, entity)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/nova/nova/objects/base.py", line 545, in _process_object
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver
>  e.kwargs['supported'])
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/nova/nova/conductor/api.py", line 280, in object_backport
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     return
> self._manager.object_backport(context, objinst, target_version)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/nova/nova/conductor/rpcapi.py", line 435, in object_backport
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver
>  target_version=target_version)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py",
> line 155, in call
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     return
> self.serializer.deserialize_entity(ctxt, result)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/nova/nova/rpc.py", line 111, in deserialize_entity
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     return
> self._base.deserialize_entity(context, entity)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/nova/nova/objects/base.py", line 575, in deserialize_entity
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     entity =
> self._process_object(context, entity)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/nova/nova/objects/base.py", line 545, in _process_object
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver
>  e.kwargs['supported'])
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/nova/nova/conductor/api.py", line 280, in object_backport
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     return
> self._manager.object_backport(context, objinst, target_version)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/nova/nova/conductor/rpcapi.py", line 435, in object_backport
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver
>  target_version=target_version)
> …. REPEATED….
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py",
> line 152, in call
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver
>  retry=self.retry)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/transport.py",
> line 90, in _send
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver
>  timeout=timeout, retry=retry)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py",
> line 436, in send
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver
>  retry=retry)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py",
> line 422, in _send
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver
>  retry=retry)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py",
> line 746, in topic_send
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver
>  exchange_name=exchange_name, retry=retry)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py",
> line 717, in publisher_send
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver
>  self.ensure(_error_callback, _publish, retry=retry)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py",
> line 617, in ensure
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     ret,
> channel = autoretry_method()
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/kombu/connection.py", line
> 436, in _ensured
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     return
> fun(*args, **kwargs)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/kombu/connection.py", line
> 508, in __call__
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     return
> fun(*args, channel=channels[0], **kwargs), channels[0]
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py",
> line 714, in _publish
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     publisher
> = cls(self.conf, channel, topic=topic, **kwargs)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py",
> line 381, in __init__
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     **options)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py",
> line 329, in __init__
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver
>  self.reconnect(channel)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py",
> line 337, in reconnect
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver
>  routing_key=self.routing_key)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line
> 85, in __init__
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver
>  self.revive(self._channel)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line
> 218, in revive
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver
>  self.declare()
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line
> 105, in declare
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver
>  self.exchange.declare()
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/kombu/entity.py", line
> 166, in declare
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver
>  nowait=nowait, passive=passive,
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/amqp/channel.py", line
> 620, in exchange_declare
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     (40,
> 11),  # Channel.exchange_declare_ok
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/amqp/abstract_channel.py",
> line 67, in wait
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver
>  self.channel_id, allowed_methods)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/amqp/connection.py", line
> 240, in _wait_method
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver
>  self.method_reader.read_method()
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File
> "/opt/icehouse/local/lib/python2.7/site-packages/amqp/method_framing.py",
> line 189, in read_method
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     raise m
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver RuntimeError:
> maximum recursion depth exceeded while calling a Python object
>
>
> In conductor I see the following (after adding a debug log message):
> 2015-01-07 12:06:50.860 9781 DEBUG nova.conductor.manager
> [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport
> Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z),
> target_version=1.2 object_backport
> /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439
> 2015-01-07 12:06:50.889 9781 DEBUG nova.conductor.manager
> [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport
> Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z),
> target_version=1.3 object_backport
> /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439
> 2015-01-07 12:06:50.914 9781 DEBUG nova.conductor.manager
> [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport
> Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z),
> target_version=1.2 object_backport
> /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439
> 2015-01-07 12:06:50.937 9781 DEBUG nova.conductor.manager
> [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport
> Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z),
> target_version=1.3 object_backport
> /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439
> (repeats a bunch more times)
>
> What I can’t figure out is why it’s asking for a target version of 1.3,
> icehouse service object is 1.2 and juno service object is 1.4
>
> Anyone come across this?
>

We haven't seen this in any other bug reports or in our automated testing.


>
> Cheers,
> Sam
>
>
>
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150107/a12a1328/attachment.html>