Open Stack

I’m working on upgrading our nova to Juno.

First I upgrade the control infrastructure, conductor,api,scheduler etc. and migrate the DB
I also set the following on the control hosts
[upgrade_levels]
compute=icehouse

I don’t touch the nodes running nova-compute.

Things seem to work however when I start a nova-compute I get the following when it registers its service:

Line in question:
nova/virt/libvirt/driver.py 
def _set_host_enabled()
service = service_obj.Service.get_by_compute_host(ctx, CONF.host)


2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 155, in call
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     return self.serializer.deserialize_entity(ctxt, result)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/nova/nova/rpc.py", line 111, in deserialize_entity
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     return self._base.deserialize_entity(context, entity)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/nova/nova/objects/base.py", line 575, in deserialize_entity
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     entity = self._process_object(context, entity)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/nova/nova/objects/base.py", line 545, in _process_object
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     e.kwargs['supported'])
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/nova/nova/conductor/api.py", line 280, in object_backport
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     return self._manager.object_backport(context, objinst, target_version)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/nova/nova/conductor/rpcapi.py", line 435, in object_backport
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     target_version=target_version)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 155, in call
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     return self.serializer.deserialize_entity(ctxt, result)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/nova/nova/rpc.py", line 111, in deserialize_entity
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     return self._base.deserialize_entity(context, entity)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/nova/nova/objects/base.py", line 575, in deserialize_entity
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     entity = self._process_object(context, entity)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/nova/nova/objects/base.py", line 545, in _process_object
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     e.kwargs['supported'])
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/nova/nova/conductor/api.py", line 280, in object_backport
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     return self._manager.object_backport(context, objinst, target_version)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/nova/nova/conductor/rpcapi.py", line 435, in object_backport
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     target_version=target_version)
…. REPEATED….
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     retry=self.retry)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     timeout=timeout, retry=retry)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 436, in send
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     retry=retry)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 422, in _send
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     retry=retry)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 746, in topic_send
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     exchange_name=exchange_name, retry=retry)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 717, in publisher_send
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     self.ensure(_error_callback, _publish, retry=retry)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 617, in ensure
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     ret, channel = autoretry_method()
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     return fun(*args, **kwargs)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/connection.py", line 508, in __call__
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     return fun(*args, channel=channels[0], **kwargs), channels[0]
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 714, in _publish
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     publisher = cls(self.conf, channel, topic=topic, **kwargs)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 381, in __init__
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     **options)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 329, in __init__
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     self.reconnect(channel)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 337, in reconnect
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     routing_key=self.routing_key)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line 85, in __init__
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     self.revive(self._channel)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line 218, in revive
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     self.declare()
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line 105, in declare
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     self.exchange.declare()
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/entity.py", line 166, in declare
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     nowait=nowait, passive=passive,
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/channel.py", line 620, in exchange_declare
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     (40, 11),  # Channel.exchange_declare_ok
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     self.channel_id, allowed_methods)
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/connection.py", line 240, in _wait_method
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     self.method_reader.read_method()
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver   File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/method_framing.py", line 189, in read_method
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver     raise m
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver RuntimeError: maximum recursion depth exceeded while calling a Python object


In conductor I see the following (after adding a debug log message):
2015-01-07 12:06:50.860 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.2 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439
2015-01-07 12:06:50.889 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.3 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439
2015-01-07 12:06:50.914 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.2 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439
2015-01-07 12:06:50.937 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.3 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439
(repeats a bunch more times)

What I can’t figure out is why it’s asking for a target version of 1.3, icehouse service object is 1.2 and juno service object is 1.4

Anyone come across this?

Cheers,
Sam