<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 6, 2015 at 5:12 PM, Sam Morrison <span dir="ltr"><<a href="mailto:sorrison@gmail.com" target="_blank">sorrison@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I’m working on upgrading our nova to Juno.<br>
<br>
First I upgrade the control infrastructure, conductor,api,scheduler etc. and migrate the DB<br>
I also set the following on the control hosts<br>
[upgrade_levels]<br>
compute=icehouse<br>
<br>
I don’t touch the nodes running nova-compute.<br>
<br>
Things seem to work however when I start a nova-compute I get the following when it registers its service:<br></blockquote><div><br></div><div>Are you restarting existing icehouse nova-compute and not juno?</div><div><br></div><div>Can you share sanitized copies of your config file as not sure what the issue is based on the stacktrace alone.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Line in question:<br>
nova/virt/libvirt/driver.py<br>
def _set_host_enabled()<br>
service = service_obj.Service.get_by_compute_host(ctx, CONF.host)<br>
<br>
<br>
2015-01-07 12:06:<a href="tel:51.377%2026596" value="+15137726596">51.377 26596</a> TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 155, in call<br>
2015-01-07 12:06:<a href="tel:51.377%2026596" value="+15137726596">51.377 26596</a> TRACE nova.virt.libvirt.driver return self.serializer.deserialize_entity(ctxt, result)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/rpc.py", line 111, in deserialize_entity<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self._base.deserialize_entity(context, entity)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/objects/base.py", line 575, in deserialize_entity<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver entity = self._process_object(context, entity)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/objects/base.py", line 545, in _process_object<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver e.kwargs['supported'])<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/conductor/api.py", line 280, in object_backport<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self._manager.object_backport(context, objinst, target_version)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/conductor/rpcapi.py", line 435, in object_backport<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver target_version=target_version)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 155, in call<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self.serializer.deserialize_entity(ctxt, result)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/rpc.py", line 111, in deserialize_entity<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self._base.deserialize_entity(context, entity)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/objects/base.py", line 575, in deserialize_entity<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver entity = self._process_object(context, entity)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/objects/base.py", line 545, in _process_object<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver e.kwargs['supported'])<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/conductor/api.py", line 280, in object_backport<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self._manager.object_backport(context, objinst, target_version)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/conductor/rpcapi.py", line 435, in object_backport<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver target_version=target_version)<br>
…. REPEATED….<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver retry=self.retry)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver timeout=timeout, retry=retry)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 436, in send<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver retry=retry)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 422, in _send<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver retry=retry)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 746, in topic_send<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver exchange_name=exchange_name, retry=retry)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 717, in publisher_send<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.ensure(_error_callback, _publish, retry=retry)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 617, in ensure<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver ret, channel = autoretry_method()<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return fun(*args, **kwargs)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/connection.py", line 508, in __call__<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return fun(*args, channel=channels[0], **kwargs), channels[0]<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 714, in _publish<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver publisher = cls(self.conf, channel, topic=topic, **kwargs)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 381, in __init__<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver **options)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 329, in __init__<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.reconnect(channel)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 337, in reconnect<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver routing_key=self.routing_key)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line 85, in __init__<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.revive(self._channel)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line 218, in revive<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.declare()<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line 105, in declare<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.exchange.declare()<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/entity.py", line 166, in declare<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver nowait=nowait, passive=passive,<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/channel.py", line 620, in exchange_declare<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver (40, 11), # Channel.exchange_declare_ok<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.channel_id, allowed_methods)<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/connection.py", line 240, in _wait_method<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.method_reader.read_method()<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/method_framing.py", line 189, in read_method<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver raise m<br>
2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver RuntimeError: maximum recursion depth exceeded while calling a Python object<br>
<br>
<br>
In conductor I see the following (after adding a debug log message):<br>
2015-01-07 12:06:50.860 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.2 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439<br>
2015-01-07 12:06:50.889 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.3 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439<br>
2015-01-07 12:06:50.914 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.2 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439<br>
2015-01-07 12:06:50.937 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.3 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439<br>
(repeats a bunch more times)<br>
<br>
What I can’t figure out is why it’s asking for a target version of 1.3, icehouse service object is 1.2 and juno service object is 1.4<br>
<br>
Anyone come across this?<br></blockquote><div><br></div><div>We haven't seen this in any other bug reports or in our automated testing.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Cheers,<br>
Sam<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
</blockquote></div><br></div></div>