<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Joe,<div class=""><br class=""></div><div class="">I’ve managed to track this down. When Juno conductor sends back the service object, it also has an embedded compute node object. </div><div class="">This compute node object is too new for Icehouse and so it attempts to back port it.</div><div class=""><br class=""></div><div class="">However for some reason instead of asking to back port the compute node object it tries to back port the service object again at the intended compute node version.</div><div class="">This is where the loop is. There must be a bug when trying to back port embedded objects.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">The fix is quite simple if you are using plain juno and icehouse as there is a bug in the service object [0], it has a check when back porting the service and downgrading the compute node but it changes it to the wrong version.</div><div class=""><br class=""></div><div class="">There is also a bug in the fixed_ip object with the embedded network object which means deletes don’t work. [1]</div><div class=""><br class=""></div><div class="">I have a fix for that too, looking at master the code around this has changed a lot so I don’t know if this has been fixed here.</div><div class="">I’m wondering if my patches can be applied straight to Juno? [2],[3]</div><div class=""><br class=""></div><div class="">Cheers,</div><div class="">Sam</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">[0] <a href="https://bugs.launchpad.net/nova/+bug/1408496" class="">https://bugs.launchpad.net/nova/+bug/1408496</a></div><div class="">[1] <a href="https://bugs.launchpad.net/nova/+bug/1408498" class="">https://bugs.launchpad.net/nova/+bug/1408498</a></div><div class="">[2] <a href="https://github.com/NeCTAR-RC/nova/commit/5c9b340b391cdbb8e3a97fc01258e59b827cb2a9" class="">https://github.com/NeCTAR-RC/nova/commit/5c9b340b391cdbb8e3a97fc01258e59b827cb2a9</a></div><div class="">[3] <a href="https://github.com/NeCTAR-RC/nova/commit/cc20ae961578ede313add592ba81b215e9791816" class="">https://github.com/NeCTAR-RC/nova/commit/cc20ae961578ede313add592ba81b215e9791816</a></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 8 Jan 2015, at 4:47 am, Joe Gordon <<a href="mailto:joe.gordon0@gmail.com" class="">joe.gordon0@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><br class="Apple-interchange-newline"><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">On Tue, Jan 6, 2015 at 5:12 PM, Sam Morrison<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:sorrison@gmail.com" target="_blank" class="">sorrison@gmail.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">I’m working on upgrading our nova to Juno.<br class=""><br class="">First I upgrade the control infrastructure, conductor,api,scheduler etc. and migrate the DB<br class="">I also set the following on the control hosts<br class="">[upgrade_levels]<br class="">compute=icehouse<br class=""><br class="">I don’t touch the nodes running nova-compute.<br class=""><br class="">Things seem to work however when I start a nova-compute I get the following when it registers its service:<br class=""></blockquote><div class=""><br class=""></div><div class="">Are you restarting existing icehouse nova-compute and not juno?</div><div class=""><br class=""></div><div class="">Can you share sanitized copies of your config file as not sure what the issue is based on the stacktrace alone.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><br class="">Line in question:<br class="">nova/virt/libvirt/driver.py<br class="">def _set_host_enabled()<br class="">service = service_obj.Service.get_by_compute_host(ctx, CONF.host)<br class=""><br class=""><br class="">2015-01-07 12:06:<a href="tel:51.377%2026596" value="+15137726596" class="">51.377 26596</a><span class="Apple-converted-space"> </span>TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 155, in call<br class="">2015-01-07 12:06:<a href="tel:51.377%2026596" value="+15137726596" class="">51.377 26596</a><span class="Apple-converted-space"> </span>TRACE nova.virt.libvirt.driver return self.serializer.deserialize_entity(ctxt, result)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/rpc.py", line 111, in deserialize_entity<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self._base.deserialize_entity(context, entity)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/objects/base.py", line 575, in deserialize_entity<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver entity = self._process_object(context, entity)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/objects/base.py", line 545, in _process_object<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver e.kwargs['supported'])<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/conductor/api.py", line 280, in object_backport<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self._manager.object_backport(context, objinst, target_version)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/conductor/rpcapi.py", line 435, in object_backport<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver target_version=target_version)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 155, in call<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self.serializer.deserialize_entity(ctxt, result)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/rpc.py", line 111, in deserialize_entity<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self._base.deserialize_entity(context, entity)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/objects/base.py", line 575, in deserialize_entity<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver entity = self._process_object(context, entity)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/objects/base.py", line 545, in _process_object<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver e.kwargs['supported'])<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/conductor/api.py", line 280, in object_backport<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self._manager.object_backport(context, objinst, target_version)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/conductor/rpcapi.py", line 435, in object_backport<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver target_version=target_version)<br class="">…. REPEATED….<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver retry=self.retry)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver timeout=timeout, retry=retry)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 436, in send<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver retry=retry)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 422, in _send<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver retry=retry)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 746, in topic_send<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver exchange_name=exchange_name, retry=retry)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 717, in publisher_send<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.ensure(_error_callback, _publish, retry=retry)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 617, in ensure<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver ret, channel = autoretry_method()<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return fun(*args, **kwargs)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/connection.py", line 508, in __call__<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return fun(*args, channel=channels[0], **kwargs), channels[0]<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 714, in _publish<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver publisher = cls(self.conf, channel, topic=topic, **kwargs)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 381, in __init__<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver **options)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 329, in __init__<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.reconnect(channel)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 337, in reconnect<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver routing_key=self.routing_key)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line 85, in __init__<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.revive(self._channel)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line 218, in revive<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.declare()<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line 105, in declare<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.exchange.declare()<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/entity.py", line 166, in declare<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver nowait=nowait, passive=passive,<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/channel.py", line 620, in exchange_declare<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver (40, 11), # Channel.exchange_declare_ok<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.channel_id, allowed_methods)<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/connection.py", line 240, in _wait_method<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.method_reader.read_method()<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/method_framing.py", line 189, in read_method<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver raise m<br class="">2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver RuntimeError: maximum recursion depth exceeded while calling a Python object<br class=""><br class=""><br class="">In conductor I see the following (after adding a debug log message):<br class="">2015-01-07 12:06:50.860 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.2 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439<br class="">2015-01-07 12:06:50.889 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.3 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439<br class="">2015-01-07 12:06:50.914 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.2 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439<br class="">2015-01-07 12:06:50.937 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.3 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439<br class="">(repeats a bunch more times)<br class=""><br class="">What I can’t figure out is why it’s asking for a target version of 1.3, icehouse service object is 1.2 and juno service object is 1.4<br class=""><br class="">Anyone come across this?<br class=""></blockquote><div class=""><br class=""></div><div class="">We haven't seen this in any other bug reports or in our automated testing.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><br class="">Cheers,<br class="">Sam<br class=""><br class=""><br class=""><br class=""><br class=""><br class="">_______________________________________________<br class="">Mailing list:<span class="Apple-converted-space"> </span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br class="">Post to :<span class="Apple-converted-space"> </span><a href="mailto:openstack@lists.openstack.org" class="">openstack@lists.openstack.org</a><br class="">Unsubscribe :<span class="Apple-converted-space"> </span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a></blockquote></div></div></blockquote></div><br class=""></div></body></html>