[Openstack] [nova] running icehouse computes under juno
Sam Morrison
sorrison at gmail.com
Thu Jan 8 00:22:47 UTC 2015
Hi Joe,
I’ve managed to track this down. When Juno conductor sends back the service object, it also has an embedded compute node object.
This compute node object is too new for Icehouse and so it attempts to back port it.
However for some reason instead of asking to back port the compute node object it tries to back port the service object again at the intended compute node version.
This is where the loop is. There must be a bug when trying to back port embedded objects.
The fix is quite simple if you are using plain juno and icehouse as there is a bug in the service object [0], it has a check when back porting the service and downgrading the compute node but it changes it to the wrong version.
There is also a bug in the fixed_ip object with the embedded network object which means deletes don’t work. [1]
I have a fix for that too, looking at master the code around this has changed a lot so I don’t know if this has been fixed here.
I’m wondering if my patches can be applied straight to Juno? [2],[3]
Cheers,
Sam
[0] https://bugs.launchpad.net/nova/+bug/1408496 <https://bugs.launchpad.net/nova/+bug/1408496>
[1] https://bugs.launchpad.net/nova/+bug/1408498
[2] https://github.com/NeCTAR-RC/nova/commit/5c9b340b391cdbb8e3a97fc01258e59b827cb2a9
[3] https://github.com/NeCTAR-RC/nova/commit/cc20ae961578ede313add592ba81b215e9791816
> On 8 Jan 2015, at 4:47 am, Joe Gordon <joe.gordon0 at gmail.com> wrote:
>
>
>
> On Tue, Jan 6, 2015 at 5:12 PM, Sam Morrison <sorrison at gmail.com <mailto:sorrison at gmail.com>> wrote:
> I’m working on upgrading our nova to Juno.
>
> First I upgrade the control infrastructure, conductor,api,scheduler etc. and migrate the DB
> I also set the following on the control hosts
> [upgrade_levels]
> compute=icehouse
>
> I don’t touch the nodes running nova-compute.
>
> Things seem to work however when I start a nova-compute I get the following when it registers its service:
>
> Are you restarting existing icehouse nova-compute and not juno?
>
> Can you share sanitized copies of your config file as not sure what the issue is based on the stacktrace alone.
>
>
> Line in question:
> nova/virt/libvirt/driver.py
> def _set_host_enabled()
> service = service_obj.Service.get_by_compute_host(ctx, CONF.host)
>
>
> 2015-01-07 12:06:51.377 26596 <tel:51.377%2026596> TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 155, in call
> 2015-01-07 12:06:51.377 26596 <tel:51.377%2026596> TRACE nova.virt.libvirt.driver return self.serializer.deserialize_entity(ctxt, result)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/rpc.py", line 111, in deserialize_entity
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self._base.deserialize_entity(context, entity)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/objects/base.py", line 575, in deserialize_entity
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver entity = self._process_object(context, entity)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/objects/base.py", line 545, in _process_object
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver e.kwargs['supported'])
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/conductor/api.py", line 280, in object_backport
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self._manager.object_backport(context, objinst, target_version)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/conductor/rpcapi.py", line 435, in object_backport
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver target_version=target_version)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 155, in call
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self.serializer.deserialize_entity(ctxt, result)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/rpc.py", line 111, in deserialize_entity
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self._base.deserialize_entity(context, entity)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/objects/base.py", line 575, in deserialize_entity
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver entity = self._process_object(context, entity)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/objects/base.py", line 545, in _process_object
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver e.kwargs['supported'])
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/conductor/api.py", line 280, in object_backport
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return self._manager.object_backport(context, objinst, target_version)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/nova/nova/conductor/rpcapi.py", line 435, in object_backport
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver target_version=target_version)
> …. REPEATED….
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 152, in call
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver retry=self.retry)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver timeout=timeout, retry=retry)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 436, in send
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver retry=retry)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 422, in _send
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver retry=retry)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 746, in topic_send
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver exchange_name=exchange_name, retry=retry)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 717, in publisher_send
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.ensure(_error_callback, _publish, retry=retry)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 617, in ensure
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver ret, channel = autoretry_method()
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return fun(*args, **kwargs)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/connection.py", line 508, in __call__
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver return fun(*args, channel=channels[0], **kwargs), channels[0]
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 714, in _publish
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver publisher = cls(self.conf, channel, topic=topic, **kwargs)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 381, in __init__
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver **options)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 329, in __init__
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.reconnect(channel)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 337, in reconnect
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver routing_key=self.routing_key)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line 85, in __init__
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.revive(self._channel)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line 218, in revive
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.declare()
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/messaging.py", line 105, in declare
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.exchange.declare()
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/kombu/entity.py", line 166, in declare
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver nowait=nowait, passive=passive,
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/channel.py", line 620, in exchange_declare
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver (40, 11), # Channel.exchange_declare_ok
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.channel_id, allowed_methods)
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/connection.py", line 240, in _wait_method
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver self.method_reader.read_method()
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver File "/opt/icehouse/local/lib/python2.7/site-packages/amqp/method_framing.py", line 189, in read_method
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver raise m
> 2015-01-07 12:06:51.377 26596 TRACE nova.virt.libvirt.driver RuntimeError: maximum recursion depth exceeded while calling a Python object
>
>
> In conductor I see the following (after adding a debug log message):
> 2015-01-07 12:06:50.860 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.2 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439
> 2015-01-07 12:06:50.889 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.3 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439
> 2015-01-07 12:06:50.914 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.2 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439
> 2015-01-07 12:06:50.937 9781 DEBUG nova.conductor.manager [req-2fffdbd0-fbcd-426d-adc5-9f02e1104652 None] Call to backport Service(availability_zone=<?>,binary='nova-compute',compute_node=ComputeNode(2),created_at=2015-01-06T03:33:50Z,deleted=False,deleted_at=None,disabled=False,disabled_reason=None,host='nova-compute-i2',id=8,report_count=6911,topic='compute',updated_at=2015-01-07T01:06:10Z), target_version=1.3 object_backport /opt/juno/local/lib/python2.7/site-packages/nova/conductor/manager.py:439
> (repeats a bunch more times)
>
> What I can’t figure out is why it’s asking for a target version of 1.3, icehouse service object is 1.2 and juno service object is 1.4
>
> Anyone come across this?
>
> We haven't seen this in any other bug reports or in our automated testing.
>
>
> Cheers,
> Sam
>
>
>
>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack>
> Post to : openstack at lists.openstack.org <mailto:openstack at lists.openstack.org>
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150108/52d30385/attachment.html>
More information about the Openstack
mailing list