Open Stack

Yeah I’m not sure what’s going on, I removed my hacks and tried it using the conductor rpcapi service and got what I think is a recursive call in nova-conductor.

Added more details to https://bugs.launchpad.net/nova/+bug/1308805

I’m thinking there maybe something missing in the stable/havana branch or else cells is doing something different when it comes to objects.
I don’t think it is a cells issue though as debugging it, it seems like it just can’t back port a 1.13 object to 1.9.

Cheers,
Sam


On 23 Apr 2014, at 1:01 am, Chris Behrens <cbehrens at codestud.com> wrote:

> 
> On Apr 19, 2014, at 11:08 PM, Sam Morrison <sorrison at gmail.com> wrote:
> 
>> Thanks for the info Chris, I’ve actually managed to get things working. Haven’t tested everything fully but seems to be working pretty good.
>> 
>> On 19 Apr 2014, at 7:26 am, Chris Behrens <cbehrens at codestud.com> wrote:
>> 
>>> The problem here is that Havana is not going to know how to backport the Icehouse object, even if had the conductor methods to do so… unless you’re running the Icehouse conductor. But yes, your nova-computes would also need the code to understand to hit conductor to do the backport, which we must not have in Havana?
>> 
>> OK this conductor api method was actually back ported to Havana, it kept it’s 1.62 version for the method but in Havana conductor manager it is set to 1.58.
>> That is easily fixed but then it gets worse. I may be missing something but the object_backport method doesn’t work at all and looking at the signature never worked?
>> I’ve raised a bug: https://bugs.launchpad.net/nova/+bug/1308805
> 
> (CCing openstack-dev and Dan Smith)
> 
> That looked wrong to me as well, and then I talked with Dan Smith and he reminded me the RPC deserializer would turn that primitive into a an object on the conductor side. The primitive there is the full primitive we use to wrap the object with the versioning information, etc.
> 
> Does your backport happen to not pass the full object primitive?  Or maybe missing the object RPC deserializer on conductor? (I would think that would have to be set in Havana)  nova/service.py would have:
> 
> 194         serializer = objects_base.NovaObjectSerializer()
> 195
> 196         self.rpcserver = rpc.get_server(target, endpoints, serializer)
> 197         self.rpcserver.start()
> 
> I’m guessing that’s there… so I would think maybe the object_backport call you have is not passing the full primitive.
> 
> I don’t have the time to peak at your code on github right this second, but maybe later. :)
> 
> - Chris
> 
> 
>> 
>>> This also means that if you don’t want your computes on Icehouse yet, you must actually be using nova-conductor and not use_local=True for it. (I saw the patch go up to fix the objects use of conductor API… so I’m guessing you must be using local right now?)
>> 
>> Yeah we still haven’t moved to use conductor so if you also don’t use conductor you’ll need the simple fix at bug: https://bugs.launchpad.net/nova/+bug/1308811
>> 
>>> So, I think an upgrade process could be:
>>> 
>>> 1) Backport the ‘object backport’ code into Havana.
>>> 2) Set up *Icehouse* nova-conductor in your child cells and use_local=False on your nova-computes
>>> 3) Restart your nova-computes.
>>> 4) Update *all* nova-cells processes (in all cells) to Icehouse. You can keep use_local=False on these, but you’ll need that object conductor API patch.
>>> 
>>> At this point you’d have all nova-cells and all nova-conductors on Icehouse and everything else on Havana. If the Havana computes are able to talk to the Icehouse conductors, they should be able to backport any newer object versions. Same with nova-cells receiving older objects from nova-api. It should be able to backport them.
>>> 
>>> After this, you should be able to upgrade nova-api… and then probably upgrade your nova-computes on a cell-by-cell basis.
>>> 
>>> I don’t *think* nova-scheduler is getting objects yet, especially if you’re somehow magically able to get builds to work in what you tested so far. :) But if it is, you may find that you need to insert an upgrade of your nova-schedulers to Icehouse between steps 3 and 4 above…or maybe just after #4… so that it can backport objects, also.
>>> 
>>> I still doubt this will work 100%… but I dunno. :)  And I could be missing something… but… I wonder if that makes sense?
>> 
>> What I have is an Icehouse API cell and a Havana compute cell and havana compute nodes with the following changes:
>> 
>> Change the method signature of attach_volume to match icehouse, the additional arguments are optional and don’t seem to break things if you ignore them.
>> https://bugs.launchpad.net/nova/+bug/1308846
>> 
>> Needed a small fix for unlocking, there is a race condition that I have a fix for but haven’t pushed up.
>> 
>> Then I hacked up a fix for object back porting.
>> The code is at https://github.com/NeCTAR-RC/nova/commits/nectar/havana-icehouse-compat
>> The last three commits are the fixes needed. 
>> I still need to push up the unlocking one and also a minor fix for metadata syncing with deleting and notifications.
>> 
>> Would love to get the object back porting stuff fixed properly from someone who knows how all the object stuff works.
>> 
>> Cheers,
>> Sam
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140424/b8cb14e7/attachment.html>