<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Yeah I’m not sure what’s going on, I removed my hacks and tried it using the conductor rpcapi service and got what I think is a recursive call in nova-conductor.<div><br></div><div>Added more details to <a href="https://bugs.launchpad.net/nova/+bug/1308805">https://bugs.launchpad.net/nova/+bug/1308805</a></div><div><br></div><div>I’m thinking there maybe something missing in the stable/havana branch or else cells is doing something different when it comes to objects.</div><div>I don’t think it is a cells issue though as debugging it, it seems like it just can’t back port a 1.13 object to 1.9.</div><div><br></div><div>Cheers,</div><div>Sam</div><div><br></div><div><br><div><div>On 23 Apr 2014, at 1:01 am, Chris Behrens <<a href="mailto:cbehrens@codestud.com">cbehrens@codestud.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><meta http-equiv="Content-Type" content="text/html charset=windows-1252"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><br><div><div>On Apr 19, 2014, at 11:08 PM, Sam Morrison <<a href="mailto:sorrison@gmail.com">sorrison@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">Thanks for the info Chris, I’ve actually managed to get things working. Haven’t tested everything fully but seems to be working pretty good.<br><br>On 19 Apr 2014, at 7:26 am, Chris Behrens <<a href="mailto:cbehrens@codestud.com">cbehrens@codestud.com</a>> wrote:<br><br><blockquote type="cite">The problem here is that Havana is not going to know how to backport the Icehouse object, even if had the conductor methods to do so… unless you’re running the Icehouse conductor. But yes, your nova-computes would also need the code to understand to hit conductor to do the backport, which we must not have in Havana?<br></blockquote><br>OK this conductor api method was actually back ported to Havana, it kept it’s 1.62 version for the method but in Havana conductor manager it is set to 1.58.<br>That is easily fixed but then it gets worse. I may be missing something but the object_backport method doesn’t work at all and looking at the signature never worked?<br>I’ve raised a bug:<span class="Apple-converted-space"> </span><a href="https://bugs.launchpad.net/nova/+bug/1308805">https://bugs.launchpad.net/nova/+bug/1308805</a><br></div></blockquote><div><br></div><div>(CCing openstack-dev and Dan Smith)</div><div><br></div><div>That looked wrong to me as well, and then I talked with Dan Smith and he reminded me the RPC deserializer would turn that primitive into a an object on the conductor side. The primitive there is the full primitive we use to wrap the object with the versioning information, etc.</div><div><br></div><div>Does your backport happen to not pass the full object primitive? Or maybe missing the object RPC deserializer on conductor? (I would think that would have to be set in Havana) nova/service.py would have:</div><div><br></div><div><div>194 serializer = objects_base.NovaObjectSerializer()</div><div>195</div><div>196 self.rpcserver = rpc.get_server(target, endpoints, serializer)</div><div>197 self.rpcserver.start()</div><div><br></div><div>I’m guessing that’s there… so I would think maybe the object_backport call you have is not passing the full primitive.</div><div><br></div></div><div>I don’t have the time to peak at your code on github right this second, but maybe later. :)</div><div><br></div><div>- Chris</div><div><br></div><br><blockquote type="cite"><div style="font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br><blockquote type="cite">This also means that if you don’t want your computes on Icehouse yet, you must actually be using nova-conductor and not use_local=True for it. (I saw the patch go up to fix the objects use of conductor API… so I’m guessing you must be using local right now?)<br></blockquote><br>Yeah we still haven’t moved to use conductor so if you also don’t use conductor you’ll need the simple fix at bug:<span class="Apple-converted-space"> </span><a href="https://bugs.launchpad.net/nova/+bug/1308811">https://bugs.launchpad.net/nova/+bug/1308811</a><br><br><blockquote type="cite">So, I think an upgrade process could be:<br><br>1) Backport the ‘object backport’ code into Havana.<br>2) Set up *Icehouse* nova-conductor in your child cells and use_local=False on your nova-computes<br>3) Restart your nova-computes.<br>4) Update *all* nova-cells processes (in all cells) to Icehouse. You can keep use_local=False on these, but you’ll need that object conductor API patch.<br><br>At this point you’d have all nova-cells and all nova-conductors on Icehouse and everything else on Havana. If the Havana computes are able to talk to the Icehouse conductors, they should be able to backport any newer object versions. Same with nova-cells receiving older objects from nova-api. It should be able to backport them.<br><br>After this, you should be able to upgrade nova-api… and then probably upgrade your nova-computes on a cell-by-cell basis.<br><br>I don’t *think* nova-scheduler is getting objects yet, especially if you’re somehow magically able to get builds to work in what you tested so far. :) But if it is, you may find that you need to insert an upgrade of your nova-schedulers to Icehouse between steps 3 and 4 above…or maybe just after #4… so that it can backport objects, also.<br><br>I still doubt this will work 100%… but I dunno. :) And I could be missing something… but… I wonder if that makes sense?<br></blockquote><br>What I have is an Icehouse API cell and a Havana compute cell and havana compute nodes with the following changes:<br><br>Change the method signature of attach_volume to match icehouse, the additional arguments are optional and don’t seem to break things if you ignore them.<br><a href="https://bugs.launchpad.net/nova/+bug/1308846">https://bugs.launchpad.net/nova/+bug/1308846</a><br><br>Needed a small fix for unlocking, there is a race condition that I have a fix for but haven’t pushed up.<br><br>Then I hacked up a fix for object back porting.<br>The code is at<span class="Apple-converted-space"> </span><a href="https://github.com/NeCTAR-RC/nova/commits/nectar/havana-icehouse-compat">https://github.com/NeCTAR-RC/nova/commits/nectar/havana-icehouse-compat</a><br>The last three commits are the fixes needed.<span class="Apple-converted-space"> </span><br>I still need to push up the unlocking one and also a minor fix for metadata syncing with deleting and notifications.<br><br>Would love to get the object back porting stuff fixed properly from someone who knows how all the object stuff works.<br><br>Cheers,<br>Sam</div></blockquote></div><br></div></blockquote></div><br></div></body></html>