[Openstack] Upgrade nova-network without downtime?

Tom Fifield tom at openstack.org
Tue Aug 4 01:51:38 UTC 2015


On 03/08/15 22:17, Andrew Bogott wrote:
> On 7/28/15 1:32 PM, Andrew Bogott wrote:
>> I'm running Icehouse with the 'legacy' nova-network service and a
>> single network node.
>>
>> Everything in my cluster is running Ubuntu Trusty and ready for an
>> upgrade to Juno, except for my network node, which is still running
>> Precise.
>> <snip>
> It turns out that I now need to do rolling restarts of my compute nodes
> for an unrelated issue (see: "kvm suspend vs. the kernel") so I have a
> bit more leeway with this problem.  After a bit of reading I'm convinced
> that while I'm messing with things it's probably a good idea to switch
> to multi_host anyway.  So, I propose the following steps:
>
> 1) Install nova-network on all compute nodes
> 2) nova-network delete (current network setting)
> 3) nova-network create (same network settings as in step 2 except with
> multi_host on)
> 4) restart nova-network on all compute nodes
>
> My supposition is that after doing that, old instances will continue to
> use my old, standalone network node (as specified in their cached dhcp
> settings) until they are restarted at which point they'll switch over to
> using their local compute host as a router thanks to the multi_host
> setting.
>
> Can anyone confirm that this is a sensible plan, or warn me off of it?
> The main danger I see is that at that after step 2, nova-compute
> probably can't be safely restarted on my original compute node -- that's
> definitely a bit fragile :(

Seems sane-ish. Though, the delete/create step will probably change a 
network ID, which could cause badness - you might want to consider 
manually updating the field in the DB instead.

Check also your DHCP timeout values - you can be lulled into a false 
sense of security that things are working, then dhcp expires and all 
hell breaks loose.

I'm assuming you were going to test this before touching prod, of course 
:) A simple test plan would be a 3 node test cluster - two computes and 
a network node. Set it up as you have your original cluster, using the 
network node. Start a few VMs, then try your steps :)

Regards,


Tom




More information about the Openstack mailing list