[Openstack-docs] [Openstack] After expanding fixed ip range instances no longer have network

Jonathan Proulx jon at jonproulx.com
Mon Sep 10 13:15:52 UTC 2012


Not sure if Disaster recovery is the right place, in a disaster scenario
you'd be starting fresh so the issues are likely a bit different.  I should
really do a DR drill to see what they are (and to see that I can recover
from a disaster) but it should go somewhere, perhaps in the nascent
operations manual?

Couple things that didn't make it onto the list

The new network (obviously) gets a new network ID, when initially starting
testing after deleting the old network and creating  the new one instances
we failing to launch because something was looking for a network with ID
1.  This was before I'd discovered dnsmasq needed to be killed by hand so
that might have been it (nova-network and nova-compute had been restarted),
but not really sure.  What I did (which worked but may not be the best
thing) was to delete the old network from the database network table,
delete all the fixed IPs from the fixed_ips table that had that network_id,
update the id of the new network to be 1 then update the network_id of all
the fixed_ips that had the new network_id to be 1. This actually replaces
the old network with the new one, but looses the history of the old one.

Also despite cleanly stopping all running instances before messing with the
networks some floating IP we still associated with old fixed ip's.  Again
not sure why this was a problem since the new network was a superset of the
old.  The symptom was tenants that had IPs in this state could not
enumerate floating IPs  'nova floating-ip-list' and equivalent API calls
(which breaks Horizon in obscure ways).  My quick and dirty fix was just to
clear all those associations since no instances were running and no
floating ips we in use:

 mysql> update floating_ips set updated_at=now(),fixed_ip_id=NULL,host=NULL
where fixed_ip_id is not NULL;

-Jon

On Mon, Sep 10, 2012 at 3:43 AM, Razique Mahroua
<razique.mahroua at gmail.com>wrote:

> I'd add the same steps apply for vlan mode,
> though after you tore down the bridge, you need then to remove the vlan :
> $ vconfig rem vlan100
>
> Razique
>
> *Nuage & Co - Razique Mahrou*
> razique.mahroua at gmail.com
>
>
> Le 10 sept. 2012 à 01:31, Tom Fifield <fifieldt at unimelb.edu.au> a écrit :
>
> This might be useful for the disaster recovery section
>
>
> -------- Original Message --------
> Subject: Re: [Openstack] After expanding fixed ip range instances no
> longer have network
> Date: Fri, 7 Sep 2012 09:12:02 -0700
> From: Vishvananda Ishaya <vishvananda at gmail.com>
> To: Jonathan Proulx <jon at jonproulx.com>
> CC: <openstack at lists.launchpad.net>
>
>
> VERY IMPORTANT FIRST STEP: nova will have moved the ip from eth0 (or
> whichever device br100 is on) to br100, so if that ip address is needed,
> you will have to move it back to eth0. If you are connecting over that ip
> you will have to do it in a script so your connection doesn't drop. Check
> the <ip>/<netmask> by using ip addr show.
>
> DISCLAIMER: writing this from memory so check for typos
> (if br100 was bridged into a device without an ip address you can skip
> moving the ips but i would still delete br100)
>
> On each node running nova-network:
> ip addr del <ip>/<netmask> dev br100
> ifconfig br100 down
> brctl delbr br100
> ip addr add <ip>/<netmask> dev eth0
>
>
> Now you should be able to make nova recreate everything for you
> On each node running nova-network:
> stop nova-network
> killall dnsmasq
> rm /var/lib/nova/networks/*
> start nova-network
>
> Vish
>
> On Sep 7, 2012, at 8:33 AM, Jonathan Proulx <jon at jonproulx.com> wrote:
>
> Hi All,
>
> Running Essex on Ununtu 12.04 using multi-host  FlatDHCP nova-networking
>
> I ran out of IPs on my fixed_ip range so I shut everything (instances,
> nova-network, nova-compute) down deleted the old network and recreated
> it with a smaller netmask.  This seems to have almost worked.
>
> I can start more instances than I previously had fixed ip's, the right
> ones seem to be being assigned and the mask on the recreated bridge
> interfaces is correct, but the (ubuntu-cloudimage) instances can't
> seem to see their NIC's any more, or perhaps aren't getting dhcp
> properly I'm still trying to force my way in as our instances rather
> rely on net access for accounts.
>
> On the compute node things seem OK the bridge is up and the right
> things are connected:
>
> root at nova-5:/var/log/nova# brctl show br100
> bridge name bridge id STP enabled interfaces
> br100 8000.60eb69d22521 no eth1
>    vnet0
>    vnet1
>    vnet2
>
>
> I do notice this in iptables:
>
> Chain nova-network-POSTROUTING (1 references)
> target     prot opt source               destination
> ACCEPT     all  --  10.0.0.0/16          nova-5.csail.mit.edu
> ACCEPT     all  --  10.0.0.0/16          10.128.0.0/24
> ACCEPT     all  --  10.0.0.0/16          10.0.0.0/16          ! ctstate
> DNAT
>
>
> my fixed range is 10.0.0.0/16 not sure where 10.128.0.0/24 comes into
> it as I don't use that network, but can't see that as a problem.
>
> Can any one tell me what I've looked?
>
> -Jon
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
>
>
> _______________________________________________
> Openstack-docs mailing list
> Openstack-docs at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs
>
>
>
> _______________________________________________
> Openstack-docs mailing list
> Openstack-docs at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-docs/attachments/20120910/1b6788d7/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: NUAGECO-LOGO-Fblan_petit.jpg
Type: image/jpeg
Size: 10122 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-docs/attachments/20120910/1b6788d7/attachment-0001.jpg>


More information about the Openstack-docs mailing list