Not sure if Disaster recovery is the right place, in a disaster scenario you'd be starting fresh so the issues are likely a bit different. I should really do a DR drill to see what they are (and to see that I can recover from a disaster) but it should go somewhere, perhaps in the nascent operations manual?<br>
<br>Couple things that didn't make it onto the list <br><br>The new network (obviously) gets a new network ID, when initially starting testing after deleting the old network and creating the new one instances we failing to launch because something was looking for a network with ID 1. This was before I'd discovered dnsmasq needed to be killed by hand so that might have been it (nova-network and nova-compute had been restarted), but not really sure. What I did (which worked but may not be the best thing) was to delete the old network from the database network table, delete all the fixed IPs from the fixed_ips table that had that network_id, update the id of the new network to be 1 then update the network_id of all the fixed_ips that had the new network_id to be 1. This actually replaces the old network with the new one, but looses the history of the old one.<br>
<br>Also despite cleanly stopping all running instances before messing with the networks some floating IP we still associated with old fixed ip's. Again not sure why this was a problem since the new network was a superset of the old. The symptom was tenants that had IPs in this state could not enumerate floating IPs 'nova floating-ip-list' and equivalent API calls (which breaks Horizon in obscure ways). My quick and dirty fix was just to clear all those associations since no instances were running and no floating ips we in use:<br>
<br> mysql> update floating_ips set updated_at=now(),fixed_ip_id=NULL,host=NULL where fixed_ip_id is not NULL;<br><br>-Jon<br><br><div class="gmail_quote">On Mon, Sep 10, 2012 at 3:43 AM, Razique Mahroua <span dir="ltr"><<a href="mailto:razique.mahroua@gmail.com" target="_blank">razique.mahroua@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">I'd add the same steps apply for vlan mode,<div>though after you tore down the bridge, you need then to remove the vlan : </div>
<div><font face="Consolas">$ vconfig rem vlan100</font></div><div><br></div><div>Razique</div><div><br><div>
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:'Lucida Grande';word-spacing:0px"><span style="font-family:Helvetica"><span style="font-family:Helvetica"><b style="color:rgb(19,112,138)">Nuage & Co - Razique Mahrou</b></span><span style="color:rgb(19,112,138);font-family:Helvetica"></span><span style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:medium"><span style="border-collapse:separate;font-variant:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="border-collapse:separate;font-variant:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="font-style:normal;font-size:medium;font-family:Helvetica;font-weight:normal">
<font color="#13708a"><a href="mailto:razique.mahroua@gmail.com" target="_blank">razique.mahroua@gmail.com</a></font></div></span><br><span></span><span style="font-style:normal;font-size:medium;font-family:Helvetica;font-weight:normal"></span><span style="font-weight:normal"></span></span><span><img src="cid:00E07450-433E-43E6-9005-48B8FD5C669F@la.fabrique" height="125" width="125"></span>
</span></span></span></div>
<br><div><div>Le 10 sept. 2012 à 01:31, Tom Fifield <<a href="mailto:fifieldt@unimelb.edu.au" target="_blank">fifieldt@unimelb.edu.au</a>> a écrit :</div><div><div class="h5"><br><blockquote type="cite">This might be useful for the disaster recovery section<br>
<br><br>-------- Original Message --------<br>Subject: Re: [Openstack] After expanding fixed ip range instances no longer have network<br>Date: Fri, 7 Sep 2012 09:12:02 -0700<br>From: Vishvananda Ishaya <<a href="mailto:vishvananda@gmail.com" target="_blank">vishvananda@gmail.com</a>><br>
To: Jonathan Proulx <<a href="mailto:jon@jonproulx.com" target="_blank">jon@jonproulx.com</a>><br>CC: <<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>><br><br><br>
VERY IMPORTANT FIRST STEP: nova will have moved the ip from eth0 (or whichever device br100 is on) to br100, so if that ip address is needed, you will have to move it back to eth0. If you are connecting over that ip you will have to do it in a script so your connection doesn't drop. Check the <ip>/<netmask> by using ip addr show.<br>
<br>DISCLAIMER: writing this from memory so check for typos<br>(if br100 was bridged into a device without an ip address you can skip moving the ips but i would still delete br100)<br><br>On each node running nova-network:<br>
ip addr del <ip>/<netmask> dev br100<br>ifconfig br100 down<br>brctl delbr br100<br>ip addr add <ip>/<netmask> dev eth0<br><br><br>Now you should be able to make nova recreate everything for you<br>
On each node running nova-network:<br>stop nova-network<br>killall dnsmasq<br>rm /var/lib/nova/networks/*<br>start nova-network<br><br>Vish<br><br>On Sep 7, 2012, at 8:33 AM, Jonathan Proulx <<a href="mailto:jon@jonproulx.com" target="_blank">jon@jonproulx.com</a>> wrote:<br>
<br><blockquote type="cite">Hi All,<br><br>Running Essex on Ununtu 12.04 using multi-host FlatDHCP nova-networking<br><br>I ran out of IPs on my fixed_ip range so I shut everything (instances,<br>nova-network, nova-compute) down deleted the old network and recreated<br>
it with a smaller netmask. This seems to have almost worked.<br><br>I can start more instances than I previously had fixed ip's, the right<br>ones seem to be being assigned and the mask on the recreated bridge<br>interfaces is correct, but the (ubuntu-cloudimage) instances can't<br>
seem to see their NIC's any more, or perhaps aren't getting dhcp<br>properly I'm still trying to force my way in as our instances rather<br>rely on net access for accounts.<br><br>On the compute node things seem OK the bridge is up and the right<br>
things are connected:<br><br>root@nova-5:/var/log/nova# brctl show br100<br>bridge name<span style="white-space:pre-wrap"> </span>bridge id<span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span>STP enabled<span style="white-space:pre-wrap"> </span>interfaces<br>
br100<span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span>8000.60eb69d22521<span style="white-space:pre-wrap"> </span>no<span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span>eth1<br>
<span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span>vnet0<br>
<span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span>vnet1<br>
<span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span><span style="white-space:pre-wrap"> </span>vnet2<br>
<br><br>I do notice this in iptables:<br><br>Chain nova-network-POSTROUTING (1 references)<br>target prot opt source destination<br>ACCEPT all -- <a href="http://10.0.0.0/16" target="_blank">10.0.0.0/16</a> <a href="http://nova-5.csail.mit.edu" target="_blank">nova-5.csail.mit.edu</a><br>
ACCEPT all -- <a href="http://10.0.0.0/16" target="_blank">10.0.0.0/16</a> <a href="http://10.128.0.0/24" target="_blank">10.128.0.0/24</a><br>ACCEPT all -- <a href="http://10.0.0.0/16" target="_blank">10.0.0.0/16</a> <a href="http://10.0.0.0/16" target="_blank">10.0.0.0/16</a> ! ctstate DNAT<br>
<br><br>my fixed range is <a href="http://10.0.0.0/16" target="_blank">10.0.0.0/16</a> not sure where <a href="http://10.128.0.0/24" target="_blank">10.128.0.0/24</a> comes into<br>it as I don't use that network, but can't see that as a problem.<br>
<br>Can any one tell me what I've looked?<br><br>-Jon<br><br>_______________________________________________<br>Mailing list: <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>Unsubscribe : <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br></blockquote><br><br>_______________________________________________<br>Mailing list: <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>Unsubscribe : <a href="https://launchpad.net/%7Eopenstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br><br><br><br><br>_______________________________________________<br>Openstack-docs mailing list<br><a href="mailto:Openstack-docs@lists.openstack.org" target="_blank">Openstack-docs@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs</a><br></blockquote></div></div></div><br></div></div><br>_______________________________________________<br>
Openstack-docs mailing list<br>
<a href="mailto:Openstack-docs@lists.openstack.org">Openstack-docs@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs</a><br>
<br></blockquote></div><br>