Open Stack

On 11/07/2012 11:47 AM, Aniruddha Khadkikar wrote:
> Hi Stackers,
>
> We have a small Openstack lab using three servers. The components are
> distributed as:
> 1. Network controller - Quantum L3&  DHCP, L2 agent, Nova, Openvswitch
> 2. Cloud controller - Quantum server, L2 agent, Nova, Openvswitch,
> Dashboard, API, MySQL, Rabbitmq
> 3. Compute node - Nova, Openvswitch, L2 agent
>
> The network is setup in the following way:
> 1. Each server has 4 nics. We are using only one public IP and one
> private IP for the openstack setup. We have a private switch for
> inter-vm communication
> 2. We are using gre tunnelling and openvswitch
> 3. br-int is assigned an IP address
> 4. br-ex is configured for floating IP allocation
>
> Everything works perfectly when we are setting it up from scratch!!!!
>
> Each vm is able to get the private IP's assigned and the NAT based
> floating IP is also assigned and we are able to SSH into it.
> The VM's also get created on all the three hosts.
>
> So we are confident that we have the right configurations in place as
> we have fully operational Openstack implementation using gre-tunnels.
>
> In order to test the resilience of the setup, we decided to reboot the
> servers to see if everything comes up again. We faced some dependency
> of services errors and after server reboot we restarted the services
> in the proper order i.e. on cloud controller we have mysql, rabbitmq,
> keystone, openvswitch and quantum-server started. This was followed by
> starting openvswitch, L3, dhcp and L2 agent. After which we started L2
> agents on all the remaining servers and followed by nova. There is
> some confusion on how to orchestrate the right order of services. This
> could possibly be something we will need to work upon in future.
>
> After this, we have nova working properly i.e. we are able to create
> vm's and the pre-existing ones are also started (virsh list also shows
> the vm's). ovsctl shows all the interfaces as earlier. However we are
> unable to access the vm's. On logging into the vm we do not see any IP
> address being assigned as the VM is unable to contact the dhcp server.
>
> The questions that come up are:
> * What could change after a reboot that would compromise a running
> network configuration?
> * Could there be issues with the TAP interfaces created? What is the
> best way to troubleshoot such a situation?
> * Has anyone seen a similar behaviour and is it specific to when we
> use gre-tunnels? Is it then specific to openvswitch which we are
> using?
> * On reboot of the network controller are any steps required to ensure
> that Openstack continues to function properly?

Can you please look in the log files for Quantum and see if there are 
any errors?

There is an open issue with Quantum and QPID after rebooting - the 
Quantum service hangs? On the host for Quantum is you do "netstat -an 
|grep 9696" do you see anything?

>
> The setup has failed twice on reboot. For the second iteration we are
> assigning the IP on startup to br-int so that openvswitch does not
> give errors.
>
> Regards
> Aniruddha
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp