Open Stack

Hello,
I'm running OpenStack RDO with the following setup.

- 1x Controller Node ( Neutron - Nova -   glance - Horizon - Keystone - GRE
+   FlatDHCPManager)
-  10x Compute Nodes ( Nova Compute  - KVM )
- OS : CentOS 6 - 64bit

Suddenly all vm's on the compute nodes can't get new IP address from the
controller node. It was working properly hours ago without any
modifications from me. This log from the controller node - **many errors**
 http://paste.openstack.org/show/88016/

And this from the compute node
 http://paste.openstack.org/show/88017/

Installed packages on Controller Node :
# rpm -qa | grep -i openstack

    openstack-selinux-0.1.3-2.el6ost.noarch
    openstack-puppet-modules-2013.2-9.1.el6.noarch
    openstack-ceilometer-api-2013.2.3-2.el6.noarch
    openstack-packstack-2013.2.1-0.36.dev1013.el6.noarch
    openstack-nova-scheduler-2013.2.3-1.el6.noarch
    openstack-ceilometer-common-2013.2.3-2.el6.noarch
    python-django-openstack-auth-1.1.2-1.el6.noarch
    openstack-ceilometer-central-2013.2.3-2.el6.noarch
    openstack-ceilometer-collector-2013.2.3-2.el6.noarch
    openstack-neutron-openvswitch-2013.2.3-9.el6.noarch
    openstack-nova-common-2013.2.3-1.el6.noarch
    openstack-packstack-puppet-2013.2.1-0.36.dev1013.el6.noarch
    openstack-glance-2013.2.3-2.el6.noarch
    openstack-nova-conductor-2013.2.3-1.el6.noarch
    openstack-nova-novncproxy-2013.2.3-1.el6.noarch
    openstack-nova-cert-2013.2.3-1.el6.noarch
    openstack-keystone-2013.2.3-3.el6.noarch
    openstack-neutron-2013.2.3-9.el6.noarch
    openstack-ceilometer-alarm-2013.2.3-2.el6.noarch
    openstack-dashboard-2013.2.3-1.el6.noarch
    openstack-nova-api-2013.2.3-1.el6.noarch
    openstack-nova-console-2013.2.3-1.el6.noarch
    openstack-utils-2013.2-2.el6.noarch

On Controller Node
 # ps aux | grep -i dhcp
    nobody   12639  0.0  0.0  12884   864 ?        S    Jul19   0:00
dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces
--interface=tap7139b265-41 --except-interface=lo
--pid-file=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/pid
--dhcp-hostsfile=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/host
--dhcp-optsfile=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/opts
--leasefile-ro --dhcp-range=tag0,10.0.0.0,static,86400s
--dhcp-lease-max=256 --conf-file= --domain=openstacklocal
    neutron  24884  0.0  0.1 273748 32140 ?        S    Jul24   0:00
/usr/bin/python /usr/bin/neutron-dhcp-agent --log-file
/var/log/neutron/dhcp-agent.log --config-file
/usr/share/neutron/neutron-dist.conf --config-file
/etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini

# ip netns
    qrouter-id
    qdhcp-id

# ip netns exec qdhcp-network-id ip a
    16: tap7139b265-41: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UNKNOWN
        link/ether fa:16:3e:2e:18:35 brd ff:ff:ff:ff:ff:ff
        inet 10.0.0.3/24 brd 10.0.0.255 scope global tap7139b265-41
        inet6 fe80::f816:3eff:fe2e:1835/64 scope link
           valid_lft forever preferred_lft forever
    17: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    >> ip netns exec qdhcp-network-id ifconfig
    lo        Link encap:Local Loopback
              inet addr:127.0.0.1  Mask:255.0.0.0
              inet6 addr: ::1/128 Scope:Host
              UP LOOPBACK RUNNING  MTU:16436  Metric:1
              RX packets:4 errors:0 dropped:0 overruns:0 frame:0
              TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:2304 (2.2 KiB)  TX bytes:2304 (2.2 KiB)

    tap7139b265-41 Link encap:Ethernet  HWaddr FA:16:3E:2E:18:35
              inet addr:10.0.0.3  Bcast:10.0.0.255  Mask:255.255.255.0
              inet6 addr: fe80::f816:3eff:fe2e:1835/64 Scope:Link
              UP BROADCAST RUNNING  MTU:1500  Metric:1
              RX packets:91007 errors:0 dropped:0 overruns:0 frame:0
              TX packets:727 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:4633459 (4.4 MiB)  TX bytes:193767 (189.2 KiB)

This is a production environment and i need an urgent help please.

Best Regards,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20140725/dbb3e987/attachment.html>