<div dir="ltr">Hello,<div><div>I'm running OpenStack RDO with the following setup.</div><div><br></div><div>- 1x Controller Node ( Neutron - Nova - glance - Horizon - Keystone - GRE + FlatDHCPManager)</div><div>- 10x Compute Nodes ( Nova Compute - KVM ) </div>
<div>- OS : CentOS 6 - 64bit</div><div><br></div><div>Suddenly all vm's on the compute nodes can't get new IP address from the controller node. It was working properly hours ago without any modifications from me. This log from the controller node - **many errors** </div>
<div> <a href="http://paste.openstack.org/show/88016/">http://paste.openstack.org/show/88016/</a><br></div><div><br></div><div>And this from the compute node</div><div> <a href="http://paste.openstack.org/show/88017/">http://paste.openstack.org/show/88017/</a></div>
<div><br></div><div>Installed packages on Controller Node :</div><div># rpm -qa | grep -i openstack</div><div><br></div><div> openstack-selinux-0.1.3-2.el6ost.noarch</div><div> openstack-puppet-modules-2013.2-9.1.el6.noarch</div>
<div> openstack-ceilometer-api-2013.2.3-2.el6.noarch</div><div> openstack-packstack-2013.2.1-0.36.dev1013.el6.noarch</div><div> openstack-nova-scheduler-2013.2.3-1.el6.noarch</div><div> openstack-ceilometer-common-2013.2.3-2.el6.noarch</div>
<div> python-django-openstack-auth-1.1.2-1.el6.noarch</div><div> openstack-ceilometer-central-2013.2.3-2.el6.noarch</div><div> openstack-ceilometer-collector-2013.2.3-2.el6.noarch</div><div> openstack-neutron-openvswitch-2013.2.3-9.el6.noarch</div>
<div> openstack-nova-common-2013.2.3-1.el6.noarch</div><div> openstack-packstack-puppet-2013.2.1-0.36.dev1013.el6.noarch</div><div> openstack-glance-2013.2.3-2.el6.noarch</div><div> openstack-nova-conductor-2013.2.3-1.el6.noarch</div>
<div> openstack-nova-novncproxy-2013.2.3-1.el6.noarch</div><div> openstack-nova-cert-2013.2.3-1.el6.noarch</div><div> openstack-keystone-2013.2.3-3.el6.noarch</div><div> openstack-neutron-2013.2.3-9.el6.noarch</div>
<div> openstack-ceilometer-alarm-2013.2.3-2.el6.noarch</div><div> openstack-dashboard-2013.2.3-1.el6.noarch</div><div> openstack-nova-api-2013.2.3-1.el6.noarch</div><div> openstack-nova-console-2013.2.3-1.el6.noarch</div>
<div> openstack-utils-2013.2-2.el6.noarch</div><div><br></div><div>On Controller Node</div><div> # ps aux | grep -i dhcp</div><div> nobody 12639 0.0 0.0 12884 864 ? S Jul19 0:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap7139b265-41 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/host --dhcp-optsfile=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/opts --leasefile-ro --dhcp-range=tag0,10.0.0.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal</div>
<div> neutron 24884 0.0 0.1 273748 32140 ? S Jul24 0:00 /usr/bin/python /usr/bin/neutron-dhcp-agent --log-file /var/log/neutron/dhcp-agent.log --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini</div>
<div><br></div><div># ip netns</div><div> qrouter-id</div><div> qdhcp-id</div><div><br></div><div># ip netns exec qdhcp-network-id ip a</div><div> 16: tap7139b265-41: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN </div>
<div> link/ether fa:16:3e:2e:18:35 brd ff:ff:ff:ff:ff:ff</div><div> inet <a href="http://10.0.0.3/24">10.0.0.3/24</a> brd 10.0.0.255 scope global tap7139b265-41</div><div> inet6 fe80::f816:3eff:fe2e:1835/64 scope link </div>
<div> valid_lft forever preferred_lft forever</div><div> 17: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN </div><div> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00</div>
<div> inet <a href="http://127.0.0.1/8">127.0.0.1/8</a> scope host lo</div><div> inet6 ::1/128 scope host </div><div> valid_lft forever preferred_lft forever</div><div> >> ip netns exec qdhcp-network-id ifconfig</div>
<div> lo Link encap:Local Loopback </div><div> inet addr:127.0.0.1 Mask:255.0.0.0</div><div> inet6 addr: ::1/128 Scope:Host</div><div> UP LOOPBACK RUNNING MTU:16436 Metric:1</div>
<div> RX packets:4 errors:0 dropped:0 overruns:0 frame:0</div><div> TX packets:4 errors:0 dropped:0 overruns:0 carrier:0</div><div> collisions:0 txqueuelen:0 </div><div> RX bytes:2304 (2.2 KiB) TX bytes:2304 (2.2 KiB)</div>
<div> </div><div> tap7139b265-41 Link encap:Ethernet HWaddr FA:16:3E:2E:18:35 </div><div> inet addr:10.0.0.3 Bcast:10.0.0.255 Mask:255.255.255.0</div><div> inet6 addr: fe80::f816:3eff:fe2e:1835/64 Scope:Link</div>
<div> UP BROADCAST RUNNING MTU:1500 Metric:1</div><div> RX packets:91007 errors:0 dropped:0 overruns:0 frame:0</div><div> TX packets:727 errors:0 dropped:0 overruns:0 carrier:0</div>
<div> collisions:0 txqueuelen:0 </div><div> RX bytes:4633459 (4.4 MiB) TX bytes:193767 (189.2 KiB)</div><div><br></div></div><div>This is a production environment and i need an urgent help please.</div>
<div><br></div><div>Best Regards,</div></div>