<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>Unsure if you got this fixed or not. </div>
<div><br>
</div>
<div>But the errors that you have in your paste’s indicate that your amqp, looks like qpid in this case, is not available to your services. I would restart qpid, then restart all the other openstack services and verify they can successfully connect. We run
rabbitmq for amqp – so I can’t provide anymore help with qpid.</div>
<div>
<div>
<div>____________________________________________</div>
<div> </div>
<div>Kris Lindgren</div>
<div>Senior Linux Systems Engineer</div>
<div>GoDaddy, LLC.</div>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Ramy Allam <<a href="mailto:linux@ramyallam.com">linux@ramyallam.com</a>><br>
<span style="font-weight:bold">Date: </span>Thursday, July 24, 2014 at 4:09 PM<br>
<span style="font-weight:bold">To: </span><<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>[Openstack-operators] VM Can't get DHCP IP<br>
</div>
<div><br>
</div>
<div dir="ltr">Hello,
<div>
<div>I'm running OpenStack RDO with the following setup.</div>
<div><br>
</div>
<div>- 1x Controller Node ( Neutron - Nova - glance - Horizon - Keystone - GRE + FlatDHCPManager)</div>
<div>- 10x Compute Nodes ( Nova Compute - KVM ) </div>
<div>- OS : CentOS 6 - 64bit</div>
<div><br>
</div>
<div>Suddenly all vm's on the compute nodes can't get new IP address from the controller node. It was working properly hours ago without any modifications from me. This log from the controller node - **many errors** </div>
<div> <a href="http://paste.openstack.org/show/88016/">http://paste.openstack.org/show/88016/</a><br>
</div>
<div><br>
</div>
<div>And this from the compute node</div>
<div> <a href="http://paste.openstack.org/show/88017/">http://paste.openstack.org/show/88017/</a></div>
<div><br>
</div>
<div>Installed packages on Controller Node :</div>
<div># rpm -qa | grep -i openstack</div>
<div><br>
</div>
<div> openstack-selinux-0.1.3-2.el6ost.noarch</div>
<div> openstack-puppet-modules-2013.2-9.1.el6.noarch</div>
<div> openstack-ceilometer-api-2013.2.3-2.el6.noarch</div>
<div> openstack-packstack-2013.2.1-0.36.dev1013.el6.noarch</div>
<div> openstack-nova-scheduler-2013.2.3-1.el6.noarch</div>
<div> openstack-ceilometer-common-2013.2.3-2.el6.noarch</div>
<div> python-django-openstack-auth-1.1.2-1.el6.noarch</div>
<div> openstack-ceilometer-central-2013.2.3-2.el6.noarch</div>
<div> openstack-ceilometer-collector-2013.2.3-2.el6.noarch</div>
<div> openstack-neutron-openvswitch-2013.2.3-9.el6.noarch</div>
<div> openstack-nova-common-2013.2.3-1.el6.noarch</div>
<div> openstack-packstack-puppet-2013.2.1-0.36.dev1013.el6.noarch</div>
<div> openstack-glance-2013.2.3-2.el6.noarch</div>
<div> openstack-nova-conductor-2013.2.3-1.el6.noarch</div>
<div> openstack-nova-novncproxy-2013.2.3-1.el6.noarch</div>
<div> openstack-nova-cert-2013.2.3-1.el6.noarch</div>
<div> openstack-keystone-2013.2.3-3.el6.noarch</div>
<div> openstack-neutron-2013.2.3-9.el6.noarch</div>
<div> openstack-ceilometer-alarm-2013.2.3-2.el6.noarch</div>
<div> openstack-dashboard-2013.2.3-1.el6.noarch</div>
<div> openstack-nova-api-2013.2.3-1.el6.noarch</div>
<div> openstack-nova-console-2013.2.3-1.el6.noarch</div>
<div> openstack-utils-2013.2-2.el6.noarch</div>
<div><br>
</div>
<div>On Controller Node</div>
<div> # ps aux | grep -i dhcp</div>
<div> nobody 12639 0.0 0.0 12884 864 ? S Jul19 0:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap7139b265-41 --except-interface=lo --pid-file=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/pid
--dhcp-hostsfile=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/host --dhcp-optsfile=/var/lib/neutron/dhcp/c19ca2ea-8278-4069-bfea-dadd92961cac/opts --leasefile-ro --dhcp-range=tag0,10.0.0.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal</div>
<div> neutron 24884 0.0 0.1 273748 32140 ? S Jul24 0:00 /usr/bin/python /usr/bin/neutron-dhcp-agent --log-file /var/log/neutron/dhcp-agent.log --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file
/etc/neutron/dhcp_agent.ini</div>
<div><br>
</div>
<div># ip netns</div>
<div> qrouter-id</div>
<div> qdhcp-id</div>
<div><br>
</div>
<div># ip netns exec qdhcp-network-id ip a</div>
<div> 16: tap7139b265-41: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN </div>
<div> link/ether fa:16:3e:2e:18:35 brd ff:ff:ff:ff:ff:ff</div>
<div> inet <a href="http://10.0.0.3/24">10.0.0.3/24</a> brd 10.0.0.255 scope global tap7139b265-41</div>
<div> inet6 fe80::f816:3eff:fe2e:1835/64 scope link </div>
<div> valid_lft forever preferred_lft forever</div>
<div> 17: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN </div>
<div> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00</div>
<div> inet <a href="http://127.0.0.1/8">127.0.0.1/8</a> scope host lo</div>
<div> inet6 ::1/128 scope host </div>
<div> valid_lft forever preferred_lft forever</div>
<div> >> ip netns exec qdhcp-network-id ifconfig</div>
<div> lo Link encap:Local Loopback </div>
<div> inet addr:127.0.0.1 Mask:255.0.0.0</div>
<div> inet6 addr: ::1/128 Scope:Host</div>
<div> UP LOOPBACK RUNNING MTU:16436 Metric:1</div>
<div> RX packets:4 errors:0 dropped:0 overruns:0 frame:0</div>
<div> TX packets:4 errors:0 dropped:0 overruns:0 carrier:0</div>
<div> collisions:0 txqueuelen:0 </div>
<div> RX bytes:2304 (2.2 KiB) TX bytes:2304 (2.2 KiB)</div>
<div> </div>
<div> tap7139b265-41 Link encap:Ethernet HWaddr FA:16:3E:2E:18:35 </div>
<div> inet addr:10.0.0.3 Bcast:10.0.0.255 Mask:255.255.255.0</div>
<div> inet6 addr: fe80::f816:3eff:fe2e:1835/64 Scope:Link</div>
<div> UP BROADCAST RUNNING MTU:1500 Metric:1</div>
<div> RX packets:91007 errors:0 dropped:0 overruns:0 frame:0</div>
<div> TX packets:727 errors:0 dropped:0 overruns:0 carrier:0</div>
<div> collisions:0 txqueuelen:0 </div>
<div> RX bytes:4633459 (4.4 MiB) TX bytes:193767 (189.2 KiB)</div>
<div><br>
</div>
</div>
<div>This is a production environment and i need an urgent help please.</div>
<div><br>
</div>
<div>Best Regards,</div>
</div>
</span>
</body>
</html>