Hey all,<div><br></div><div>it now just happened twice again, both just today. and the last at 22:00 UTC, with</div><div>the following in the nova-network's syslog:</div><div><br></div><div><div>root@gw1:/var/log# grep 'dnsmasq.*10889' daemon.log</div>
<div>Jun 15 17:39:32 cesar1 dnsmasq[10889]: started, version v2.62-7-g4ce4f37 cachesize 150</div><div>Jun 15 17:39:32 cesar1 dnsmasq[10889]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack</div>
<div>Jun 15 17:39:32 cesar1 dnsmasq-dhcp[10889]: DHCP, static leases only on 10.10.40.3, lease time 3d</div><div>Jun 15 17:39:32 cesar1 dnsmasq[10889]: reading /etc/resolv.conf</div><div>Jun 15 17:39:32 cesar1 dnsmasq[10889]: using nameserver 4.2.2.1#53</div>
<div>Jun 15 17:39:32 cesar1 dnsmasq[10889]: using nameserver 178.63.26.173#53</div><div>Jun 15 17:39:32 cesar1 dnsmasq[10889]: using nameserver 192.168.2.122#53</div><div>Jun 15 17:39:32 cesar1 dnsmasq[10889]: using nameserver 192.168.2.121#53</div>
<div>Jun 15 17:39:32 cesar1 dnsmasq[10889]: read /etc/hosts - 519 addresses</div><div>Jun 15 17:39:32 cesar1 dnsmasq-dhcp[10889]: read /var/lib/nova/networks/nova-br100.conf</div><div>Jun 15 21:59:41 cesar1 dnsmasq-dhcp[10889]: DHCPREQUEST(br100) 10.10.40.16 fa:16:3e:3d:ff:f3 </div>
<div>Jun 15 21:59:41 cesar1 dnsmasq-dhcp[10889]: DHCPACK(br100) 10.10.40.16 fa:16:3e:3d:ff:f3 redis-appdata1</div><div><br></div><div>it seemed that this once VM was the only one who sent a dhcp request over the past 5 hours,</div>
<div>and that first wone got replied with dhcp ack, and that is it.</div><div>That's been the time the host behind that IP (redis-appdata1) stopped functioning.</div><div><br></div><div>However, I now actually did update dnsmasq on our gateway note, to latest trunk</div>
<div>of dnsmasq git repository, killed dnsmasq, restarted nova-network (which auto-starts dnsmasq per </div><div>device).</div><div><br></div><div>Now, I really hoped that this one particular bug fix was the cause of the downtime,</div>
<div>but appearently, thet MIGHT be another factor.</div><div><br></div><div>There is unfortunately nothing to read in the VM's syslog.</div><div>What else could cause the VM to forget its IP?</div><div>Can this also be caused by send_arp_for_ha=True?</div>
<div><br></div><div>Regards,</div><div>Christian.</div><div><br></div>Christian.<br><div class="gmail_quote">On Fri, Jun 15, 2012 at 2:50 AM, Nathanael Burton <span dir="ltr"><<a href="mailto:nathanael.i.burton@gmail.com" target="_blank">nathanael.i.burton@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p>FWIW I haven't run across the dnsmasq bug in our environment using EPEL packages.</p>
<p>Nate</p><div class="HOEnZb"><div class="h5">
<div class="gmail_quote">On Jun 14, 2012 7:20 PM, "Vishvananda Ishaya" <<a href="mailto:vishvananda@gmail.com" target="_blank">vishvananda@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">Are you running in VLAN mode? If so, you probably need to update to a new version of dnsmasq. See this message for reference:<div><br></div><div><a href="http://osdir.com/ml/openstack-cloud-computing/2012-05/msg00785.html" target="_blank">http://osdir.com/ml/openstack-cloud-computing/2012-05/msg00785.html</a></div>
<div><br></div><div>Vish</div><div><br><div><div>On Jun 14, 2012, at 1:41 PM, Christian Parpart wrote:</div><br><blockquote type="cite">Hey all,<div><br></div><div>I feel really sad with saying this, now, that we have quite a few instances in producgtion</div>
<div>since about 5 days at least, I now have encountered the second instance loosing its</div><div>
IP address due to "No DHCPOFFER" (as of syslog in the instance).</div><div><br></div><div>I checked the logs in the central nova-network and gateway node and found</div><div>dnsmasq still to reply on requests from all the other instances and it even</div>
<div>got the request from the instance in question and even sent an OFFER, as of what</div><div>I can tell by now (i'm investigating / posting logs asap), but while it seemed</div><div>that the dnsmasq sends an offer, the instances says it didn't receive one - wtf?</div>
<div><br></div><div>Please tell me what I can do to actually *fix* this issue, since this is by far very fatal.</div><div><br></div><div>One chance I'd see (as a workaround) is, to let created instanced retrieve</div>
<div>its IP via dhcp, but then reconfigure /etc/network/instances to continue with</div><div>static networking setup. However, I'd just like the dhcp thingy to get fixed.</div><div><br></div><div>I'm very open to any kind of helping comments, :)</div>
<div><br></div><div>So long,</div><div>Christian.</div><div><br></div>
_______________________________________________<br>Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
</blockquote></div><br></div></div><br>_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br></blockquote></div>
</div></div></blockquote></div><br></div>