<div dir="ltr">I'm having difficulty reproducing the issue. The bug that Neil referenced (<a href="https://bugs.launchpad.net/neutron/+bug/1192381">https://bugs.launchpad.net/neutron/+bug/1192381</a>) looks like it was in Icehouse well before the 2014.1.3 release that looks like Fuel 5.1.1 is using.<div><br></div><div>I tried setting the agent report interval to something higher than the downtime to make it seem like the agent is failing sporadically to the server, but it's not impacting the notifications.</div><div><br></div><div>Neil, does your testing where you saw something similar have a lot of concurrent creation/deletion?</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 8, 2015 at 12:21 PM, Andrew Woodward <span dir="ltr"><<a href="mailto:awoodward@mirantis.com" target="_blank">awoodward@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Daniel,<br><br><div>This sounds familiar, see if this matches [1]. IIRC, there was another issue like this that was might already address this in the updates into Fuel 5.1.2 packages repo [2]. You can either update the neutron packages from [2] Or try one of community builds for 5.1.2 [3]. If this doesn't resolve the issue, open a bug against MOS dev [4].</div><div><br></div><div>[1] <a href="https://bugs.launchpad.net/bugs/1295715" target="_blank">https://bugs.launchpad.net/bugs/1295715</a><br></div><div>[2] <a href="http://fuel-repository.mirantis.com/fwm/5.1.2/ubuntu/pool/main/" target="_blank">http://fuel-repository.mirantis.com/fwm/5.1.2/ubuntu/pool/main/</a></div><div>[3] <a href="https://ci.fuel-infra.org/" target="_blank">https://ci.fuel-infra.org/</a> </div><div>[4] <a href="https://bugs.launchpad.net/mos/+filebug" target="_blank">https://bugs.launchpad.net/mos/+filebug</a></div></div><br><div class="gmail_quote"><div><div class="h5"><div dir="ltr">On Mon, Jun 8, 2015 at 10:15 AM Neil Jerram <<a href="mailto:Neil.Jerram@metaswitch.com" target="_blank">Neil.Jerram@metaswitch.com</a>> wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">Two further thoughts on this:<br>
<br>
1. Another DHCP agent problem that my team noticed is that it<br>
call_driver('reload_allocations') takes a bit of time (to regenerate the<br>
Dnsmasq config files, and to spawn a shell that sends a HUP signal) -<br>
enough so that if there is a fast steady rate of port-create and<br>
port-delete notifications coming from the Neutron server, these can<br>
build up in DHCPAgent's RPC queue, and then they still only get<br>
dispatched one at a time. So the queue and the time delay become longer<br>
and longer.<br>
<br>
I have a fix pending for this, which uses an extra thread to read those<br>
notifications off the RPC queue onto an internal queue, and then batches<br>
the call_driver('reload_allocations') processing when there is a<br>
contiguous sequence of such notifications - i.e. only does the config<br>
regeneration and HUP once, instead of lots of times.<br>
<br>
I don't think this is directly related to what you are seeing - but<br>
perhaps there actually is some link that I am missing.<br>
<br>
2. There is an interesting and vaguely similar thread currently being<br>
discussed about the L3 agent (subject "L3 agent rescheduling issue") -<br>
about possible RPC/threading issues between the agent and the Neutron<br>
server. You might like to review that thread and see if it describes<br>
any problems analogous to your DHCP one.<br>
<br>
Regards,<br>
Neil<br>
<br>
<br>
On 08/06/15 17:53, Neil Jerram wrote:<br>
> My team has seen a problem that could be related: in a churn test where<br>
> VMs are created and terminated at a constant rate - but so that the<br>
> number of active VMs should remain roughly constant - the size of the<br>
> host and addn_hosts files keeps increasing.<br>
><br>
> In other words, it appears that the config for VMs that have actually<br>
> been terminated is not being removed from the config file. Clearly, if<br>
> you have a limited pool of IP addresses, this can eventually lead to the<br>
> problem that you have described.<br>
><br>
> For your case - i.e. with Icehouse - the problem might be<br>
> <a href="https://bugs.launchpad.net/neutron/+bug/1192381" target="_blank">https://bugs.launchpad.net/neutron/+bug/1192381</a>. I'm not sure if the<br>
> fix for that problem - i.e. sending port-create and port-delete<br>
> notifications to DHCP agents even when the server thinks they are down -<br>
> was merged before the Icehouse release, or not.<br>
><br>
> But there must be at least one other cause as well, because my team was<br>
> seeing this with Juno-level code.<br>
><br>
> Therefore I, too, would be interested in any other insights about this<br>
> problem.<br>
><br>
> Regards,<br>
> Neil<br>
><br>
><br>
><br>
> On 08/06/15 16:26, Daniel Comnea wrote:<br>
>> Any help, ideas please?<br>
>><br>
>> Thx,<br>
>> Dani<br>
>><br>
>> On Mon, Jun 8, 2015 at 9:25 AM, Daniel Comnea <<a href="mailto:comnea.dani@gmail.com" target="_blank">comnea.dani@gmail.com</a><br>
>> <mailto:<a href="mailto:comnea.dani@gmail.com" target="_blank">comnea.dani@gmail.com</a>>> wrote:<br>
>><br>
>> + Operators<br>
>><br>
>> Much thanks in advance,<br>
>> Dani<br>
>><br>
>><br>
>><br>
>><br>
>> On Sun, Jun 7, 2015 at 6:31 PM, Daniel Comnea <<a href="mailto:comnea.dani@gmail.com" target="_blank">comnea.dani@gmail.com</a><br>
>> <mailto:<a href="mailto:comnea.dani@gmail.com" target="_blank">comnea.dani@gmail.com</a>>> wrote:<br>
>><br>
>> Hi all,<br>
>><br>
>> I'm running IceHouse (build using Fuel 5.1.1) on Ubuntu where<br>
>> dnsmask version 2.59-4.<br>
>> I have a very basic network layout where i have a private net<br>
>> which has 2 subnets<br>
>><br>
>> 2fb7de9d-d6df-481f-acca-2f7860cffa60 | private-net<br>
>> |<br>
>> e79c3477-d3e5-471c-a728-8d881cf31bee <a href="http://192.168.110.0/24" target="_blank">192.168.110.0/24</a><br>
>> <<a href="http://192.168.110.0/24" target="_blank">http://192.168.110.0/24</a>> |<br>
>> |<br>
>> | |<br>
>> f48c3223-8507-455c-9c13-8b727ea5f441 <a href="http://192.168.111.0/24" target="_blank">192.168.111.0/24</a><br>
>> <<a href="http://192.168.111.0/24" target="_blank">http://192.168.111.0/24</a>> |<br>
>><br>
>> and i'm creating VMs via HEAT.<br>
>> What is happening is that sometimes i get duplicated entries in<br>
>> [1] and because of that the VM which was spun up doesn't get<br>
>> an ip.<br>
>> The Dnsmask processes are running okay [2] and i can't see<br>
>> anything special/ wrong in it.<br>
>><br>
>> Any idea why this is happening? Or are you aware of any bugs<br>
>> around this area? Do you see a problems with having 2 subnets<br>
>> mapped to 1 private-net?<br>
>><br>
>><br>
>><br>
>> Thanks,<br>
>> Dani<br>
>><br>
>> [1]<br>
>><br>
>> /var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/addn_hosts<br>
>><br>
>> [2]<br>
>><br>
>> nobody 5664 1 0 Jun02 ? 00:00:08 dnsmasq<br>
>> --no-hosts --no-resolv --strict-order --bind-interfaces<br>
>> --interface=tapc9164734-0c --except-interface=lo<br>
>><br>
>> --pid-file=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/pid<br>
>><br>
>> --dhcp-hostsfile=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/host<br>
>><br>
>><br>
>> --addn-hosts=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/addn_hosts<br>
>><br>
>><br>
>> --dhcp-optsfile=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/opts<br>
>><br>
>> --leasefile-ro --dhcp-authoritative<br>
>> --dhcp-range=set:tag0,192.168.110.0,static,86400s<br>
>> --dhcp-range=set:tag1,192.168.111.0,static,86400s<br>
>> --dhcp-lease-max=512 --conf-file= --server=10.0.0.31<br>
>> --server=10.0.0.32 --domain=openstacklocal<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> OpenStack-operators mailing list<br>
>> <a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
>><br>
><br>
> _______________________________________________<br>
> OpenStack-operators mailing list<br>
> <a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br></div></div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><span class="HOEnZb"><font color="#888888"><br>
</font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><div dir="ltr">-- <br></div><div dir="ltr">--<div>Andrew Woodward</div><div>Mirantis</div><div>Fuel Community Ambassador</div><div>Ceph Community </div></div>
</font></span><br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div>Kevin Benton</div></div>
</div>