<div dir="ltr">Daniel,<br><br><div>This sounds familiar, see if this matches [1]. IIRC, there was another issue like this that was might already address this in the updates into Fuel 5.1.2 packages repo [2]. You can either update the neutron packages from [2] Or try one of community builds for 5.1.2 [3]. If this doesn't resolve the issue, open a bug against MOS dev [4].</div><div><br></div><div>[1] <a href="https://bugs.launchpad.net/bugs/1295715">https://bugs.launchpad.net/bugs/1295715</a><br></div><div>[2] <a href="http://fuel-repository.mirantis.com/fwm/5.1.2/ubuntu/pool/main/">http://fuel-repository.mirantis.com/fwm/5.1.2/ubuntu/pool/main/</a></div><div>[3] <a href="https://ci.fuel-infra.org/">https://ci.fuel-infra.org/</a> </div><div>[4] <a href="https://bugs.launchpad.net/mos/+filebug">https://bugs.launchpad.net/mos/+filebug</a></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Jun 8, 2015 at 10:15 AM Neil Jerram <<a href="mailto:Neil.Jerram@metaswitch.com">Neil.Jerram@metaswitch.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Two further thoughts on this:<br>
<br>
1. Another DHCP agent problem that my team noticed is that it<br>
call_driver('reload_allocations') takes a bit of time (to regenerate the<br>
Dnsmasq config files, and to spawn a shell that sends a HUP signal) -<br>
enough so that if there is a fast steady rate of port-create and<br>
port-delete notifications coming from the Neutron server, these can<br>
build up in DHCPAgent's RPC queue, and then they still only get<br>
dispatched one at a time. So the queue and the time delay become longer<br>
and longer.<br>
<br>
I have a fix pending for this, which uses an extra thread to read those<br>
notifications off the RPC queue onto an internal queue, and then batches<br>
the call_driver('reload_allocations') processing when there is a<br>
contiguous sequence of such notifications - i.e. only does the config<br>
regeneration and HUP once, instead of lots of times.<br>
<br>
I don't think this is directly related to what you are seeing - but<br>
perhaps there actually is some link that I am missing.<br>
<br>
2. There is an interesting and vaguely similar thread currently being<br>
discussed about the L3 agent (subject "L3 agent rescheduling issue") -<br>
about possible RPC/threading issues between the agent and the Neutron<br>
server. You might like to review that thread and see if it describes<br>
any problems analogous to your DHCP one.<br>
<br>
Regards,<br>
Neil<br>
<br>
<br>
On 08/06/15 17:53, Neil Jerram wrote:<br>
> My team has seen a problem that could be related: in a churn test where<br>
> VMs are created and terminated at a constant rate - but so that the<br>
> number of active VMs should remain roughly constant - the size of the<br>
> host and addn_hosts files keeps increasing.<br>
><br>
> In other words, it appears that the config for VMs that have actually<br>
> been terminated is not being removed from the config file. Clearly, if<br>
> you have a limited pool of IP addresses, this can eventually lead to the<br>
> problem that you have described.<br>
><br>
> For your case - i.e. with Icehouse - the problem might be<br>
> <a href="https://bugs.launchpad.net/neutron/+bug/1192381" target="_blank">https://bugs.launchpad.net/neutron/+bug/1192381</a>. I'm not sure if the<br>
> fix for that problem - i.e. sending port-create and port-delete<br>
> notifications to DHCP agents even when the server thinks they are down -<br>
> was merged before the Icehouse release, or not.<br>
><br>
> But there must be at least one other cause as well, because my team was<br>
> seeing this with Juno-level code.<br>
><br>
> Therefore I, too, would be interested in any other insights about this<br>
> problem.<br>
><br>
> Regards,<br>
> Neil<br>
><br>
><br>
><br>
> On 08/06/15 16:26, Daniel Comnea wrote:<br>
>> Any help, ideas please?<br>
>><br>
>> Thx,<br>
>> Dani<br>
>><br>
>> On Mon, Jun 8, 2015 at 9:25 AM, Daniel Comnea <<a href="mailto:comnea.dani@gmail.com" target="_blank">comnea.dani@gmail.com</a><br>
>> <mailto:<a href="mailto:comnea.dani@gmail.com" target="_blank">comnea.dani@gmail.com</a>>> wrote:<br>
>><br>
>> + Operators<br>
>><br>
>> Much thanks in advance,<br>
>> Dani<br>
>><br>
>><br>
>><br>
>><br>
>> On Sun, Jun 7, 2015 at 6:31 PM, Daniel Comnea <<a href="mailto:comnea.dani@gmail.com" target="_blank">comnea.dani@gmail.com</a><br>
>> <mailto:<a href="mailto:comnea.dani@gmail.com" target="_blank">comnea.dani@gmail.com</a>>> wrote:<br>
>><br>
>> Hi all,<br>
>><br>
>> I'm running IceHouse (build using Fuel 5.1.1) on Ubuntu where<br>
>> dnsmask version 2.59-4.<br>
>> I have a very basic network layout where i have a private net<br>
>> which has 2 subnets<br>
>><br>
>> 2fb7de9d-d6df-481f-acca-2f7860cffa60 | private-net<br>
>> |<br>
>> e79c3477-d3e5-471c-a728-8d881cf31bee <a href="http://192.168.110.0/24" target="_blank">192.168.110.0/24</a><br>
>> <<a href="http://192.168.110.0/24" target="_blank">http://192.168.110.0/24</a>> |<br>
>> |<br>
>> | |<br>
>> f48c3223-8507-455c-9c13-8b727ea5f441 <a href="http://192.168.111.0/24" target="_blank">192.168.111.0/24</a><br>
>> <<a href="http://192.168.111.0/24" target="_blank">http://192.168.111.0/24</a>> |<br>
>><br>
>> and i'm creating VMs via HEAT.<br>
>> What is happening is that sometimes i get duplicated entries in<br>
>> [1] and because of that the VM which was spun up doesn't get<br>
>> an ip.<br>
>> The Dnsmask processes are running okay [2] and i can't see<br>
>> anything special/ wrong in it.<br>
>><br>
>> Any idea why this is happening? Or are you aware of any bugs<br>
>> around this area? Do you see a problems with having 2 subnets<br>
>> mapped to 1 private-net?<br>
>><br>
>><br>
>><br>
>> Thanks,<br>
>> Dani<br>
>><br>
>> [1]<br>
>><br>
>> /var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/addn_hosts<br>
>><br>
>> [2]<br>
>><br>
>> nobody 5664 1 0 Jun02 ? 00:00:08 dnsmasq<br>
>> --no-hosts --no-resolv --strict-order --bind-interfaces<br>
>> --interface=tapc9164734-0c --except-interface=lo<br>
>><br>
>> --pid-file=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/pid<br>
>><br>
>> --dhcp-hostsfile=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/host<br>
>><br>
>><br>
>> --addn-hosts=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/addn_hosts<br>
>><br>
>><br>
>> --dhcp-optsfile=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/opts<br>
>><br>
>> --leasefile-ro --dhcp-authoritative<br>
>> --dhcp-range=set:tag0,192.168.110.0,static,86400s<br>
>> --dhcp-range=set:tag1,192.168.111.0,static,86400s<br>
>> --dhcp-lease-max=512 --conf-file= --server=10.0.0.31<br>
>> --server=10.0.0.32 --domain=openstacklocal<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> OpenStack-operators mailing list<br>
>> <a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
>><br>
><br>
> _______________________________________________<br>
> OpenStack-operators mailing list<br>
> <a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div><div dir="ltr">-- <br></div><div dir="ltr">--<div>Andrew Woodward</div><div>Mirantis</div><div>Fuel Community Ambassador</div><div>Ceph Community </div></div>