[openstack-dev] [Openstack-operators] [openstack-operators][neutron[dhcp][dnsmask]: duplicate entries in addn_hosts causing no IP allocation

Neil Jerram Neil.Jerram at metaswitch.com
Tue Jun 9 10:34:04 UTC 2015


On 09/06/15 01:15, Kevin Benton wrote:
> I'm having difficulty reproducing the issue. The bug that Neil
> referenced (https://bugs.launchpad.net/neutron/+bug/1192381) looks like
> it was in Icehouse well before the 2014.1.3 release that looks like Fuel
> 5.1.1 is using.

Just to be sure, I assume we're focussing here on the issue that Daniel 
reported (IP appears twice in Dnsmasq config), and for which I described 
a possible corollary (Dnsmasq config size keeps growing), and NOT on the 
"Another DHCP agent problem" that I mentioned below. :-)

BTW, now that I've reviewed the history of when my team saw this, I can 
say that it was actually first reported to us with the 'IP appears twice 
in Dnsmasq config' symptom - i.e. exactly the same as Daniel's case. 
The fact of the Dnsmasq config increasing in size was noticed later.

> I tried setting the agent report interval to something higher than the
> downtime to make it seem like the agent is failing sporadically to the
> server, but it's not impacting the notifications.

Makes sense - that's the effect of the fix for 1192381.

To be clear, though, what code are you trying to reproduce on?  Current 
master?

> Neil, does your testing where you saw something similar have a lot of
> concurrent creation/deletion?

It was a test of continuously deleting and creating VMs, with this 
pseudocode:

thread_pool = new_thread_pool(size=30)
for x in range(0,30):
     thread_pool.submit(create_vm)
thread_pool.wait_for_all_threads_to_complete()
while True:
      time.sleep(5)
      for x in range(0,int(random.random()*5)):
           thread_pool.submit(randomly_delete_a_vm_and_create_a_new_one)

I'm not clear whether that would qualify as 'concurrent', in the sense 
that you have in mind.

Regards,
	Neil

> On Mon, Jun 8, 2015 at 12:21 PM, Andrew Woodward <awoodward at mirantis.com
> <mailto:awoodward at mirantis.com>> wrote:
>
>     Daniel,
>
>     This sounds familiar, see if this matches [1]. IIRC, there was
>     another issue like this that was might already address this in the
>     updates into Fuel 5.1.2 packages repo [2]. You can either update the
>     neutron packages from [2] Or try one of community builds for 5.1.2
>     [3]. If this doesn't resolve the issue, open a bug against MOS dev [4].
>
>     [1] https://bugs.launchpad.net/bugs/1295715
>     [2] http://fuel-repository.mirantis.com/fwm/5.1.2/ubuntu/pool/main/
>     [3] https://ci.fuel-infra.org/
>     [4] https://bugs.launchpad.net/mos/+filebug
>
>     On Mon, Jun 8, 2015 at 10:15 AM Neil Jerram
>     <Neil.Jerram at metaswitch.com <mailto:Neil.Jerram at metaswitch.com>> wrote:
>
>         Two further thoughts on this:
>
>         1. Another DHCP agent problem that my team noticed is that it
>         call_driver('reload_allocations') takes a bit of time (to
>         regenerate the
>         Dnsmasq config files, and to spawn a shell that sends a HUP
>         signal) -
>         enough so that if there is a fast steady rate of port-create and
>         port-delete notifications coming from the Neutron server, these can
>         build up in DHCPAgent's RPC queue, and then they still only get
>         dispatched one at a time.  So the queue and the time delay
>         become longer
>         and longer.
>
>         I have a fix pending for this, which uses an extra thread to
>         read those
>         notifications off the RPC queue onto an internal queue, and then
>         batches
>         the call_driver('reload_allocations') processing when there is a
>         contiguous sequence of such notifications - i.e. only does the
>         config
>         regeneration and HUP once, instead of lots of times.
>
>         I don't think this is directly related to what you are seeing - but
>         perhaps there actually is some link that I am missing.
>
>         2. There is an interesting and vaguely similar thread currently
>         being
>         discussed about the L3 agent (subject "L3 agent rescheduling
>         issue") -
>         about possible RPC/threading issues between the agent and the
>         Neutron
>         server.  You might like to review that thread and see if it
>         describes
>         any problems analogous to your DHCP one.
>
>         Regards,
>                  Neil
>
>
>         On 08/06/15 17:53, Neil Jerram wrote:
>          > My team has seen a problem that could be related: in a churn
>         test where
>          > VMs are created and terminated at a constant rate - but so
>         that the
>          > number of active VMs should remain roughly constant - the
>         size of the
>          > host and addn_hosts files keeps increasing.
>          >
>          > In other words, it appears that the config for VMs that have
>         actually
>          > been terminated is not being removed from the config file.
>         Clearly, if
>          > you have a limited pool of IP addresses, this can eventually
>         lead to the
>          > problem that you have described.
>          >
>          > For your case - i.e. with Icehouse - the problem might be
>          > https://bugs.launchpad.net/neutron/+bug/1192381.  I'm not
>         sure if the
>          > fix for that problem - i.e. sending port-create and port-delete
>          > notifications to DHCP agents even when the server thinks they
>         are down -
>          > was merged before the Icehouse release, or not.
>          >
>          > But there must be at least one other cause as well, because
>         my team was
>          > seeing this with Juno-level code.
>          >
>          > Therefore I, too, would be interested in any other insights
>         about this
>          > problem.
>          >
>          > Regards,
>          >      Neil
>          >
>          >
>          >
>          > On 08/06/15 16:26, Daniel Comnea wrote:
>          >> Any help, ideas please?
>          >>
>          >> Thx,
>          >> Dani
>          >>
>          >> On Mon, Jun 8, 2015 at 9:25 AM, Daniel Comnea
>         <comnea.dani at gmail.com <mailto:comnea.dani at gmail.com>
>          >> <mailto:comnea.dani at gmail.com
>         <mailto:comnea.dani at gmail.com>>> wrote:
>          >>
>          >>     + Operators
>          >>
>          >>     Much thanks in advance,
>          >>     Dani
>          >>
>          >>
>          >>
>          >>
>          >>     On Sun, Jun 7, 2015 at 6:31 PM, Daniel Comnea
>         <comnea.dani at gmail.com <mailto:comnea.dani at gmail.com>
>          >>     <mailto:comnea.dani at gmail.com
>         <mailto:comnea.dani at gmail.com>>> wrote:
>          >>
>          >>         Hi all,
>          >>
>          >>         I'm running IceHouse (build using Fuel 5.1.1) on
>         Ubuntu where
>          >>         dnsmask version 2.59-4.
>          >>         I have a very basic network layout where i have a
>         private net
>          >>         which has 2 subnets
>          >>
>          >>           2fb7de9d-d6df-481f-acca-2f7860cffa60 | private-net
>          >>                                     |
>          >>         e79c3477-d3e5-471c-a728-8d881cf31bee
>         192.168.110.0/24 <http://192.168.110.0/24>
>          >>         <http://192.168.110.0/24> |
>          >>         |
>          >>         |
>               |
>          >>         f48c3223-8507-455c-9c13-8b727ea5f441
>         192.168.111.0/24 <http://192.168.111.0/24>
>          >>         <http://192.168.111.0/24> |
>          >>
>          >>         and i'm creating VMs via HEAT.
>          >>         What is happening is that sometimes i get duplicated
>         entries in
>          >>         [1] and because of that the VM which was spun up
>         doesn't get
>          >> an ip.
>          >>         The Dnsmask processes are running okay [2] and i
>         can't see
>          >>         anything special/ wrong in it.
>          >>
>          >>         Any idea why this is happening? Or are you aware of
>         any bugs
>          >>         around this area? Do you see a problems with having
>         2 subnets
>          >>         mapped to 1 private-net?
>          >>
>          >>
>          >>
>          >>         Thanks,
>          >>         Dani
>          >>
>          >>         [1]
>          >>
>          >>
>         /var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/addn_hosts
>          >>
>          >>         [2]
>          >>
>          >>         nobody    5664     1  0 Jun02 ?        00:00:08 dnsmasq
>          >>         --no-hosts --no-resolv --strict-order --bind-interfaces
>          >>         --interface=tapc9164734-0c --except-interface=lo
>          >>
>          >>
>         --pid-file=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/pid
>          >>
>          >>
>         --dhcp-hostsfile=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/host
>          >>
>          >>
>          >>
>         --addn-hosts=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/addn_hosts
>          >>
>          >>
>          >>
>         --dhcp-optsfile=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/opts
>          >>
>          >>         --leasefile-ro --dhcp-authoritative
>          >>         --dhcp-range=set:tag0,192.168.110.0,static,86400s
>          >>         --dhcp-range=set:tag1,192.168.111.0,static,86400s
>          >>         --dhcp-lease-max=512 --conf-file= --server=10.0.0.31
>          >>         --server=10.0.0.32 --domain=openstacklocal
>          >>
>          >>
>          >>
>          >>
>          >>
>          >> _______________________________________________
>          >> OpenStack-operators mailing list
>          >> OpenStack-operators at lists.openstack.org
>         <mailto:OpenStack-operators at lists.openstack.org>
>          >>
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>          >>
>          >
>          > _______________________________________________
>          > OpenStack-operators mailing list
>          > OpenStack-operators at lists.openstack.org
>         <mailto:OpenStack-operators at lists.openstack.org>
>          >
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>         __________________________________________________________________________
>         OpenStack Development Mailing List (not for usage questions)
>         Unsubscribe:
>         OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>         <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>     --
>     --
>     Andrew Woodward
>     Mirantis
>     Fuel Community Ambassador
>     Ceph Community
>
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> --
> Kevin Benton
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>



More information about the OpenStack-dev mailing list