Open Stack

Daniel,

This sounds familiar, see if this matches [1]. IIRC, there was another
issue like this that was might already address this in the updates into
Fuel 5.1.2 packages repo [2]. You can either update the neutron packages
from [2] Or try one of community builds for 5.1.2 [3]. If this doesn't
resolve the issue, open a bug against MOS dev [4].

[1] https://bugs.launchpad.net/bugs/1295715
[2] http://fuel-repository.mirantis.com/fwm/5.1.2/ubuntu/pool/main/
[3] https://ci.fuel-infra.org/
[4] https://bugs.launchpad.net/mos/+filebug

On Mon, Jun 8, 2015 at 10:15 AM Neil Jerram <Neil.Jerram at metaswitch.com>
wrote:

> Two further thoughts on this:
>
> 1. Another DHCP agent problem that my team noticed is that it
> call_driver('reload_allocations') takes a bit of time (to regenerate the
> Dnsmasq config files, and to spawn a shell that sends a HUP signal) -
> enough so that if there is a fast steady rate of port-create and
> port-delete notifications coming from the Neutron server, these can
> build up in DHCPAgent's RPC queue, and then they still only get
> dispatched one at a time.  So the queue and the time delay become longer
> and longer.
>
> I have a fix pending for this, which uses an extra thread to read those
> notifications off the RPC queue onto an internal queue, and then batches
> the call_driver('reload_allocations') processing when there is a
> contiguous sequence of such notifications - i.e. only does the config
> regeneration and HUP once, instead of lots of times.
>
> I don't think this is directly related to what you are seeing - but
> perhaps there actually is some link that I am missing.
>
> 2. There is an interesting and vaguely similar thread currently being
> discussed about the L3 agent (subject "L3 agent rescheduling issue") -
> about possible RPC/threading issues between the agent and the Neutron
> server.  You might like to review that thread and see if it describes
> any problems analogous to your DHCP one.
>
> Regards,
>         Neil
>
>
> On 08/06/15 17:53, Neil Jerram wrote:
> > My team has seen a problem that could be related: in a churn test where
> > VMs are created and terminated at a constant rate - but so that the
> > number of active VMs should remain roughly constant - the size of the
> > host and addn_hosts files keeps increasing.
> >
> > In other words, it appears that the config for VMs that have actually
> > been terminated is not being removed from the config file.  Clearly, if
> > you have a limited pool of IP addresses, this can eventually lead to the
> > problem that you have described.
> >
> > For your case - i.e. with Icehouse - the problem might be
> > https://bugs.launchpad.net/neutron/+bug/1192381.  I'm not sure if the
> > fix for that problem - i.e. sending port-create and port-delete
> > notifications to DHCP agents even when the server thinks they are down -
> > was merged before the Icehouse release, or not.
> >
> > But there must be at least one other cause as well, because my team was
> > seeing this with Juno-level code.
> >
> > Therefore I, too, would be interested in any other insights about this
> > problem.
> >
> > Regards,
> >      Neil
> >
> >
> >
> > On 08/06/15 16:26, Daniel Comnea wrote:
> >> Any help, ideas please?
> >>
> >> Thx,
> >> Dani
> >>
> >> On Mon, Jun 8, 2015 at 9:25 AM, Daniel Comnea <comnea.dani at gmail.com
> >> <mailto:comnea.dani at gmail.com>> wrote:
> >>
> >>     + Operators
> >>
> >>     Much thanks in advance,
> >>     Dani
> >>
> >>
> >>
> >>
> >>     On Sun, Jun 7, 2015 at 6:31 PM, Daniel Comnea <
> comnea.dani at gmail.com
> >>     <mailto:comnea.dani at gmail.com>> wrote:
> >>
> >>         Hi all,
> >>
> >>         I'm running IceHouse (build using Fuel 5.1.1) on Ubuntu where
> >>         dnsmask version 2.59-4.
> >>         I have a very basic network layout where i have a private net
> >>         which has 2 subnets
> >>
> >>           2fb7de9d-d6df-481f-acca-2f7860cffa60 | private-net
> >>                                     |
> >>         e79c3477-d3e5-471c-a728-8d881cf31bee 192.168.110.0/24
> >>         <http://192.168.110.0/24> |
> >>         |
> >>         |                                                         |
> >>         f48c3223-8507-455c-9c13-8b727ea5f441 192.168.111.0/24
> >>         <http://192.168.111.0/24> |
> >>
> >>         and i'm creating VMs via HEAT.
> >>         What is happening is that sometimes i get duplicated entries in
> >>         [1] and because of that the VM which was spun up doesn't get
> >> an ip.
> >>         The Dnsmask processes are running okay [2] and i can't see
> >>         anything special/ wrong in it.
> >>
> >>         Any idea why this is happening? Or are you aware of any bugs
> >>         around this area? Do you see a problems with having 2 subnets
> >>         mapped to 1 private-net?
> >>
> >>
> >>
> >>         Thanks,
> >>         Dani
> >>
> >>         [1]
> >>
> >> /var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/addn_hosts
> >>
> >>         [2]
> >>
> >>         nobody    5664     1  0 Jun02 ?        00:00:08 dnsmasq
> >>         --no-hosts --no-resolv --strict-order --bind-interfaces
> >>         --interface=tapc9164734-0c --except-interface=lo
> >>
> >>
> --pid-file=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/pid
> >>
> >>
> --dhcp-hostsfile=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/host
> >>
> >>
> >>
> --addn-hosts=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/addn_hosts
> >>
> >>
> >>
> --dhcp-optsfile=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/opts
> >>
> >>         --leasefile-ro --dhcp-authoritative
> >>         --dhcp-range=set:tag0,192.168.110.0,static,86400s
> >>         --dhcp-range=set:tag1,192.168.111.0,static,86400s
> >>         --dhcp-lease-max=512 --conf-file= --server=10.0.0.31
> >>         --server=10.0.0.32 --domain=openstacklocal
> >>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> OpenStack-operators mailing list
> >> OpenStack-operators at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >>
> >
> > _______________________________________________
> > OpenStack-operators mailing list
> > OpenStack-operators at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 
--
Andrew Woodward
Mirantis
Fuel Community Ambassador
Ceph Community
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150608/8b4c955c/attachment.html>