Open Stack

Two further thoughts on this:

1. Another DHCP agent problem that my team noticed is that it 
call_driver('reload_allocations') takes a bit of time (to regenerate the 
Dnsmasq config files, and to spawn a shell that sends a HUP signal) - 
enough so that if there is a fast steady rate of port-create and 
port-delete notifications coming from the Neutron server, these can 
build up in DHCPAgent's RPC queue, and then they still only get 
dispatched one at a time.  So the queue and the time delay become longer 
and longer.

I have a fix pending for this, which uses an extra thread to read those 
notifications off the RPC queue onto an internal queue, and then batches 
the call_driver('reload_allocations') processing when there is a 
contiguous sequence of such notifications - i.e. only does the config 
regeneration and HUP once, instead of lots of times.

I don't think this is directly related to what you are seeing - but 
perhaps there actually is some link that I am missing.

2. There is an interesting and vaguely similar thread currently being 
discussed about the L3 agent (subject "L3 agent rescheduling issue") - 
about possible RPC/threading issues between the agent and the Neutron 
server.  You might like to review that thread and see if it describes 
any problems analogous to your DHCP one.

Regards,
	Neil


On 08/06/15 17:53, Neil Jerram wrote:
> My team has seen a problem that could be related: in a churn test where
> VMs are created and terminated at a constant rate - but so that the
> number of active VMs should remain roughly constant - the size of the
> host and addn_hosts files keeps increasing.
>
> In other words, it appears that the config for VMs that have actually
> been terminated is not being removed from the config file.  Clearly, if
> you have a limited pool of IP addresses, this can eventually lead to the
> problem that you have described.
>
> For your case - i.e. with Icehouse - the problem might be
> https://bugs.launchpad.net/neutron/+bug/1192381.  I'm not sure if the
> fix for that problem - i.e. sending port-create and port-delete
> notifications to DHCP agents even when the server thinks they are down -
> was merged before the Icehouse release, or not.
>
> But there must be at least one other cause as well, because my team was
> seeing this with Juno-level code.
>
> Therefore I, too, would be interested in any other insights about this
> problem.
>
> Regards,
>      Neil
>
>
>
> On 08/06/15 16:26, Daniel Comnea wrote:
>> Any help, ideas please?
>>
>> Thx,
>> Dani
>>
>> On Mon, Jun 8, 2015 at 9:25 AM, Daniel Comnea <comnea.dani at gmail.com
>> <mailto:comnea.dani at gmail.com>> wrote:
>>
>>     + Operators
>>
>>     Much thanks in advance,
>>     Dani
>>
>>
>>
>>
>>     On Sun, Jun 7, 2015 at 6:31 PM, Daniel Comnea <comnea.dani at gmail.com
>>     <mailto:comnea.dani at gmail.com>> wrote:
>>
>>         Hi all,
>>
>>         I'm running IceHouse (build using Fuel 5.1.1) on Ubuntu where
>>         dnsmask version 2.59-4.
>>         I have a very basic network layout where i have a private net
>>         which has 2 subnets
>>
>>           2fb7de9d-d6df-481f-acca-2f7860cffa60 | private-net
>>                                     |
>>         e79c3477-d3e5-471c-a728-8d881cf31bee 192.168.110.0/24
>>         <http://192.168.110.0/24> |
>>         |
>>         |                                                         |
>>         f48c3223-8507-455c-9c13-8b727ea5f441 192.168.111.0/24
>>         <http://192.168.111.0/24> |
>>
>>         and i'm creating VMs via HEAT.
>>         What is happening is that sometimes i get duplicated entries in
>>         [1] and because of that the VM which was spun up doesn't get
>> an ip.
>>         The Dnsmask processes are running okay [2] and i can't see
>>         anything special/ wrong in it.
>>
>>         Any idea why this is happening? Or are you aware of any bugs
>>         around this area? Do you see a problems with having 2 subnets
>>         mapped to 1 private-net?
>>
>>
>>
>>         Thanks,
>>         Dani
>>
>>         [1]
>>
>> /var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/addn_hosts
>>
>>         [2]
>>
>>         nobody    5664     1  0 Jun02 ?        00:00:08 dnsmasq
>>         --no-hosts --no-resolv --strict-order --bind-interfaces
>>         --interface=tapc9164734-0c --except-interface=lo
>>
>> --pid-file=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/pid
>>
>> --dhcp-hostsfile=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/host
>>
>>
>> --addn-hosts=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/addn_hosts
>>
>>
>> --dhcp-optsfile=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/opts
>>
>>         --leasefile-ro --dhcp-authoritative
>>         --dhcp-range=set:tag0,192.168.110.0,static,86400s
>>         --dhcp-range=set:tag1,192.168.111.0,static,86400s
>>         --dhcp-lease-max=512 --conf-file= --server=10.0.0.31
>>         --server=10.0.0.32 --domain=openstacklocal
>>
>>
>>
>>
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators