<div dir="ltr">another way is to have a large agent_down_time, by default it is 9 secs.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Dec 4, 2013 at 7:55 AM, Carl Baldwin <span dir="ltr"><<a href="mailto:carl@ecbaldwin.net" target="_blank">carl@ecbaldwin.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Stephen, all,<br>
<br>
I agree that there may be some opportunity to split things out a bit.<br>
However, I'm not sure what the best way will be. I recall that Mark<br>
mentioned breaking out the processes that handle API requests and RPC<br>
from each other at the summit. Anyway, it is something that has been<br>
discussed.<br>
<br>
I actually wanted to point out that the neutron server now has the<br>
ability to run a configurable number of sub-processes to handle a<br>
heavier load. Introduced with this commit:<br>
<br>
<a href="https://review.openstack.org/#/c/37131/" target="_blank">https://review.openstack.org/#/c/37131/</a><br>
<br>
Set api_workers to something > 1 and restart the server.<br>
<br>
The server can also be run on more than one physical host in<br>
combination with multiple child processes.<br>
<span class="HOEnZb"><font color="#888888"><br>
Carl<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
On Tue, Dec 3, 2013 at 9:47 AM, Stephen Gran<br>
<<a href="mailto:stephen.gran@theguardian.com">stephen.gran@theguardian.com</a>> wrote:<br>
> On 03/12/13 16:08, Maru Newby wrote:<br>
>><br>
>> I've been investigating a bug that is preventing VM's from receiving IP<br>
>> addresses when a Neutron service is under high load:<br>
>><br>
>> <a href="https://bugs.launchpad.net/neutron/+bug/1192381" target="_blank">https://bugs.launchpad.net/neutron/+bug/1192381</a><br>
>><br>
>> High load causes the DHCP agent's status updates to be delayed, causing<br>
>> the Neutron service to assume that the agent is down. This results in the<br>
>> Neutron service not sending notifications of port addition to the DHCP<br>
>> agent. At present, the notifications are simply dropped. A simple fix is<br>
>> to send notifications regardless of agent status. Does anybody have any<br>
>> objections to this stop-gap approach? I'm not clear on the implications of<br>
>> sending notifications to agents that are down, but I'm hoping for a simple<br>
>> fix that can be backported to both havana and grizzly (yes, this bug has<br>
>> been with us that long).<br>
>><br>
>> Fixing this problem for real, though, will likely be more involved. The<br>
>> proposal to replace the current wsgi framework with Pecan may increase the<br>
>> Neutron service's scalability, but should we continue to use a 'fire and<br>
>> forget' approach to notification? Being able to track the success or<br>
>> failure of a given action outside of the logs would seem pretty important,<br>
>> and allow for more effective coordination with Nova than is currently<br>
>> possible.<br>
><br>
><br>
> It strikes me that we ask an awful lot of a single neutron-server instance -<br>
> it has to take state updates from all the agents, it has to do scheduling,<br>
> it has to respond to API requests, and it has to communicate about actual<br>
> changes with the agents.<br>
><br>
> Maybe breaking some of these out the way nova has a scheduler and a<br>
> conductor and so on might be a good model (I know there are things people<br>
> are unhappy about with nova-scheduler, but imagine how much worse it would<br>
> be if it was built into the API).<br>
><br>
> Doing all of those tasks, and doing it largely single threaded, is just<br>
> asking for overload.<br>
><br>
> Cheers,<br>
> --<br>
> Stephen Gran<br>
> Senior Systems Integrator - <a href="http://theguardian.com" target="_blank">theguardian.com</a><br>
> Please consider the environment before printing this email.<br>
> ------------------------------------------------------------------<br>
> Visit <a href="http://theguardian.com" target="_blank">theguardian.com</a><br>
> On your mobile, download the Guardian iPhone app <a href="http://theguardian.com/iphone" target="_blank">theguardian.com/iphone</a> and<br>
> our iPad edition <a href="http://theguardian.com/iPad" target="_blank">theguardian.com/iPad</a> Save up to 33% by subscribing to the<br>
> Guardian and Observer - choose the papers you want and get full digital<br>
> access.<br>
> Visit <a href="http://subscribe.theguardian.com" target="_blank">subscribe.theguardian.com</a><br>
><br>
> This e-mail and all attachments are confidential and may also<br>
> be privileged. If you are not the named recipient, please notify<br>
> the sender and delete the e-mail and all attachments immediately.<br>
> Do not disclose the contents to another person. You may not use<br>
> the information for any purpose, or store, or copy, it in any way.<br>
><br>
> Guardian News & Media Limited is not liable for any computer<br>
> viruses or other material transmitted with or as part of this<br>
> e-mail. You should employ virus checking software.<br>
><br>
> Guardian News & Media Limited<br>
><br>
> A member of Guardian Media Group plc<br>
> Registered Office<br>
> PO Box 68164<br>
> Kings Place<br>
> 90 York Way<br>
> London<br>
> N1P 2AP<br>
><br>
> Registered in England Number 908396<br>
><br>
> --------------------------------------------------------------------------<br>
><br>
><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>