<div dir="ltr">Thanks, I'll give it a try.<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Nov 22, 2013 at 2:35 AM, Carl Baldwin <span dir="ltr"><<a href="mailto:carl@ecbaldwin.net" target="_blank">carl@ecbaldwin.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<br>
Please tell me if your experience is similar to what I experienced:<br>
<br>
1. I would see *at most one* "MySQL server has gone away" error for<br>
each process that was spawned as an API worker. I saw them within a<br>
minute of spawning the workers and then I did not see these errors<br>
anymore until I restarted the server and spawned new processes.<br>
<br>
2. I noted in patch set 7 the line of code that completely fixed this<br>
for me. Please confirm that you have applied a patch that includes<br>
this fix.<br>
<br>
<a href="https://review.openstack.org/#/c/37131/7/neutron/wsgi.py" target="_blank">https://review.openstack.org/#/c/37131/7/neutron/wsgi.py</a><br>
<br>
3. I did not change anything with pool_recycle or idle_interval in my<br>
config files. All I did was set api_workers to the number of workers<br>
that I wanted to spawn. The line of code with my comment in it above<br>
was sufficient for me.<br>
<br>
It could be that there is another cause for the errors that you're<br>
seeing. For example, is there a max connections setting in mysql that<br>
might be exceeded when you spawn multiple workers? More detail would<br>
be helpful.<br>
<br>
Cheers,<br>
Carl<br>
<div class="HOEnZb"><div class="h5"><br>
On Wed, Nov 20, 2013 at 7:40 PM, Zhongyue Luo <<a href="mailto:zhongyue.nah@intel.com">zhongyue.nah@intel.com</a>> wrote:<br>
> Carl,<br>
><br>
> By 2006 I mean the "MySQL server has gong away" error code.<br>
><br>
> The error message was still appearing when idle_timeout is set to 1 and the<br>
> quantum API server did not work in my case.<br>
><br>
> Could you perhaps share your conf file when applying this patch?<br>
><br>
> Thanks.<br>
><br>
><br>
><br>
> On Thu, Nov 21, 2013 at 3:34 AM, Carl Baldwin <<a href="mailto:carl@ecbaldwin.net">carl@ecbaldwin.net</a>> wrote:<br>
>><br>
>> Hi, sorry for the delay in response. I'm glad to look at it.<br>
>><br>
>> Can you be more specific about the error? Maybe paste the error your<br>
>> seeing in <a href="http://paste.openstack.org" target="_blank">paste.openstack.org</a>? I don't find any reference to "2006".<br>
>> Maybe I'm missing something.<br>
>><br>
>> Also, is the patch that you applied the most recent? With the final<br>
>> version of the patch it was no longer necessary for me to set<br>
>> pool_recycle or idle_interval.<br>
>><br>
>> Thanks,<br>
>> Carl<br>
>><br>
>> On Tue, Nov 19, 2013 at 7:14 PM, Zhongyue Luo <<a href="mailto:zhongyue.nah@intel.com">zhongyue.nah@intel.com</a>><br>
>> wrote:<br>
>> > Carl, Yingjun,<br>
>> ><br>
>> > I'm still getting the 2006 error even by configuring idle_interval to 1.<br>
>> ><br>
>> > I applied the patch to the RDO havana dist on centos 6.4.<br>
>> ><br>
>> > Are there any other options I should be considering such as min/max pool<br>
>> > size or use_tpool?<br>
>> ><br>
>> > Thanks.<br>
>> ><br>
>> ><br>
>> ><br>
>> > On Sat, Sep 7, 2013 at 3:33 AM, Baldwin, Carl (HPCS Neutron)<br>
>> > <<a href="mailto:carl.baldwin@hp.com">carl.baldwin@hp.com</a>> wrote:<br>
>> >><br>
>> >> This pool_recycle parameter is already configurable using the<br>
>> >> idle_timeout<br>
>> >> configuration variable in neutron.conf. I tested this with a value of<br>
>> >> 1<br>
>> >> as suggested and it did get rid of the mysql server gone away messages.<br>
>> >><br>
>> >> This is a great clue but I think I would like a long-term solution that<br>
>> >> allows the end-user to still configure this like they were before.<br>
>> >><br>
>> >> I'm currently thinking along the lines of calling something like<br>
>> >> pool.dispose() in each child immediately after it is spawned. I think<br>
>> >> this should invalidate all of the existing connections so that when a<br>
>> >> connection is checked out of the pool a new one will be created fresh.<br>
>> >><br>
>> >> Thoughts? I'll be testing. Hopefully, I'll have a fixed patch up<br>
>> >> soon.<br>
>> >><br>
>> >> Cheers,<br>
>> >> Carl<br>
>> >><br>
>> >> From: Yingjun Li <<a href="mailto:liyingjun1988@gmail.com">liyingjun1988@gmail.com</a>><br>
>> >> Reply-To: OpenStack Development Mailing List<br>
>> >> <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
>> >> Date: Thursday, September 5, 2013 8:28 PM<br>
>> >> To: OpenStack Development Mailing List<br>
>> >> <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
>> >> Subject: Re: [openstack-dev] [Neutron] The three API server<br>
>> >> multi-worker<br>
>> >> process patches.<br>
>> >><br>
>> >><br>
>> >> +1 for Carl's patch, and i have abandoned my patch..<br>
>> >><br>
>> >> About the `MySQL server gone away` problem, I fixed it by set<br>
>> >> 'pool_recycle' to 1 in db/api.py.<br>
>> >><br>
>> >> 在 2013年9月6日星期五，Nachi Ueno 写道：<br>
>> >><br>
>> >> Hi Folks<br>
>> >><br>
>> >> We choose <a href="https://review.openstack.org/#/c/37131/" target="_blank">https://review.openstack.org/#/c/37131/</a> <-- This patch to go<br>
>> >> on.<br>
>> >> We are also discussing in this patch.<br>
>> >><br>
>> >> Best<br>
>> >> Nachi<br>
>> >><br>
>> >><br>
>> >><br>
>> >> 2013/9/5 Baldwin, Carl (HPCS Neutron) <<a href="mailto:carl.baldwin@hp.com">carl.baldwin@hp.com</a>>:<br>
>> >> > Brian,<br>
>> >> ><br>
>> >> > As far as I know, no consensus was reached.<br>
>> >> ><br>
>> >> > A problem was discovered that happens when spawning multiple<br>
>> >> > processes.<br>
>> >> > The mysql connection seems to "go away" after between 10-60 seconds<br>
>> >> > in<br>
>> >> > my<br>
>> >> > testing causing a seemingly random API call to fail. After that, it<br>
>> >> > is<br>
>> >> > okay. This must be due to some interaction between forking the<br>
>> >> > process<br>
>> >> > and the mysql connection pool. This needs to be solved but I haven't<br>
>> >> > had<br>
>> >> > the time to look in to it this week.<br>
>> >> ><br>
>> >> > I'm not sure if the other proposal suffers from this problem.<br>
>> >> ><br>
>> >> > Carl<br>
>> >> ><br>
>> >> > On 9/4/13 3:34 PM, "Brian Cline" <<a href="mailto:bcline@softlayer.com">bcline@softlayer.com</a>> wrote:<br>
>> >> ><br>
>> >> >>Was any consensus on this ever reached? It appears both reviews are<br>
>> >> >> still<br>
>> >> >>open. I'm partial to review 37131 as it attacks the problem a more<br>
>> >> >>concisely, and, as mentioned, combined the efforts of the two more<br>
>> >> >>effective patches. I would echo Carl's sentiments that it's an easy<br>
>> >> >>review minus the few minor behaviors discussed on the review thread<br>
>> >> >>today.<br>
>> >> >><br>
>> >> >>We feel very strongly about these making it into Havana -- being<br>
>> >> >> confined<br>
>> >> >>to a single neutron-server instance per cluster or region is a huge<br>
>> >> >>bottleneck--essentially the only controller process with massive CPU<br>
>> >> >>churn in environments with constant instance churn, or sudden large<br>
>> >> >>batches of new instance requests.<br>
>> >> >><br>
>> >> >>In Grizzly, this behavior caused addresses not to be issued to some<br>
>> >> >>instances during boot, due to quantum-server thinking the DHCP agents<br>
>> >> >>timed out and were no longer available, when in reality they were<br>
>> >> >> just<br>
>> >> >>backlogged (waiting on quantum-server, it seemed).<br>
>> >> >><br>
>> >> >>Is it realistically looking like this patch will be cut for h3?<br>
>> >> >><br>
>> >> >>--<br>
>> >> >>Brian Cline<br>
>> >> >>Software Engineer III, Product Innovation<br>
>> >> >><br>
>> >> >>SoftLayer, an IBM Company<br>
>> >> >>4849 Alpha Rd, Dallas, TX 75244<br>
>> >> >><a href="tel:214.782.7876" value="+12147827876">214.782.7876</a> direct | <a href="mailto:bcline@softlayer.com">bcline@softlayer.com</a><br>
>> >> >><br>
>> >> >><br>
>> >> >>-----Original Message-----<br>
>> >> >>From: Baldwin, Carl (HPCS Neutron) [mailto:<a href="mailto:carl.baldwin@hp.com">carl.baldwin@hp.com</a>]<br>
>> >> >>Sent: Wednesday, August 28, 2013 3:04 PM<br>
>> >> >>To: Mark McClain<br>
>> >> >>Cc: OpenStack Development Mailing List<br>
>> >> >>Subject: [openstack-dev] [Neutron] The three API server multi-worker<br>
>> >> >>process patches.<br>
>> >> >><br>
>> >> >>All,<br>
>> >> >><br>
>> >> >>We've known for a while now that some duplication of work happened<br>
>> >> >> with<br>
>> >> >>respect to adding multiple worker processes to the neutron-server.<br>
>> >> >> There<br>
>> >> >>were a few mistakes made which led to three patches being done<br>
>> >> >>independently of each other.<br>
>> >> >><br>
>> >> >>Can we settle on one and accept it?<br>
>> >> >><br>
>> >> >>I have changed my patch at the suggestion of one of the other 2<br>
>> >> >> authors,<br>
>> >> >>Peter Feiner, in attempt to find common ground. It now uses<br>
>> >> >> openstack<br>
>> >> >>common code and therefore it is more concise than any of the original<br>
>> >> >>three and should be pretty easy to review. I'll admit to some bias<br>
>> >> >>toward<br>
>> >> >>my own implementation but most importantly, I would like for one of<br>
>> >> >> these<br>
>> >> >>implementations to land and start seeing broad usage in the community<br>
>> >> >>earlier than later.<br>
>> >> >><br>
>> >> >>Carl Baldwin<br>
>> >> >><br>
>> >> >>PS Here are the two remaining patches. The third has been abandoned.<br>
>> >> >><br>
>> >> >><a href="https://review.openstack.org/#/c/37131/" target="_blank">https://review.openstack.org/#/c/37131/</a><br>
>> >> >><a href="https://review.openstack.org/#/c/36487/" target="_blank">https://review.openstack.org/#/c/36487/</a><br>
>> >> >><br>
>> >> >><br>
>> >> >>_______________________________________________<br>
>> >> >>OpenStack-dev mailing list<br>
>> >> >><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> >> >><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>> >> ><br>
>> >> ><br>
>> >> > _______________________________________________<br>
>> >> > OpenStack-dev mailing list<br>
>> >> > <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> >><br>
>> >><br>
>> >> _______________________________________________<br>
>> >> OpenStack-dev mailing list<br>
>> >> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> > --<br>
>> > Intel SSG/STO/DCST/CIT<br>
>> > 880 Zixing Road, Zizhu Science Park, Minhang District, 200241, Shanghai,<br>
>> > China<br>
>> > <a href="tel:%2B862161166500" value="+862161166500">+862161166500</a><br>
>> ><br>
>> > _______________________________________________<br>
>> > OpenStack-dev mailing list<br>
>> > <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>> ><br>
>><br>
>> _______________________________________________<br>
>> OpenStack-dev mailing list<br>
>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
><br>
> --<br>
> Intel SSG/STO/DCST/CIT<br>
> 880 Zixing Road, Zizhu Science Park, Minhang District, 200241, Shanghai,<br>
> China<br>
> <a href="tel:%2B862161166500" value="+862161166500">+862161166500</a><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div dir="ltr"><div><b>Intel SSG/STO/DCST/CIT</b></div>
<div>880 Zixing Road, Zizhu Science Park, Minhang District, 200241, Shanghai,
China<br></div>
<div>+862161166500</div></div>
</div>