<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le mar. 17 sept. 2019 à 19:55, Albert Braden <<a href="mailto:Albert.Braden@synopsys.com">Albert.Braden@synopsys.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I had not heard about the eventlet heartbeat issue. Where can I read more about it?<br></blockquote><div><br></div>Under apache and mod_wsgi eventlet green thread doesn't work properly.<br>Nova faced this issue few months ago through the use of oslo.messaging and especially through the heartbeat's rabbitmq driver.<br><br>The heartbeat was runned by using a green thread under apache and mod_wsgi, so after few secondes/minutes the heartbeat thread became idle and so the connection with the rabbitmq server was closed and re-opened etc... Hence, that introduced a lot of connections opened and closed between the client and the server.</div><div class="gmail_quote"><br></div><div class="gmail_quote">You can find more discuss about there:<br></div><div class="gmail_quote"><div>- <a href="http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005822.html">http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005822.html</a></div><div><br></div><div>And the oslo.messaging fix related to this issue :<br></div><div>- <a href="https://github.com/openstack/oslo.messaging/commit/22f240b82fffbd62be8568a7d0d3369134596ace">https://github.com/openstack/oslo.messaging/commit/22f240b82fffbd62be8568a7d0d3369134596ace</a></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
The [wsgi] section of my nova.conf is default; nothing is uncommented.<br>
<br>
-----Original Message-----<br>
From: Sean Mooney <<a href="mailto:smooney@redhat.com" target="_blank">smooney@redhat.com</a>> <br>
Sent: Tuesday, September 17, 2019 9:50 AM<br>
To: Albert Braden <<a href="mailto:albertb@synopsys.com" target="_blank">albertb@synopsys.com</a>>; <a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a><br>
Cc: Ben Nemec <<a href="mailto:openstack@nemebean.com" target="_blank">openstack@nemebean.com</a>>; Chris Hoge <<a href="mailto:chris@openstack.org" target="_blank">chris@openstack.org</a>><br>
Subject: Re: [oslo][nova] Nova causes MySQL timeouts<br>
<br>
On Tue, 2019-09-17 at 16:36 +0000, Albert Braden wrote:<br>
> I thought I had figured out that the solution was to increase the MySQL wait_timeout so that it is longer than the<br>
> nova (and glance, neutron, etc.) connection_recycle_time (3600). I increased my MySQL wait_timeout to 6000:<br>
> <br>
> root@us01odc-qa-ctrl1:~# mysqladmin variables|grep wait_timeout|grep -v _wait<br>
> > wait_timeout | 6000 <br>
> <br>
> But I still see the MySQL errors. There's no LB; we are pointing to a single MySQL host. <br>
> <br>
> Sep 11 14:59:56 us01odc-qa-ctrl1 mysqld[1052956]: 2019-09-11 14:59:56 8016 [Warning] Aborted connection 8016 to db:<br>
> 'nova' user: 'nova' host: '<a href="http://us01odc-qa-ctrl2.internal.synopsys.com" rel="noreferrer" target="_blank">us01odc-qa-ctrl2.internal.synopsys.com</a>' (Got timeout reading communication packets)<br>
> Sep 11 14:59:57 us01odc-qa-ctrl1 mysqld[1052956]: 2019-09-11 14:59:57 8019 [Warning] Aborted connection 8019 to db:<br>
> 'glance' user: 'glance' host: '<a href="http://us01odc-qa-ctrl1.internal.synopsys.com" rel="noreferrer" target="_blank">us01odc-qa-ctrl1.internal.synopsys.com</a>' (Got timeout reading communication packets)<br>
> Sep 11 14:59:57 us01odc-qa-ctrl1 mysqld[1052956]: 2019-09-11 14:59:57 8018 [Warning] Aborted connection 8018 to db:<br>
> 'nova_api' user: 'nova' host: '<a href="http://us01odc-qa-ctrl2.internal.synopsys.com" rel="noreferrer" target="_blank">us01odc-qa-ctrl2.internal.synopsys.com</a>' (Got timeout reading communication packets)<br>
> Sep 11 15:00:50 us01odc-qa-ctrl1 mysqld[1052956]: 2019-09-11 15:00:50 8022 [Warning] Aborted connection 8022 to db:<br>
> 'nova_api' user: 'nova' host: '<a href="http://us01odc-qa-ctrl1.internal.synopsys.com" rel="noreferrer" target="_blank">us01odc-qa-ctrl1.internal.synopsys.com</a>' (Got timeout reading communication packets)<br>
> <br>
> The errors come from nova, neutron, glance and keystone; it appears that all default to 3600. So it appears that, even<br>
> with wait_timeout > connection_recycle_time we still see mysql timeout errors.<br>
> <br>
> Just for fun I tried setting the MySQL wait_timeout to 86400 and restarting MySQL. I expected that this would pause<br>
> the "Aborted connection" errors for 24 hours, but they started again after an hour. So it looks like my original<br>
> assumption was incorrect. I thought nova was keeping connections open until the MySQL server timed them out, but now<br>
> it appears that something else is happening.<br>
> <br>
> Has anyone successfully stopped these MySQL error messages?<br>
<br>
could this be related to the eventlet heartbeat issue we see for rabbitmq when running the api under mod_wsgi/uwsgi?<br>
<br>
e.g. hav eyou confirmed that you wsgi serer is configure to use 1 thread and multiple processes for concurancy<br>
multiple thread in one process might have issues.<br>
> -----Original Message-----<br>
> From: Ben Nemec <<a href="mailto:openstack@nemebean.com" target="_blank">openstack@nemebean.com</a>> <br>
> Sent: Monday, September 9, 2019 9:50 AM<br>
> To: Chris Hoge <<a href="mailto:chris@openstack.org" target="_blank">chris@openstack.org</a>>; <a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a><br>
> Subject: Re: [oslo][nova] Nova causes MySQL timeouts<br>
> <br>
> <br>
> <br>
> On 9/9/19 11:38 AM, Chris Hoge wrote:<br>
> > In my personal experience, running Nova on a four core machine without<br>
> > limiting the number of database connections will easily exhaust the<br>
> > available connections to MySQL/MariaDB. Keep in mind that the limit<br>
> > applies to every instance of a service, so if Nova starts 'm' services<br>
> > replicated for 'n' cores with 'd' possible connections you'll be up to<br>
> > ‘m x n x d' connections. It gets big fast.<br>
> > <br>
> > The default setting of '0' (that is, unlimited) does not make for a good<br>
> > first-run experience, IMO.<br>
> <br>
> We don't default to 0. We default to 5: <br>
> <br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_oslo.db_stein_reference_opts.html-23database.max-5Fpool-5Fsize&d=DwIDaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=W7apBhYbgfvGgB46HWLe-By9d_MYg6RB_eU3C2mARRY&s=p7bBYcuhnDR_J08MWFBj8XLiRUUV8JfruAIcl0zF234&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_oslo.db_stein_reference_opts.html-23database.max-5Fpool-5Fsize&d=DwIDaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=W7apBhYbgfvGgB46HWLe-By9d_MYg6RB_eU3C2mARRY&s=p7bBYcuhnDR_J08MWFBj8XLiRUUV8JfruAIcl0zF234&e=</a><br>
> <br>
> <br>
> > <br>
> > This issue comes up every few years or so, and the consensus previously<br>
> > is that 200-2000 connections is recommended based on your needs. Your<br>
> > database has to be configured to handle the load and looking at the<br>
> > configuration value across all your services and setting them<br>
> > consistently and appropriately is important.<br>
> > <br>
> > <br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_pipermail_openstack-2Ddev_2015-2DApril_061808.html&d=DwIDaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=W7apBhYbgfvGgB46HWLe-By9d_MYg6RB_eU3C2mARRY&s=FGLfZK5eHj7z_xL-5DJsPgHkOt_T131ugvicMvcMDbc&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_pipermail_openstack-2Ddev_2015-2DApril_061808.html&d=DwIDaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=W7apBhYbgfvGgB46HWLe-By9d_MYg6RB_eU3C2mARRY&s=FGLfZK5eHj7z_xL-5DJsPgHkOt_T131ugvicMvcMDbc&e=</a><br>
> > <br>
> <br>
> Thanks, I did not recall that discussion.<br>
> <br>
> If I'm reading it correctly, Jay is suggesting that for MySQL we should <br>
> just disable connection pooling. As I noted earlier, I don't think we <br>
> expose the ability to do that in oslo.db (patches welcome!), but setting <br>
> max_pool_size to 1 would get you pretty close. Maybe we should add that <br>
> to the help text for the option in oslo.db?<br>
> <br>
> > <br>
> > > On Sep 6, 2019, at 7:34 AM, Ben Nemec <<a href="mailto:openstack@nemebean.com" target="_blank">openstack@nemebean.com</a>> wrote:<br>
> > > <br>
> > > Tagging with oslo as this sounds related to oslo.db.<br>
> > > <br>
> > > On 9/5/19 7:37 PM, Albert Braden wrote:<br>
> > > > After more googling it appears that max_pool_size is a maximum limit on the number of connections that can stay<br>
> > > > open, and max_overflow is a maximum limit on the number of connections that can be temporarily opened when the<br>
> > > > pool has been consumed. It looks like the defaults are 5 and 10 which would keep 5 connections open all the time<br>
> > > > and allow 10 temp.<br>
> > > > Do I need to set max_pool_size to 0 and max_overflow to the number of connections that I want to allow? Is that<br>
> > > > a reasonable and correct configuration? Intuitively that doesn't seem right, to have a pool size of 0, but if<br>
> > > > the "pool" is a group of connections that will remain open until they time out, then maybe 0 is correct?<br>
> > > <br>
> > > I don't think so. According to [0] and [1], a pool_size of 0 means unlimited. You could probably set it to 1 to<br>
> > > minimize the number of connections kept open, but then I expect you'll have overhead from having to re-open<br>
> > > connections frequently.<br>
> > > <br>
> > > It sounds like you could use a NullPool to eliminate connection pooling entirely, but I don't think we support<br>
> > > that in oslo.db. Based on the error message you're seeing, I would take a look at connection_recycle_time[2]. I<br>
> > > seem to recall seeing a comment that the recycle time needs to be shorter than any of the timeouts in the path<br>
> > > between the service and the db (so anything like haproxy or mysql itself). Shortening that, or lengthening<br>
> > > intervening timeouts, might get rid of these disconnection messages.<br>
> > > <br>
> > > 0: <br>
> > > <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_oslo.db_stein_reference_opts.html-23database.max-5Fpool-5Fsize&d=DwIDaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=W7apBhYbgfvGgB46HWLe-By9d_MYg6RB_eU3C2mARRY&s=p7bBYcuhnDR_J08MWFBj8XLiRUUV8JfruAIcl0zF234&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_oslo.db_stein_reference_opts.html-23database.max-5Fpool-5Fsize&d=DwIDaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=W7apBhYbgfvGgB46HWLe-By9d_MYg6RB_eU3C2mARRY&s=p7bBYcuhnDR_J08MWFBj8XLiRUUV8JfruAIcl0zF234&e=</a><br>
> > > <br>
> > > 1: <br>
> > > <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.sqlalchemy.org_en_13_core_pooling.html-23sqlalchemy.pool.QueuePool.-5F-5Finit-5F-5F&d=DwIDaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=W7apBhYbgfvGgB46HWLe-By9d_MYg6RB_eU3C2mARRY&s=_EIhQyyj1gSM0PrX7de3yJr8hNi7tD8-tnfPo2VV_LU&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.sqlalchemy.org_en_13_core_pooling.html-23sqlalchemy.pool.QueuePool.-5F-5Finit-5F-5F&d=DwIDaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=W7apBhYbgfvGgB46HWLe-By9d_MYg6RB_eU3C2mARRY&s=_EIhQyyj1gSM0PrX7de3yJr8hNi7tD8-tnfPo2VV_LU&e=</a><br>
> > > <br>
> > > 2: <br>
> > > <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_oslo.db_stein_reference_opts.html-23database.connection-5Frecycle-5Ftime&d=DwIDaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=W7apBhYbgfvGgB46HWLe-By9d_MYg6RB_eU3C2mARRY&s=xDnj80EQrxXwenOLgmKEaJbF3VRIylapDgqyMs81pSY&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_oslo.db_stein_reference_opts.html-23database.connection-5Frecycle-5Ftime&d=DwIDaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=W7apBhYbgfvGgB46HWLe-By9d_MYg6RB_eU3C2mARRY&s=xDnj80EQrxXwenOLgmKEaJbF3VRIylapDgqyMs81pSY&e=</a><br>
> > > <br>
> > > <br>
> > > > *From:* Albert Braden <<a href="mailto:Albert.Braden@synopsys.com" target="_blank">Albert.Braden@synopsys.com</a>><br>
> > > > *Sent:* Wednesday, September 4, 2019 10:19 AM<br>
> > > > *To:* <a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a><br>
> > > > *Cc:* Gaëtan Trellu <<a href="mailto:gaetan.trellu@incloudus.com" target="_blank">gaetan.trellu@incloudus.com</a>><br>
> > > > *Subject:* RE: Nova causes MySQL timeouts<br>
> > > > We’re not setting max_pool_size nor max_overflow option presently. I googled around and found this document:<br>
> > > > <br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_keystone_stein_configuration_config-2Doptions.html&d=DwIDaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=W7apBhYbgfvGgB46HWLe-By9d_MYg6RB_eU3C2mARRY&s=NXcUpNTYGd6ZP-1oOUaQXsF7rHQ0mAt4e9uL8zzd0KA&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_keystone_stein_configuration_config-2Doptions.html&d=DwIDaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=W7apBhYbgfvGgB46HWLe-By9d_MYg6RB_eU3C2mARRY&s=NXcUpNTYGd6ZP-1oOUaQXsF7rHQ0mAt4e9uL8zzd0KA&e=</a><br>
> > > > = <<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_keystone_stein_configuration_config-" rel="noreferrer" target="_blank">https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.openstack.org_keystone_stein_configuration_config-</a><br>
> > > > 2Doptions.html&d=DwMGaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=3eF4Bv1HRQW6gl7<br>
> > > > II12rTTSKj_A9_LDISS6hU0nP-R0&s=0EGWx9qW60G1cxoPFCIv_G1-iXX20jKcC5-AwlCWk8g&e=><br>
> > > > Document says:<br>
> > > > [api_database]<br>
> > > > connection_recycle_time = 3600 (Integer) Timeout before idle SQL connections are reaped.<br>
> > > > max_overflow = None (Integer) If set, use this value for max_overflow with<br>
> > > > SQLAlchemy.<br>
> > > > max_pool_size = None (Integer) Maximum number of SQL connections to keep open<br>
> > > > in a pool.<br>
> > > > [database]<br>
> > > > connection_recycle_time = 3600 (Integer) Timeout before idle SQL connections are reaped.<br>
> > > > min_pool_size = 1 (Integer) Minimum number of SQL connections to keep<br>
> > > > open in a pool.<br>
> > > > max_overflow = 50 (Integer) If set, use this value for max_overflow<br>
> > > > with SQLAlchemy.<br>
> > > > max_pool_size = None (Integer) Maximum number of SQL connections to keep open<br>
> > > > in a pool.<br>
> > > > If min_pool_size is >0, would that cause at least 1 connection to remain open until it times out? What are the<br>
> > > > recommended values for these, to allow unused connections to close before they time out? Is “min_pool_size = 0”<br>
> > > > an acceptable setting?<br>
> > > > My settings are default:<br>
> > > > [api_database]:<br>
> > > > #connection_recycle_time = 3600<br>
> > > > #max_overflow = <None><br>
> > > > #max_pool_size = <None><br>
> > > > [database]:<br>
> > > > #connection_recycle_time = 3600<br>
> > > > #min_pool_size = 1<br>
> > > > #max_overflow = 50<br>
> > > > #max_pool_size = 5<br>
> > > > It’s not obvious what max_overflow does. Where can I find a document that explains more about these settings?<br>
> > > > *From:* Gaëtan Trellu <<a href="mailto:gaetan.trellu@incloudus.com" target="_blank">gaetan.trellu@incloudus.com</a> <mailto:<a href="mailto:gaetan.trellu@incloudus.com" target="_blank">gaetan.trellu@incloudus.com</a>>><br>
> > > > *Sent:* Tuesday, September 3, 2019 1:37 PM<br>
> > > > *To:* Albert Braden <<a href="mailto:albertb@synopsys.com" target="_blank">albertb@synopsys.com</a> <mailto:<a href="mailto:albertb@synopsys.com" target="_blank">albertb@synopsys.com</a>>><br>
> > > > *Cc:* <a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a> <mailto:<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
> > > > *Subject:* Re: Nova causes MySQL timeouts<br>
> > > > Hi Albert,<br>
> > > > It is a configuration issue, have a look to max_pool_size and max_overflow options under [database] section.<br>
> > > > Keep in mind than more workers you will have more connections will be opened on the database.<br>
> > > > Gaetan (goldyfruit)<br>
> > > > On Sep 3, 2019 4:31 PM, Albert Braden <<a href="mailto:Albert.Braden@synopsys.com" target="_blank">Albert.Braden@synopsys.com</a> <mailto:<a href="mailto:Albert.Braden@synopsys.com" target="_blank">Albert.Braden@synopsys.com</a>>> wrote:<br>
> > > > It looks like nova is keeping mysql connections open until they time<br>
> > > > out. How are others responding to this issue? Do you just ignore the<br>
> > > > mysql errors, or is it possible to change configuration so that nova<br>
> > > > closes and reopens connections before they time out? Or is there a<br>
> > > > way to stop mysql from logging these aborted connections without<br>
> > > > hiding real issues?<br>
> > > > Aborted connection 10726 to db: 'nova' user: 'nova' host: 'asdf'<br>
> > > > (Got timeout reading communication packets)<br>
> > <br>
> > <br>
> <br>
> <br>
<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Hervé Beraud</div><div>Senior Software Engineer<br></div><div>Red Hat - Openstack Oslo</div><div>irc: hberaud</div><div>-----BEGIN PGP SIGNATURE-----<br><br>wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+<br>Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+<br>RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP<br>F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G<br>5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g<br>glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw<br>m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ<br>hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0<br>qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y<br>F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3<br>B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O<br>v6rDpkeNksZ9fFSyoY2o<br>=ECSj<br>-----END PGP SIGNATURE-----<br><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div>