<div dir="ltr">Hello all.<div><br></div><div>After discussion in IRC, I agree, that we should take care about interaction with eventlet at least because right now eventlet is the OpenStack production configuration. So we must be sure, that we'll get no issues, when we work with eventlet. </div>
<div><br></div><div>So I agree, that it makes a sense - to add a test for eventlet and sqlalchemy interaction to oslo.db. You convinced me :)</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 10, 2014 at 7:05 PM, Mike Bayer <span dir="ltr"><<a href="mailto:mbayer@redhat.com" target="_blank">mbayer@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><div class="">
<br>
<div>On 7/10/14, 7:47 AM, Sean Dague wrote:<br>
</div>
<blockquote type="cite">
<pre>Honestly, that seems weird to me.
oslo.db is built as a common layer for OpenStack services.
eventlet is used by most OpenStack services.
There are lots of known issues with eventlet vs. our db access patterns.
Knowing that the db layer works in the common OpenStack pattern seems
really important. And something that seems to make sense very close to
the db code itself.</pre>
</blockquote>
<br></div>
Yeah I am +1 on this, the use of eventlet is very prominent
throughout openstack components, and oslo.db is intended as a "glue"
layer between SQLAlchemy and those apps. The patterns that are
used with eventlet should be tested at the oslo.db level, as oslo.db
is responsible for configuration of the driver and additionally IMO
should be taking on a much greater role in establishing
transactional patterns which also have an impact on these issues.<br>
<br>
SQLAlchemy itself never spawns any threads. However, we certainly
have a crap-ton of tests that test the connection pool and other
concurrency-sensitive areas in the context of many threads being
run. It's a critical use case so we test against it.<br>
<br>
oslo.db should at every turn be attempting to remove redundancy from
downstream projects. If ten projects all use eventlet, they
shouldn't all have to replicate the same test over and over that
should just be upwards of them.<div><div class="h5"><br>
<br>
<br>
<br>
<br>
<br>
<blockquote type="cite">
<pre>
-Sean
On 07/10/2014 07:42 AM, Victor Sergeyev wrote:
</pre>
<blockquote type="cite">
<pre>Hello Angus!
IMO, the simple answer on your question is - tests for eventlet and
oslo.db interaction should be in the same place, where eventlet and
oslo.db interact. :)
A little digression - we suppose, that oslo.db should neither know, nor
take care whether target projects use eventlet/gevent/OS
threads/multiple processes/callbacks/etc for handling concurrency -
oslo.db just can't (and should not) make such decisions for users. For
the very same reason SQLAlchemy doesn't do that.
Thanks,
Victor
On Thu, Jul 10, 2014 at 10:55 AM, Angus Lees <<a href="mailto:guslees@gmail.com" target="_blank">guslees@gmail.com</a>
<a href="mailto:guslees@gmail.com" target="_blank"><mailto:guslees@gmail.com></a>> wrote:
We have an issue with neutron (and presumably elsewhere), where
mysqldb and eventlet may deadlock, until the mysqldb deadlock timer
fires.
I believe it's responsible for ~all of these failures:
<a href="http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkxvY2sgd2FpdCB0aW1lb3V0IGV4Y2VlZGVkOyB0cnkgcmVzdGFydGluZyB0cmFuc2FjdGlvblwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDA0OTcwMzgwMjc0fQ==" target="_blank">http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkxvY2sgd2FpdCB0aW1lb3V0IGV4Y2VlZGVkOyB0cnkgcmVzdGFydGluZyB0cmFuc2FjdGlvblwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDA0OTcwMzgwMjc0fQ==</a>
Now, the fix is one thing and is underway (the current favourite
option is just switching to a different mysql client library) - my
question here is instead about this test:
<a href="https://review.openstack.org/#/c/104436/" target="_blank">https://review.openstack.org/#/c/104436/</a>
This test (as written) is against oslo.db and drives eventlet +
sqlalchemy to confirm that the current sqlalchemy driver does _not_
have the above deadlock observed with mysqldb. I think it (or some
version of it) is an important test, but the oslo.db guys don't want
it in their testsuite since they've purged every explicit mention of
eventlet. I'm sympathetic to this pov.
I think we should have something like this test *somewhere*, at
least as long as we're using eventlet frequently.
I'm a bit new to openstack, so I'm lost in a maze of testing
options. Could some kind member of the TC point to where this test
*should* go?
--
- Gus
_______________________________________________
OpenStack-dev mailing list
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank"><mailto:OpenStack-dev@lists.openstack.org></a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
_______________________________________________
OpenStack-dev mailing list
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<pre>
</pre>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
OpenStack-dev mailing list
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</div></div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>