Open Stack

Ok thanks again for the input.

Corey

On Wed, Jul 25, 2018 at 2:15 PM, Joshua Harlow <harlowja at fastmail.com>
wrote:

> So the only diff is that GreenThreadPoolExecutor was customized to work
> for eventlet (with a similar/same api as ThreadPoolExecutor); as for
> performance I would expect (under eventlet) that GreenThreadPoolExecutor
> would have better performance because it can use the native eventlet green
> objects (which it does try to use) instead of having to go threw the layers
> that ThreadPoolExecutor would have to use to achieve the same (and in this
> case as you found out it looks like those layers are not patched correctly
> in the newest ThreadPoolExecutor).
>
> Otherwise yes, under eventlet imho swap out the executor (assuming you can
> do this) and under threading swap in threadpool executor (ideally if done
> correctly the same stuff should 'just work').
>
> Corey Bryant wrote:
>
>> Josh,
>>
>> Thanks for the input. GreenThreadPoolExecutor does not have the deadlock
>> issue, so that is promising (at least with futurist 1.6.0).
>>
>> Does ThreadPoolExecutor have better performance than
>> GreenThreadPoolExecutor? Curious if we could just swap out
>> ThreadPoolExecutor for GreenThreadPoolExecutor.
>>
>> Thanks,
>> Corey
>>
>> On Wed, Jul 25, 2018 at 12:54 PM, Joshua Harlow <harlowja at fastmail.com
>> <mailto:harlowja at fastmail.com>> wrote:
>>
>>     Have you tried the following instead of threadpoolexecutor (which
>>     honestly should work as well, even under eventlet + eventlet
>> patching).
>>
>>     https://docs.openstack.org/futurist/latest/reference/index.
>> html#futurist.GreenThreadPoolExecutor
>>     <https://docs.openstack.org/futurist/latest/reference/index.
>> html#futurist.GreenThreadPoolExecutor>
>>
>>     If you have the ability to specify which executor your code is
>>     using, and you are running under eventlet I'd give preference to the
>>     green thread pool executor under that situation (and if not running
>>     under eventlet then prefer the threadpool executor variant).
>>
>>     As for @tomoto question; honestly openstack was created before
>>     asyncio was a thing so that was a reason and assuming eventlet
>>     patching is actually working then all the existing stdlib stuff
>>     should keep on working under eventlet (including
>>     concurrent.futures); otherwise eventlet.monkey_patch isn't working
>>     and that's breaking the eventlet api. If their contract is that only
>>     certain things work when monkey patched, that's fair, but that needs
>>     to be documented somewhere (honestly it's time imho to get the hell
>>     off eventlet everywhere but that likely requires rewrites of a lot
>>     of things, oops...).
>>
>>     -Josh
>>
>>     Corey Bryant wrote:
>>
>>         Hi All,
>>
>>         I'm trying to add Py3 packaging support for Ubuntu Rocky and
>>         while there
>>         are a lot of issues involved with supporting Py3.7, this is one
>>         of the
>>         big ones that I could use a hand with.
>>
>>         With py3.7, there's a deadlock when eventlet monkeypatch of stdlib
>>         thread modules is combined with use of ThreadPoolExecutor. I
>>         know this
>>         affects at least designate. The same or similar also affects heat
>>         (though I've not dug into the code the traceback after canceling
>>         tests
>>         matches that seen with designate). And it may affect other
>>         projects that
>>         I haven't touched yet.
>>
>>         How to recreate [1]:
>>         * designate: Add a tox.ini py37 target and run
>>         designate.tests.test_workers.test_processing.TestProcessingE
>> xecutor.test_execute_multiple_tasks
>>         * heat: Add a tox.ini py37 target and run tests
>>         * general: Run bpo34173-recreate.py
>>         <https://bugs.python.org/file47709/bpo34173-recreate.py
>>         <https://bugs.python.org/file47709/bpo34173-recreate.py>> from
>> issue
>>         34173 (see below).
>>         [1] ubuntu cosmic has py3.7
>>
>>         In issue 508 (see below) @tomoto asks "Eventlet and asyncio
>>         solve same
>>         problem. Why would you want concurrent.futures and eventlet in
>> same
>>         application?"
>>
>>         I told @tomoto that I'd seek input to that question from
>> upstream. I
>>         know there've been efforts to move away from eventlet but I just
>>         don't
>>         have the knowledge to  provide a good answer to him.
>>
>>         Here are the bugs/issues I currently have open for this:
>>         https://github.com/eventlet/eventlet/issues/508
>>         <https://github.com/eventlet/eventlet/issues/508>
>>         <https://github.com/eventlet/eventlet/issues/508
>>         <https://github.com/eventlet/eventlet/issues/508>>
>>         https://bugs.launchpad.net/designate/+bug/1782647
>>         <https://bugs.launchpad.net/designate/+bug/1782647>
>>         <https://bugs.launchpad.net/designate/+bug/1782647
>>         <https://bugs.launchpad.net/designate/+bug/1782647>>
>>         https://bugs.python.org/issue34173
>>         <https://bugs.python.org/issue34173>
>>         <https://bugs.python.org/issue34173
>>         <https://bugs.python.org/issue34173>>
>>
>>         Any help with this would be greatly appreciated!
>>
>>         Thanks,
>>         Corey
>>
>>         ____________________________________________________________
>> ______________
>>         OpenStack Development Mailing List (not for usage questions)
>>         Unsubscribe:
>>         OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>         <http://OpenStack-dev-request@lists.openstack.org?subject:un
>> subscribe>
>>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>>     ____________________________________________________________
>> ______________
>>     OpenStack Development Mailing List (not for usage questions)
>>     Unsubscribe:
>>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
>> >
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>> ____________________________________________________________
>> ______________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180725/e615b886/attachment.html>