Open Stack

On Thu, May 18, 2017 at 03:16:20PM -0400, Mike Bayer wrote:
>
>
>On 05/18/2017 02:37 PM, Julien Danjou wrote:
>>On Thu, May 18 2017, Mike Bayer wrote:
>>
>>>I'm not understanding this?  do you mean this?
>>
>>In the long run, yes. Unfortunately, we're not happy with the way Oslo
>>libraries are managed and too OpenStack centric. I've tried for the last
>>couple of years to move things on, but it's barely possible to deprecate
>>anything and contribute, so I feel it's safer to start fresh and better
>>alternative. Cotyledon by Mehdi is a good example of what can be
>>achieved.
>
>
>here's cotyledon:
>
>https://cotyledon.readthedocs.io/en/latest/
>
>
>replaces oslo.service with a multiprocessing approach that doesn't use 
>eventlet.  great!  any openstack service that rides on oslo.service 
>would like to be able to transparently switch from eventlet to 
>multiprocessing the same way they can more or less switch to mod_wsgi 
>at the moment.   IMO this should be part of oslo.service itself.   

I have quickly presented cotyledon some summit ago, we said we will wait
to see if other projects want to get rid of eventlet before adopting
such new lib (or merge it with oslo.service).

But for now, the lib is still under telemetry umbrella.

Keeping the current API and supporting both are (I think) impossible.
The current API is too eventlet centric. And some applications rely
on implicit internal contract/behavior/assumption.

Dealing about concurrent/thread/signal safety in multithreading app or
eventlet app is already hard enough. So having the lib that deals with
both is even harder. We already have oslo.messaging that deals with
3 threads models, this is just an unending story of race conditions.

Since a new API is needed, why not writing a new lib. Anyways when you
get rid of eventlet you have so many thing to change to ensure your
performance will not drop. Changing from oslo.service to cotyledon is
really easy on the side.

>Docs state: "oslo.service being impossible to fix and bringing an 
>heavy dependency on eventlet, "  is there a discussion thread on that?

Not really, I just put some comments on reviews and discus this on IRC.
Since nobody except Telemetry have expressed/try to get rid of eventlet.

For the story we first get rid of eventlet in Telemetry, fixes couple of
performance issue due to using threading/process instead
greenlet/greenthread.

Then we fall into some weird issue due to oslo.service internal
implementation. Process not exiting properly, signals not received,
deadlock when signal are received, unkillable process,
tooz/oslo.messaging heartbeat not scheduled correctly, worker not
restarted when they are dead. All of what we expect from oslo.service
was not working correctly anymore because we remove the line
'eventlet.monkeypatch()'.

For example, when oslo.service receive a signal, it can arrive on any
thread, this thread is paused, the callback is run in this thread
context, but if the callback try to discus to your code in this thread,
the process lockup, because your code is paused. Python
offers tool to avoid that (signal.set_wakeup_fd), but oslo.service don't
use it. I have tried to run callbacks only on the main thread with
set_wakeup_fd, to avoid this kind of issue but I fail. The whole
oslo.service code is clearly not designed to be threadsafe/signalsafe.
Well, it works for eventlet because you have only one real thread.

And this is just one example on complicated thing I have tried to fix,
before starting cotyledon.

>I'm finding it hard to believe that only a few years ago, everyone saw 
>the wisdom of not re-implementing everything in their own projects and 
>using a common layer like oslo, and already that whole situation is 
>becoming forgotten - not just for consistency, but also when a bug is 
>found, if fixed in oslo it gets fixed for everyone.

Because the internal of cotyledon and oslo.service are so different.
Having the code in oslo or not doesn't help for maintenance anymore.
Cotyledon is a lib, code and bugs :) can already be shared between
projects that doesn't want eventlet.

>An increase in the scope of oslo is essential to dealing with the 
>issue of "complexity" in openstack. 

Increasing the scope of oslo works only if libs have maintainers. But
most of them lack of people today. Most of oslo libs are in maintenance
mode. But that another subject.

> The state of openstack as dozens 
>of individual software projects each with their own idiosyncratic 
>quirks, CLIs, process and deployment models, and everything else that 
>is visible to operators is ground zero for perceived operator 
>complexity.

Cotyledon have been written to be Openstack agnostic. But I have also
write an optional module within the library to glue oslo.config and
cotyledon. Mainly to mimic the oslo.config options/reload of
oslo.service and make operators experience unchanged for Openstack
people.

--
Mehdi Abaakouk
mail: sileht at sileht.net
irc: sileht
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 884 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170519/88485ecc/attachment.sig>