Open Stack

On 05/19/2017 04:23 AM, Mehdi Abaakouk wrote:
> 
And some applications rely
> on implicit internal contract/behavior/assumption.

IMO that's a bug for them.    I'm inspired to see that Keystone, Nova 
etc. are able to move between and eventlet backend and a mod_wsgi 
backend.    IMO eventlet is really not needed for those services that 
present a REST interface.   Although for a message queue with lots of 
long-running connections that receive events, that's a place where I 
*would* want to use a polling / non-blocking model.  But I'd use it 
explicitly, not with monkeypatching.

> 
> Since a new API is needed, why not writing a new lib. Anyways when you
> get rid of eventlet you have so many thing to change to ensure your
> performance will not drop. 

While I don't know the specifics for your project(s), I don't buy that 
in general because IMO eventlet is not giving us any performance boost 
in the majority of cases.   most of our IO is blocking on the database 
and all the applications have DB connections throttled at about 50 per 
process at the max, and that's only recently, it used to be just 15.



Changing from oslo.service to cotyledon is
> really easy on the side.

I'd ask why not oslo.cotyledon but it seems there's a faction here that 
is overall moving out of the Openstack umbrella in any case.


> 
>> Docs state: "oslo.service being impossible to fix and bringing an 
>> heavy dependency on eventlet, "  is there a discussion thread on that?
> 
> Not really, I just put some comments on reviews and discus this on IRC.
> Since nobody except Telemetry have expressed/try to get rid of eventlet.

Many (most?) of the web services can run under mod_wsgi with threads, 
Keystone seems to be standard on this now and I get the impression Nova 
is going in that direction too.    (anyone correct me if I'm wrong on 
any of that, I looked to ask around on IRC but it's too late in the day).



> 
> For the story we first get rid of eventlet in Telemetry, fixes couple of
> performance issue due to using threading/process instead
> greenlet/greenthread.
> 
> Then we fall into some weird issue due to oslo.service internal
> implementation. Process not exiting properly, signals not received,
> deadlock when signal are received, unkillable process,
> tooz/oslo.messaging heartbeat not scheduled correctly, worker not
> restarted when they are dead. All of what we expect from oslo.service
> was not working correctly anymore because we remove the line
> 'eventlet.monkeypatch()'.

So, I've used gevent more than eventlet in my own upstream non-blocking 
work, and while this opinion is like spilling water in the ocean, I 
think applications should never use monkeypatching.   They should call 
into the eventlet/gevent greenlet API directly if that's what they want 
to do.

Of course this means that database logic has to move out of greenlets 
entirely since none of the DBAPIs use non-blocking IO.  That's fine. 
Database-oriented business logic should not be in greenlets.    I've 
written about this as well.    If one is familiar enough with greenlets 
and threads you can write an application that makes explicit use of 
both.   However, that's application level stuff.   Web service apps like 
Nova conductor  / Neutron / Keystone should not be aware of any of that. 
   They should be coded to assume nothing about context switching.  IMO 
the threading model is "safer" to code towards since you have to handle 
locking and concurrency contingencies explicitly without hardwiring that 
to your assumptions about when context switching is to take place and 
when it's not.


> 
> For example, when oslo.service receive a signal, it can arrive on any
> thread, this thread is paused, the callback is run in this thread
> context, but if the callback try to discus to your code in this thread,
> the process lockup, because your code is paused. Python
> offers tool to avoid that (signal.set_wakeup_fd), but oslo.service don't
> use it. I have tried to run callbacks only on the main thread with
> set_wakeup_fd, to avoid this kind of issue but I fail. The whole
> oslo.service code is clearly not designed to be threadsafe/signalsafe.
> Well, it works for eventlet because you have only one real thread.
> 
> And this is just one example on complicated thing I have tried to fix,
> before starting cotyledon.

I've no doubt oslo.service has major eventlet problems baked in, I've 
looked at it a little bit but didn't go too far with it.   That still 
doesn't mean there shouldn't be an "oslo.service2" that can effectively 
produce a concurrency-agnostic platform.    It of course would have the 
goal in mind of moving projects off eventlet since as I mentioned, 
eventlet monkeypatching should not be used which means our services 
should do most of their "implicitly concurrent" work within threads.

Basically I think openstack should be getting off eventlet in a big way 
so I guess my sentiment here is that the Gnocchi / Cotyledon /etc. 
faction is just splitting off rather than serving as any kind of 
direction for the rest of Openstack to start looking.  But that's only 
an impression, maybe projects will use Cotyledon anyway.   If every 
project goes off and uses something completely different though, then I 
think we're losing.   The point of oslo was to prevent that.

> 
>> I'm finding it hard to believe that only a few years ago, everyone saw 
>> the wisdom of not re-implementing everything in their own projects and 
>> using a common layer like oslo, and already that whole situation is 
>> becoming forgotten - not just for consistency, but also when a bug is 
>> found, if fixed in oslo it gets fixed for everyone.
> 
> Because the internal of cotyledon and oslo.service are so different.
> Having the code in oslo or not doesn't help for maintenance anymore.
> Cotyledon is a lib, code and bugs :) can already be shared between
> projects that doesn't want eventlet.
> 
>> An increase in the scope of oslo is essential to dealing with the 
>> issue of "complexity" in openstack. 
> 
> Increasing the scope of oslo works only if libs have maintainers. But
> most of them lack of people today. Most of oslo libs are in maintenance
> mode. But that another subject.
> 
>> The state of openstack as dozens of individual software projects each 
>> with their own idiosyncratic quirks, CLIs, process and deployment 
>> models, and everything else that is visible to operators is ground 
>> zero for perceived operator complexity.
> 
> Cotyledon have been written to be Openstack agnostic. But I have also
> write an optional module within the library to glue oslo.config and
> cotyledon. Mainly to mimic the oslo.config options/reload of
> oslo.service and make operators experience unchanged for Openstack
> people.

OK, so that would be your oslo.cotyledon.   That works.



> 
> -- 
> Mehdi Abaakouk
> mail: sileht at sileht.net
> irc: sileht
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>