Open Stack

Mike,

Thank you for saying all that you said above.

Best regards,
Boris Pavlovic

On Tue, May 12, 2015 at 2:35 AM, Clint Byrum <clint at fewbar.com> wrote:

> Excerpts from Mike Bayer's message of 2015-05-11 15:44:30 -0700:
> >
> > On 5/11/15 5:25 PM, Robert Collins wrote:
> > >
> > > Details: Skip over this bit if you know it all already.
> > >
> > > The GIL plays a big factor here: if you want to scale the amount of
> > > CPU available to a Python service, you have two routes:
> > > A) move work to a different process through some RPC - be that DB's
> > > using SQL, other services using oslo.messaging or HTTP - whatever.
> > > B) use C extensions to perform work in threads - e.g. openssl context
> > > processing.
> > >
> > > To increase concurrency you can use threads, eventlet, asyncio,
> > > twisted etc - because within a single process *all* Python bytecode
> > > execution happens inside the GIL lock, so you get at most one CPU for
> > > a CPU bound workload. For an IO bound workload, you can fit more work
> > > in by context switching within that one CPU capacity. And - the GIL is
> > > a poor scheduler, so at the limit - an IO bound workload where the IO
> > > backend has more capacity than we have CPU to consume it within our
> > > process, you will run into priority inversion and other problems.
> > > [This varies by Python release too].
> > >
> > > request_duration = time_in_cpu + time_blocked
> > > request_cpu_utilisation = time_in_cpu/request_duration
> > > cpu_utilisation = concurrency * request_cpu_utilisation
> > >
> > > Assuming that we don't want any one process to spend a lot of time at
> > > 100% - to avoid such at-the-limit issues, lets pick say 80%
> > > utilisation, or a safety factor of 0.2. If a single request consumes
> > > 50% of its duration waiting on IO, and 50% of its duration executing
> > > bytecode, we can only run one such request concurrently without
> > > hitting 100% utilisations. (2*0.5 CPU == 1). For a request that spends
> > > 75% of its duration waiting on IO and 25% on CPU, we can run 3 such
> > > requests concurrently without exceeding our target of 80% utilisation:
> > > (3*0.25=0.75).
> > >
> > > What we have today in our standard architecture for OpenStack is
> > > optimised for IO bound workloads: waiting on the
> > > network/subprocesses/disk/libvirt etc. Running high numbers of
> > > eventlet handlers in a single process only works when the majority of
> > > the work being done by a handler is IO.
> >
> > Everything stated here is great, however in our situation there is one
> > unfortunate fact which renders it completely incorrect at the moment.
> > I'm still puzzled why we are getting into deep think sessions about the
> > vagaries of the GIL and async when there is essentially a full-on
> > red-alert performance blocker rendering all of this discussion useless,
> > so I must again remind us: what we have *today* in Openstack is *as
> > completely un-optimized as you can possibly be*.
> >
> > The most GIL-heavy nightmare CPU bound task you can imagine running on
> > 25 threads on a ten year old Pentium will run better than the Openstack
> > we have today, because we are running a C-based, non-eventlet patched DB
> > library within a single OS thread that happens to use eventlet, but the
> > use of eventlet is totally pointless because right now it blocks
> > completely on all database IO.   All production Openstack applications
> > today are fully serialized to only be able to emit a single query to the
> > database at a time; for each message sent, the entire application blocks
> > an order of magnitude more than it would under the GIL waiting for the
> > database library to send a message to MySQL, waiting for MySQL to send a
> > response including the full results, waiting for the database to unwrap
> > the response into Python structures, and finally back to the Python
> > space, where we can send another database message and block the entire
> > application and all greenlets while this single message proceeds.
> >
> > To share a link I've already shared about a dozen times here, here's
> > some tests under similar conditions which illustrate what that
> > concurrency looks like:
> >
> http://www.diamondtin.com/2014/sqlalchemy-gevent-mysql-python-drivers-comparison/
> .
> > MySQLdb takes *20 times longer* to handle the work of 100 sessions than
> > PyMySQL when it's inappropriately run under gevent, when there is
> > modestly high concurrency happening.   When I talk about moving to
> > threads, this is not a "won't help or hurt" kind of issue, at the moment
> > it's a change that will immediately allow massive improvement to the
> > performance of all Openstack applications instantly.  We need to change
> > the DB library or dump eventlet.
> >
> > As far as if we should dump eventlet or use a pure-Python DB library, my
> > contention is that a thread based + C database library will outperform
> > an eventlet + Python-based database library. Additionally, if we make
> > either change, when we do so we may very well see all kinds of new
> > database-concurrency related bugs in our apps too, because we will be
> > talking to the database much more intensively all the sudden; it is my
> > opinion that a traditional threading model will be an easier environment
> > to handle working out the approach to these issues; we have to assume
> > "concurrency at any time" in any case because we run multiple instances
> > of Nova etc. at the same time.  At the end of the day, we aren't going
> > to see wildly better performance with one approach over the other in any
> > case, so we should pick the one that is easier to develop, maintain, and
> > keep stable.
> >
>
> Mike, I agree with the entire paragraph above, and I've been surprised to
> see the way this thread has gone with so much speculation.  Optimization
> can be such a divisive thing, I think we need to be mindful of that.
>
> Anyway, there is additional thought that might change the decision
> a bit. There is one "pro" to changing to use pymsql vs. changing to
> use threads, and that is that it isolates the change to only database
> access. Switching to threading means introducing threads to every piece
> of code we might touch while multiple threads are active.
>
> It really seems worth it to see if I/O bound portions of OpenStack
> become more responsive with pymysql before embarking on a change to the
> concurrency model. If it doesn't, not much harm done, and if it does,
> but makes us CPU bound, well then we have even more of a reason to set
> out on such a large task.
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150512/98ffb39f/attachment.html>