Open Stack

On Sun, 2014-07-06 at 09:28 -0400, Eoghan Glynn wrote:
> 
> > This is an attempt to summarize a really useful discussion that Victor,
> > Flavio and I have been having today. At the bottom are some background
> > links - basically what I have open in my browser right now thinking
> > through all of this.
> 
> Thanks for the detailed summary, it puts a more flesh on the bones
> than a brief conversation on the fringes of the Paris mid-cycle.
> 
> Just a few clarifications and suggestions inline to add into the
> mix.
> 
> > We're attempting to take baby-steps towards moving completely from
> > eventlet to asyncio/trollius. The thinking is for Ceilometer to be the
> > first victim.
> 
> First beneficiary, I hope :)
>  
> > Ceilometer's code is run in response to various I/O events like REST API
> > requests, RPC calls, notifications received, etc. We eventually want the
> > asyncio event loop to be what schedules Ceilometer's code in response to
> > these events. Right now, it is eventlet doing that.
> 
> Yes.
> 
> And there is one other class of stimulus, also related to eventlet,
> that is very important for triggering the execution of ceilometer
> logic. That would be the timed tasks that drive polling of:
> 
>  * REST APIs provided by other openstack services 
>  * the local hypervisor running on each compute node
>  * the SNMP daemons running at host-level etc.
> 
> and also trigger periodic alarm evaluation.
> 
> IIUC these tasks are all mediated via the oslo threadgroup's
> usage of eventlet.greenpool[1]. Would this logic also be replaced
> as part of this effort?

As part of the broader "switch from eventlet to asyncio" effort, yes
absolutely.

At the core of any event loop is code to do select() (or equivalents)
waiting for file descriptors to become readable or writable, or timers
to expire. We want to switch from the eventlet event loop to the asyncio
event loop.

The ThreadGroup abstraction from oslo-incubator is an interface to the
eventlet event loop. When you do:

  self.tg.add_timer(interval, self._evaluate_assigned_alarms)

You're saying "run evaluate_assigned_alarms() every $interval seconds,
using select() to sleep between executions".

When you do:

  self.tg.add_thread(self.start_udp)

you're saying "run some code which will either run to completion or set
wait for fd or timer events using select()".

The asyncio versions of those will be:

  event_loop.call_later(delay, callback)
  event_loop.call_soon(callback)

where the supplied callbacks will be asyncio 'coroutines' which rather
than doing:

  def foo(...):
      buf = read(fd)

and rely on eventlet's monkey patch to cause us to enter the event
loop's select() when the read() blocks, we instead do:

  @asyncio.coroutine
  def foo(...):
      buf = yield from read(fd)

which shows exactly where we might yield to the event loop.

The challenge is that porting code like the foo() function above is
pretty invasive and we can't simply port an entire service at once. So,
we need to be able to support a service using both eventlet-reliant code
and asyncio coroutines.

In your example of the openstack.common.threadgroup API - we would
initially need to add support for scheduling asyncio coroutine callback
arguments as eventlet greenthreads in add_timer() and add_thread(), and
later we would port threadgroup itself to rely completely on asyncio.

> > Now, because we're using eventlet, the code that is run in response to
> > these events looks like synchronous code that makes a bunch of
> > synchronous calls. For example, the code might do some_sync_op() and
> > that will cause a context switch to a different greenthread (within the
> > same native thread) where we might handle another I/O event (like a REST
> > API request)
> 
> Just to make the point that most of the agents in the ceilometer
> zoo tend to react to just a single type of stimulus, as opposed
> to a mix of dispatching from both message bus and the REST API.
> 
> So to classify, we'd have:
> 
>  * compute-agent: timer tasks for polling
>  * central-agent: timer tasks for polling
>  * notification-agent: dispatch of "external" notifications from
>    the message bus
>  * collector: dispatch of "internal" metering messages from the
>    message bus
>  * api-service: dispatch of REST API calls
>  * alarm-evaluator: timer tasks for alarm evaluation
>  * alarm-notifier: dispatch of "internal" alarm notifications
> 
> IIRC, the only case where there's a significant mix of trigger
> styles is the partitioned alarm evaluator, where assignments of
> alarm subsets for evaluation is driven over RPC, whereas the
> actual thresholding is triggered by a timer.

Cool, that's helpful. I think the key thing is deciding which stimulus
(and hence agent) we should start with.

> > Porting from eventlet's implicit async approach to asyncio's explicit
> > async API will be seriously time consuming and we need to be able to do
> > it piece-by-piece.
> 
> Yes, I agree, a step-wise approach is the key here.
> 
> So I'd love to have some sense of the time horizon for this
> effort. It clearly feels like a multi-cycle effort, so the main
> question in my mind right now is whether we should be targeting
> the first deliverables for juno-3?
> 
> That would provide a proof-point in advance of the K* summit,
> where I presume the task would be get wider buy-in for the idea.
> 
> If it makes sense to go ahead and aim the first baby steps for
> juno-3, then we'd need to have a ceilometer-spec detailing these
> changes. This would need to be proposed by say EoW and then
> landed before the spec acceptance deadline for juno (~July 21st).
> 
> We could use this spec proposal to dig into the perceived benefits
> of this effort:
> 
>  * the obvious win around getting rid of the eventlet black-magic
>  * plus possibly other benefits such as code clarity and ease of
>    maintenance
> 
> and OTOH get a heads-up on the risks:
> 
>  * possible immaturity in the new framework?
>  * overhead involved in contributors getting to grips with the
>    new coroutine model

My sense is that until someone does a prototype with a single callback
function (for whichever stimulus), we really don't have much of an idea
what's involved.

> > The question then becomes what do we need to do in order to port a
> > single oslo.messaging RPC endpoint method in Ceilometer to asyncio's
> > explicit async approach?
> 
> One approach would be to select one well-defined area of ceilometer
> as an initial test-bed for these ideas.
> 
> And one potential candidate for that would be the partitioned alarm
> evaluator, which uses:
> 
>  1. fan-out RPC for the heartbeats underpinning master-slave
>     coordination
>  2. RPC calls for alarm allocations and assignments
> 
> I spoke to Cyril Roelandt at the mid-cycle, who is interested in:
> 
>  * replacing #1 with the tooz distributed co-ordination library[2]
>  * and also possibly replacing #2 with taskflow
> 
> The benefit of using taskflow for "sticky" task assignments isn't
> 100% clear, so it may actually make better sense to just use tooz
> for the leadership election, and the new asyncio model for #2.
> 
> Starting there would have the advantage of being out on the side
> of the main ceilometer pipeline.
> 
> However, if we do decide to go ahead with taskflow, then we could
> fine another good starting point for asyncio as an alternative.

No opinion from me on where to start, really. I'd expect the ability to
use the underlying libraries called from the callback in a sane way with
asyncio would be a significant factor.

> >   - when all of ceilometer has been ported over to asyncio coroutines,
> >     we can stop monkey patching, stop using greenio and switch to the
> >     asyncio event loop
> 
> ... kick back and light a cigar! :)

Indeed :)

Thanks,
Mark.

> [1] https://github.com/openstack/oslo-incubator/blob/master/openstack/common/threadgroup.py#L72
> [2] https://github.com/stackforge/tooz
> [3] https://wiki.openstack.org/wiki/TaskFlow