Open Stack

So just to clear this up, my understanding is that asyncio and replacing
PRC calls with taskflow's job concept are two very different things. The
asyncio change would be retaining the RPC layer while the job concept[1]
would be something entirely different. I'm not a ceilometer expert though
so my understanding might be incorrect.

Overall the taskflow job mechanism is a lot like RQ[2] in concept which is
an abstraction around jobs, and doesn't mandate RPC or redis, or zookeeper
or ... as a job is performed. My biased
not-so-knowledgeable-about-ceilometer opinion is that a job mechanism
suits ceilometer more than a RPC one does (and since a job processing
mechanism is higher level abstraction it hopefully is more flexible with
regards to asyncio or other...).

[1] http://docs.openstack.org/developer/taskflow/jobs.html
[2] http://python-rq.org/

-----Original Message-----
From: Eoghan Glynn <eglynn at redhat.com>
Reply-To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Date: Sunday, July 6, 2014 at 6:28 AM
To: "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [oslo] Asyncio and oslo.messaging

>
>
>> This is an attempt to summarize a really useful discussion that Victor,
>> Flavio and I have been having today. At the bottom are some background
>> links - basically what I have open in my browser right now thinking
>> through all of this.
>
>Thanks for the detailed summary, it puts a more flesh on the bones
>than a brief conversation on the fringes of the Paris mid-cycle.
>
>Just a few clarifications and suggestions inline to add into the
>mix.
>
>> We're attempting to take baby-steps towards moving completely from
>> eventlet to asyncio/trollius. The thinking is for Ceilometer to be the
>> first victim.
>
>First beneficiary, I hope :)
> 
>> Ceilometer's code is run in response to various I/O events like REST API
>> requests, RPC calls, notifications received, etc. We eventually want the
>> asyncio event loop to be what schedules Ceilometer's code in response to
>> these events. Right now, it is eventlet doing that.
>
>Yes.
>
>And there is one other class of stimulus, also related to eventlet,
>that is very important for triggering the execution of ceilometer
>logic. That would be the timed tasks that drive polling of:
>
> * REST APIs provided by other openstack services
> * the local hypervisor running on each compute node
> * the SNMP daemons running at host-level etc.
>
>and also trigger periodic alarm evaluation.
>
>IIUC these tasks are all mediated via the oslo threadgroup's
>usage of eventlet.greenpool[1]. Would this logic also be replaced
>as part of this effort?
>
>> Now, because we're using eventlet, the code that is run in response to
>> these events looks like synchronous code that makes a bunch of
>> synchronous calls. For example, the code might do some_sync_op() and
>> that will cause a context switch to a different greenthread (within the
>> same native thread) where we might handle another I/O event (like a REST
>> API request)
>
>Just to make the point that most of the agents in the ceilometer
>zoo tend to react to just a single type of stimulus, as opposed
>to a mix of dispatching from both message bus and the REST API.
>
>So to classify, we'd have:
>
> * compute-agent: timer tasks for polling
> * central-agent: timer tasks for polling
> * notification-agent: dispatch of "external" notifications from
>   the message bus
> * collector: dispatch of "internal" metering messages from the
>   message bus
> * api-service: dispatch of REST API calls
> * alarm-evaluator: timer tasks for alarm evaluation
> * alarm-notifier: dispatch of "internal" alarm notifications
>
>IIRC, the only case where there's a significant mix of trigger
>styles is the partitioned alarm evaluator, where assignments of
>alarm subsets for evaluation is driven over RPC, whereas the
>actual thresholding is triggered by a timer.
>
>> Porting from eventlet's implicit async approach to asyncio's explicit
>> async API will be seriously time consuming and we need to be able to do
>> it piece-by-piece.
>
>Yes, I agree, a step-wise approach is the key here.
>
>So I'd love to have some sense of the time horizon for this
>effort. It clearly feels like a multi-cycle effort, so the main
>question in my mind right now is whether we should be targeting
>the first deliverables for juno-3?
>
>That would provide a proof-point in advance of the K* summit,
>where I presume the task would be get wider buy-in for the idea.
>
>If it makes sense to go ahead and aim the first baby steps for
>juno-3, then we'd need to have a ceilometer-spec detailing these
>changes. This would need to be proposed by say EoW and then
>landed before the spec acceptance deadline for juno (~July 21st).
>
>We could use this spec proposal to dig into the perceived benefits
>of this effort:
>
> * the obvious win around getting rid of the eventlet black-magic
> * plus possibly other benefits such as code clarity and ease of
>   maintenance
>
>and OTOH get a heads-up on the risks:
>
> * possible immaturity in the new framework?
> * overhead involved in contributors getting to grips with the
>   new coroutine model
>
>> The question then becomes what do we need to do in order to port a
>> single oslo.messaging RPC endpoint method in Ceilometer to asyncio's
>> explicit async approach?
>
>One approach would be to select one well-defined area of ceilometer
>as an initial test-bed for these ideas.
>
>And one potential candidate for that would be the partitioned alarm
>evaluator, which uses:
>
> 1. fan-out RPC for the heartbeats underpinning master-slave
>    coordination
> 2. RPC calls for alarm allocations and assignments
>
>I spoke to Cyril Roelandt at the mid-cycle, who is interested in:
>
> * replacing #1 with the tooz distributed co-ordination library[2]
> * and also possibly replacing #2 with taskflow
>
>The benefit of using taskflow for "sticky" task assignments isn't
>100% clear, so it may actually make better sense to just use tooz
>for the leadership election, and the new asyncio model for #2.
>
>Starting there would have the advantage of being out on the side
>of the main ceilometer pipeline.
>
>However, if we do decide to go ahead with taskflow, then we could
>fine another good starting point for asyncio as an alternative.
>
>>   - when all of ceilometer has been ported over to asyncio coroutines,
>>     we can stop monkey patching, stop using greenio and switch to the
>>     asyncio event loop
>
>... kick back and light a cigar! :)
>
>Cheers,
>Eoghan
>
>[1] 
>https://github.com/openstack/oslo-incubator/blob/master/openstack/common/t
>hreadgroup.py#L72
>[2] https://github.com/stackforge/tooz
>[3] https://wiki.openstack.org/wiki/TaskFlow
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev