Open Stack

> Most stuff in OpenStack gets around this by doing synchronous calls across oslo.messaging, where there is an end-to-end ack. We don't want that here > >though. We'll probably have to make do with having ways to recover after a failure (kick off another update with the same data is always an option). The hard >part is that if something dies we don't really want to wait until the stack timeout to start recovering.

We should be able to address this in convergence without having to wait for stack timeout.  This scenario would be similar to initiating the stack update while another large stack update is still progress.  We are looking into addressing this scenario.

Thanks,
Divakar

-----Original Message-----
From: Zane Bitter [mailto:zbitter at redhat.com] 
Sent: Thursday, November 13, 2014 11:26 PM
To: openstack-dev at lists.openstack.org
Subject: Re: [openstack-dev] [Heat] Using Job Queues for timeout ops

On 13/11/14 09:58, Clint Byrum wrote:
> Excerpts from Zane Bitter's message of 2014-11-13 05:54:03 -0800:
>> On 13/11/14 03:29, Murugan, Visnusaran wrote:
>>> Hi all,
>>>
>>> Convergence-POC distributes stack operations by sending resource 
>>> actions over RPC for any heat-engine to execute. Entire stack 
>>> lifecycle will be controlled by worker/observer notifications. This 
>>> distributed model has its own advantages and disadvantages.
>>>
>>> Any stack operation has a timeout and a single engine will be 
>>> responsible for it. If that engine goes down, timeout is lost along 
>>> with it. So a traditional way is for other engines to recreate 
>>> timeout from scratch. Also a missed resource action notification 
>>> will be detected only when stack operation timeout happens.
>>>
>>> To overcome this, we will need the following capability:
>>>
>>> 1.Resource timeout (can be used for retry)
>>
>> I don't believe this is strictly needed for phase 1 (essentially we 
>> don't have it now, so nothing gets worse).
>>
>
> We do have a stack timeout, and it stands to reason that we won't have 
> a single box with a timeout greenthread after this, so a strategy is 
> needed.

Right, that was 2, but I was talking specifically about the resource retry. I think we agree on both points.

>> For phase 2, yes, we'll want it. One thing we haven't discussed much 
>> is that if we used Zaqar for this then the observer could claim a 
>> message but not acknowledge it until it had processed it, so we could 
>> have guaranteed delivery.
>>
>
> Frankly, if oslo.messaging doesn't support reliable delivery then we 
> need to add it.

That is straight-up impossible with AMQP. Either you ack the message and risk losing it if the worker dies before processing is complete, or you don't ack the message until it's processed and you become a blocker for every other worker trying to pull jobs off the queue. It works fine when you have only one worker; otherwise not so much. This is the crux of the whole "why isn't Zaqar just Rabbit" debate.

Most stuff in OpenStack gets around this by doing synchronous calls across oslo.messaging, where there is an end-to-end ack. We don't want that here though. We'll probably have to make do with having ways to recover after a failure (kick off another update with the same data is always an option). The hard part is that if something dies we don't really want to wait until the stack timeout to start recovering.



> Zaqar should have nothing to do with this and is, IMO, a poor choice 
> at this stage, though I like the idea of using it in the future so 
> that we can make Heat more of an outside-the-cloud app.

I'm inclined to agree that it would be hard to force operators to deploy Zaqar in order to be able to deploy Heat, and that we should probably be cautious for that reason.

That said, from a purely technical point of view it's not a poor choice at all - it has *exactly* the semantics we want (unlike AMQP), and at least to the extent that the operator wants to offer Zaqar to users anyway it completely eliminates a whole backend that they would otherwise have to deploy. It's a tragedy that all of OpenStack has not been designed to build upon itself in this way and it causes me physical pain to know that we're about to perpetuate it.

>>> 2.Recover from engine failure (loss of stack timeout, resource 
>>> action
>>> notification)
>>>
>>> Suggestion:
>>>
>>> 1.Use task queue like celery to host timeouts for both stack and resource.
>>
>> I believe Celery is more or less a non-starter as an OpenStack 
>> dependency because it uses Kombu directly to talk to the queue, vs.
>> oslo.messaging which is an abstraction layer over Kombu, Qpid, ZeroMQ 
>> and maybe others in the future. i.e. requiring Celery means that some 
>> users would be forced to install Rabbit for the first time.
>>
>> One option would be to fork Celery and replace Kombu with 
>> oslo.messaging as its abstraction layer. Good luck getting that 
>> maintained though, since Celery _invented_ Kombu to be it's abstraction layer.
>>
>
> A slight side point here: Kombu supports Qpid and ZeroMQ. 
> Oslo.messaging

You're right about Kombu supporting Qpid, it appears they added it. I don't see ZeroMQ on the list though:

http://kombu.readthedocs.org/en/latest/userguide/connections.html#transport-comparison

> is more about having a unified API than a set of magic backends. It 
> actually boggles my mind why we didn't just use kombu (cue 20 
> reactions with people saying it wasn't EXACTLY right), but I think 
> we're committed

Well, we also have to take into account the fact that Qpid support was added only during the last 9 months, whereas oslo.messaging was implemented 3 years ago and time travel hasn't been invented yet (for any definition of 'yet').

> to oslo.messaging now. Anyway, celery would need no such refactor, as 
> kombu would be able to access the same bus as everything else just fine.

Interesting, so that would make it easier to get Celery added to the global requirements, although we'd likely still have headaches to deal with around configuration.

>>> 2.Poll database for engine failures and restart timers/ retrigger 
>>> resource retry (IMHO: This would be a traditional and weighs heavy)
>>>
>>> 3.Migrate heat to use TaskFlow. (Too many code change)
>>
>> If it's just handling timed triggers (maybe this is closer to #2) and 
>> not migrating the whole code base, then I don't see why it would be a 
>> big change (or even a change at all - it's basically new functionality).
>> I'm not sure if TaskFlow has something like this already. If not we 
>> could also look at what Mistral is doing with timed tasks and see if 
>> we could spin some of it out into an Oslo library.
>>
>
> I feel like it boils down to something running periodically checking 
> for scheduled tasks that are due to run but have not run yet. I wonder 
> if we can actually look at Ironic for how they do this, because Ironic 
> polls power state of machines constantly, and uses a hash ring to make 
> sure only one conductor is polling any one machine at a time. If we 
> broke stacks up into a hash ring like that for the purpose of 
> singleton tasks like timeout checking, that might work out nicely.

+1 for something like this, and +2 if we can get it from a library we
don't have to write ourselves (whether it be TaskFlow or something spun out of Mistral or Ironic into Oslo).

cheers,
Zane.

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev