Open Stack

On 13/11/14 09:58, Clint Byrum wrote:
> Excerpts from Zane Bitter's message of 2014-11-13 05:54:03 -0800:
>> On 13/11/14 03:29, Murugan, Visnusaran wrote:
>>> Hi all,
>>>
>>> Convergence-POC distributes stack operations by sending resource actions
>>> over RPC for any heat-engine to execute. Entire stack lifecycle will be
>>> controlled by worker/observer notifications. This distributed model has
>>> its own advantages and disadvantages.
>>>
>>> Any stack operation has a timeout and a single engine will be
>>> responsible for it. If that engine goes down, timeout is lost along with
>>> it. So a traditional way is for other engines to recreate timeout from
>>> scratch. Also a missed resource action notification will be detected
>>> only when stack operation timeout happens.
>>>
>>> To overcome this, we will need the following capability:
>>>
>>> 1.Resource timeout (can be used for retry)
>>
>> I don't believe this is strictly needed for phase 1 (essentially we
>> don't have it now, so nothing gets worse).
>>
>
> We do have a stack timeout, and it stands to reason that we won't have a
> single box with a timeout greenthread after this, so a strategy is
> needed.

Right, that was 2, but I was talking specifically about the resource 
retry. I think we agree on both points.

>> For phase 2, yes, we'll want it. One thing we haven't discussed much is
>> that if we used Zaqar for this then the observer could claim a message
>> but not acknowledge it until it had processed it, so we could have
>> guaranteed delivery.
>>
>
> Frankly, if oslo.messaging doesn't support reliable delivery then we
> need to add it.

That is straight-up impossible with AMQP. Either you ack the message and 
risk losing it if the worker dies before processing is complete, or you 
don't ack the message until it's processed and you become a blocker for 
every other worker trying to pull jobs off the queue. It works fine when 
you have only one worker; otherwise not so much. This is the crux of the 
whole "why isn't Zaqar just Rabbit" debate.

Most stuff in OpenStack gets around this by doing synchronous calls 
across oslo.messaging, where there is an end-to-end ack. We don't want 
that here though. We'll probably have to make do with having ways to 
recover after a failure (kick off another update with the same data is 
always an option). The hard part is that if something dies we don't 
really want to wait until the stack timeout to start recovering.

> Zaqar should have nothing to do with this and is, IMO, a
> poor choice at this stage, though I like the idea of using it in the
> future so that we can make Heat more of an outside-the-cloud app.

I'm inclined to agree that it would be hard to force operators to deploy 
Zaqar in order to be able to deploy Heat, and that we should probably be 
cautious for that reason.

That said, from a purely technical point of view it's not a poor choice 
at all - it has *exactly* the semantics we want (unlike AMQP), and at 
least to the extent that the operator wants to offer Zaqar to users 
anyway it completely eliminates a whole backend that they would 
otherwise have to deploy. It's a tragedy that all of OpenStack has not 
been designed to build upon itself in this way and it causes me physical 
pain to know that we're about to perpetuate it.

>>> 2.Recover from engine failure (loss of stack timeout, resource action
>>> notification)
>>>
>>> Suggestion:
>>>
>>> 1.Use task queue like celery to host timeouts for both stack and resource.
>>
>> I believe Celery is more or less a non-starter as an OpenStack
>> dependency because it uses Kombu directly to talk to the queue, vs.
>> oslo.messaging which is an abstraction layer over Kombu, Qpid, ZeroMQ
>> and maybe others in the future. i.e. requiring Celery means that some
>> users would be forced to install Rabbit for the first time.
>>
>> One option would be to fork Celery and replace Kombu with oslo.messaging
>> as its abstraction layer. Good luck getting that maintained though,
>> since Celery _invented_ Kombu to be it's abstraction layer.
>>
>
> A slight side point here: Kombu supports Qpid and ZeroMQ. Oslo.messaging

You're right about Kombu supporting Qpid, it appears they added it. I 
don't see ZeroMQ on the list though:

http://kombu.readthedocs.org/en/latest/userguide/connections.html#transport-comparison

> is more about having a unified API than a set of magic backends. It
> actually boggles my mind why we didn't just use kombu (cue 20 reactions
> with people saying it wasn't EXACTLY right), but I think we're committed

Well, we also have to take into account the fact that Qpid support was 
added only during the last 9 months, whereas oslo.messaging was 
implemented 3 years ago and time travel hasn't been invented yet (for 
any definition of 'yet').

> to oslo.messaging now. Anyway, celery would need no such refactor, as
> kombu would be able to access the same bus as everything else just fine.

Interesting, so that would make it easier to get Celery added to the 
global requirements, although we'd likely still have headaches to deal 
with around configuration.

>>> 2.Poll database for engine failures and restart timers/ retrigger
>>> resource retry (IMHO: This would be a traditional and weighs heavy)
>>>
>>> 3.Migrate heat to use TaskFlow. (Too many code change)
>>
>> If it's just handling timed triggers (maybe this is closer to #2) and
>> not migrating the whole code base, then I don't see why it would be a
>> big change (or even a change at all - it's basically new functionality).
>> I'm not sure if TaskFlow has something like this already. If not we
>> could also look at what Mistral is doing with timed tasks and see if we
>> could spin some of it out into an Oslo library.
>>
>
> I feel like it boils down to something running periodically checking for
> scheduled tasks that are due to run but have not run yet. I wonder if we
> can actually look at Ironic for how they do this, because Ironic polls
> power state of machines constantly, and uses a hash ring to make sure
> only one conductor is polling any one machine at a time. If we broke
> stacks up into a hash ring like that for the purpose of singleton tasks
> like timeout checking, that might work out nicely.

+1 for something like this, and +2 if we can get it from a library we 
don't have to write ourselves (whether it be TaskFlow or something spun 
out of Mistral or Ironic into Oslo).

cheers,
Zane.