Open Stack

Zane,

We do follow shardy's suggestion of having worker/observer as eventlet in heat-engine. No new process. The timer will be executed under an engine's worker.

Question:
1. heat-engine processing resource-action failed (process killed)
2. heat-engine processing timeout for a stack fails (process killed)

In the above mentioned cases, I thought celery tasks would come to our rescue.

Convergence-poc implementation can recover from error and retry if there is a notification available.


-Vishnu

-----Original Message-----
From: Zane Bitter [mailto:zbitter at redhat.com] 
Sent: Thursday, November 13, 2014 7:05 PM
To: openstack-dev at lists.openstack.org
Subject: Re: [openstack-dev] [Heat] Using Job Queues for timeout ops

On 13/11/14 06:52, Angus Salkeld wrote:
> On Thu, Nov 13, 2014 at 6:29 PM, Murugan, Visnusaran 
> <visnusaran.murugan at hp.com <mailto:visnusaran.murugan at hp.com>> wrote:
>
>     Hi all,____
>
>     __ __
>
>     Convergence-POC distributes stack operations by sending resource
>     actions over RPC for any heat-engine to execute. Entire stack
>     lifecycle will be controlled by worker/observer notifications. This
>     distributed model has its own advantages and disadvantages.____
>
>     __ __
>
>     Any stack operation has a timeout and a single engine will be
>     responsible for it. If that engine goes down, timeout is lost along
>     with it. So a traditional way is for other engines to recreate
>     timeout from scratch. Also a missed resource action notification
>     will be detected only when stack operation timeout happens. __ __
>
>     __ __
>
>     To overcome this, we will need the following capability:____
>
>     __1.__Resource timeout (can be used for retry)
>
> We will shortly have a worker job, can't we have a job that just 
> sleeps that gets started in parallel with the job that is doing the work?
> It gets to the end of the sleep and runs a check.

What if that worker dies too? There's no guarantee that it'd even be a different worker. In fact, there's not even a guarantee that we'd have multiple workers.

BTW Steve Hardy's suggestion, which I have more or less come around to, is that the engines themselves should be the workers in convergence, to save operators deploying two types of processes. (The observers will still be a separate process though, in phase 2.)

>     ____
>
>     __2.__Recover from engine failure (loss of stack timeout, resource
>     action notification)____
>
>     __
>
>
> My suggestion above could catch failures as long as it was run in a 
> different process.
>
> -Angus
>
>     __
>
>     __ __
>
>     Suggestion:____
>
>     __1.__Use task queue like celery to host timeouts for both stack and
>     resource.____
>
>     __2.__Poll database for engine failures and restart timers/
>     retrigger resource retry (IMHO: This would be a traditional and
>     weighs heavy)____
>
>     __3.__Migrate heat to use TaskFlow. (Too many code change)____
>
>     __ __
>
>     I am not suggesting we use Task Flow. Using celery will have very
>     minimum code change. (decorate appropriate functions) ____
>
>     __ __
>
>     __ __
>
>     Your thoughts.____
>
>     __ __
>
>     -Vishnu____
>
>     IRC: ckmvishnu____
>
>
>     _______________________________________________
>     OpenStack-dev mailing list
>     OpenStack-dev at lists.openstack.org
>     <mailto:OpenStack-dev at lists.openstack.org>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev