Open Stack

On 22/05/16 22:38, Lingxian Kong wrote:
> Hi, Zane, I think you must be interested in this:
> https://review.openstack.org/#/c/308664/

Oh! Yes that is very interesting. I'm glad that other people are 
thinking about these kinds of problems also.

My blueprint has a different focus, in that it's more about allowing the 
_user_ to configure these actions. Listening to oslo_messaging 
notifications instead of to Zaqar messages limits you to operator use 
cases only. (It's also likely much more efficient - listening to a large 
number of queues simultaneously is not the use case Zaqar is optimised 
for, which is essentially why I changed my mind about whether this 
should be implemented within Mistral.)

At least two projects that I know of (Ceilometer and Searchlight) are 
looking at implementing proxying of oslo_messaging notifications into 
Zaqar queues, and when that is combined with the Zaqar notification 
trigger we'll open up this functionality to everyone - other OpenStack 
services, operators and end-users.

e.g. a user will be able to subscribe (via Ceilometer or Searchlight) to 
a notification about a particular server from Nova in a Zaqar queue and 
have that trigger a Mistral workflow that marks the server unhealthy in 
Heat and triggers a reconvergence of the stack.

Unfortunately, there's always a tension between getting something done 
and in users' hands right now - which is easier to do if you stay within 
the confines of one project - and building flexible, orthogonal, 
user-pluggable abstractions, which are IMHO in the long-term best 
interests of users but generally require the co-operation, co-ordination 
and deployment of multiple projects.

It seems like we, the OpenStack community, are failing not only at 
co-ordinating these kinds of features across projects, but in even know 
what is being planned in other projects. Perhaps we need some sort of 
"Autonomous Applications working group"?

cheers,
Zane.

>
> Regards!
> -----------------------------------
> Lingxian Kong
>
>
> On Fri, May 20, 2016 at 9:49 AM, Zane Bitter <zbitter at redhat.com> wrote:
>> On 19/05/16 04:20, Thomas Herve wrote:
>>>
>>> On Wed, May 18, 2016 at 8:49 PM, Zane Bitter <zbitter at redhat.com> wrote:
>>>>
>>>> I've been lobbying the Mistral developers for $SUBJECT since, basically,
>>>> forever.[1][2][3] I like to think after a couple of years I succeeded in
>>>> changing their view on it from "crazy" to merely "unrealistic".[4] In the
>>>> last few months I've had a couple of realisations though:
>>>>
>>>> 1) The 'pull' model I've been suggesting is the wrong one,
>>>> architecturally
>>>> speaking. It's asking Mistral to do too much to poll Zaqar queues.
>>>> 2) A 'push' model is the correct architecture and it already exists in
>>>> the
>>>> form of Zaqar's Notifications, which suddenly makes this goal look very
>>>> realistic indeed.
>>>>
>>>> I've posted a Zaqar spec for this here:
>>>>
>>>> https://review.openstack.org/#/c/318202/
>>>
>>>
>>> Commented. I certainly need some more time to think about it.
>>
>>
>> Thanks, I updated the spec based on your comments.
>>
>>>> Not being super familiar with either project myself, I think this needs
>>>> close scrutiny from Mistral developers as well as Zaqar developers to
>>>> make
>>>> sure I haven't got any of the details wrong. I'd also welcome any
>>>> volunteers
>>>> interested in implementing this :)
>>>>
>>>>
>>>> One more long-term thing that I did *not* mention in the spec: there are
>>>> both Zaqar notifications and Mistral actions for sending email and
>>>> hitting
>>>> webhooks. These are two of the hardest things for a cloud operator to
>>>> secure. It would be highly advantageous if there were only _one_ place in
>>>> OpenStack where these were implemented. Either project would potentially
>>>> work - Zaqar notifications could call a simple, operator defined workflow
>>>> behind the scenes for email/webhook notifications; alternatively the
>>>> Mistral
>>>> email/webhook actions could drop a message on a Zaqar queue connected to
>>>> a
>>>> notification - although the former sounds easier to me. (And of course
>>>> clouds with only one of the services available could fall back to the
>>>> current plugins.) Something to think about for the future...
>>>
>>>
>>> And you're forgetting about Aodh alarm notifications, which are
>>> webhooks as well.
>>
>>
>> Yeah, those just need to go away :) Aodh can already post the notifications
>> to Zaqar instead, and hopefully we'll be able to migrate everything to that
>> method over time.
>>
>>> Unfortunately, I think none of them do a
>>> particularly great job in terms of robustness and security.
>>
>>
>> Agree. The best you can do is still middling, and we're not there yet.
>>
>>> I
>>> suggested moving away from doing webhook in Zaqar to Flavio in the
>>> past, and I think he responded that he thought it was part of the core
>>> functionality. It's hard to delegate such operations and create a
>>> dependency on another project. Maybe we can start by creating some
>>> library code.
>>
>>
>> I guess that's a start, but if I were running a large public cloud I'd
>> probably want to isolate this on its own network surrounded by some pretty
>> custom firewalls. A library doesn't really help make that easier.
>>
>> cheers,
>> Zane.
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>