Open Stack

Flavio Percoco wrote:
> On 10/04/15 10:00 -0400, Ryan Brown wrote:
>> On 04/09/2015 09:40 PM, Angus Salkeld wrote:
>>> On Fri, Apr 10, 2015 at 6:39 AM, Zane Bitter <zbitter at redhat.com
>>> <mailto:zbitter at redhat.com>> wrote:
>>>
>>> On 06/04/15 22:55, Angus Salkeld wrote:
>>>
>>> Hi all
>>>
>>> For quite some time we (Heat team) have wanted to be able to send
>>> messages to our
>>> users (by user I do not mean the Operator, but the User that is
>>> interacting with the client).
>>>
>>> What do I mean by "user messages", and how do they differ from our
>>> current log messages
>>> and notifications?
>>> - Our current logs are for the operator and have information
>>> that the
>>> user should not have
>>> (ip addresses, hostnames, configuration options, other tenant
>>> info etc..)
>>> - Our notifications (that Ceilometer uses) *could* be used, but
>>> I am not
>>> sure if it quite fits.
>>> (they seem a bit heavy weight for a log message and aimed at
>>> higher
>>> level events)
>>>
>>> These messages could be (based on Heat's use case):
>>>
>>> - Specific user oriented log messages (distinct from our normal
>>> operator
>>> logs)
>>> - Deprecation messages (if they are using old resource
>>> properties/template features)
>>> - Progress and resource state changes (an application doesn't
>>> want to
>>> poll an api for a state change)
>>> - Automated actions (autoscaling events, time based actions)
>>> - Potentially integrated server logs (from in guest agents)
>>>
>>> I wanted to raise this to "[all]" as it would be great to have a
>>> general
>>> solution that
>>> all projects can make use of.
>>>
>>> What do we have now:
>>> - The user can not get any kind of log message from services. The
>>> closest thing
>>> ATM is the notifications in Ceilometer, but I have the
>>> feeling that
>>> these have a different aim.
>>> - nova console log
>>> - Heat has a DB "event" table for users (we have long wanted to
>>> get rid
>>> of this)
>>>
>>> What do other clouds provide:
>>> - https://devcenter.heroku.com/__articles/logging
>>> <https://devcenter.heroku.com/articles/logging>
>>> - https://cloud.google.com/__logging/docs/
>>> <https://cloud.google.com/logging/docs/>
>>> - https://aws.amazon.com/blogs/__aws/cloudwatch-log-service/
>>> <https://aws.amazon.com/blogs/aws/cloudwatch-log-service/>
>>> - http://aws.amazon.com/__cloudtrail/
>>> <http://aws.amazon.com/cloudtrail/>
>>> (other examples...)
>>>
>>> What are some options we could investigate:
>>> 1. remote syslog
>>> The user provides a rsyslog server IP/port and we send their
>>> messages to that.
>>> [pros] simple, and the user could also send their server's log
>>> messages to the same
>>> rsyslog - great visibility into what is going on.
>>>
>>> There are great tools like
>>> loggly/logstash/papertrailapp
>>> that source logs from remote syslog
>>> It leaves the user in control of what tools they
>>> get to use.
>>>
>>> [cons] Would we become a spam agent (just sending traffic to an
>>> IP/Port) - I guess that's how remote syslog
>>> works. I am not sure if this is an issue or not?
>>>
>>> This might be a lesser solution for the use case
>>> of "an
>>> application doesn't want to poll an api for a state change"
>>>
>>> I am not sure how we would integrate this with
>>> horizon.
>>>
>>> 2. Zaqar
>>> We send the messages to a queue in Zaqar.
>>> [pros] multi tenant OpenStack project for messaging!
>>>
>>> [cons] I don't think Zaqar is installed in most
>>> installations (tho'
>>> please correct me here if this
>>> is wrong). I know Mirantis does not currently
>>> support
>>>
>>>
>>> I think you're correct for now, but I also think that the ability to
>>> send messages to the user side is a valuable enough feature to
>>> convince many cloud operators to deploy it. Everybody wins in that case.
>>>
>>> Zaqar, so that would be a problem for me.
>>>
>>> There is not the level of external tooling like
>>> in option
>>> "1" (logstash and friends)
>>>
>>>
>>> IMO whatever solution we choose is going to end up requiring the
>>> same semantics as Zaqar: durable storage, timeouts of stale
>>> messages, arbitrary scale-out, multi-tenancy with Keystone auth,
>>> pub-sub, and so on. That leaves us with 3 options:
>>>
>>> 1) Use Zaqar
>>>
>>> 2) Write a ground-up replacement for Zaqar. I hope we can agree that
>>> this is insane. Particularly if our reason for not using Zaqar is
>>> that it isn't widely deployed enough yet.
>>>
>>> 3) Write or make use of something much simpler than Zaqar that
>>> implements only the exact subset of Zaqar's semantics that we need.
>>> However, IMHO that is very likely to turn out to be substantially
>>> all of them, and once again this is unlikely to solve the problem of
>>> wide deployment before Zaqar.
>>>
>>> So, in conclusion, +1 Zaqar.
>>>
>>> However, there are some other interesting questions posed by your email.
>>>
>>> - Do we need a separate service to condition the output to a
>>> different protocol? IMHO no - or, rather, not beyond the ones
>>> already proposed for Zaqar (long polling, WebSockets, SNS-style
>>> notifications). Even if there was some better protocol (in principle
>>> I'm not opposed to Atom, for example), I think we'd benefit more by
>>> just adding support for it in Zaqar - if it works for this use case
>>> then it will work for others that users commonly have.
>>>
>>> - What should be feeding inputs into the queue? Some service that
>>> consumes the existing oslo messaging notifications and sanitises
>>> them for the user?
>>>
>>>
>>> I think if we use Zaqar, it would be madness to not do this. *But* we
>>> have to be really careful we don't end up sending sensitive deployment
>>> information
>>> to the user.
>>>
>>>
>>> Or would every service publish its user notifications directly to
>>> the queue? I think this might vary on a case-by-case basis. For
>>> things like log messages, warnings and the like, I can see that
>>> having a single place to configure it would be valuable. For other
>>> things, like the hooks we just added in Heat to allow the user to
>>> insert their own actions into Heat's workflow, it may make more
>>> sense for the user to explicitly provide the Zaqar topic they want
>>> notification on. Which leads to a final question:
>>>
>>> - Are there different use cases here that require different
>>> solutions? I'm pretty sure that Zaqar is the Right Thing for things
>>> like Heat workflow hooks, Ceilometer alarms, triggering Mistral
>>> workflows, &c. However we should at least consider the possibility
>>> that there's a better fit for the more logging-like things. As I
>>> mentioned, I think on the back end they will likely look quite
>>> different. I'd like to think that the user-facing part could be the
>>> same, but it would be interesting to hear from anyone who disagrees.
>>> In particular, if we were to go as far as allowing users to use the
>>> same mechanism for their own application logging, then Zaqar would
>>> probably not be the best channel for it.
>>>
>>>
>>> Use cases:
>>> 1. structured-logging/events/notifications that the end user can access
>>> (the main use case I am trying to solve here)
>>> - for heat this would be a replacement to our event table and enable
>>> us to send more events (so we don't have to worry about burning up our
>>> db with events)
>>> - for any other service this would be a great new feature that allows
>>> the user more insight into what is happening with their resources.
>>>
>>> 2. Heat's two way comms with the user (hooks?)
>>> - I am not at all covering that use case in this discussion - I think
>>> a completely different issue.
>>>
>>> 3. Ceilometer/Monasca alarms
>>> (the assumption is that besides webhooks we add other transports)
>>> - Again I wasn't trying to solve this problem - but could be in-scope.
>>>
>>> 4. integration with in-server logging
>>> (your server applications log to syslog, the question is can we get that
>>> logging "nicely" to the user
>>> and potentially integrate with "1." - this was more of a nice-to-have)
>>> - for a nice example of how this can be
>>> done: https://devcenter.heroku.com/articles/logging
>>>
>>> Could Zaqar fulfil all of these use case? - Maybe.
>>
>> Zaqar as it presently exists, maybe. Zaqar *wants* to fit these cases,
>> and they want our feedback if they are missing anything to support these.
>
> Yes, please. I'd go a bit further than this and say we're willing to
> help on getting this done as that'll be one of the things the team
> will focus on in the next cycle.
>
>>
>> The biggest stretch here IMO is (4) because not all Zaqar storage
>> backends support strict ordering, but that can be fixed at the
>> application layer.
>
> Correct, right now the mongodb storage is the only one with support
> for FIFO. This has been made optional in Kilo.

And just to bring up this painful point again ;)

http://lists.openstack.org/pipermail/openstack-dev/2014-March/030510.html

Please no mongodb ;)

>
> Cheers,
> Flavio
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev