Open Stack

Excerpts from Ryan Brown's message of 2015-04-10 07:00:27 -0700:
> On 04/09/2015 09:40 PM, Angus Salkeld wrote:
> > On Fri, Apr 10, 2015 at 6:39 AM, Zane Bitter <zbitter at redhat.com
> > <mailto:zbitter at redhat.com>> wrote:
> > 
> >     On 06/04/15 22:55, Angus Salkeld wrote:
> > 
> >         Hi all
> > 
> >         For quite some time we (Heat team) have wanted to be able to send
> >         messages to our
> >         users (by user I do not mean the Operator, but the User that is
> >         interacting with the client).
> > 
> >         What do I mean by "user messages", and how do they differ from our
> >         current log messages
> >         and notifications?
> >         - Our current logs are for the operator and have information
> >         that the
> >         user should not have
> >            (ip addresses, hostnames, configuration options, other tenant
> >         info etc..)
> >         - Our notifications (that Ceilometer uses) *could* be used, but
> >         I am not
> >         sure if it quite fits.
> >            (they seem a bit heavy weight for a log message and aimed at
> >         higher
> >         level events)
> > 
> >         These messages could be (based on Heat's use case):
> > 
> >         - Specific user oriented log messages (distinct from our normal
> >         operator
> >         logs)
> >         - Deprecation messages (if they are using old resource
> >         properties/template features)
> >         - Progress and resource state changes (an application doesn't
> >         want to
> >         poll an api for a state change)
> >         - Automated actions (autoscaling events, time based actions)
> >         - Potentially integrated server logs (from in guest agents)
> > 
> >         I wanted to raise this to "[all]" as it would be great to have a
> >         general
> >         solution that
> >         all projects can make use of.
> > 
> >         What do we have now:
> >         - The user can not get any kind of log message from services. The
> >         closest thing
> >            ATM is the notifications in Ceilometer, but I have the
> >         feeling that
> >         these have a different aim.
> >         - nova console log
> >         - Heat has a DB "event" table for users (we have long wanted to
> >         get rid
> >         of this)
> > 
> >         What do other clouds provide:
> >         - https://devcenter.heroku.com/__articles/logging
> >         <https://devcenter.heroku.com/articles/logging>
> >         - https://cloud.google.com/__logging/docs/
> >         <https://cloud.google.com/logging/docs/>
> >         - https://aws.amazon.com/blogs/__aws/cloudwatch-log-service/
> >         <https://aws.amazon.com/blogs/aws/cloudwatch-log-service/>
> >         - http://aws.amazon.com/__cloudtrail/
> >         <http://aws.amazon.com/cloudtrail/>
> >         (other examples...)
> > 
> >         What are some options we could investigate:
> >         1. remote syslog
> >              The user provides a rsyslog server IP/port and we send their
> >         messages to that.
> >              [pros] simple, and the user could also send their server's log
> >         messages to the same
> >                        rsyslog - great visibility into what is going on.
> > 
> >                        There are great tools like
> >         loggly/logstash/papertrailapp
> >         that source logs from remote syslog
> >                        It leaves the user in control of what tools they
> >         get to use.
> > 
> >              [cons] Would we become a spam agent (just sending traffic to an
> >         IP/Port) - I guess that's how remote syslog
> >                         works. I am not sure if this is an issue or not?
> > 
> >                        This might be a lesser solution for the use case
> >         of "an
> >         application doesn't want to poll an api for a state change"
> > 
> >                        I am not sure how we would integrate this with
> >         horizon.
> > 
> >         2. Zaqar
> >              We send the messages to a queue in Zaqar.
> >              [pros] multi tenant OpenStack project for messaging!
> > 
> >              [cons] I don't think Zaqar is installed in most
> >         installations (tho'
> >         please correct me here if this
> >                         is wrong). I know Mirantis does not currently
> >         support
> > 
> > 
> >     I think you're correct for now, but I also think that the ability to
> >     send messages to the user side is a valuable enough feature to
> >     convince many cloud operators to deploy it. Everybody wins in that case.
> > 
> >         Zaqar, so that would be a problem for me.
> > 
> >                        There is not the level of external tooling like
> >         in option
> >         "1" (logstash and friends)
> > 
> > 
> >     IMO whatever solution we choose is going to end up requiring the
> >     same semantics as Zaqar: durable storage, timeouts of stale
> >     messages, arbitrary scale-out, multi-tenancy with Keystone auth,
> >     pub-sub, and so on. That leaves us with 3 options:
> > 
> >     1) Use Zaqar
> > 
> >     2) Write a ground-up replacement for Zaqar. I hope we can agree that
> >     this is insane. Particularly if our reason for not using Zaqar is
> >     that it isn't widely deployed enough yet.
> > 
> >     3) Write or make use of something much simpler than Zaqar that
> >     implements only the exact subset of Zaqar's semantics that we need.
> >     However, IMHO that is very likely to turn out to be substantially
> >     all of them, and once again this is unlikely to solve the problem of
> >     wide deployment before Zaqar.
> > 
> >     So, in conclusion, +1 Zaqar.
> > 
> >     However, there are some other interesting questions posed by your email.
> > 
> >     - Do we need a separate service to condition the output to a
> >     different protocol? IMHO no - or, rather, not beyond the ones
> >     already proposed for Zaqar (long polling, WebSockets, SNS-style
> >     notifications). Even if there was some better protocol (in principle
> >     I'm not opposed to Atom, for example), I think we'd benefit more by
> >     just adding support for it in Zaqar - if it works for this use case
> >     then it will work for others that users commonly have.
> > 
> >     - What should be feeding inputs into the queue? Some service that
> >     consumes the existing oslo messaging notifications and sanitises
> >     them for the user? 
> > 
> > 
> > I think if we use Zaqar, it would be madness to not do this. *But* we
> > have to be really careful we don't end up sending sensitive deployment
> > information
> > to the user.
> >  
> > 
> >     Or would every service publish its user notifications directly to
> >     the queue? I think this might vary on a case-by-case basis. For
> >     things like log messages, warnings and the like, I can see that
> >     having a single place to configure it would be valuable. For other
> >     things, like the hooks we just added in Heat to allow the user to
> >     insert their own actions into Heat's workflow, it may make more
> >     sense for the user to explicitly provide the Zaqar topic they want
> >     notification on. Which leads to a final question:
> > 
> >     - Are there different use cases here that require different
> >     solutions? I'm pretty sure that Zaqar is the Right Thing for things
> >     like Heat workflow hooks, Ceilometer alarms, triggering Mistral
> >     workflows, &c. However we should at least consider the possibility
> >     that there's a better fit for the more logging-like things. As I
> >     mentioned, I think on the back end they will likely look quite
> >     different. I'd like to think that the user-facing part could be the
> >     same, but it would be interesting to hear from anyone who disagrees.
> >     In particular, if we were to go as far as allowing users to use the
> >     same mechanism for their own application logging, then Zaqar would
> >     probably not be the best channel for it.
> > 
> > 
> > Use cases:
> > 1. structured-logging/events/notifications that the end user can access
> > (the main use case I am trying to solve here)
> >    - for heat this would be a replacement to our event table and enable
> > us to send more events (so we don't have to worry about burning up our
> > db with events)
> >    - for any other service this would be a great new feature that allows
> > the user more insight into what is happening with their resources.
> > 
> > 2. Heat's two way comms with the user (hooks?)
> >   - I am not at all covering that use case in this discussion - I think
> > a completely different issue.
> > 
> > 3. Ceilometer/Monasca alarms
> > (the assumption is that besides webhooks we add other transports)
> > - Again I wasn't trying to solve this problem - but could be in-scope.
> > 
> > 4. integration with in-server logging
> > (your server applications log to syslog, the question is can we get that
> > logging "nicely" to the user
> >  and potentially integrate with "1." - this was more of a nice-to-have)
> > - for a nice example of how this can be
> > done: https://devcenter.heroku.com/articles/logging
> > 
> > Could Zaqar fulfil all of these use case? - Maybe.
> 
> Zaqar as it presently exists, maybe. Zaqar *wants* to fit these cases,
> and they want our feedback if they are missing anything to support these.
> 
> The biggest stretch here IMO is (4) because not all Zaqar storage
> backends support strict ordering, but that can be fixed at the
> application layer.
> 

I'd recommend strongly to avoid requiring FIFO. It's a performance
killer. Best effort is fine, and let the user sort through them by
timestamp if they pop out in an odd order.

So to summarize, I think this would make sense as a plan to get this
into a usable state for all services to be able to offer:

* Create a micro-service whose sole purpose in life is to filter
  notifications. A user would interact with this service by submitting
  requests for subscriptions, something like this:
  
  POST https://service/subscription
  {
   "service": "compute",
   "topic": {"server": {"id": "uuid-of-server"}}
   "destination": "zaqar:https://zaqar/url/for/sending"
  }

  ... and all the other lovely restful things

* Ensure notifications exist for user-interesting things and that they
  don't overload ceilometer and the like with unnecessary notifications.

If Zaqar becomes the piece that prevents adoption, add more destination
drivers, such as atom feeds, logstash, AMQP queues, or syslog services.

Horizon people, can you comment on whether this would be helpful for you
too? It seems like it would!