Open Stack

On 06/04/15 22:55, Angus Salkeld wrote:
> Hi all
>
> For quite some time we (Heat team) have wanted to be able to send
> messages to our
> users (by user I do not mean the Operator, but the User that is
> interacting with the client).
>
> What do I mean by "user messages", and how do they differ from our
> current log messages
> and notifications?
> - Our current logs are for the operator and have information that the
> user should not have
>    (ip addresses, hostnames, configuration options, other tenant info etc..)
> - Our notifications (that Ceilometer uses) *could* be used, but I am not
> sure if it quite fits.
>    (they seem a bit heavy weight for a log message and aimed at higher
> level events)
>
> These messages could be (based on Heat's use case):
>
> - Specific user oriented log messages (distinct from our normal operator
> logs)
> - Deprecation messages (if they are using old resource
> properties/template features)
> - Progress and resource state changes (an application doesn't want to
> poll an api for a state change)
> - Automated actions (autoscaling events, time based actions)
> - Potentially integrated server logs (from in guest agents)
>
> I wanted to raise this to "[all]" as it would be great to have a general
> solution that
> all projects can make use of.
>
> What do we have now:
> - The user can not get any kind of log message from services. The
> closest thing
>    ATM is the notifications in Ceilometer, but I have the feeling that
> these have a different aim.
> - nova console log
> - Heat has a DB "event" table for users (we have long wanted to get rid
> of this)
>
> What do other clouds provide:
> - https://devcenter.heroku.com/articles/logging
> - https://cloud.google.com/logging/docs/
> - https://aws.amazon.com/blogs/aws/cloudwatch-log-service/
> - http://aws.amazon.com/cloudtrail/
> (other examples...)
>
> What are some options we could investigate:
> 1. remote syslog
>      The user provides a rsyslog server IP/port and we send their
> messages to that.
>      [pros] simple, and the user could also send their server's log
> messages to the same
>                rsyslog - great visibility into what is going on.
>
>                There are great tools like loggly/logstash/papertrailapp
> that source logs from remote syslog
>                It leaves the user in control of what tools they get to use.
>
>      [cons] Would we become a spam agent (just sending traffic to an
> IP/Port) - I guess that's how remote syslog
>                 works. I am not sure if this is an issue or not?
>
>                This might be a lesser solution for the use case of "an
> application doesn't want to poll an api for a state change"
>
>                I am not sure how we would integrate this with horizon.
>
> 2. Zaqar
>      We send the messages to a queue in Zaqar.
>      [pros] multi tenant OpenStack project for messaging!
>
>      [cons] I don't think Zaqar is installed in most installations (tho'
> please correct me here if this
>                 is wrong). I know Mirantis does not currently support

I think you're correct for now, but I also think that the ability to 
send messages to the user side is a valuable enough feature to convince 
many cloud operators to deploy it. Everybody wins in that case.

> Zaqar, so that would be a problem for me.
>
>                There is not the level of external tooling like in option
> "1" (logstash and friends)

IMO whatever solution we choose is going to end up requiring the same 
semantics as Zaqar: durable storage, timeouts of stale messages, 
arbitrary scale-out, multi-tenancy with Keystone auth, pub-sub, and so 
on. That leaves us with 3 options:

1) Use Zaqar

2) Write a ground-up replacement for Zaqar. I hope we can agree that 
this is insane. Particularly if our reason for not using Zaqar is that 
it isn't widely deployed enough yet.

3) Write or make use of something much simpler than Zaqar that 
implements only the exact subset of Zaqar's semantics that we need. 
However, IMHO that is very likely to turn out to be substantially all of 
them, and once again this is unlikely to solve the problem of wide 
deployment before Zaqar.

So, in conclusion, +1 Zaqar.

However, there are some other interesting questions posed by your email.

- Do we need a separate service to condition the output to a different 
protocol? IMHO no - or, rather, not beyond the ones already proposed for 
Zaqar (long polling, WebSockets, SNS-style notifications). Even if there 
was some better protocol (in principle I'm not opposed to Atom, for 
example), I think we'd benefit more by just adding support for it in 
Zaqar - if it works for this use case then it will work for others that 
users commonly have.

- What should be feeding inputs into the queue? Some service that 
consumes the existing oslo messaging notifications and sanitises them 
for the user? Or would every service publish its user notifications 
directly to the queue? I think this might vary on a case-by-case basis. 
For things like log messages, warnings and the like, I can see that 
having a single place to configure it would be valuable. For other 
things, like the hooks we just added in Heat to allow the user to insert 
their own actions into Heat's workflow, it may make more sense for the 
user to explicitly provide the Zaqar topic they want notification on. 
Which leads to a final question:

- Are there different use cases here that require different solutions? 
I'm pretty sure that Zaqar is the Right Thing for things like Heat 
workflow hooks, Ceilometer alarms, triggering Mistral workflows, &c. 
However we should at least consider the possibility that there's a 
better fit for the more logging-like things. As I mentioned, I think on 
the back end they will likely look quite different. I'd like to think 
that the user-facing part could be the same, but it would be interesting 
to hear from anyone who disagrees. In particular, if we were to go as 
far as allowing users to use the same mechanism for their own 
application logging, then Zaqar would probably not be the best channel 
for it.

cheers,
Zane.

> 3. Other options:
>     Please chip in with suggestions/links!
>
>
> https://blueprints.launchpad.net/heat/+spec/user-visible-logs
>
>
> Regards
> Angus