Open Stack

On 10/04/15 10:00 -0400, Ryan Brown wrote:
>On 04/09/2015 09:40 PM, Angus Salkeld wrote:
>> On Fri, Apr 10, 2015 at 6:39 AM, Zane Bitter <zbitter at redhat.com
>> <mailto:zbitter at redhat.com>> wrote:
>>
>>     On 06/04/15 22:55, Angus Salkeld wrote:
>>
>>         Hi all
>>
>>         For quite some time we (Heat team) have wanted to be able to send
>>         messages to our
>>         users (by user I do not mean the Operator, but the User that is
>>         interacting with the client).
>>
>>         What do I mean by "user messages", and how do they differ from our
>>         current log messages
>>         and notifications?
>>         - Our current logs are for the operator and have information
>>         that the
>>         user should not have
>>            (ip addresses, hostnames, configuration options, other tenant
>>         info etc..)
>>         - Our notifications (that Ceilometer uses) *could* be used, but
>>         I am not
>>         sure if it quite fits.
>>            (they seem a bit heavy weight for a log message and aimed at
>>         higher
>>         level events)
>>
>>         These messages could be (based on Heat's use case):
>>
>>         - Specific user oriented log messages (distinct from our normal
>>         operator
>>         logs)
>>         - Deprecation messages (if they are using old resource
>>         properties/template features)
>>         - Progress and resource state changes (an application doesn't
>>         want to
>>         poll an api for a state change)
>>         - Automated actions (autoscaling events, time based actions)
>>         - Potentially integrated server logs (from in guest agents)
>>
>>         I wanted to raise this to "[all]" as it would be great to have a
>>         general
>>         solution that
>>         all projects can make use of.
>>
>>         What do we have now:
>>         - The user can not get any kind of log message from services. The
>>         closest thing
>>            ATM is the notifications in Ceilometer, but I have the
>>         feeling that
>>         these have a different aim.
>>         - nova console log
>>         - Heat has a DB "event" table for users (we have long wanted to
>>         get rid
>>         of this)
>>
>>         What do other clouds provide:
>>         - https://devcenter.heroku.com/__articles/logging
>>         <https://devcenter.heroku.com/articles/logging>
>>         - https://cloud.google.com/__logging/docs/
>>         <https://cloud.google.com/logging/docs/>
>>         - https://aws.amazon.com/blogs/__aws/cloudwatch-log-service/
>>         <https://aws.amazon.com/blogs/aws/cloudwatch-log-service/>
>>         - http://aws.amazon.com/__cloudtrail/
>>         <http://aws.amazon.com/cloudtrail/>
>>         (other examples...)
>>
>>         What are some options we could investigate:
>>         1. remote syslog
>>              The user provides a rsyslog server IP/port and we send their
>>         messages to that.
>>              [pros] simple, and the user could also send their server's log
>>         messages to the same
>>                        rsyslog - great visibility into what is going on.
>>
>>                        There are great tools like
>>         loggly/logstash/papertrailapp
>>         that source logs from remote syslog
>>                        It leaves the user in control of what tools they
>>         get to use.
>>
>>              [cons] Would we become a spam agent (just sending traffic to an
>>         IP/Port) - I guess that's how remote syslog
>>                         works. I am not sure if this is an issue or not?
>>
>>                        This might be a lesser solution for the use case
>>         of "an
>>         application doesn't want to poll an api for a state change"
>>
>>                        I am not sure how we would integrate this with
>>         horizon.
>>
>>         2. Zaqar
>>              We send the messages to a queue in Zaqar.
>>              [pros] multi tenant OpenStack project for messaging!
>>
>>              [cons] I don't think Zaqar is installed in most
>>         installations (tho'
>>         please correct me here if this
>>                         is wrong). I know Mirantis does not currently
>>         support
>>
>>
>>     I think you're correct for now, but I also think that the ability to
>>     send messages to the user side is a valuable enough feature to
>>     convince many cloud operators to deploy it. Everybody wins in that case.
>>
>>         Zaqar, so that would be a problem for me.
>>
>>                        There is not the level of external tooling like
>>         in option
>>         "1" (logstash and friends)
>>
>>
>>     IMO whatever solution we choose is going to end up requiring the
>>     same semantics as Zaqar: durable storage, timeouts of stale
>>     messages, arbitrary scale-out, multi-tenancy with Keystone auth,
>>     pub-sub, and so on. That leaves us with 3 options:
>>
>>     1) Use Zaqar
>>
>>     2) Write a ground-up replacement for Zaqar. I hope we can agree that
>>     this is insane. Particularly if our reason for not using Zaqar is
>>     that it isn't widely deployed enough yet.
>>
>>     3) Write or make use of something much simpler than Zaqar that
>>     implements only the exact subset of Zaqar's semantics that we need.
>>     However, IMHO that is very likely to turn out to be substantially
>>     all of them, and once again this is unlikely to solve the problem of
>>     wide deployment before Zaqar.
>>
>>     So, in conclusion, +1 Zaqar.
>>
>>     However, there are some other interesting questions posed by your email.
>>
>>     - Do we need a separate service to condition the output to a
>>     different protocol? IMHO no - or, rather, not beyond the ones
>>     already proposed for Zaqar (long polling, WebSockets, SNS-style
>>     notifications). Even if there was some better protocol (in principle
>>     I'm not opposed to Atom, for example), I think we'd benefit more by
>>     just adding support for it in Zaqar - if it works for this use case
>>     then it will work for others that users commonly have.
>>
>>     - What should be feeding inputs into the queue? Some service that
>>     consumes the existing oslo messaging notifications and sanitises
>>     them for the user?
>>
>>
>> I think if we use Zaqar, it would be madness to not do this. *But* we
>> have to be really careful we don't end up sending sensitive deployment
>> information
>> to the user.
>>
>>
>>     Or would every service publish its user notifications directly to
>>     the queue? I think this might vary on a case-by-case basis. For
>>     things like log messages, warnings and the like, I can see that
>>     having a single place to configure it would be valuable. For other
>>     things, like the hooks we just added in Heat to allow the user to
>>     insert their own actions into Heat's workflow, it may make more
>>     sense for the user to explicitly provide the Zaqar topic they want
>>     notification on. Which leads to a final question:
>>
>>     - Are there different use cases here that require different
>>     solutions? I'm pretty sure that Zaqar is the Right Thing for things
>>     like Heat workflow hooks, Ceilometer alarms, triggering Mistral
>>     workflows, &c. However we should at least consider the possibility
>>     that there's a better fit for the more logging-like things. As I
>>     mentioned, I think on the back end they will likely look quite
>>     different. I'd like to think that the user-facing part could be the
>>     same, but it would be interesting to hear from anyone who disagrees.
>>     In particular, if we were to go as far as allowing users to use the
>>     same mechanism for their own application logging, then Zaqar would
>>     probably not be the best channel for it.
>>
>>
>> Use cases:
>> 1. structured-logging/events/notifications that the end user can access
>> (the main use case I am trying to solve here)
>>    - for heat this would be a replacement to our event table and enable
>> us to send more events (so we don't have to worry about burning up our
>> db with events)
>>    - for any other service this would be a great new feature that allows
>> the user more insight into what is happening with their resources.
>>
>> 2. Heat's two way comms with the user (hooks?)
>>   - I am not at all covering that use case in this discussion - I think
>> a completely different issue.
>>
>> 3. Ceilometer/Monasca alarms
>> (the assumption is that besides webhooks we add other transports)
>> - Again I wasn't trying to solve this problem - but could be in-scope.
>>
>> 4. integration with in-server logging
>> (your server applications log to syslog, the question is can we get that
>> logging "nicely" to the user
>>  and potentially integrate with "1." - this was more of a nice-to-have)
>> - for a nice example of how this can be
>> done: https://devcenter.heroku.com/articles/logging
>>
>> Could Zaqar fulfil all of these use case? - Maybe.
>
>Zaqar as it presently exists, maybe. Zaqar *wants* to fit these cases,
>and they want our feedback if they are missing anything to support these.

Yes, please. I'd go a bit further than this and say we're willing to
help on getting this done as that'll be one of the things the team
will focus on in the next cycle.

>
>The biggest stretch here IMO is (4) because not all Zaqar storage
>backends support strict ordering, but that can be fixed at the
>application layer.

Correct, right now the mongodb storage is the only one with support
for FIFO. This has been made optional in Kilo.

Cheers,
Flavio

-- 
@flaper87
Flavio Percoco
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150410/c49354b5/attachment.pgp>