Open Stack

On Fri, Apr 10, 2015 at 6:39 AM, Zane Bitter <zbitter at redhat.com> wrote:

> On 06/04/15 22:55, Angus Salkeld wrote:
>
>> Hi all
>>
>> For quite some time we (Heat team) have wanted to be able to send
>> messages to our
>> users (by user I do not mean the Operator, but the User that is
>> interacting with the client).
>>
>> What do I mean by "user messages", and how do they differ from our
>> current log messages
>> and notifications?
>> - Our current logs are for the operator and have information that the
>> user should not have
>>    (ip addresses, hostnames, configuration options, other tenant info
>> etc..)
>> - Our notifications (that Ceilometer uses) *could* be used, but I am not
>> sure if it quite fits.
>>    (they seem a bit heavy weight for a log message and aimed at higher
>> level events)
>>
>> These messages could be (based on Heat's use case):
>>
>> - Specific user oriented log messages (distinct from our normal operator
>> logs)
>> - Deprecation messages (if they are using old resource
>> properties/template features)
>> - Progress and resource state changes (an application doesn't want to
>> poll an api for a state change)
>> - Automated actions (autoscaling events, time based actions)
>> - Potentially integrated server logs (from in guest agents)
>>
>> I wanted to raise this to "[all]" as it would be great to have a general
>> solution that
>> all projects can make use of.
>>
>> What do we have now:
>> - The user can not get any kind of log message from services. The
>> closest thing
>>    ATM is the notifications in Ceilometer, but I have the feeling that
>> these have a different aim.
>> - nova console log
>> - Heat has a DB "event" table for users (we have long wanted to get rid
>> of this)
>>
>> What do other clouds provide:
>> - https://devcenter.heroku.com/articles/logging
>> - https://cloud.google.com/logging/docs/
>> - https://aws.amazon.com/blogs/aws/cloudwatch-log-service/
>> - http://aws.amazon.com/cloudtrail/
>> (other examples...)
>>
>> What are some options we could investigate:
>> 1. remote syslog
>>      The user provides a rsyslog server IP/port and we send their
>> messages to that.
>>      [pros] simple, and the user could also send their server's log
>> messages to the same
>>                rsyslog - great visibility into what is going on.
>>
>>                There are great tools like loggly/logstash/papertrailapp
>> that source logs from remote syslog
>>                It leaves the user in control of what tools they get to
>> use.
>>
>>      [cons] Would we become a spam agent (just sending traffic to an
>> IP/Port) - I guess that's how remote syslog
>>                 works. I am not sure if this is an issue or not?
>>
>>                This might be a lesser solution for the use case of "an
>> application doesn't want to poll an api for a state change"
>>
>>                I am not sure how we would integrate this with horizon.
>>
>> 2. Zaqar
>>      We send the messages to a queue in Zaqar.
>>      [pros] multi tenant OpenStack project for messaging!
>>
>>      [cons] I don't think Zaqar is installed in most installations (tho'
>> please correct me here if this
>>                 is wrong). I know Mirantis does not currently support
>>
>
> I think you're correct for now, but I also think that the ability to send
> messages to the user side is a valuable enough feature to convince many
> cloud operators to deploy it. Everybody wins in that case.
>
>  Zaqar, so that would be a problem for me.
>>
>>                There is not the level of external tooling like in option
>> "1" (logstash and friends)
>>
>
> IMO whatever solution we choose is going to end up requiring the same
> semantics as Zaqar: durable storage, timeouts of stale messages, arbitrary
> scale-out, multi-tenancy with Keystone auth, pub-sub, and so on. That
> leaves us with 3 options:
>
> 1) Use Zaqar
>
> 2) Write a ground-up replacement for Zaqar. I hope we can agree that this
> is insane. Particularly if our reason for not using Zaqar is that it isn't
> widely deployed enough yet.
>
> 3) Write or make use of something much simpler than Zaqar that implements
> only the exact subset of Zaqar's semantics that we need. However, IMHO that
> is very likely to turn out to be substantially all of them, and once again
> this is unlikely to solve the problem of wide deployment before Zaqar.
>
> So, in conclusion, +1 Zaqar.
>
> However, there are some other interesting questions posed by your email.
>
> - Do we need a separate service to condition the output to a different
> protocol? IMHO no - or, rather, not beyond the ones already proposed for
> Zaqar (long polling, WebSockets, SNS-style notifications). Even if there
> was some better protocol (in principle I'm not opposed to Atom, for
> example), I think we'd benefit more by just adding support for it in Zaqar
> - if it works for this use case then it will work for others that users
> commonly have.
>
> - What should be feeding inputs into the queue? Some service that consumes
> the existing oslo messaging notifications and sanitises them for the user?


I think if we use Zaqar, it would be madness to not do this. *But* we have
to be really careful we don't end up sending sensitive deployment
information
to the user.


> Or would every service publish its user notifications directly to the
> queue? I think this might vary on a case-by-case basis. For things like log
> messages, warnings and the like, I can see that having a single place to
> configure it would be valuable. For other things, like the hooks we just
> added in Heat to allow the user to insert their own actions into Heat's
> workflow, it may make more sense for the user to explicitly provide the
> Zaqar topic they want notification on. Which leads to a final question:
>
> - Are there different use cases here that require different solutions? I'm
> pretty sure that Zaqar is the Right Thing for things like Heat workflow
> hooks, Ceilometer alarms, triggering Mistral workflows, &c. However we
> should at least consider the possibility that there's a better fit for the
> more logging-like things. As I mentioned, I think on the back end they will
> likely look quite different. I'd like to think that the user-facing part
> could be the same, but it would be interesting to hear from anyone who
> disagrees. In particular, if we were to go as far as allowing users to use
> the same mechanism for their own application logging, then Zaqar would
> probably not be the best channel for it.
>

Use cases:
1. structured-logging/events/notifications that the end user can access
(the main use case I am trying to solve here)
   - for heat this would be a replacement to our event table and enable us
to send more events (so we don't have to worry about burning up our db with
events)
   - for any other service this would be a great new feature that allows
the user more insight into what is happening with their resources.

2. Heat's two way comms with the user (hooks?)
  - I am not at all covering that use case in this discussion - I think a
completely different issue.

3. Ceilometer/Monasca alarms
(the assumption is that besides webhooks we add other transports)
- Again I wasn't trying to solve this problem - but could be in-scope.

4. integration with in-server logging
(your server applications log to syslog, the question is can we get that
logging "nicely" to the user
 and potentially integrate with "1." - this was more of a nice-to-have)
- for a nice example of how this can be done:
https://devcenter.heroku.com/articles/logging

Could Zaqar fulfil all of these use case? - Maybe.


-Angus


>
> cheers,
> Zane.
>
>  3. Other options:
>>     Please chip in with suggestions/links!
>>
>>
>> https://blueprints.launchpad.net/heat/+spec/user-visible-logs
>>
>>
>> Regards
>> Angus
>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150410/3ca3ea7c/attachment.html>