<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, Apr 10, 2015 at 6:39 AM, Zane Bitter <span dir="ltr"><<a href="mailto:zbitter@redhat.com" target="_blank">zbitter@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div>On 06/04/15 22:55, Angus Salkeld wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Hi all<br>
<br>
For quite some time we (Heat team) have wanted to be able to send<br>
messages to our<br>
users (by user I do not mean the Operator, but the User that is<br>
interacting with the client).<br>
<br>
What do I mean by "user messages", and how do they differ from our<br>
current log messages<br>
and notifications?<br>
- Our current logs are for the operator and have information that the<br>
user should not have<br>
(ip addresses, hostnames, configuration options, other tenant info etc..)<br>
- Our notifications (that Ceilometer uses) *could* be used, but I am not<br>
sure if it quite fits.<br>
(they seem a bit heavy weight for a log message and aimed at higher<br>
level events)<br>
<br>
These messages could be (based on Heat's use case):<br>
<br>
- Specific user oriented log messages (distinct from our normal operator<br>
logs)<br>
- Deprecation messages (if they are using old resource<br>
properties/template features)<br>
- Progress and resource state changes (an application doesn't want to<br>
poll an api for a state change)<br>
- Automated actions (autoscaling events, time based actions)<br>
- Potentially integrated server logs (from in guest agents)<br>
<br>
I wanted to raise this to "[all]" as it would be great to have a general<br>
solution that<br>
all projects can make use of.<br>
<br>
What do we have now:<br>
- The user can not get any kind of log message from services. The<br>
closest thing<br>
ATM is the notifications in Ceilometer, but I have the feeling that<br>
these have a different aim.<br>
- nova console log<br>
- Heat has a DB "event" table for users (we have long wanted to get rid<br>
of this)<br>
<br>
What do other clouds provide:<br>
- <a href="https://devcenter.heroku.com/articles/logging" target="_blank">https://devcenter.heroku.com/<u></u>articles/logging</a><br>
- <a href="https://cloud.google.com/logging/docs/" target="_blank">https://cloud.google.com/<u></u>logging/docs/</a><br>
- <a href="https://aws.amazon.com/blogs/aws/cloudwatch-log-service/" target="_blank">https://aws.amazon.com/blogs/<u></u>aws/cloudwatch-log-service/</a><br>
- <a href="http://aws.amazon.com/cloudtrail/" target="_blank">http://aws.amazon.com/<u></u>cloudtrail/</a><br>
(other examples...)<br>
<br>
What are some options we could investigate:<br>
1. remote syslog<br>
The user provides a rsyslog server IP/port and we send their<br>
messages to that.<br>
[pros] simple, and the user could also send their server's log<br>
messages to the same<br>
rsyslog - great visibility into what is going on.<br>
<br>
There are great tools like loggly/logstash/papertrailapp<br>
that source logs from remote syslog<br>
It leaves the user in control of what tools they get to use.<br>
<br>
[cons] Would we become a spam agent (just sending traffic to an<br>
IP/Port) - I guess that's how remote syslog<br>
works. I am not sure if this is an issue or not?<br>
<br>
This might be a lesser solution for the use case of "an<br>
application doesn't want to poll an api for a state change"<br>
<br>
I am not sure how we would integrate this with horizon.<br>
<br>
2. Zaqar<br>
We send the messages to a queue in Zaqar.<br>
[pros] multi tenant OpenStack project for messaging!<br>
<br>
[cons] I don't think Zaqar is installed in most installations (tho'<br>
please correct me here if this<br>
is wrong). I know Mirantis does not currently support<br>
</blockquote>
<br></div></div>
I think you're correct for now, but I also think that the ability to send messages to the user side is a valuable enough feature to convince many cloud operators to deploy it. Everybody wins in that case.<span><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Zaqar, so that would be a problem for me.<br>
<br>
There is not the level of external tooling like in option<br>
"1" (logstash and friends)<br>
</blockquote>
<br></span>
IMO whatever solution we choose is going to end up requiring the same semantics as Zaqar: durable storage, timeouts of stale messages, arbitrary scale-out, multi-tenancy with Keystone auth, pub-sub, and so on. That leaves us with 3 options:<br>
<br>
1) Use Zaqar<br>
<br>
2) Write a ground-up replacement for Zaqar. I hope we can agree that this is insane. Particularly if our reason for not using Zaqar is that it isn't widely deployed enough yet.<br>
<br>
3) Write or make use of something much simpler than Zaqar that implements only the exact subset of Zaqar's semantics that we need. However, IMHO that is very likely to turn out to be substantially all of them, and once again this is unlikely to solve the problem of wide deployment before Zaqar.<br>
<br></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">So, in conclusion, +1 Zaqar.<br>
<br>
However, there are some other interesting questions posed by your email.<br>
<br>
- Do we need a separate service to condition the output to a different protocol? IMHO no - or, rather, not beyond the ones already proposed for Zaqar (long polling, WebSockets, SNS-style notifications). Even if there was some better protocol (in principle I'm not opposed to Atom, for example), I think we'd benefit more by just adding support for it in Zaqar - if it works for this use case then it will work for others that users commonly have.<br>
<br>
- What should be feeding inputs into the queue? Some service that consumes the existing oslo messaging notifications and sanitises them for the user? </blockquote><div><br></div><div>I think if we use Zaqar, it would be madness to not do this. *But* we have to be really careful we don't end up sending sensitive deployment information</div><div>to the user.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Or would every service publish its user notifications directly to the queue? I think this might vary on a case-by-case basis. For things like log messages, warnings and the like, I can see that having a single place to configure it would be valuable. For other things, like the hooks we just added in Heat to allow the user to insert their own actions into Heat's workflow, it may make more sense for the user to explicitly provide the Zaqar topic they want notification on. Which leads to a final question:<br>
<br>
- Are there different use cases here that require different solutions? I'm pretty sure that Zaqar is the Right Thing for things like Heat workflow hooks, Ceilometer alarms, triggering Mistral workflows, &c. However we should at least consider the possibility that there's a better fit for the more logging-like things. As I mentioned, I think on the back end they will likely look quite different. I'd like to think that the user-facing part could be the same, but it would be interesting to hear from anyone who disagrees. In particular, if we were to go as far as allowing users to use the same mechanism for their own application logging, then Zaqar would probably not be the best channel for it.<br></blockquote><div><br></div><div>Use cases:</div><div>1. structured-logging/events/notifications that the end user can access (the main use case I am trying to solve here)</div><div> - for heat this would be a replacement to our event table and enable us to send more events (so we don't have to worry about burning up our db with events)<br></div><div> - for any other service this would be a great new feature that allows the user more insight into what is happening with their resources.</div><div><br></div><div>2. Heat's two way comms with the user (hooks?)</div><div> - I am not at all covering that use case in this discussion - I think a completely different issue.</div><div><br></div><div>3. Ceilometer/Monasca alarms</div><div>(the assumption is that besides webhooks we add other transports)</div><div>- Again I wasn't trying to solve this problem - but could be in-scope.</div><div><br></div><div>4. integration with in-server logging</div><div>(your server applications log to syslog, the question is can we get that logging "nicely" to the user</div><div> and potentially integrate with "1." - this was more of a nice-to-have)</div><div>- for a nice example of how this can be done: <a href="https://devcenter.heroku.com/articles/logging">https://devcenter.heroku.com/articles/logging</a><br></div><div><br></div><div>Could Zaqar fulfil all of these use case? - Maybe.</div><div><br></div><div><br></div><div>-Angus</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
cheers,<br>
Zane.<span><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
3. Other options:<br>
Please chip in with suggestions/links!<br>
<br>
<br>
<a href="https://blueprints.launchpad.net/heat/+spec/user-visible-logs" target="_blank">https://blueprints.launchpad.<u></u>net/heat/+spec/user-visible-<u></u>logs</a><br>
<br>
<br>
Regards<br>
Angus<br>
</blockquote>
<br>
<br></span>
______________________________<u></u>______________________________<u></u>______________<span><br>
OpenStack Development Mailing List (not for usage questions)<br></span><div><div>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.<u></u>openstack.org?subject:<u></u>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>