<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Aug 12, 2013 at 4:11 PM, Clint Byrum <span dir="ltr"><<a href="mailto:clint@fewbar.com" target="_blank">clint@fewbar.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Excerpts from Doug Hellmann's message of 2013-08-12 12:08:58 -0700:<br>
<div><div class="h5">> On Fri, Aug 9, 2013 at 11:56 AM, Clint Byrum <<a href="mailto:clint@fewbar.com">clint@fewbar.com</a>> wrote:<br>
><br>
> > Excerpts from Sandy Walsh's message of 2013-08-09 06:16:55 -0700:<br>
> > ><br>
> > > On 08/08/2013 11:36 PM, Angus Salkeld wrote:<br>
> > > > On 08/08/13 13:16 -0700, Clint Byrum wrote:<br>
> > > >> Last night while reviewing a feature which would add more events to<br>
> > the<br>
> > > >> event table, it dawned on me that the event table really must be<br>
> > removed.<br>
> > > ><br>
> > > ><br>
> > > >><br>
> > > >> <a href="https://bugs.launchpad.net/heat/+bug/1209492" target="_blank">https://bugs.launchpad.net/heat/+bug/1209492</a><br>
> > > >><br>
> > > >> tl;dr: users can write an infinite number of rows to the event table<br>
> > at<br>
> > > >> a fairly alarming rate just by creating and updating a very large<br>
> > stack<br>
> > > >> that has no resources that cost any time or are even billable (like an<br>
> > > >> autoscaling launch configuration).<br>
> > > >><br>
> > > >> The table has no purge function, so the only way to clear out old<br>
> > events<br>
> > > >> is to delete the stack, or manually remove them directly in the<br>
> > database.<br>
> > > >><br>
> > > >> We've all been through this before, logging to a database seems great<br>
> > > >> until you actually do it.<br>
> > > >><br>
> > > >> I have some ideas for how to solve it, but I wanted to get a wider<br>
> > > >> audience:<br>
> > > >><br>
> > > >> 1) Make the event list a ring buffer. Have rows 0 - $MAX_BUFFER_SIZE<br>
> > in<br>
> > > >> each stack, and simply write each new event to the next open position,<br>
> > > >> wrapping at $MAX_BUFFER_SIZE. Pros: little change to current code,<br>
> > > >> just need an offset column added and code that will properly wrap to 0<br>
> > > >> at $MAX_BUFFER_SIZE. Cons: still can incur heavy transactional load on<br>
> > > >> the database server.A<br>
> > > >><br>
> > > >> 1.b) Same, but instead of rows, just maintain a blob and append the<br>
> > rows<br>
> > > >> as json list. Lowers transactional load but would push some load onto<br>
> > > >> the API servers and such to parse these out, and would make pagination<br>
> > > >> challenging. Blobs also can be a drain on DB server performance.<br>
> > > >><br>
> > > >> 2) Write a purge script. Delete old ones. Pros: No code change, just<br>
> > > >> new code to do purging. Cons: same as 1, plus more vulnerability to an<br>
> > > >> aggressive attacker who can fit a lot of data in between purges. Also<br>
> > > >> large scale deletes can be really painful (see: keystone sql token<br>
> > > >> backend).<br>
> > > >><br>
> > > >> 3) Log events to Swift. I can't seem to find information on how/if<br>
> > > >> appending works there. Tons of tiny single-row files is an option,<br>
> > but I<br>
> > > >> want to hear from people with more swift knowledge if that is a<br>
> > viable,<br>
> > > >> performant option. Pros: Scale to the moon. Can charge tenant for<br>
> > usage<br>
> > > >> and let them purge events as needed. Cons: Adds swift as a requirement<br>
> > > >> of Heat.<br>
> > > >><br>
> > > >> 4) Provide a way for users to receive logs via HTTP POST. Pros: Simple<br>
> > > >> and punts the problem to the users. Cons: users will be SoL if they<br>
> > > >> don't have a place to have logs posted to.<br>
> > > >><br>
> > > >> 5) Provide a way for users to receive logs via messaging service like<br>
> > > >> Marconi. Pros/Cons: same as HTTP, but perhaps a little more confusing<br>
> > > >> and ambitious given Marconi's short existence.<br>
> > > >><br>
> > > >> 6) Provide a pluggable backend for logging. This seems like the way<br>
> > most<br>
> > > >> OpenStack projects solve these issues, which is to let the deployers<br>
> > > >> choose and/or provide their own way to handle a sticky problem. Pros:<br>
> > > >> Simple and flexible for the future. Cons: Would require writing at<br>
> > least<br>
> > > >> one backend provider that does what the previous 5 options suggest.<br>
> > > >><br>
> > > >> To be clear: Heat cannot really exist without this, as it is the only<br>
> > way<br>
> > > >> to find out what your stack is doing or has done.<br>
> > > ><br>
> > > > btw Clint I have ditched that "Recorder" patch as Ceilometer is<br>
> > > > getting a Alarm History api soon, so we can defer to that for that<br>
> > > > functionality (alarm transitions).<br>
> > > ><br>
> > > > But we still need a better way to record events/logs for the user.<br>
> > > > So I make this blueprint a while ago:<br>
> > > > <a href="https://blueprints.launchpad.net/heat/+spec/user-visible-logs" target="_blank">https://blueprints.launchpad.net/heat/+spec/user-visible-logs</a><br>
> > > ><br>
> > > > I am becomming more in favor of user options rather than deployer<br>
> > > > options if possible. So provide resources for Marconi, Meniscus and<br>
> > > > what ever...<br>
> > > > Although what is nice about Marconi is you could then hook up what<br>
> > > > ever you want to it.<br>
> > ><br>
> > > Logs are one thing (and Meniscus is a great choice for that), but events<br>
> > > are the very thing CM is designed to handle. Wouldn't it make sense to<br>
> > > push them back into there?<br>
> > ><br>
> ><br>
> > I'm not sure these events make sense in the current Ceilometer (I assume<br>
> > that is "CM" above) context. These events are:<br>
> ><br>
> > ... Creating stack A<br>
> > ... Creating stack A resource A<br>
> > ... Created stack A resource A<br>
> > ... Created stack A<br>
> ><br>
> > Users will want to be able to see all of the events for a stack, and<br>
> > likely we need to be able to paginate through them as well.<br>
> ><br>
> > They are fundamental and low level enough for Heat that I'm not sure<br>
> > putting them in Ceilometer makes much sense, but maybe I don't understand<br>
> > Ceilometer.. or "CM" is something else entirely. :)<br>
> ><br>
><br>
> CM is indeed ceilometer.<br>
><br>
> The plan for the event API there is to make it admin-only (at least for<br>
> now). If this is data the user wants to see, that may change the plan for<br>
> the API or may mean storing it in ceilometer isn't a good fit.<br>
><br>
<br>
</div></div>Visibility into these events is critical to tracking the progress of<br>
any action done to a Heat stack:<br>
<br>
+---------------------+----+------------------------+--------------------+----------------------+<br>
| logical_resource_id | id | resource_status_reason | resource_status | event_time |<br>
+---------------------+----+------------------------+--------------------+----------------------+<br>
| AccessPolicy | 24 | state changed | CREATE_IN_PROGRESS | 2013-08-12T19:45:36Z |<br>
| AccessPolicy | 25 | state changed | CREATE_COMPLETE | 2013-08-12T19:45:36Z |<br>
| User | 26 | state changed | CREATE_IN_PROGRESS | 2013-08-12T19:45:36Z |<br>
| Key | 28 | state changed | CREATE_IN_PROGRESS | 2013-08-12T19:45:38Z |<br>
| User | 27 | state changed | CREATE_COMPLETE | 2013-08-12T19:45:38Z |<br>
| Key | 29 | state changed | CREATE_COMPLETE | 2013-08-12T19:45:39Z |<br>
| notcompute | 30 | state changed | CREATE_IN_PROGRESS | 2013-08-12T19:45:40Z |<br>
+---------------------+----+------------------------+--------------------+----------------------+<br>
<br>
So unless there is a plan to make this a user centric service, it does<br>
not seem like a good fit.<br>
<div class="im"><br>
> Are these "events" transmitted in the same way as notifications? If so, we<br>
> may already have the data.<br>
><br>
<br>
</div>The Heat engine records them while working on the stack. They have a<br>
fairly narrow, well defined interface, so it should be fairly easy to<br>
address the storage issue with a backend abstraction.<br></blockquote><div><br></div><div>OK. The term "event" frequently means "notification" for ceilometer, but it sounds like it's completely different in this case.</div>
<div><br>Doug</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5"><br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>