Open Stack

On Fri, Aug 9, 2013 at 11:56 AM, Clint Byrum <clint at fewbar.com> wrote:

> Excerpts from Sandy Walsh's message of 2013-08-09 06:16:55 -0700:
> >
> > On 08/08/2013 11:36 PM, Angus Salkeld wrote:
> > > On 08/08/13 13:16 -0700, Clint Byrum wrote:
> > >> Last night while reviewing a feature which would add more events to
> the
> > >> event table, it dawned on me that the event table really must be
> removed.
> > >
> > >
> > >>
> > >> https://bugs.launchpad.net/heat/+bug/1209492
> > >>
> > >> tl;dr: users can write an infinite number of rows to the event table
> at
> > >> a fairly alarming rate just by creating and updating a very large
> stack
> > >> that has no resources that cost any time or are even billable (like an
> > >> autoscaling launch configuration).
> > >>
> > >> The table has no purge function, so the only way to clear out old
> events
> > >> is to delete the stack, or manually remove them directly in the
> database.
> > >>
> > >> We've all been through this before, logging to a database seems great
> > >> until you actually do it.
> > >>
> > >> I have some ideas for how to solve it, but I wanted to get a wider
> > >> audience:
> > >>
> > >> 1) Make the event list a ring buffer. Have rows 0 - $MAX_BUFFER_SIZE
> in
> > >> each stack, and simply write each new event to the next open position,
> > >> wrapping at $MAX_BUFFER_SIZE. Pros: little change to current code,
> > >> just need an offset column added and code that will properly wrap to 0
> > >> at $MAX_BUFFER_SIZE. Cons: still can incur heavy transactional load on
> > >> the database server.A
> > >>
> > >> 1.b) Same, but instead of rows, just maintain a blob and append the
> rows
> > >> as json list. Lowers transactional load but would push some load onto
> > >> the API servers and such to parse these out, and would make pagination
> > >> challenging. Blobs also can be a drain on DB server performance.
> > >>
> > >> 2) Write a purge script. Delete old ones. Pros: No code change, just
> > >> new code to do purging. Cons: same as 1, plus more vulnerability to an
> > >> aggressive attacker who can fit a lot of data in between purges. Also
> > >> large scale deletes can be really painful (see: keystone sql token
> > >> backend).
> > >>
> > >> 3) Log events to Swift. I can't seem to find information on how/if
> > >> appending works there. Tons of tiny single-row files is an option,
> but I
> > >> want to hear from people with more swift knowledge if that is a
> viable,
> > >> performant option. Pros: Scale to the moon. Can charge tenant for
> usage
> > >> and let them purge events as needed. Cons: Adds swift as a requirement
> > >> of Heat.
> > >>
> > >> 4) Provide a way for users to receive logs via HTTP POST. Pros: Simple
> > >> and punts the problem to the users. Cons: users will be SoL if they
> > >> don't have a place to have logs posted to.
> > >>
> > >> 5) Provide a way for users to receive logs via messaging service like
> > >> Marconi.  Pros/Cons: same as HTTP, but perhaps a little more confusing
> > >> and ambitious given Marconi's short existence.
> > >>
> > >> 6) Provide a pluggable backend for logging. This seems like the way
> most
> > >> OpenStack projects solve these issues, which is to let the deployers
> > >> choose and/or provide their own way to handle a sticky problem. Pros:
> > >> Simple and flexible for the future. Cons: Would require writing at
> least
> > >> one backend provider that does what the previous 5 options suggest.
> > >>
> > >> To be clear: Heat cannot really exist without this, as it is the only
> way
> > >> to find out what your stack is doing or has done.
> > >
> > > btw Clint I have ditched that "Recorder" patch as Ceilometer is
> > > getting a Alarm History api soon, so we can defer to that for that
> > > functionality (alarm transitions).
> > >
> > > But we still need a better way to record events/logs for the user.
> > > So I make this blueprint a while ago:
> > > https://blueprints.launchpad.net/heat/+spec/user-visible-logs
> > >
> > > I am becomming more in favor of user options rather than deployer
> > > options if possible. So provide resources for Marconi, Meniscus and
> > > what ever...
> > > Although what is nice about Marconi is you could then hook up what
> > > ever you want to it.
> >
> > Logs are one thing (and Meniscus is a great choice for that), but events
> > are the very thing CM is designed to handle. Wouldn't it make sense to
> > push them back into there?
> >
>
> I'm not sure these events make sense in the current Ceilometer (I assume
> that is "CM" above) context. These events are:
>
> ... Creating stack A
> ... Creating stack A resource A
> ... Created stack A resource A
> ... Created stack A
>
> Users will want to be able to see all of the events for a stack, and
> likely we need to be able to paginate through them as well.
>
> They are fundamental and low level enough for Heat that I'm not sure
> putting them in Ceilometer makes much sense, but maybe I don't understand
> Ceilometer..  or "CM" is something else entirely. :)
>

CM is indeed ceilometer.

The plan for the event API there is to make it admin-only (at least for
now). If this is data the user wants to see, that may change the plan for
the API or may mean storing it in ceilometer isn't a good fit.

Are these "events" transmitted in the same way as notifications? If so, we
may already have the data.

Doug


>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130812/edda5c80/attachment.html>