[openstack-dev] Event reporter usage

Joshua Harlow harlowja at yahoo-inc.com
Tue Mar 5 20:34:12 UTC 2013


Very welcome, it seems like yet another event notification/tracking
mechanism (YANM) may not be the best approach.

I understand how its nice to keep track of 'states/events' that a compute
node is going through, and I definitely see how it is nice to have it for
each compute instance (especially in regards to a orchestration layer
using said information to know what last happened, and where/what can be
resumed from...). It just catches me by surprise a little that another
mechanism got in, instead of maybe asking what is the '1' true way that
can work (is there one?).

I do want said information myself, for orchestration, and state
refactoring in nova, but would like to know how other people want to get
this. Perhaps a summit session is in order (this topic seems to repeat at
every summit, so what the heck, might as well repeat it again).

IMHO, and possibly just a rant, so feel free to skip this and carry on.
Nova I think has a problem. I would almost call it the 'microsoft office'
problem, where there are lots of new features, but unsure side-effects of
those features, side-effects being new routes messages can go, new
database entries that duplicates other database/MQ activity. It concerns
me greatly that this is or has happened, as I think most of our 99%
use-case is just making sure VM's come up correctly (aka I just want to
use 'microsoft office' to write a simple doc, not have clippy added in,
super spell check, super duper XYZ feature...). I just haven't been seeing
a large focus on making sure the 'fundamentals' are done the best there
can be (orchestration of states in a structured & controlled manner being
one example), and anything extra is only added in under 'experimental'
(which seems to be occurring, so that¹s good). Just we have to fight off
the 'microsoft office' effect, even though it is very tough. From helping
out with a orchestration prototype I know personally that nova has a
'feature-bloat' problem. There seems to be fundamental changes that need
to occur to right the ship in my opinion of what a 'righted' ship is.

PS: no offense to microsoft people meant :-)

On 3/5/13 4:49 AM, "Sandy Walsh" <sandy.walsh at rackspace.com> wrote:

>
>
>On 03/05/2013 01:10 AM, Joshua Harlow wrote:
>> Hi all,
>> 
>> I was just looking through the grizzly code and came upon a new goodie:
>> 
>> /compute_utils.EventReporter/
>> 
>> Is there a blueprint for how this is going to be used, it looks like its
>> being used to save 'states' for later 'resuming' for an orchestration
>> unit (conductor?)?
>> 
>> Is that correct? I haven't been following this to much so was wondering
>> as yahoo! and nttdata 'centralized' orchestration unit prototype (aka
>> https://etherpad.openstack.org/the-future-of-orch) also needs such state
>> transition data, but we would like to keep usage of said states and
>> recording of states starting/finishing and such in 1 place (and not
>> dispersed throughout the code). I also see stuff
>> like 
>>https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L24
>>7 which
>> is creating records based off of function names, is there a reason we
>> can't actually solidify on what the different states are beforehand and
>> not creating them on the fly. Isn't that a more sane and longer-term
>> approach where said states and there transitions are in 1 place and not
>> all over?
>> 
>> Just a personal concern but has there been any any determination of how
>> writing all this data into the DB (via the MQ) will affect DB size, its
>> a pretty big change from writing nothing in the DB (or vm_state and
>> task_state) to writing detailed information about when each function
>> starts and stops, has there been any size estimation on the change so
>> that deployers can ensure they adjust correctly? Maybe in general, but
>> is there any good design docs on the conductor, its features in grizzly
>> and such. If people are going to use it, I'd like to know what all its
>> doing in grizzly and what new database tables it adds, what new MQ
>> dependencies it adds, and so on. Does anyone else think that would be
>> useful?
>> 
>> -Josh
>
>Josh, thanks for writing this. I've been meaning to do something similar
>myself. We clearly have an issue with the myriad of ways notification
>events are being handled in Nova.
>
>We have:
>1. The AMQP notifications,
>2. The DB InstanceFault table,
>3. Exception handling decorators,
>4. The conductor mechanisms you've highlighted,
>5. and, to a lesser extent, logging
>
>Recently I was asked about what it would take to get some of the data
>stored in the InstanceFaults into the notifications. (why do it twice?)
>Then there's the problem of squashing a private data leakage problem in
>one place to have it appear in some other place.
>
>We are doing work on the Ceilometer side to shore up #1 to provide a
>meaningful data package, but there is clearly duplicated effort.
>
>Personally I think storing ever-growing InstanceFault data in the
>production database is a bad idea. Our rule should be: "Get archival
>data away from production Nova as quickly as possible."
>
>Sounds like we need a summit session to brainstorm ideas on unifying
>these things (logging excluded, perhaps)?
>
>-S
>
>
>> 
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




More information about the OpenStack-dev mailing list