<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Jun 14, 2014 at 7:33 AM, Eoghan Glynn <span dir="ltr"><<a href="mailto:eglynn@redhat.com" target="_blank">eglynn@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><br>
<br>
----- Original Message -----<br>
> On 11 June 2014 20:07, Joe Gordon <<a href="mailto:joe.gordon0@gmail.com">joe.gordon0@gmail.com</a>> wrote:<br>
> > On Wed, Jun 11, 2014 at 11:38 AM, Matt Riedemann<br>
> > <<a href="mailto:mriedem@linux.vnet.ibm.com">mriedem@linux.vnet.ibm.com</a>> wrote:<br>
> >> On 6/11/2014 10:01 AM, Eoghan Glynn wrote:<br>
> >>> Thanks for bringing this to the list Matt, comments inline ...<br>
> >>><br>
> >>>> tl;dr: some pervasive changes were made to nova to enable polling in<br>
> >>>> ceilometer which broke some things and in my opinion shouldn't have been<br>
> >>>> merged as a bug fix but rather should have been a blueprint.<br>
> >>>><br>
> >>>> ===<br>
> >>>><br>
> >>>> The detailed version:<br>
> >>>><br>
> >>>> I opened bug 1328694 [1] yesterday and found that came back to some<br>
> >>>> changes made in ceilometer for bug 1262124 [2].<br>
> >>>><br>
> >>>> Upon further inspection, the original ceilometer bug 1262124 made some<br>
> >>>> changes to the nova os-floating-ips API extension and the database API<br>
> >>>> [3], and changes to python-novaclient [4] to enable ceilometer to use<br>
> >>>> the new API changes (basically pass --all-tenants when listing floating<br>
> >>>> IPs).<br>
> >>>><br>
> >>>> The original nova change introduced bug 1328694 which spams the nova-api<br>
> >>>> logs due to the ceilometer change [5] which does the polling, and right<br>
> >>>> now in the gate ceilometer is polling every 15 seconds.<br>
> >>><br>
> >>><br>
> >>> IIUC that polling cadence in the gate is in the process of being reverted<br>
> >>> to the out-of-the-box default of 600s.<br>
> >>><br>
> >>>> I pushed a revert in ceilometer to fix the spam bug and a separate patch<br>
> >>>> was pushed to nova to fix the problem in the network API.<br>
> >>><br>
> >>><br>
> >>> Thank you for that. The revert is just now approved on the ceilometer<br>
> >>> side,<br>
> >>> and is wending its merry way through the gate.<br>
> >>><br>
> >>>> The bigger problem I see here is that these changes were all made under<br>
> >>>> the guise of a bug when I think this is actually a blueprint. We have<br>
> >>>> changes to the nova API, changes to the nova database API, CLI changes,<br>
> >>>> potential performance impacts (ceilometer can be hitting the nova<br>
> >>>> database a lot when polling here), security impacts (ceilometer needs<br>
> >>>> admin access to the nova API to list floating IPs for all tenants),<br>
> >>>> documentation impacts (the API and CLI changes are not documented), etc.<br>
> >>>><br>
> >>>> So right now we're left with, in my mind, two questions:<br>
> >>>><br>
> >>>> 1. Do we just fix the spam bug 1328694 and move on, or<br>
> >>>> 2. Do we revert the nova API/CLI changes and require this goes through<br>
> >>>> the nova-spec blueprint review process, which should have happened in<br>
> >>>> the first place.<br>
> >>><br>
> >>><br>
> >>> So just to repeat the points I made on the unlogged #os-nova IRC channel<br>
> >>> earlier, for posterity here ...<br>
> >>><br>
> >>> Nova already exposed an all_tenants flag in multiple APIs (servers,<br>
> >>> volumes,<br>
> >>> security-groups etc.) and these would have:<br>
> >>><br>
> >>> (a) generally pre-existed ceilometer's usage of the corresponding APIs<br>
> >>><br>
> >>> and:<br>
> >>><br>
> >>> (b) been tracked and proposed at the time via straight-forward LP<br>
> >>> bugs,<br>
> >>> as opposed to being considered blueprint material<br>
> >>><br>
> >>> So the manner of the addition of the all_tenants flag to the floating_ips<br>
> >>> API looks like it just followed existing custom & practice.<br>
> >>><br>
> >>> Though that said, the blueprint process and in particular the nova-specs<br>
> >>> aspect, has been tightened up since then.<br>
> >>><br>
> >>> My preference would be to fix the issue in the underlying API, but to use<br>
> >>> this as "a teachable moment" ... i.e. to require more oversight (in the<br>
> >>> form of a reviewed & approved BP spec) when such API changes are proposed<br>
> >>> in the future.<br>
> >>><br>
> >>> Cheers,<br>
> >>> Eoghan<br>
> >>><br>
> >>>> Are there other concerns here? If there are no major objections to the<br>
> >>>> code that's already merged, then #2 might be excessive but we'd still<br>
> >>>> need docs changes.<br>
> >>>><br>
> >>>> I've already put this on the nova meeting agenda for tomorrow.<br>
> >>>><br>
> >>>> [1] <a href="https://bugs.launchpad.net/ceilometer/+bug/1328694" target="_blank">https://bugs.launchpad.net/ceilometer/+bug/1328694</a><br>
> >>>> [2] <a href="https://bugs.launchpad.net/nova/+bug/1262124" target="_blank">https://bugs.launchpad.net/nova/+bug/1262124</a><br>
> >>>> [3] <a href="https://review.openstack.org/#/c/81429/" target="_blank">https://review.openstack.org/#/c/81429/</a><br>
> >>>> [4] <a href="https://review.openstack.org/#/c/83660/" target="_blank">https://review.openstack.org/#/c/83660/</a><br>
> >>>> [5] <a href="https://review.openstack.org/#/c/83676/" target="_blank">https://review.openstack.org/#/c/83676/</a><br>
> >>>><br>
> >>>> --<br>
> >>>><br>
> >>>> Thanks,<br>
> >>>><br>
> >>>> Matt Riedemann<br>
> >>>><br>
> >>>><br>
> >>>> _______________________________________________<br>
> >>>> OpenStack-dev mailing list<br>
> >>>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> >>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> >>>><br>
> >>><br>
> >>> _______________________________________________<br>
> >>> OpenStack-dev mailing list<br>
> >>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> >>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> >>><br>
> >><br>
> >> While there is precedent for --all-tenants with some of the other APIs,<br>
> >> I'm concerned about where this stops. When ceilometer wants polling on<br>
> >> some<br>
> >> other resources that the nova API exposes, will it need the same thing?<br>
> >> Doing all of this polling for resources in all tenants in nova puts an<br>
> >> undue<br>
> >> burden on the nova API and the database.<br>
> >><br>
> >> Can we do something with notifications here instead? That's where the<br>
> >> nova-spec process would have probably caught this.<br>
> ><br>
> > ++ to notifications and not polling.<br>
><br>
> Yeah, I think we need to revert this, and go through the specs<br>
> process. Its been released in Juno-1 now, so this revert feels bad,<br>
> but perhaps its the best of a bad situation?<br>
><br>
> Word of caution, we need to get notifications versioned correctly if<br>
> we want this as a more formal external "API". I think Heat have<br>
> similar issues in this area, efficiently knowing about something<br>
> happening in Nova. So we do need to "solve this".<br>
><br>
> Some kind of callback to ceilometer's REST API sounds attractive, but<br>
> messes up dependencies.<br>
<br>
</div></div>Yeah, I don't think that would be a workable approach, as it breaks<br>
the separation of concerns ... i.e. nova would then need to become<br>
aware of the format of ceilometer metering samples.<br>
<br>
Instead, the established model is that nova (or any other openstack<br>
service in general) either:<br>
<br>
* emits resource lifecycle data in the form of notifications (which<br>
generally contains a super-set of the data that ceilometer is<br>
specifically interested in),<br>
<br>
or else:<br>
<br>
* allows these data to be surfaced via a RESTful API that ceilometer<br>
can call into<br>
<div class=""><br>
> Maybe implemented as a plugin to a Nova<br>
> "notification system", could work better somehow? A versioned<br>
> interface that issues "server life cycle" events (some versioned<br>
> subset of notifications), and allow services to implement their own<br>
> plugins to that interface. But maybe we should probably just version<br>
> the current notifications, rather than re-invent another interface.<br>
<br>
</div>I don't fully understand the concept of services implementing their<br>
own plugins to this versioned interface.<br>
<br>
Would the implementing service in this case be nova or ceilometer?<br>
<br>
If the latter, why re-invent the existing notifications mechanism<br>
layered over AMQP (i.e. oslo.messaging or the old openstack.common<br>
nofications, now deprecated)?<br>
<br>
In terms of versioning, it would be great to protect ceilometer from<br>
unilateral changes to or inconsistencies in the notification formats<br>
used by the services, as this has been found to be brittle in practice.<br></blockquote><div><br></div><div>++, right now nova only produces one stable type of API: REST. Notifications aren't versioned and we don't really gate on their format (or document them). Without versioning and proper gating, notifications will never become a stable contractual API.</div>
<div><br></div><div>While I would like to see a way to protect ceilometer from unilateral changes, I think we should focus on making sure notifications become a contractual stable API, just like REST. As that is the more general issue is at play here.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Cheers,<br>
Eoghan<br>
<div class="im HOEnZb"><br>
> Thanks,<br>
> John<br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
</div><div class="HOEnZb"><div class="h5">_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>