<br><br><div class="gmail_quote">On Mon, Nov 26, 2012 at 12:11 PM, Eoghan Glynn <span dir="ltr"><<a href="mailto:eglynn@redhat.com" target="_blank">eglynn@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5"><br>
<br>
> Eglynn and I discussed about how to handle the cumulative counter on<br>
> IRC and we want to continue the discussion in the ML to get more<br>
> feedback. The IRC log is in <a href="http://pastebin.com/byKvhNcJ" target="_blank">http://pastebin.com/byKvhNcJ</a>. Eglynn,<br>
> please correct me if I make any mistake.<br>
><br>
> The issue is the cpu_info[cpu_time] is reset to 0 when a domain is<br>
> suspend/resume. There are already some discussion on it already,<br>
> and several potential solution raised.<br>
> Jd discussed this firstly in<br>
> <a href="https://bugs.launchpad.net/ceilometer/+bug/1061817" target="_blank">https://bugs.launchpad.net/ceilometer/+bug/1061817</a> and then it's<br>
> discussed in ML at<br>
> <a href="http://www.mail-archive.com/openstack@lists.launchpad.net/msg17781.html" target="_blank">http://www.mail-archive.com/openstack@lists.launchpad.net/msg17781.html</a>.<br>
><br>
> There are several method on this, like JD's MAX()- FIRST(), or<br>
> asalkeld's idea of store the offset. Both try to detect if there is<br>
> reset happen by checking if there is counter reverse. (again,<br>
> Eglynn, please correct me if I'm wrong).<br>
><br>
> However, during the discussion, we think that if the suspend/resume<br>
> cycle is very rapid, and the ceilometer polling frequency is set up<br>
> so that the stats appear to be in a monotonic sequence, then<br>
> ceilometer will failed to detect the reset. A malicious user can<br>
> exploit such situation.<br>
><br>
> For example, the ceilometer query every 4 minutes, and<br>
> suspend/resume take 1 minutes (not exact length, just example).<br>
> Then if user try to reset every 4 min, right after the ceilometer<br>
> polling, and then execute 3 minutes. With such arrangement,<br>
> ceilometer can't detect the reset. Yes, it's difficult to achieve<br>
> this detect, but once achieved it, it will save a lot of cost.<br>
><br>
> My proposal is to fix this issue from nova side as it's nova bug.<br>
> Per my understanding, the reason is different view of<br>
> openstack/libvirt. For openstack, the instance is a logic object,<br>
> thus still exist after suspend/resumt, while for libvirt, it's a<br>
> real object, qemu process, which does not exist after<br>
> suspend/resume. Nova should translate libvirt's view to openstack's<br>
> view. And this issue happens to other functionality, like nova's<br>
> get_diagnostic(). But I agree with Eglynn that there are several<br>
> implementation difficulties to fix it from nova side.<br>
<br>
<br>
</div></div>Here's another datapoint to consider, the bandwidth usage accounting<br>
code in nova (currently only implemented for the xenapi driver) uses<br>
a fairly basic scheme to detect resets simply by remembering the last<br>
value and checking if the sequence appears monotonic:<br>
<br>
<a href="https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2928" target="_blank">https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2928</a><br>
<br>
I don't know if this code is intended to produce rock-solid or just<br>
"advisory" stats, but it's interesting that it has taken this simple<br>
approach.<br>
<br>
As discussed on IRC earlier, I'd be leery of going down the road of<br>
enforcing a "fix" on the nova side, such that resets were detected and<br>
adjusted for prior to being reported (apart from the complexity of<br>
reliably storing the previous value, there are multiple different<br>
counters that could potentially be reset in different ways, possibly<br>
depending on the hypervisor driver etc.)<br>
<br>
However doing the adjustment within the metering store would always<br>
be possible (either by storing a running offset as well as the reported<br>
values, or switching to a delta-based measurement), as would isolating<br>
the local maxima for max-min style queries (though we'd need to bound<br>
the computational cost of that approach).<br></blockquote><div><br></div><div>I understand Julien's reluctance to have the pollsters emitting computed data, but tracking values inside the pollster so we can send a delta instead of the cumulative value we have now seems like a relatively straightforward fix for this.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Of course that doesn't help in terms of detecting the edge case<br>
where the reset is occurring so rapidly that the post-reset value<br>
overtakes the last known value *before* the next sample is<br>
taken. AFAICS this could only be addressed in code that's aware<br>
of all the possible cases where a reset can occur, and is also in<br>
a position to trigger the appropriate actions to enable the<br>
adjustment (e.g. nova-compute persisting the latest stats when it<br>
receives a 'suspend_instance' RPC message).<br></blockquote><div><br></div><div>We're already plugging in to the notifier API to have nova do some polling when instances are deleted. We could do the same on suspend, reboot, etc. -- whatever introduces this race condition.</div>
<div><br></div><div>Doug</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Cheers,<br>
Eoghan<br>
<div class="HOEnZb"><div class="h5"><br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br>