Open Stack

On Mon, Nov 26, 2012 at 12:11 PM, Eoghan Glynn <eglynn at redhat.com> wrote:

>
>
> >       Eglynn and I discussed about how to handle the cumulative counter
> on
> >       IRC and we want to continue the discussion in the ML to get more
> >       feedback. The IRC log is in http://pastebin.com/byKvhNcJ. Eglynn,
> >       please correct me if I make any mistake.
> >
> >       The issue is the cpu_info[cpu_time] is reset to 0 when a domain is
> >       suspend/resume. There are already some discussion on it already,
> >       and several potential solution raised.
> >       Jd discussed this firstly in
> >       https://bugs.launchpad.net/ceilometer/+bug/1061817 and then it's
> >       discussed in ML at
> >
> http://www.mail-archive.com/openstack@lists.launchpad.net/msg17781.html.
> >
> >       There are several method on this, like JD's MAX()- FIRST(), or
> >       asalkeld's idea of store the offset. Both try to detect if there is
> >       reset happen by checking if there is counter reverse. (again,
> >       Eglynn, please correct me if I'm wrong).
> >
> >       However, during the discussion, we think that if the suspend/resume
> >       cycle is very rapid, and the ceilometer polling frequency is set up
> >       so that the stats appear to be in a monotonic sequence, then
> >       ceilometer will failed to detect the reset. A malicious user can
> >       exploit such situation.
> >
> >       For example, the ceilometer query every 4 minutes, and
> >       suspend/resume take 1 minutes (not exact length, just example).
> >       Then if user try to reset every 4 min, right after the ceilometer
> >       polling, and then execute 3 minutes. With such arrangement,
> >       ceilometer can't detect the reset. Yes, it's difficult to achieve
> >       this detect, but once achieved it, it will save a lot of cost.
> >
> >       My proposal is to fix this issue from nova side as it's nova bug.
> >       Per my understanding, the reason is different view of
> >       openstack/libvirt. For openstack, the instance is a logic object,
> >       thus still exist after suspend/resumt, while for libvirt, it's a
> >       real object, qemu process, which does not exist after
> >       suspend/resume. Nova should translate libvirt's view to openstack's
> >       view.  And this issue happens to other functionality, like nova's
> >       get_diagnostic().  But I agree with Eglynn that there are several
> >       implementation difficulties to fix it from nova side.
>
>
> Here's another datapoint to consider, the bandwidth usage accounting
> code in nova (currently only implemented for the xenapi driver) uses
> a fairly basic scheme to detect resets simply by remembering the last
> value and checking if the sequence appears monotonic:
>
>
> https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2928
>
> I don't know if this code is intended to produce rock-solid or just
> "advisory" stats, but it's interesting that it has taken this simple
> approach.
>
> As discussed on IRC earlier, I'd be leery of going down the road of
> enforcing a "fix" on the nova side, such that resets were detected and
> adjusted for prior to being reported (apart from the complexity of
> reliably storing the previous value, there are multiple different
> counters that could potentially be reset in different ways, possibly
> depending on the hypervisor driver etc.)
>
> However doing the adjustment within the metering store would always
> be possible (either by storing a running offset as well as the reported
> values, or switching to a delta-based measurement), as would isolating
> the local maxima for max-min style queries (though we'd need to bound
> the computational cost of that approach).
>

I understand Julien's reluctance to have the pollsters emitting computed
data, but tracking values inside the pollster so we can send a delta
instead of the cumulative value we have now seems like a relatively
straightforward fix for this.


>
> Of course that doesn't help in terms of detecting the edge case
> where the reset is occurring so rapidly that the post-reset value
> overtakes the last known value *before* the next sample is
> taken. AFAICS this could only be addressed in code that's aware
> of all the possible cases where a reset can occur, and is also in
> a position to trigger the appropriate actions to enable the
> adjustment (e.g. nova-compute persisting the latest stats when it
> receives a 'suspend_instance' RPC message).
>

We're already plugging in to the notifier API to have nova do some polling
when instances are deleted. We could do the same on suspend, reboot, etc.
-- whatever introduces this race condition.

Doug


>
> Cheers,
> Eoghan
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121126/5177015b/attachment.html>