Open Stack

On Wed, Jul 16, 2014 at 07:44:38PM +0400, Dina Belova wrote:
> Ildiko, thanks for starting this discussion.
> 
> Really, that is quite painful problem for Ceilometer and QA team. As far as
> I know, currently there is some kind of tendency of making integration
> Tempest tests quicker and less resource consuming - that's quite logical
> IMHO. Polling as a way of information collecting from different services
> and projects is quite consuming speaking about load on Nova API, etc. -
> that's why I completely understand the wish of QA team to get rid of it,
> although polling still makes lots work inside Ceilometer, and that's why
> integration testing for this feature is really important for me as
> Ceilometer contributor - without pollsters testing we have no way to check
> its workability.
> 
> That's why I'll be really glad if Ildiko's (or whatever other) solution
> that will allow polling testing in the gate will be found and accepted.
> 
> Problem with described above solution requires some kind of change in what
> do we call "environment preparing" for the integration testing - and we
> really need QA crew help here. Afair polling deprecation was suggested in
> some of the IRC discussions (by only notifications usage), but that's not
> the solution that might be just used right now - but we need way of
> Ceilometer workability verification right now to continue work on its
> improvement.
> 
> So any suggestions and comments are welcome here :)
> 
> Thanks!
> Dina
> 
> 
> On Wed, Jul 16, 2014 at 7:06 PM, Ildikó Váncsa <ildiko.vancsa at ericsson.com>
> wrote:
> 
> >  Hi Folks,
> >
> >
> >
> > We’ve faced with some problems during running Ceilometer integration tests
> > on the gate. The main issue is that we cannot test the polling mechanism,
> > as if we use a small polling interval, like 1 min, then it puts a high
> > pressure on Nova API. If we use a longer interval, like 10 mins, then we
> > will not be able to execute any tests successfully, because it would run
> > too long.
> >
> >
> >
> > The idea, to solve this issue,  is to reconfigure Ceilometer, when the
> > polling is tested. Which would mean to change the polling interval from the
> > default 10 mins to 1 min at the beginning of the test, restart the service
> > and when the test is finished, the polling interval should be changed back
> > to 10 mins, which will require one more service restart. The downside of
> > this idea is, that it needs service restart today. It is on the list of
> > plans to support dynamic re-configuration of Ceilometer, which would mean
> > the ability to change the polling interval without restarting the service.
> >
> >
> >
> > I know that this idea isn’t ideal from the PoV that the system
> > configuration is changed during running the tests, but this is an expected
> > scenario even in a production environment. We would change a parameter that
> > can be changed by a user any time in a way as users do it too. Later on,
> > when we can reconfigure the polling interval without restarting the
> > service, this approach will be even simpler.

So your saying that you expect users to be able to manually reconfigure
Ceilometer on the fly to be able to use polling, that seems far from ideal.

> >
> >
> >
> > This idea would make it possible to test the polling mechanism of
> > Ceilometer without any radical change in the ordering of test cases or any
> > other things that would be strange in integration tests. We couldn’t find
> > any better way to solve the issue of the load on the APIs caused by polling.
> >
> >
> >
> > What’s your opinion about this scenario? Do you think it could be a viable
> > solution to the above described problem?
> >
> >
> >

Umm, so frankly this approach seems kind of crazy to me. Aside from the project
level implications of saying that as a user to ensure you can't use polling data
reliably unless you adjust the polling frequency of Ceilometer. The bigger issue
is that you're not necessarily solving the problem. The test will still have an
inherent race condition because there is still no guarantee on the polling
happening during the test window.

So assume this were implemented and you decrease the polling rate to 1 min. and
restart ceilometer during the test setUp(). You'll still dependent on an
internal ceilometer event occurring during the wait period of the test. There's
actually no guarantee that everything will happen in the timeout interval for
the test, your just hoping it will. It's still just a best guess that will
probably work in the general case, but will just cause race bugs in the gate
when things get slow for random reasons. (which increasing the poll rate,
even temporarily, will contribute to)

The other thing to consider is how would this be implemented, changing a config
and restarting a service is *way* outside the scope of tempest. I'd be -2 on
anything proposed to tempest that mucks around with a projects config file like
this or anything that restarts a service. If it were exposed through a rest API
command that'd be a different story, but then you'd still have the race issue I
described above.

IMO, a far better approach would be to just implement a flush command, or
something of that ilk, in the rest API to force a poll. That way the test can
just say: poll now, and then collect the results after the action is complete. I
can also see that being useful for admins/operators who want to collect the
results periodically, for billing or what have you. They can write a tool that
does a flush to get up to the latest data and the periodic collect everything.
Instead of now where writing such a utility is at the whim of whenever the most
recent poll occurred.

-Matt Treinish
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140724/20568e77/attachment.pgp>