Open Stack

> > Note that the notifications that capture these resource state transitions
> > are a long-standing mechanism in openstack that ceilometer has depended
> > on from the very outset. I don't think it's realistic to envisage these
> > interactions will be replaced by REST APIs any time soon.
> 
> I wasn't advocating doing everything over a REST API. (API is an
> overloaded term) I just meant that if we're depending on
> notifications for communication between projects then we should
> enforce a stability contract on them. Similar to what we already
> have with the API stability guidelines for the REST APIs. The fact
> that there is no direct enforcement on notifications, either through
> social policy or testing, is what I was taking issue with.
>
> I also think if we decide to have a policy of enforcing notification
> stability then we should directly test the notifications from an
> external repo to block slips. But, that's a discussion for later, if
> at all.

A-ha, OK, got it.

I've discussed enforcing such stability with jogo on IRC last night, and
kicked off a separate thread to capture that:

  http://lists.openstack.org/pipermail/openstack-dev/2014-July/039858.html

However the time-horizon for that effort would be quite a bit into the
future, compared to the test coverage that we're aiming to have in place
for juno-2.

> > > > The branchless Tempest spec envisages new features will be added and
> > > > need to be skipped when testing stable/previous, but IIUC requires
> > > > that the presence of new behaviors is externally discoverable[5].
> > > 
> > > I think the test case you proposed is fine. I know some people will
> > > argue that it is expanding the scope of tempest to include more
> > > whitebox like testing, because the notification are an internal
> > > side-effect of the api call, but I don't see it that way. It feels
> > > more like exactly what tempest is there to enable testing, a
> > > cross-project interaction using the api.
> > 
> > In my example, APIs are only used to initiate the action in cinder
> > and then to check the metering data in ceilometer.
> > 
> > But the middle-piece, i.e. the interaction between cinder & ceilometer,
> > is not mediated by an API. Rather, its carried via an unversioned
> > notification.
> 
> Yeah, exactly, that's why I feel it's a valid Tempest test case.

Just to clarify: you meant to type "it's a valid Tempest test case"
as opposed to "it's *not* a valid Tempest test case", right?
 
> What I was referring to as the counter argument, and where the
> difference of opinion was, is that the test will be making REST API
> calls to both trigger a nominally internal mechanism (the
> notification) from the services and then using the ceilometer api to
> validate the notification worked. 

Yes, that's exactly the idea.

> But, it's arguably the real intent of these tests is to validate
> that internal mechanism, which is basically a whitebox test. The
> argument was that by testing it in tempes we're testing
> notifications poorly because of it's black box limitation
> notifications will just be tested indirectly. Which I feel is a
> valid point, but not a sufficient reason to exclude the notification
> tests from tempest.

Agreed.

> I think the best way to move forward is to have functional whitebox
> tests for the notifications as part of the individual projects
> generating them, and that way we can direct validation of the
> notification. But, I also feel there should be tempest tests on top
> of that that verify the ceilometer side of consuming the
> notification and the api exposing that information.

Excellent. So, indeed, more fullsome coverage of the notification
logic with in-tree tests on the producer side would definitely
to welcome, and could be seen as a "phase zero" of an overall to
fix/impove the notification mechanism.
  
> > > But, they're is also a slight misunderstanding here. Having a
> > > feature be externally discoverable isn't a hard requirement for a
> > > config option in tempest, it's just *strongly* recommended. Mostly,
> > > because if there isn't a way to discover it how are end users
> > > expected to know what will work.
> > 
> > A-ha, I missed the subtle distinction there and thought that this
> > discoverability was a *strict* requirement. So how bad a citizen would
> > a project be considered to be if it chose not to meet that strong
> > recommendation?
> 
> You'd be far from the only ones who are doing that, for an existing example
> look at anything on the nova driver feature matrix. Most of those aren't
> discoverable from the API. So I think it would be ok to do that, but when we
> have efforts like:
> 
> https://review.openstack.org/#/c/94473/
> 
> it'll make that more difficult. Which is why I think having discoverability
> through the API is important. (it's the same public cloud question)

So for now, would it suffice for the master versus stable/icehouse
config to be checked-in in static form pending the completion of that
BP on tempest-conf-autogen?

Then the assumption is that this static config is replaced by auto-
generating the tempest config using some project-specific discovery
mechanisms?

> > > For this specific case I think it's definitely fair to have an
> > > option for which notifications services are expected to be
> > > generated. That's something that is definitely a configurable option
> > > when setting up a deployment, and is something that feels like a
> > > valid tempest config option, so we know which tests will work. We
> > > already have similar feature flags for config time options in the
> > > services, and having options like that would also get you out of
> > > that backport mess you have right now.
> > 
> > So would this test configuration option would have a semantic like:
> > 
> >  "a wildcarded list of notification event types that ceilometer consumes"
> > 
> > then tests could be skipped on the basis of the notifications that
> > they depend on being unavailable, in the manner of say:
> > 
> >   @testtools.skipUnless(
> >       matchesAll(CONF.telemetry_consumed_notifications.volume,
> >                  ['snapshot.exists',
> >                   'snapshot.create.*',
> >                   'snapshot.delete.*',
> >                   'snapshot.resize.*',]
> >                 )
> >   )
> >   @test.services('volume')
> >   def test_check_volume_notification(self):
> i>       ...
> > 
> > Is something of that ilk what you envisaged above?
> 
> Yeah that was my idea more or less, but I'd probably move the logic into a
> separate
> decorator to make a bit cleaner. Like:
> 
> @test.consumed_notifications('volumes', 'snapshot.exists',
> 'snapshot.create.*',
>                              'snapshot.delete.*', 'snapshot.resize.*')
> 
> and you can just double them up if the test requires notifications from other
> services.

OK, so the remaining thing I wanted to confirm is that it's acceptable
for the skip/no-skip logic of that decorator to be driven by static
(as opposed to discoverable) config?

> > > However, it does raise the question of being an end user how am I
> > > expected to know which notifications get counted? Which is why having
> > > the feature discoverability is generally a really good idea.
> > 
> > So certain things we could potentially make discoverable through the
> > ceilometer capabilities API. But there's a limit to how fine-grained
> > we can make that. Also it was primarily intended to surface lack of
> > feature-parity in the storage driver layer (e.g. one driver supports
> > sdtdev, but another doesn't) as opposed to the notification-handling
> > layer.
> 
> I think we should probably decouple the discoverability question
> from the discussion about testing, because to a certain degree it's
> a separate problem.  It might be worth splitting the API
> discoverability discussion off as a separate thread so we don't
> cloud either topic. (pun intended :) )

I'm confused as how we can completely split off the discoverability
question, if its a *strong* recommendation (but not a strict requirement)
of the testing framework.

> I guess it really depends on how we expect people to be consuming
> the API, and whether we're presenting that expectation to
> people. I'll have to admit I'm not as familiar with ceilometer as I
> should be, which is probably where some of my confusion is coming
> from.

The intent of capabilities API was initially purely to give users
insight into the storage-driver-specific functionality exposed in
the primary API.

I guess we've been kicking around on the idea of re-purposing that
API, so that it also allows surfacing of the list of meters being
captured.

But that idea hasn't yet been firmed up to the extent that we've
figured out how *users* might consume it.

> The naive real world example I have in my head is: someone wants to
> build a tool that will generate pretty pictures from data queried
> from the ceilometer API, and if they want to run this on 2+ distinct
> OpenStack deployments how will they know only through the API which
> notifications are getting counted, etc. If it's doable through the
> API then the tool can adjust what gets generated based on an API
> call, (either up front or with an expected error response, like
> unsupported request. (of course a new response like unsupported
> would probably mean a new API rev) Without the discoverability
> manual intervention would be required each time it's used with a new
> cloud. This probably isn't a valid use case and I'm missing
> something critical here. In which case the discoverability
> discussion is moot.

That would be a case where discoverability of meters would be useful
(i.e. not just the meters currently in the metering store, also all
the meter that this deployment of ceilometer is capable of gathering
samples for). However that feature will take a bit more thought &
discussion to firm up.

> > > > One approach mooted for allowing these kind of scenarios to be tested
> > > > was to split off the pure-API aspects of Tempest so that it can be used
> > > > for probing public-cloud-capabilities as well as upstream CI, and then
> > > > build project-specific mini-Tempests to test integration with other
> > > > projects.
> > > > 
> > > > Personally, I'm not a fan of that approach as it would require a lot
> > > > of QA expertise in each project, lead to inefficient use of CI
> > > > nodepool resources to run all the mini-Tempests, and probably lead to
> > > > a divergent hotchpotch of per-project approaches.
> > > 
> > > I think the proposal here was for people interested in doing
> > > whitebox testing, where there is a desire to test an internal
> > > project mechanism. I could see the argument for testing
> > > notifications this way, but that would have to be for every project
> > > individually. There are already several projects that have
> > > functional testing like this in tree and run them as a gating
> > > job. There are definitely certain classes of testing where doing
> > > this makes sense.
> > 
> > I'm not sure that this would be realistic to test individually (if by
> > that you meant just with the ceilometer agents running alone) as it
> > depends on a notification emitted from cinder.
> 
> I wasn't actually referring to use it for this specific case, just a
> general thought. Although maybe it would be useful to add to add
> notification functional tests to Cinder. I think Sean's response on
> the OP outlined some interesting alternative testing strategies
> which could would definitely fit this use case. I still need to
> finish forming my opinions about that before I respond to it.

OK. (And so do I).

> So I wrote this in one pass, sorry if bits of it are incoherent. :)

No, it was all good, thanks!

Cheers,
Eoghan