Open Stack

On Thu, Jul 10, 2014 at 08:37:40AM -0400, Eoghan Glynn wrote:
> 
> > > Note that the notifications that capture these resource state transitions
> > > are a long-standing mechanism in openstack that ceilometer has depended
> > > on from the very outset. I don't think it's realistic to envisage these
> > > interactions will be replaced by REST APIs any time soon.
> > 
> > I wasn't advocating doing everything over a REST API. (API is an
> > overloaded term) I just meant that if we're depending on
> > notifications for communication between projects then we should
> > enforce a stability contract on them. Similar to what we already
> > have with the API stability guidelines for the REST APIs. The fact
> > that there is no direct enforcement on notifications, either through
> > social policy or testing, is what I was taking issue with.
> >
> > I also think if we decide to have a policy of enforcing notification
> > stability then we should directly test the notifications from an
> > external repo to block slips. But, that's a discussion for later, if
> > at all.
> 
> A-ha, OK, got it.
> 
> I've discussed enforcing such stability with jogo on IRC last night, and
> kicked off a separate thread to capture that:
> 
>   http://lists.openstack.org/pipermail/openstack-dev/2014-July/039858.html
> 
> However the time-horizon for that effort would be quite a bit into the
> future, compared to the test coverage that we're aiming to have in place
> for juno-2.
> 
> > > > > The branchless Tempest spec envisages new features will be added and
> > > > > need to be skipped when testing stable/previous, but IIUC requires
> > > > > that the presence of new behaviors is externally discoverable[5].
> > > > 
> > > > I think the test case you proposed is fine. I know some people will
> > > > argue that it is expanding the scope of tempest to include more
> > > > whitebox like testing, because the notification are an internal
> > > > side-effect of the api call, but I don't see it that way. It feels
> > > > more like exactly what tempest is there to enable testing, a
> > > > cross-project interaction using the api.
> > > 
> > > In my example, APIs are only used to initiate the action in cinder
> > > and then to check the metering data in ceilometer.
> > > 
> > > But the middle-piece, i.e. the interaction between cinder & ceilometer,
> > > is not mediated by an API. Rather, its carried via an unversioned
> > > notification.
> > 
> > Yeah, exactly, that's why I feel it's a valid Tempest test case.
> 
> Just to clarify: you meant to type "it's a valid Tempest test case"
> as opposed to "it's *not* a valid Tempest test case", right?

Heh, yes I meant to say, "it is a valid test case".

>  
> > What I was referring to as the counter argument, and where the
> > difference of opinion was, is that the test will be making REST API
> > calls to both trigger a nominally internal mechanism (the
> > notification) from the services and then using the ceilometer api to
> > validate the notification worked. 
> 
> Yes, that's exactly the idea.
> 
> > But, it's arguably the real intent of these tests is to validate
> > that internal mechanism, which is basically a whitebox test. The
> > argument was that by testing it in tempes we're testing
> > notifications poorly because of it's black box limitation
> > notifications will just be tested indirectly. Which I feel is a
> > valid point, but not a sufficient reason to exclude the notification
> > tests from tempest.
> 
> Agreed.
> 
> > I think the best way to move forward is to have functional whitebox
> > tests for the notifications as part of the individual projects
> > generating them, and that way we can direct validation of the
> > notification. But, I also feel there should be tempest tests on top
> > of that that verify the ceilometer side of consuming the
> > notification and the api exposing that information.
> 
> Excellent. So, indeed, more fullsome coverage of the notification
> logic with in-tree tests on the producer side would definitely
> to welcome, and could be seen as a "phase zero" of an overall to
> fix/impove the notification mechanism.
>   
> > > > But, they're is also a slight misunderstanding here. Having a
> > > > feature be externally discoverable isn't a hard requirement for a
> > > > config option in tempest, it's just *strongly* recommended. Mostly,
> > > > because if there isn't a way to discover it how are end users
> > > > expected to know what will work.
> > > 
> > > A-ha, I missed the subtle distinction there and thought that this
> > > discoverability was a *strict* requirement. So how bad a citizen would
> > > a project be considered to be if it chose not to meet that strong
> > > recommendation?
> > 
> > You'd be far from the only ones who are doing that, for an existing example
> > look at anything on the nova driver feature matrix. Most of those aren't
> > discoverable from the API. So I think it would be ok to do that, but when we
> > have efforts like:
> > 
> > https://review.openstack.org/#/c/94473/
> > 
> > it'll make that more difficult. Which is why I think having discoverability
> > through the API is important. (it's the same public cloud question)
> 
> So for now, would it suffice for the master versus stable/icehouse
> config to be checked-in in static form pending the completion of that
> BP on tempest-conf-autogen?

Yeah, I think that'll be fine. The auto-generation stuff is far from having
complete coverage of all the tempest config options. It's more of a best effort
approach.

> 
> Then the assumption is that this static config is replaced by auto-
> generating the tempest config using some project-specific discovery
> mechanisms?

Not exactly, tempest will still always require a static config. The
auto-generating mechanism won't ever be used for gating, it just to help
enable configuring and running tempest against an existing deployment, which
is apparently a popular use case.

> 
> > > > For this specific case I think it's definitely fair to have an
> > > > option for which notifications services are expected to be
> > > > generated. That's something that is definitely a configurable option
> > > > when setting up a deployment, and is something that feels like a
> > > > valid tempest config option, so we know which tests will work. We
> > > > already have similar feature flags for config time options in the
> > > > services, and having options like that would also get you out of
> > > > that backport mess you have right now.
> > > 
> > > So would this test configuration option would have a semantic like:
> > > 
> > >  "a wildcarded list of notification event types that ceilometer consumes"
> > > 
> > > then tests could be skipped on the basis of the notifications that
> > > they depend on being unavailable, in the manner of say:
> > > 
> > >   @testtools.skipUnless(
> > >       matchesAll(CONF.telemetry_consumed_notifications.volume,
> > >                  ['snapshot.exists',
> > >                   'snapshot.create.*',
> > >                   'snapshot.delete.*',
> > >                   'snapshot.resize.*',]
> > >                 )
> > >   )
> > >   @test.services('volume')
> > >   def test_check_volume_notification(self):
> > i>       ...
> > > 
> > > Is something of that ilk what you envisaged above?
> > 
> > Yeah that was my idea more or less, but I'd probably move the logic into a
> > separate
> > decorator to make a bit cleaner. Like:
> > 
> > @test.consumed_notifications('volumes', 'snapshot.exists',
> > 'snapshot.create.*',
> >                              'snapshot.delete.*', 'snapshot.resize.*')
> > 
> > and you can just double them up if the test requires notifications from other
> > services.
> 
> OK, so the remaining thing I wanted to confirm is that it's acceptable
> for the skip/no-skip logic of that decorator to be driven by static
> (as opposed to discoverable) config?

Yes, it'll always be from a static config in the tempest config file. For the
purposes of skip decisions it's always a static option.

The discoverability being used in that spec I mentioned before would be a
separate tool to aid in generating that config file if one wasn't available
beforehand.

> 
> > > > However, it does raise the question of being an end user how am I
> > > > expected to know which notifications get counted? Which is why having
> > > > the feature discoverability is generally a really good idea.
> > > 
> > > So certain things we could potentially make discoverable through the
> > > ceilometer capabilities API. But there's a limit to how fine-grained
> > > we can make that. Also it was primarily intended to surface lack of
> > > feature-parity in the storage driver layer (e.g. one driver supports
> > > sdtdev, but another doesn't) as opposed to the notification-handling
> > > layer.
> > 
> > I think we should probably decouple the discoverability question
> > from the discussion about testing, because to a certain degree it's
> > a separate problem.  It might be worth splitting the API
> > discoverability discussion off as a separate thread so we don't
> > cloud either topic. (pun intended :) )
> 
> I'm confused as how we can completely split off the discoverability
> question, if its a *strong* recommendation (but not a strict requirement)
> of the testing framework.

I just meant it was probably worth discussing it separately on the Ceilometer
side, as an independent topic. Because, while it's related to testing it
really can't be settled in a discussion about testing.

-Matt Treinish

> 
> > I guess it really depends on how we expect people to be consuming
> > the API, and whether we're presenting that expectation to
> > people. I'll have to admit I'm not as familiar with ceilometer as I
> > should be, which is probably where some of my confusion is coming
> > from.
> 
> The intent of capabilities API was initially purely to give users
> insight into the storage-driver-specific functionality exposed in
> the primary API.
> 
> I guess we've been kicking around on the idea of re-purposing that
> API, so that it also allows surfacing of the list of meters being
> captured.
> 
> But that idea hasn't yet been firmed up to the extent that we've
> figured out how *users* might consume it.
> 
> > The naive real world example I have in my head is: someone wants to
> > build a tool that will generate pretty pictures from data queried
> > from the ceilometer API, and if they want to run this on 2+ distinct
> > OpenStack deployments how will they know only through the API which
> > notifications are getting counted, etc. If it's doable through the
> > API then the tool can adjust what gets generated based on an API
> > call, (either up front or with an expected error response, like
> > unsupported request. (of course a new response like unsupported
> > would probably mean a new API rev) Without the discoverability
> > manual intervention would be required each time it's used with a new
> > cloud. This probably isn't a valid use case and I'm missing
> > something critical here. In which case the discoverability
> > discussion is moot.
> 
> That would be a case where discoverability of meters would be useful
> (i.e. not just the meters currently in the metering store, also all
> the meter that this deployment of ceilometer is capable of gathering
> samples for). However that feature will take a bit more thought &
> discussion to firm up.
> 
> > > > > One approach mooted for allowing these kind of scenarios to be tested
> > > > > was to split off the pure-API aspects of Tempest so that it can be used
> > > > > for probing public-cloud-capabilities as well as upstream CI, and then
> > > > > build project-specific mini-Tempests to test integration with other
> > > > > projects.
> > > > > 
> > > > > Personally, I'm not a fan of that approach as it would require a lot
> > > > > of QA expertise in each project, lead to inefficient use of CI
> > > > > nodepool resources to run all the mini-Tempests, and probably lead to
> > > > > a divergent hotchpotch of per-project approaches.
> > > > 
> > > > I think the proposal here was for people interested in doing
> > > > whitebox testing, where there is a desire to test an internal
> > > > project mechanism. I could see the argument for testing
> > > > notifications this way, but that would have to be for every project
> > > > individually. There are already several projects that have
> > > > functional testing like this in tree and run them as a gating
> > > > job. There are definitely certain classes of testing where doing
> > > > this makes sense.
> > > 
> > > I'm not sure that this would be realistic to test individually (if by
> > > that you meant just with the ceilometer agents running alone) as it
> > > depends on a notification emitted from cinder.
> > 
> > I wasn't actually referring to use it for this specific case, just a
> > general thought. Although maybe it would be useful to add to add
> > notification functional tests to Cinder. I think Sean's response on
> > the OP outlined some interesting alternative testing strategies
> > which could would definitely fit this use case. I still need to
> > finish forming my opinions about that before I respond to it.
> 
> OK. (And so do I).
> 
> > So I wrote this in one pass, sorry if bits of it are incoherent. :)
> 
> No, it was all good, thanks!
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140710/cff1c143/attachment-0001.pgp>