Open Stack

On Mon, Jul 21, 2014 at 11:15 AM, Dan Smith <dms at danplanet.com> wrote:

> > In addition to seeking a spec-freeze-exception for 95025, I would also
> > like some clarification of the requirement to test this upgrade
> > path. Some nova-core folks have pointed out that they do not want to
> > accept the nova.virt.ironic driver until the upgrade path from
> > nova.virt.baremetal *has test coverage*, but what that means is not
> > clear to me. It's been suggested that we use grenade (I am pretty sure
> > Sean suggested this at the summit, and I wrote it into my spec proposal
> > soon thereafter). After looking into grenade, I don't think it is the
> > right tool to test with, and I'm concerned that no one pointed this out
> > sooner.
>
> Grenade is our release test tool, so I think that, barring details, it's
> reasonable to use $GRENADE when talking about this sort of thing.


Grenade uses tempest to validate the "old" and "new" stacks. Unless Sean
and Matthew are willing to change that, this is a detail we can't ignore.


> I
> didn't realize that nova-bm doesn't work in devstack until you pointed
> it out in IRC last week. Since we're pretty good about requiring
> devstack support for new things like drivers, I would have expected
> nova-bm to work there, but obviously times were a bit different when
> that driver was merged.


> > Philosophically, this isn't an upgrade of one service from version X to
> > Y. It's a replacement of one nova driver with a completely different
> > driver. As I understand it, that's not what grenade is for. But maybe
> > I'm wrong on this, or maybe it's flexible.
>
> I think it's "start devstack on release X, validate, do some work,
> re-start devstack on release Y, validate". I'm not sure that it's
> ill-suited for this, but IANAGE.




> > I also have a technical objection: even if devstack can start and
> > properly configure nova.virt.baremteal (which I doubt, because it isn't
> > tested at all), it is going to fail the tempest/api/compute test suite
> > horribly. The baremetal driver never passed tempest, and never had
> > devstack-gate support. This matters because grenade uses tempest to
> > validate a stack pre- and post-upgrade. Therefore, since we know that
> > the old code is going to fail tempest, requiring grenade testing as a
> > precondition to accepting the ironic driver effectively means we need to
> > go develop the baremetal driver to a point it could pass tempest. I'm
> > going to assume no one is actually suggesting that, and instead believe
> > that none of us thought this through.
> >
> > (FWIW, Ironic doesn't pass the tempest/api/compute suite today, but
> > we're working hard on it.)
>
> Do the devstack exercises pass?


A few of them passed, once upon a time, but the whole suite? It never
passed on the baremetal driver for me. And it was never run or maintained
in the gate.


> We test things like cells today (/me
> hears sdague scream in the background), which don't pass tempest, using
> the exercises to make sure it's at least able to create an instance.


> > So, I'd like to ask for suggestions on what sort of upgrade testing is
> > reasonable here. I'll toss out two ideas:
> > - load some fake data into the nova_bm schema, run the upgrade scripts,
> > start ironic, issue some API queries, and see if the data's correct
> > - start devstack, load some real data into the nova_bm schema, run the
> > upgrade scripts, then try to deploy an instance with ironic
>
> These were my suggestions last week, so I'll own up to them now.
> Obviously I think that something using grenade that goes from a
> functional environment on release X to a functional environment on
> release Y is best. However, I of course don't think it makes sense to
> spend a ton of time getting nova-bm to pass tempest just so we can shoot
> it in the head.
>

I'm glad to hear that, since everything up to this point in your reply
seems to indicate that we should go back and add test coverage (whether
tempest or exercise.sh) for the very code we are trying to delete.

So my question remains. Even for option #2, while we can "load some real
data" into the nova_bm schema, since we can't do any functional testing on
it today, I don't think we should be expected to go fix things to make that
pass. This leaves us in the position of running tempest only once -- on the
result of the migration. Is that sufficient from your perspective?



>
> I'm not really sure what to do here. I think that we need an upgrade
> path, and that it needs to be tested. I don't think our users would
> appreciate us removing any other virt driver and replacing it with a new
> one, avoiding an upgrade path because "it's a different driver now". I
> also don't want to spend a bunch of time on nova-bm, which we have
> already neglected in our other test requirements (which is maybe part of
> the problem here).
>

Yea. Nova has kept the baremetal driver in tree with no testing whatsoever
far beyond its relevance, hoping for Ironic to come along and replace it --
except no one was maintaining it, and the only user (TripleO) is eagerly
moving to Ironic and doesn't care about a migration path.

Virt drivers have been removed from Nova for less. Like some of those,
baremetal was allowed in tree before the thrid-party CI requirements were
in place. I don't understand the resistance to removing this driver which
has never had any third-party CI, and does not have any team actively
maintaining it.


> Assuming grenade can be flexible about what it runs against the old and
> new environments to determine "workyness", then I think the second
> option above is probably a pretty good level of assurance, given where
> we are right now.
>

I would like to hear from the grenade engineers/maintainers on this one. My
read of the code suggests this "test for workyness" is not pluggable at all
-- but we can easily disable running tempest on the "old" cloud. Here's the
code:

225     # Validate the install
226     echo_summary "Running base smoke test"
227     if [[ "$BASE_RUN_SMOKE" == "True" ]]; then
228         cd $BASE_RELEASE_DIR/tempest
229         tox -esmoke -- --concurrency=$TEMPEST_CONCURRENCY
230     fi
231     stop $STOP base-smoke 110
....
336     # Validate the upgrade
337     if [[ "$TARGET_RUN_SMOKE" == "True" ]]; then
338         echo_summary "Running tempest scenario and smoke tests"
339         cd $TARGET_RELEASE_DIR/tempest
340         tox -esmoke -- --concurrency=$TEMPEST_CONCURRENCY
341         stop $STOP run-smoke 330
342     fi



Regards,
Devananda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140721/e3ed77b1/attachment.html>