<div dir="ltr"><div>On Mon, Jul 21, 2014 at 11:15 AM, Dan Smith <span dir="ltr"><<a href="mailto:dms@danplanet.com" target="_blank">dms@danplanet.com</a>></span> wrote:<br></div><div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div>> In addition to seeking a spec-freeze-exception for 95025, I would also<br>
> like some clarification of the requirement to test this upgrade<br>
> path. Some nova-core folks have pointed out that they do not want to<br>
> accept the nova.virt.ironic driver until the upgrade path from<br>
> nova.virt.baremetal *has test coverage*, but what that means is not<br>
> clear to me. It's been suggested that we use grenade (I am pretty sure<br>
> Sean suggested this at the summit, and I wrote it into my spec proposal<br>
> soon thereafter). After looking into grenade, I don't think it is the<br>
> right tool to test with, and I'm concerned that no one pointed this out<br>
> sooner.<br>
<br>
</div>Grenade is our release test tool, so I think that, barring details, it's<br>
reasonable to use $GRENADE when talking about this sort of thing. </blockquote><div><br></div><div>Grenade uses tempest to validate the "old" and "new" stacks. Unless Sean and Matthew are willing to change that, this is a detail we can't ignore.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
I<br>
didn't realize that nova-bm doesn't work in devstack until you pointed<br>
it out in IRC last week. Since we're pretty good about requiring<br>
devstack support for new things like drivers, I would have expected<br>
nova-bm to work there, but obviously times were a bit different when<br>
that driver was merged. </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><br>
> Philosophically, this isn't an upgrade of one service from version X to<br>
> Y. It's a replacement of one nova driver with a completely different<br>
> driver. As I understand it, that's not what grenade is for. But maybe<br>
> I'm wrong on this, or maybe it's flexible.<br>
<br>
</div>I think it's "start devstack on release X, validate, do some work,<br>
re-start devstack on release Y, validate". I'm not sure that it's<br>
ill-suited for this, but IANAGE. </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> </blockquote>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><br>
> I also have a technical objection: even if devstack can start and<br>
> properly configure nova.virt.baremteal (which I doubt, because it isn't<br>
> tested at all), it is going to fail the tempest/api/compute test suite<br>
> horribly. The baremetal driver never passed tempest, and never had<br>
> devstack-gate support. This matters because grenade uses tempest to<br>
> validate a stack pre- and post-upgrade. Therefore, since we know that<br>
> the old code is going to fail tempest, requiring grenade testing as a<br>
> precondition to accepting the ironic driver effectively means we need to<br>
> go develop the baremetal driver to a point it could pass tempest. I'm<br>
> going to assume no one is actually suggesting that, and instead believe<br>
> that none of us thought this through.<br>
><br>
> (FWIW, Ironic doesn't pass the tempest/api/compute suite today, but<br>
> we're working hard on it.)<br>
<br>
</div>Do the devstack exercises pass? </blockquote><div><br></div><div>A few of them passed, once upon a time, but the whole suite? It never passed on the baremetal driver for me. And it was never run or maintained in the gate.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
We test things like cells today (/me<br>
hears sdague scream in the background), which don't pass tempest, using<br>
the exercises to make sure it's at least able to create an instance. </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><br>
> So, I'd like to ask for suggestions on what sort of upgrade testing is<br>
> reasonable here. I'll toss out two ideas:<br>
> - load some fake data into the nova_bm schema, run the upgrade scripts,<br>
> start ironic, issue some API queries, and see if the data's correct<br>
> - start devstack, load some real data into the nova_bm schema, run the<br>
> upgrade scripts, then try to deploy an instance with ironic<br>
<br>
</div>These were my suggestions last week, so I'll own up to them now.<br>
Obviously I think that something using grenade that goes from a<br>
functional environment on release X to a functional environment on<br>
release Y is best. However, I of course don't think it makes sense to<br>
spend a ton of time getting nova-bm to pass tempest just so we can shoot<br>
it in the head.<br></blockquote><div><br></div><div>I'm glad to hear that, since everything up to this point in your reply seems to indicate that we should go back and add test coverage (whether tempest or exercise.sh) for the very code we are trying to delete.</div>
<div><br></div><div>So my question remains. Even for option #2, while we can "load some real data" into the nova_bm schema, since we can't do any functional testing on it today, I don't think we should be expected to go fix things to make that pass. This leaves us in the position of running tempest only once -- on the result of the migration. Is that sufficient from your perspective?</div>
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
I'm not really sure what to do here. I think that we need an upgrade<br>
path, and that it needs to be tested. I don't think our users would<br>
appreciate us removing any other virt driver and replacing it with a new<br>
one, avoiding an upgrade path because "it's a different driver now". I<br>
also don't want to spend a bunch of time on nova-bm, which we have<br>
already neglected in our other test requirements (which is maybe part of<br>
the problem here).<br></blockquote><div><br></div><div>Yea. Nova has kept the baremetal driver in tree with no testing whatsoever far beyond its relevance, hoping for Ironic to come along and replace it -- except no one was maintaining it, and the only user (TripleO) is eagerly moving to Ironic and doesn't care about a migration path.</div>
<div><br></div><div>Virt drivers have been removed from Nova for less. Like some of those, baremetal was allowed in tree before the thrid-party CI requirements were in place. I don't understand the resistance to removing this driver which has never had any third-party CI, and does not have any team actively maintaining it.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Assuming grenade can be flexible about what it runs against the old and<br>
new environments to determine "workyness", then I think the second<br>
option above is probably a pretty good level of assurance, given where<br>
we are right now.<br></blockquote><div><br></div><div>I would like to hear from the grenade engineers/maintainers on this one. My read of the code suggests this "test for workyness" is not pluggable at all -- but we can easily disable running tempest on the "old" cloud. Here's the code:</div>
<div><br></div><div><div>225 # Validate the install</div><div>226 echo_summary "Running base smoke test"</div><div>227 if [[ "$BASE_RUN_SMOKE" == "True" ]]; then</div><div>228 cd $BASE_RELEASE_DIR/tempest</div>
<div>229 tox -esmoke -- --concurrency=$TEMPEST_CONCURRENCY</div><div>230 fi</div><div>231 stop $STOP base-smoke 110</div></div><div>....</div><div><div>336 # Validate the upgrade</div><div>337 if [[ "$TARGET_RUN_SMOKE" == "True" ]]; then</div>
<div>338 echo_summary "Running tempest scenario and smoke tests"</div><div>339 cd $TARGET_RELEASE_DIR/tempest</div><div>340 tox -esmoke -- --concurrency=$TEMPEST_CONCURRENCY</div><div>341 stop $STOP run-smoke 330</div>
<div>342 fi</div></div><div><br></div><div><br></div><div><br></div><div>Regards,</div><div>Devananda</div></div></div></div>