<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 30, 2015 at 6:21 PM, Rochelle Grober <span dir="ltr"><<a href="mailto:rochelle.grober@huawei.com" target="_blank">rochelle.grober@huawei.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Top posting… I believe the main issue was a problem with snapshots that caused false negatives for most cinder drivers. But, that got fixed. Unfortunately,
we haven’t yet established a good process to notify third parties when skipped tests are fixed and should be “unskipped”. Maybe tagging the tests can help on this. But, I really do think this round was a bit of first run gotchas and rookie mistakes on all
sides. A good post mortem on how to better communicate changes and deadlines may go a long way to smooth these out in the next round.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">--Rocky<u></u><u></u></span></p><div><div class="h5">
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Tahoma,sans-serif">John Griffith on Monday, March 30, 2015 15:36 wrote:<br>
<br>
</span><u></u><u></u></p>
<div>
<p class="MsoNormal">On Mon, Mar 30, 2015 at 4:06 PM, Doug Wiegley <<a href="mailto:dougwig@parksidesoftware.com" target="_blank">dougwig@parksidesoftware.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">A few reasons, I’m sure there are others:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">- Broken tests that hardcode something about the ref implementation. The test needs to be fixed, of course, but in the meantime, a constantly failing CI is worthless (hello, lbaas scenario test.)<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:'Cambria Math',serif"></span><span style="font-family:'Courier New'">Certainly... but that's relatively easy to fix (bug/patch to Tempest). Although that's not actually the case in this particular context as
there are a handful of third party devices that run the full set of tests that the ref driver runs with no additional skips or modifications. <u></u><u></u></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:'Cambria Math',serif"></span><span style="font-family:'Courier New'"><u></u><u></u></span></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">- Test relies on some “optional” feature, like overlapping IP subnets that the backend doesn’t support. I’d argue it’s another case of broken tests if they require an optional feature, but it still needs skipping in the meantime.<u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:'Cambria Math',serif"></span><span style="font-family:'Courier New'">This may be something specific to Neutron perhaps? In Cinder LVM is pretty much the "lowest common denominator". I'm not aware of any volume
tests in Tempest that rely on optional features that don't pick this up automatically out of the config (like multi-backend for example).<u></u><u></u></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:'Cambria Math',serif"></span><span style="font-family:'Courier New'"><u></u><u></u></span></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">- Some new feature added to an interface, in the presence of shims/decomposed drivers/plugins (e.g. adding TLS termination support to lbaas.) Those implementations will lag the feature commit, by definition.<u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:'Cambria Math',serif"></span><span style="font-family:'Courier New'">Yeah, certainly I think this highlights some of the differences between Cinder and Neutron perhaps and the differences in complexity. <u></u><u></u></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:'Courier New'">Thanks for the feedback... I don't disagree per say, however Cinder is set up a bit different here in terms of expectations for base functionality requirements and compatibility but your points are
definitely well taken. </span><span style="font-family:'Cambria Math',serif"></span><span style="font-family:'Courier New'"><u></u><u></u></span></p>
</div>
</div>
<blockquote style="border-style:none none none solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">doug<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<div>
<div>
<p class="MsoNormal">On Mar 30, 2015, at 2:54 PM, John Griffith <<a href="mailto:john.griffith8@gmail.com" target="_blank">john.griffith8@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:'Courier New'">This may have already been raised/discussed, but I'm kinda confused so thought I'd ask on the ML here. The whole point of third party CI as I recall was to run the same tests that we run in the official
Gate against third party drivers. To me that would imply that a CI system/device that marks itself as "GOOD" doesn't do things like add skips locally that aren't in the tempest code already?<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:'Courier New'"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:'Courier New'">In other words, seems like cheating to say "My CI passes and all is good, except for the tests that don't work which I skip... but pay no attention to those please".<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:'Courier New'"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:'Courier New'">Did I miss something, isn't the whole point of Third Party CI to demonstrate that a third parties backend is tested and functions to the same degree that the reference implementations do? So the goal
(using Cinder for example) was to be able to say that any API call that works on the LVM reference driver will work on the drivers listed in driverlog; and that we know this because they run the same Tempest API tests?<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:'Courier New'"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:'Courier New'">Don't get me wrong, certainly not saying there's malice or things should be marked as no good... but if the practice is to skip what you can't do then maybe that should be documented in the driverlog
submission, as opposed to just stating "Yeah, we run CI successfully".<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:'Courier New'"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:'Courier New'">Thanks,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:'Courier New'">John<u></u><u></u></span></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal">__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org" target="_blank">
OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><u></u><u></u></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12pt"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div></div></div>
</div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div><div class="gmail_extra"><div class="gmail_default" style="font-family:monospace,monospace">Not top posting...</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">>> <span style="color:rgb(31,73,125);font-family:Calibri,sans-serif;font-size:14.6666669845581px">I believe the main issue was a problem with snapshots that caused false negatives for most cinder drivers. But, that got fixed</span></div><div class="gmail_default" style="font-family:monospace,monospace"><span style="color:rgb(31,73,125);font-family:Calibri,sans-serif;font-size:14.6666669845581px"><br></span></div><div class="gmail_default" style="font-family:monospace,monospace">Huh? What was the problem, where was the problem, who/what fixed it, was there a bug logged somewhere, what comprises *most* Cinder drivers? <br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">Not disputing what you say, but for me it's just raised more questions than anything else.</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><br></div></div>