<div dir="ltr"><div class="gmail_default" style="font-family:courier new,monospace">I have to agree with Duncan here. I also don't know if I fully understand the limit in options. Stress test seems like it could/should be different (again overlap isn't a horrible thing) and I don't see it as siphoning off resources so not sure of the issue. We've become quite wrapped up in projects, programs and the like lately and it seems to hinder forward progress more than anything else.</div>
<div class="gmail_default" style="font-family:courier new,monospace"><br></div><div class="gmail_default" style="font-family:courier new,monospace">I'm also not convinced that Tempest is where all things belong, in fact I've been thinking more and more that a good bit of what Tempest does today should fall more on the responsibility of the projects themselves. For example functional testing of features etc, ideally I'd love to have more of that fall on the projects and their respective teams. That might even be something as simple to start as saying "if you contribute a new feature, you have to also provide a link to a contribution to the Tempest test-suite that checks it". Sort of like we do for unit tests, cross-project tracking is difficult of course, but it's a start. The other idea is maybe functional test harnesses live in their respective projects.<br>
</div><div class="gmail_default" style="font-family:courier new,monospace"><br></div><div class="gmail_default" style="font-family:courier new,monospace">Honestly I think who better to write tests for a project than the folks building and contributing to the project. At some point IMO the QA team isn't going to scale. I wonder if maybe we should be thinking about proposals for delineating responsibility and goals in terms of functional testing?</div>
<div class="gmail_default" style="font-family:courier new,monospace"><br></div><div class="gmail_default" style="font-family:courier new,monospace"><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Wed, Aug 6, 2014 at 12:25 PM, Duncan Thomas <span dir="ltr"><<a href="mailto:duncan.thomas@gmail.com" target="_blank">duncan.thomas@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm not following here - you complain about rally being monolithic,<br>
then suggest that parts of it should be baked into tempest - a tool<br>
that is already huge and difficult to get into. I'd rather see tools<br>
that do one thing well and some overlap than one tool to rule them<br>
all.<br>
<div class="HOEnZb"><div class="h5"><br>
On 6 August 2014 14:44, Sean Dague <<a href="mailto:sean@dague.net">sean@dague.net</a>> wrote:<br>
> On 08/06/2014 09:11 AM, Russell Bryant wrote:<br>
>> On 08/06/2014 06:30 AM, Thierry Carrez wrote:<br>
>>> Hi everyone,<br>
>>><br>
>>> At the TC meeting yesterday we discussed Rally program request and<br>
>>> incubation request. We quickly dismissed the incubation request, as<br>
>>> Rally appears to be able to live happily on top of OpenStack and would<br>
>>> benefit from having a release cycle decoupled from the OpenStack<br>
>>> "integrated release".<br>
>>><br>
>>> That leaves the question of the program. OpenStack programs are created<br>
>>> by the Technical Committee, to bless existing efforts and teams that are<br>
>>> considered *essential* to the production of the "OpenStack" integrated<br>
>>> release and the completion of the OpenStack project mission. There are 3<br>
>>> ways to look at Rally and official programs at this point:<br>
>>><br>
>>> 1. Rally as an essential QA tool<br>
>>> Performance testing (and especially performance regression testing) is<br>
>>> an essential QA function, and a feature that Rally provides. If the QA<br>
>>> team is happy to use Rally to fill that function, then Rally can<br>
>>> obviously be adopted by the (already-existing) QA program. That said,<br>
>>> that would put Rally under the authority of the QA PTL, and that raises<br>
>>> a few questions due to the current architecture of Rally, which is more<br>
>>> product-oriented. There needs to be further discussion between the QA<br>
>>> core team and the Rally team to see how that could work and if that<br>
>>> option would be acceptable for both sides.<br>
>>><br>
>>> 2. Rally as an essential operator tool<br>
>>> Regular benchmarking of OpenStack deployments is a best practice for<br>
>>> cloud operators, and a feature that Rally provides. With a bit of a<br>
>>> stretch, we could consider that benchmarking is essential to the<br>
>>> completion of the OpenStack project mission. That program could one day<br>
>>> evolve to include more such "operations best practices" tools. In<br>
>>> addition to the slight stretch already mentioned, one concern here is<br>
>>> that we still want to have performance testing in QA (which is clearly<br>
>>> essential to the production of "OpenStack"). Letting Rally primarily be<br>
>>> an operational tool might make that outcome more difficult.<br>
>>><br>
>>> 3. Let Rally be a product on top of OpenStack<br>
>>> The last option is to not have Rally in any program, and not consider it<br>
>>> *essential* to the production of the "OpenStack" integrated release or<br>
>>> the completion of the OpenStack project mission. Rally can happily exist<br>
>>> as an operator tool on top of OpenStack. It is built as a monolithic<br>
>>> product: that approach works very well for external complementary<br>
>>> solutions... Also be more integrated in OpenStack or part of the<br>
>>> OpenStack programs might come at a cost (slicing some functionality out<br>
>>> of rally to make it more a framework and less a product) that might not<br>
>>> be what its authors want.<br>
>>><br>
>>> Let's explore each option to see which ones are viable, and the pros and<br>
>>> cons of each.<br>
>><br>
>> My feeling right now is that Rally is trying to accomplish too much at<br>
>> the start (both #1 and #2). I would rather see the project focus on<br>
>> doing one of them as best as it can before increasing scope.<br>
>><br>
>> It's my opinion that #1 is the most important thing that Rally can be<br>
>> doing to help ensure the success of OpenStack, so I'd like to explore<br>
>> the "Rally as a QA tool" in more detail to start with.<br>
><br>
> I want to clarify some things. I don't think that rally in it's current<br>
> form belongs in any OpenStack project. It's a giant monolythic tool,<br>
> which is apparently a design point. That's the wrong design point for an<br>
> OpenStack project.<br>
><br>
> For instance:<br>
><br>
> <a href="https://github.com/stackforge/rally/tree/master/rally/benchmark/scenarios" target="_blank">https://github.com/stackforge/rally/tree/master/rally/benchmark/scenarios</a> should<br>
> all be tests in Tempest (and actually today mostly are via API tests).<br>
> There is an existing stress framework in Tempest which does the<br>
> repetitive looping that rally does on these already. This fact has been<br>
> brought up before.<br>
><br>
> <a href="https://github.com/stackforge/rally/tree/master/rally/verification/verifiers" target="_blank">https://github.com/stackforge/rally/tree/master/rally/verification/verifiers</a><br>
> - should be baked back into Tempest (at least on the results side,<br>
> though diving in there now it looks largely duplicative from existing<br>
> subunit to html code).<br>
><br>
> <a href="https://github.com/stackforge/rally/blob/master/rally/db/api.py" target="_blank">https://github.com/stackforge/rally/blob/master/rally/db/api.py</a> - is<br>
> largely (not entirely) what we'd like from a long term trending piece<br>
> that subunit2sql is working on. Again this was just all thrown into the<br>
> Rally db instead of thinking about how to split it off. Also, notable<br>
> here is there are some fundamental testr bugs (like worker<br>
> misallocation) which mean the data is massively dirty today. It would be<br>
> good for people to actually work on fixing those things.<br>
><br>
> The parts that should stay outside of Tempest are the setup tool<br>
> (separation of concerns is that Tempest is the load runner, not the<br>
> setup environment) and any of the SLA portions.<br>
><br>
> I think rally brings forward a good point about making Tempest easier to<br>
> run. But I think that shouldn't be done outside Tempest. Making the test<br>
> tool easier to use should be done in the tool itself. If that means<br>
> adding a tempest cmd or such, so be it. Note this was a topic for<br>
> discussion at last summit:<br>
> <a href="https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#QA" target="_blank">https://wiki.openstack.org/wiki/Summit/Juno/Etherpads#QA</a><br>
><br>
>> From the TC meeting, it seems that the QA group (via sdague, at least)<br>
>> has provided some feedback to Rally over the last several months. I<br>
>> would really like to see an analysis and write-up from the QA group on<br>
>> the current state of Rally and how it may (or may not) be able to serve<br>
>> the performance QA needs.<br>
><br>
> Something that we need to figure out is given where we are in the<br>
> release cycle do we want to ask the QA team to go off and do Rally deep<br>
> dive now to try to pull it apart into the parts that make sense for<br>
> other programs to take in. There are always trade offs.<br>
><br>
> Like the fact that right now the rally team is proposing gate jobs which<br>
> have some overlap to the existing largeops jobs. Did they start a<br>
> conversation about it? Nope. They just went off to do their thing<br>
> instead. <a href="https://review.openstack.org/#/c/112251/" target="_blank">https://review.openstack.org/#/c/112251/</a><br>
><br>
> So now we're going to run 2 jobs that do very similar things, with<br>
> different teams adjusting the test loads. Which I think is basically<br>
> madness.<br>
><br>
> -Sean<br>
><br>
> --<br>
> Sean Dague<br>
> <a href="http://dague.net" target="_blank">http://dague.net</a><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
<br>
</div></div><span class="HOEnZb"><font color="#888888">--<br>
Duncan Thomas<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>