<div dir="ltr">Hi Sean,<div><br></div><div><br></div><div>I would consider "collaboration" between different programs and their "missions" as a 2 different topics. </div><div><br></div><div>To clear up the situation I made diagram that shows how new "Operations" program is aligned with OpenStack Integrated release and OpenStack QA. </div>
<div><br></div><div><img src="cid:ii_147a16d952ee1a5b" alt="Inline image 1" width="562" height="490"><br></div><div>This means that I agree that part of Rally belongs to QA, but from other side it is going to present OpenStack API, so it's as well part of integrated release. </div>
<div>Rally is quite monolithic and can't be split, that is why I think Rally should be a separated program (i.e. Rally scope is just different from QA scope). </div><div>As well, It's not clear for me, why collaboration is possible only in case of one program? In my opinion collaboration & programs are irrelevant things. </div>
<div><br></div><div><br></div><div>About collaboration between Rally & Tempest teams...</div><div>Major goal of integration Tempest in Rally is to make it simpler to use tempest on production clouds via OpenStack API.</div>
<div>This work requires a lot of collaboration between teams, as you already mention we should work on improving measuring</div><div>durations and tempest.conf generation. I fully agree that this belongs to Tempest. By the way, Rally team is already helping with this part. </div>
<div><br></div><div>In my opinion, end result should be something like: Rally just calls Tempest (or couple of scripts from tempest) and store results to its DB, </div><div>presenting to end user tempest functionality via OpenStack API. <br>
</div><div><br></div><div>To get this done, we should implement next features in tempest:</div><div>1) Auto tempest.conf generation</div><div>2) Production ready cleanup - tempest should be absolutely safe for run against cloud </div>
<div>3) Improvements related to time measurement. </div><div>4) Integration of OSprofiler & Tempest. </div><div><br></div><div>So in any case I would prefer to continue collaboration.. </div><div><br></div><div>Thoughts? </div>
<div><br></div><div><br></div><div>Best regards,</div><div>Boris Pavlovic </div>
<div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Aug 4, 2014 at 4:24 PM, Sean Dague <span dir="ltr"><<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>On 07/31/2014 06:55 AM, Angus Salkeld wrote:<br>
> On Sun, 2014-07-27 at 07:57 -0700, Sean Dague wrote:<br>
>> On 07/26/2014 05:51 PM, Hayes, Graham wrote:<br>
>>> On Tue, 2014-07-22 at 12:18 -0400, Sean Dague wrote:<br>
>>>> On 07/22/2014 11:58 AM, David Kranz wrote:<br>
>>>>> On 07/22/2014 10:44 AM, Sean Dague wrote:<br>
>>>>>> Honestly, I'm really not sure I see this as a different program, but is<br>
>>>>>> really something that should be folded into the QA program. I feel like<br>
>>>>>> a top level effort like this is going to lead to a lot of duplication in<br>
>>>>>> the data analysis that's currently going on, as well as functionality<br>
>>>>>> for better load driver UX.<br>
>>>>>><br>
>>>>>> -Sean<br>
>>>>> +1<br>
>>>>> It will also lead to pointless discussions/arguments about which<br>
>>>>> activities are part of "QA" and which are part of<br>
>>>>> "Performance and Scalability Testing".<br>
>>><br>
>>> I think that those discussions will still take place, it will just be on<br>
>>> a per repository basis, instead of a per program one.<br>
>>><br>
>>> [snip]<br>
>>><br>
>>>><br>
>>>> Right, 100% agreed. Rally would remain with it's own repo + review team,<br>
>>>> just like grenade.<br>
>>>><br>
>>>> -Sean<br>
>>>><br>
>>><br>
>>> Is the concept of a separate review team not the point of a program?<br>
>>><br>
>>> In the the thread from Designate's Incubation request Thierry said [1]:<br>
>>><br>
>>>> "Programs" just let us bless goals and teams and let them organize<br>
>>>> code however they want, with contribution to any code repo under that<br>
>>>> umbrella being considered "official" and ATC-status-granting.<br>
>>><br>
>>> I do think that this is something that needs to be clarified by the TC -<br>
>>> Rally could not get a PTL if they were part of the QA project, but every<br>
>>> time we get a program request, the same discussion happens.<br>
>>><br>
>>> I think that mission statements can be edited to fit new programs as<br>
>>> they occur, and that it is more important to let teams that have been<br>
>>> working closely together to stay as a distinct group.<br>
>><br>
>> My big concern here is that many of the things that these efforts have<br>
>> been doing are things we actually want much closer to the base. For<br>
>> instance, metrics on Tempest runs.<br>
>><br>
>> When Rally was first created it had it's own load generator. It took a<br>
>> ton of effort to keep the team from duplicating that and instead just<br>
>> use some subset of Tempest. Then when measuring showed up, we actually<br>
>> said that is something that would be great in Tempest, so whoever ran<br>
>> it, be it for Testing, Monitoring, or Performance gathering, would have<br>
>> access to that data. But the Rally team went off in a corner and did it<br>
>> otherwise. That's caused the QA team to have to go and redo this work<br>
>> from scratch with subunit2sql, in a way that can be consumed by multiple<br>
>> efforts.<br>
>><br>
>> So I'm generally -1 to this being a separate effort on the basis that so<br>
>> far the team has decided to stay in their own sandbox instead of<br>
>> participating actively where many of us thing the functions should be<br>
>> added. I also think this isn't like Designate, because this isn't<br>
>> intended to be part of the integrated release.<br>
><br>
> From reading Boris's email it seems like rally will provide a horizon<br>
> panel and api to back it (for the operator to kick of performance runs<br>
> and view stats). So this does seem like something that would be a<br>
> part of the integrated release (if I am reading things correctly).<br>
><br>
> Is the QA program happy to extend their scope to include that?<br>
> QA could become "Quality Assurance of upstream code and running<br>
> OpenStack installations". If not we need to find some other program<br>
> for rally.<br>
<br>
</div></div>I think that's realistically already the scope of the QA program, we<br>
might just need to change the governance wording.<br>
<br>
Tempest has always been intended to be run on production clouds (public<br>
or private) to ensure proper function. Many operators are doing this<br>
today as part of normal health management. And we continue to evolve it<br>
to be something which works well in that environment.<br>
<br>
All the statistics collection / analysis parts in Rally today I think<br>
are basically things that should be part of any Tempest installation /<br>
run. It's cool that Rally did a bunch of work there, but having that<br>
code outside of Tempest is sort of problematic, especially as there are<br>
huge issues with the collection of that data because of missing timing<br>
information in subunit. So realistically to get accurate results there<br>
needs to be additional events added into Tempest tests to build this<br>
correctly. If you stare at the raw results here today they have such<br>
huge accuracy problems (due to unaccounted for time in setupClass, which<br>
is a known problem) to the point of being misleading, and possibly<br>
actually harmful.<br>
<br>
These are things that are fixable, but hard to do outside of the Tempest<br>
project itself. Exporting accurate timing / stats should be a feature<br>
close to the test load, not something that's done externally with<br>
guessing and fudge factors.<br>
<br>
So every time I look at the docs in Rally -<br>
<a href="https://github.com/stackforge/rally" target="_blank">https://github.com/stackforge/rally</a> I see largely features that should<br>
be coming out of the test runners themselves.<br>
<div><div><br>
-Sean<br>
<br>
--<br>
Sean Dague<br>
<a href="http://dague.net" target="_blank">http://dague.net</a><br>
<br>
</div></div><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div></div>