Open Stack

Hi Sean,


I would consider  "collaboration" between different programs and their
"missions" as a 2 different topics.

To clear up the situation I made diagram that shows how new "Operations"
program is aligned with OpenStack Integrated release and OpenStack QA.

[image: Inline image 1]
This means that I agree that part of Rally belongs to QA, but from other
side it is going to present OpenStack API, so it's as well part of
integrated release.
Rally is quite monolithic and can't be split, that is why I think Rally
should be a separated program (i.e. Rally scope is just different from QA
scope).
As well, It's not clear for me, why collaboration is possible only in case
of one program? In my opinion collaboration & programs are irrelevant
things.


About collaboration between Rally & Tempest teams...
Major goal of integration Tempest in Rally is to make it simpler to use
tempest on production clouds via OpenStack API.
This work requires a lot of collaboration between teams, as you already
mention we should work on improving measuring
durations and tempest.conf generation. I fully agree that this belongs to
Tempest. By the way, Rally team is already helping with this part.

In my opinion, end result should be something like: Rally just calls
Tempest (or couple of scripts from tempest) and store results to its DB,
presenting to end user tempest functionality via OpenStack API.

To get this done, we should implement next features in tempest:
1) Auto  tempest.conf generation
2) Production ready cleanup  - tempest should be absolutely safe for run
against cloud
3) Improvements related to time measurement.
4) Integration of OSprofiler & Tempest.

So in any case I would prefer to continue collaboration..

Thoughts?


Best regards,
Boris Pavlovic




On Mon, Aug 4, 2014 at 4:24 PM, Sean Dague <sean at dague.net> wrote:

> On 07/31/2014 06:55 AM, Angus Salkeld wrote:
> > On Sun, 2014-07-27 at 07:57 -0700, Sean Dague wrote:
> >> On 07/26/2014 05:51 PM, Hayes, Graham wrote:
> >>> On Tue, 2014-07-22 at 12:18 -0400, Sean Dague wrote:
> >>>> On 07/22/2014 11:58 AM, David Kranz wrote:
> >>>>> On 07/22/2014 10:44 AM, Sean Dague wrote:
> >>>>>> Honestly, I'm really not sure I see this as a different program,
> but is
> >>>>>> really something that should be folded into the QA program. I feel
> like
> >>>>>> a top level effort like this is going to lead to a lot of
> duplication in
> >>>>>> the data analysis that's currently going on, as well as
> functionality
> >>>>>> for better load driver UX.
> >>>>>>
> >>>>>>  -Sean
> >>>>> +1
> >>>>> It will also lead to pointless discussions/arguments about which
> >>>>> activities are part of "QA" and which are part of
> >>>>> "Performance and Scalability Testing".
> >>>
> >>> I think that those discussions will still take place, it will just be
> on
> >>> a per repository basis, instead of a per program one.
> >>>
> >>> [snip]
> >>>
> >>>>
> >>>> Right, 100% agreed. Rally would remain with it's own repo + review
> team,
> >>>> just like grenade.
> >>>>
> >>>>    -Sean
> >>>>
> >>>
> >>> Is the concept of a separate review team not the point of a program?
> >>>
> >>> In the the thread from Designate's Incubation request Thierry said [1]:
> >>>
> >>>> "Programs" just let us bless goals and teams and let them organize
> >>>> code however they want, with contribution to any code repo under that
> >>>> umbrella being considered "official" and ATC-status-granting.
> >>>
> >>> I do think that this is something that needs to be clarified by the TC
> -
> >>> Rally could not get a PTL if they were part of the QA project, but
> every
> >>> time we get a program request, the same discussion happens.
> >>>
> >>> I think that mission statements can be edited to fit new programs as
> >>> they occur, and that it is more important to let teams that have been
> >>> working closely together to stay as a distinct group.
> >>
> >> My big concern here is that many of the things that these efforts have
> >> been doing are things we actually want much closer to the base. For
> >> instance, metrics on Tempest runs.
> >>
> >> When Rally was first created it had it's own load generator. It took a
> >> ton of effort to keep the team from duplicating that and instead just
> >> use some subset of Tempest. Then when measuring showed up, we actually
> >> said that is something that would be great in Tempest, so whoever ran
> >> it, be it for Testing, Monitoring, or Performance gathering, would have
> >> access to that data. But the Rally team went off in a corner and did it
> >> otherwise. That's caused the QA team to have to go and redo this work
> >> from scratch with subunit2sql, in a way that can be consumed by multiple
> >> efforts.
> >>
> >> So I'm generally -1 to this being a separate effort on the basis that so
> >> far the team has decided to stay in their own sandbox instead of
> >> participating actively where many of us thing the functions should be
> >> added. I also think this isn't like Designate, because this isn't
> >> intended to be part of the integrated release.
> >
> > From reading Boris's email it seems like rally will provide a horizon
> > panel and api to back it (for the operator to kick of performance runs
> > and view stats). So this does seem like something that would be a
> > part of the integrated release (if I am reading things correctly).
> >
> > Is the QA program happy to extend their scope to include that?
> > QA could become "Quality Assurance of upstream code and running
> > OpenStack installations". If not we need to find some other program
> > for rally.
>
> I think that's realistically already the scope of the QA program, we
> might just need to change the governance wording.
>
> Tempest has always been intended to be run on production clouds (public
> or private) to ensure proper function. Many operators are doing this
> today as part of normal health management. And we continue to evolve it
> to be something which works well in that environment.
>
> All the statistics collection / analysis parts in Rally today I think
> are basically things that should be part of any Tempest installation /
> run. It's cool that Rally did a bunch of work there, but having that
> code outside of Tempest is sort of problematic, especially as there are
> huge issues with the collection of that data because of missing timing
> information in subunit. So realistically to get accurate results there
> needs to be additional events added into Tempest tests to build this
> correctly. If you stare at the raw results here today they have such
> huge accuracy problems (due to unaccounted for time in setupClass, which
> is a known problem) to the point of being misleading, and possibly
> actually harmful.
>
> These are things that are fixable, but hard to do outside of the Tempest
> project itself. Exporting accurate timing / stats should be a feature
> close to the test load, not something that's done externally with
> guessing and fudge factors.
>
> So every time I look at the docs in Rally -
> https://github.com/stackforge/rally I see largely features that should
> be coming out of the test runners themselves.
>
>         -Sean
>
> --
> Sean Dague
> http://dague.net
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140804/aedd8faf/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Operation Program.png
Type: image/png
Size: 31891 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140804/aedd8faf/attachment.png>