Open Stack

On 03/11/2014 07:35 AM, Sean Dague wrote:
> On 03/11/2014 10:15 AM, Steven Dake wrote:
>> On 03/11/2014 04:04 AM, Sean Dague wrote:
>>> On 03/04/2014 12:39 PM, Steven Hardy wrote:
>>>> Hi all,
>>>>
>>>> As some of you know, I've been working on the instance-users blueprint[1].
>>>>
>>>> This blueprint implementation requires three new items to be added to the
>>>> heat.conf, or some resources (those which create keystone users) will not
>>>> work:
>>>>
>>>> https://review.openstack.org/#/c/73978/
>>>> https://review.openstack.org/#/c/76035/
>>>>
>>>> So on upgrade, the deployer must create a keystone domain and domain-admin
>>>> user, add the details to heat.conf, as already been done in devstack[2].
>>>>
>>>> The changes requried for this to work have already landed in devstack, but
>>>> it was discussed to day and Clint suggested this may be unacceptable
>>>> upgrade behavior - I'm not sure so looking for guidance/comments.
>>>>
>>>> My plan was/is:
>>>> - Make devstack work
>>>> - Talk to tripleo folks to assist in any transition (what prompted this
>>>>    discussion)
>>>> - Document the upgrade requirements in the Icehouse release notes so the
>>>>    wider community can upgrade from Havana.
>>>> - Try to give a heads-up to those maintaining downstream heat deployment
>>>>    tools (e.g stackforge/puppet-heat) that some tweaks will be required for
>>>>    Icehouse.
>>>>
>>>> However some have suggested there may be an openstack-wide policy which
>>>> requires peoples old config files to continue working indefinitely on
>>>> upgrade between versions - is this right?  If so where is it documented?
>>> This is basically enforced in code in grenade, the language for this
>>> actually got lost in the project requirements discussion in the TC, I'll
>>> bring that back in the post graduation requirements discussion we're
>>> having again.
>>>
>>> The issue is - Heat still doesn't materially participate in grenade.
>>> Heat is substantially far behind the other integrated projects in it's
>>> integration with the upstream testing. Only monday did we finally start
>>> gating on a real unit of work for Heat (the heat-slow jobs). If I was
>>> letter grading projects right now on upstream testing I'd give Nova an
>>> A, Neutron a C (still no full run, no working grenade), and Heat a D.
>> Sean,
>>
>> I agree the Heat community hasn't done a bang-up job of getting
>> integrated with Tempest.  We only have 50 functional tests implemented.
>> The community clearly needs to do more and provide better functional
>> coverage with Heat.
>>
>> It is inappropriate to say "Only monday did we finally start gating"
>> because that was a huge move in the right direction.  It took alot of
>> effort and should not be so easily dismissed.  Clearly the community,
>> and especially the core developers, are making an effort.  Keep in mind
>> we have to balance upstream development work, answering user questions,
>> staying on top of a 5 page review queue, keeping relationships and track
>> of the various integrated projects which are consuming Heat as a
>> building block, plus all of the demands of our day jobs.
> I agree it was a huge step in the right direction. It's not clear to me
> why expressing that this was very recent was inappropriate.
>
> Recent conversations have made me realize that a lot of the Heat core
> team doesn't realize that Heat's participation in upstream gating is
> below average, so I decided to be blunt about it. Because it was only
> after being blunt about that with the Neutron team in Hong Kong did we
> get any real motion on it (Neutron has seen huge gains this cycle).
>
> All the integrated projects have the same challenges.
>
> Upstream QA is really important. It not only protects heat from itself,
> it protects it from changes in other projects.
>
>> We just don't have enough bandwidth on the core team to tackle writing
>> all of the tempest test cases ourselves.  We have made an effort to
>> distribute this work to the overall heat community via wishlist bugs in
>> Heat which several new folks have picked up.  I hope to see our coverage
>> improve over time, especially with more advanced scenario tests through
>> this effort.
> Bandwidth is a problem for everyone. It's a matter of priorities. The
> fact that realistic upstream gating is considered wishlist priority in
> from a Heat perspective is something I find troubling.
Sean,

Unfortunately the root of the problem is there is no way to track in one 
place the suggested test cases for projects.  The Tempest community 
doesn't want test cases in the tempest launchpad tracker. At one point 
we were told to track the work using etherpads, which is absolutely 
ridiculous.

So we must resort to using wishlist priority.  In all cases, a user bug 
that has a negative impact on operation of Heat is higher priority then 
implementing functional testing.  I get that if we had functional 
testing, maybe that bug wouldn't have been filed in the first case.  
However, we are in a situation where we already have the bugs, and they 
already need to be addressed.

If the test cases were stored in tempest launchpad, they could be 
properly prioritized from a "upstream-testing POV".  The purpose of the 
Heat launchpad tracker is to identify "Does Heat function properly".  
Upstream gating doesn't really fit into that limited scope, unless the 
problem is somehow negatively effecting the correct operation of the 
check/gate.  I believe this was the same objection the tempest folks had 
with tracking tempest scenarios for Heat in the tempest tracker.

> Putting the investment into realistic scenarios in Tempest / gate is
> going to be a huge timesaving for the Heat team. It will ensure Heat is
> functioning at every commit (not just releases), it will protect Heat
> from chasing breaking issues in Keystone or Nova, and it will mean that
> we'll expose more subtle issues that only come with being able to do
> data analysis on 10k runs.
>
> I get it's never fun to hear that a project is below average on a metric
> that's important to the OpenStack community. But if we aren't honest and
> open about these things they never change.

I understand the value of a proper check/gate for functional testing.  I 
also understand that a global check/gate for heat tempest scenario 
testing would be a huge improvement.  I am also willing to accept 
criticism that the current state of things needs attention.  I agree 
that the current situation is not ideal.

I took offense at the implication that we aren't doing anything about 
our lack of tempest integration.  We clearly are.

Steps heat-core have taken:
1. Identify the entire feature matrix of Heat across the project
2. Break down the feature matrix of Heat into unique scenarios that need 
testing and file workable bugs for people to take on
3. Get a heat-cfntools enabled image in the gate, so it can be run with 
tempest
4. Add a gating job for Heat for the current coverage in tempest

This work didn't do itself.  If you wan't to grade us on our work, 
grading on our commitment to improving the situation would be more 
productive.

Regards,
-steve

> 	-Sean
>
> --
> Sean Dague
> Samsung Research America
> sean at dague.net / sean.dague at samsung.com
> http://dague.net
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140311/2826b381/attachment.html>