<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 03/11/2014 07:35 AM, Sean Dague
wrote:<br>
</div>
<blockquote cite="mid:531F1F42.3040907@dague.net" type="cite">
<pre wrap="">On 03/11/2014 10:15 AM, Steven Dake wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On 03/11/2014 04:04 AM, Sean Dague wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On 03/04/2014 12:39 PM, Steven Hardy wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi all,
As some of you know, I've been working on the instance-users blueprint[1].
This blueprint implementation requires three new items to be added to the
heat.conf, or some resources (those which create keystone users) will not
work:
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/73978/">https://review.openstack.org/#/c/73978/</a>
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/76035/">https://review.openstack.org/#/c/76035/</a>
So on upgrade, the deployer must create a keystone domain and domain-admin
user, add the details to heat.conf, as already been done in devstack[2].
The changes requried for this to work have already landed in devstack, but
it was discussed to day and Clint suggested this may be unacceptable
upgrade behavior - I'm not sure so looking for guidance/comments.
My plan was/is:
- Make devstack work
- Talk to tripleo folks to assist in any transition (what prompted this
discussion)
- Document the upgrade requirements in the Icehouse release notes so the
wider community can upgrade from Havana.
- Try to give a heads-up to those maintaining downstream heat deployment
tools (e.g stackforge/puppet-heat) that some tweaks will be required for
Icehouse.
However some have suggested there may be an openstack-wide policy which
requires peoples old config files to continue working indefinitely on
upgrade between versions - is this right? If so where is it documented?
</pre>
</blockquote>
<pre wrap="">This is basically enforced in code in grenade, the language for this
actually got lost in the project requirements discussion in the TC, I'll
bring that back in the post graduation requirements discussion we're
having again.
The issue is - Heat still doesn't materially participate in grenade.
Heat is substantially far behind the other integrated projects in it's
integration with the upstream testing. Only monday did we finally start
gating on a real unit of work for Heat (the heat-slow jobs). If I was
letter grading projects right now on upstream testing I'd give Nova an
A, Neutron a C (still no full run, no working grenade), and Heat a D.
</pre>
</blockquote>
<pre wrap="">Sean,
I agree the Heat community hasn't done a bang-up job of getting
integrated with Tempest. We only have 50 functional tests implemented.
The community clearly needs to do more and provide better functional
coverage with Heat.
It is inappropriate to say "Only monday did we finally start gating"
because that was a huge move in the right direction. It took alot of
effort and should not be so easily dismissed. Clearly the community,
and especially the core developers, are making an effort. Keep in mind
we have to balance upstream development work, answering user questions,
staying on top of a 5 page review queue, keeping relationships and track
of the various integrated projects which are consuming Heat as a
building block, plus all of the demands of our day jobs.
</pre>
</blockquote>
<pre wrap="">
I agree it was a huge step in the right direction. It's not clear to me
why expressing that this was very recent was inappropriate.
Recent conversations have made me realize that a lot of the Heat core
team doesn't realize that Heat's participation in upstream gating is
below average, so I decided to be blunt about it. Because it was only
after being blunt about that with the Neutron team in Hong Kong did we
get any real motion on it (Neutron has seen huge gains this cycle).
All the integrated projects have the same challenges.
Upstream QA is really important. It not only protects heat from itself,
it protects it from changes in other projects.
</pre>
<blockquote type="cite">
<pre wrap="">We just don't have enough bandwidth on the core team to tackle writing
all of the tempest test cases ourselves. We have made an effort to
distribute this work to the overall heat community via wishlist bugs in
Heat which several new folks have picked up. I hope to see our coverage
improve over time, especially with more advanced scenario tests through
this effort.
</pre>
</blockquote>
<pre wrap="">
Bandwidth is a problem for everyone. It's a matter of priorities. The
fact that realistic upstream gating is considered wishlist priority in
from a Heat perspective is something I find troubling.
</pre>
</blockquote>
Sean,<br>
<br>
Unfortunately the root of the problem is there is no way to track in
one place the suggested test cases for projects. The Tempest
community doesn't want test cases in the tempest launchpad tracker.
At one point we were told to track the work using etherpads, which
is absolutely ridiculous. <br>
<br>
So we must resort to using wishlist priority. In all cases, a user
bug that has a negative impact on operation of Heat is higher
priority then implementing functional testing. I get that if we had
functional testing, maybe that bug wouldn't have been filed in the
first case. However, we are in a situation where we already have
the bugs, and they already need to be addressed.<br>
<br>
If the test cases were stored in tempest launchpad, they could be
properly prioritized from a "upstream-testing POV". The purpose of
the Heat launchpad tracker is to identify "Does Heat function
properly". Upstream gating doesn't really fit into that limited
scope, unless the problem is somehow negatively effecting the
correct operation of the check/gate. I believe this was the same
objection the tempest folks had with tracking tempest scenarios for
Heat in the tempest tracker.<br>
<br>
<blockquote cite="mid:531F1F42.3040907@dague.net" type="cite">
<pre wrap="">
Putting the investment into realistic scenarios in Tempest / gate is
going to be a huge timesaving for the Heat team. It will ensure Heat is
functioning at every commit (not just releases), it will protect Heat
from chasing breaking issues in Keystone or Nova, and it will mean that
we'll expose more subtle issues that only come with being able to do
data analysis on 10k runs.
I get it's never fun to hear that a project is below average on a metric
that's important to the OpenStack community. But if we aren't honest and
open about these things they never change.
</pre>
</blockquote>
<br>
I understand the value of a proper check/gate for functional
testing. I also understand that a global check/gate for heat
tempest scenario testing would be a huge improvement. I am also
willing to accept criticism that the current state of things needs
attention. I agree that the current situation is not ideal.<br>
<br>
I took offense at the implication that we aren't doing anything
about our lack of tempest integration. We clearly are.<br>
<br>
Steps heat-core have taken:<br>
1. Identify the entire feature matrix of Heat across the project<br>
2. Break down the feature matrix of Heat into unique scenarios that
need testing and file workable bugs for people to take on<br>
3. Get a heat-cfntools enabled image in the gate, so it can be run
with tempest<br>
4. Add a gating job for Heat for the current coverage in tempest<br>
<br>
This work didn't do itself. If you wan't to grade us on our work,
grading on our commitment to improving the situation would be more
productive.<br>
<br>
Regards,<br>
-steve<br>
<br>
<blockquote cite="mid:531F1F42.3040907@dague.net" type="cite">
<pre wrap="">
-Sean
--
Sean Dague
Samsung Research America
<a class="moz-txt-link-abbreviated" href="mailto:sean@dague.net">sean@dague.net</a> / <a class="moz-txt-link-abbreviated" href="mailto:sean.dague@samsung.com">sean.dague@samsung.com</a>
<a class="moz-txt-link-freetext" href="http://dague.net">http://dague.net</a>
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>