<font size=2 face="sans-serif">And this guy: </font><a href="https://bugs.launchpad.net/nova/+bug/1241628"><font size=3 color=blue><u>https://bugs.launchpad.net/nova/+bug/1241628</u></font></a><font size=3>
</font><a href="https://bugs.launchpad.net/nova/+bug/1241628"></a><font size=2 face="sans-serif"><br>
</font>
<br><font size=2 face="sans-serif"><br>
</font>
<br><font size=1 face="Arial">Thanks,</font>
<br>
<br><font size=3 color=#8f8f8f face="Arial"><b>MATT RIEDEMANN</b></font><font size=1 face="Arial"><br>
Advisory Software Engineer<br>
Cloud Solutions and OpenStack Development</font>
<table width=680 style="border-collapse:collapse;">
<tr height=8>
<td width=680 colspan=2 style="border-style:solid;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;">
<hr>
<tr valign=top height=8>
<td width=418 style="border-style:solid;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><font size=1 color=#4181c0 face="Arial"><b>Phone:</b></font><font size=1 color=#5f5f5f face="Arial">
1-507-253-7622</font><font size=1 color=#4181c0 face="Arial"> | <b>Mobile:</b></font><font size=1 color=#5f5f5f face="Arial">
1-507-990-1889</font><font size=1 color=#4181c0 face="Arial"><b><br>
E-mail:</b></font><font size=1 color=#5f5f5f face="Arial"> </font><a href=mailto:mriedem@us.ibm.com target=_blank><font size=1 color=#5f5f5f face="Arial"><u>mriedem@us.ibm.com</u></font></a>
<td width=261 style="border-style:solid;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;">
<div align=right><img src=cid:_1_09BB370009BB316C004F54C486257C08 width=83 height=30 alt=IBM><font size=1 color=#5f5f5f face="Arial"><br>
<br>
3605 Hwy 52 N<br>
Rochester, MN 55901-1407<br>
United States</font></div></table>
<br>
<br>
<br>
<br>
<br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Matt Riedemann/Rochester/IBM@IBMUS</font>
<br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">OpenStack Development
Mailing List <openstack-dev@lists.openstack.org>, </font>
<br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">10/18/2013 09:25 AM</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [openstack-dev]
[nova][powervm] my notes from the meeting on powervm
CI</font>
<br>
<hr noshade>
<br>
<br>
<br><font size=2 face="sans-serif">I just opened this bug, it's going to
be one of the blockers for us to get PowerVM CI going in Icehouse:</font><font size=3>
<br>
</font><font size=3 color=blue><u><br>
</u></font><a href="https://bugs.launchpad.net/nova/+bug/1241619"><font size=3 color=blue><u>https://bugs.launchpad.net/nova/+bug/1241619</u></font></a><font size=3>
<br>
</font><font size=2 face="sans-serif"><br>
</font><font size=3><br>
</font><font size=1 face="Arial"><br>
Thanks,</font><font size=3> <br>
</font><font size=3 color=#8f8f8f face="Arial"><b><br>
MATT RIEDEMANN</b></font><font size=1 face="Arial"><br>
Advisory Software Engineer<br>
Cloud Solutions and OpenStack Development</font><font size=3> </font>
<table width=680 style="border-collapse:collapse;">
<tr height=8>
<td width=678 colspan=2 style="border-style:solid;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;">
<hr>
<tr valign=top height=8>
<td width=416 style="border-style:solid;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;"><font size=1 color=#4181c0 face="Arial"><b>Phone:</b></font><font size=1 color=#5f5f5f face="Arial">
1-507-253-7622</font><font size=1 color=#4181c0 face="Arial"> | <b>Mobile:</b></font><font size=1 color=#5f5f5f face="Arial">
1-507-990-1889</font><font size=1 color=#4181c0 face="Arial"><b><br>
E-mail:</b></font><font size=1 color=#5f5f5f face="Arial"> </font><a href=mailto:mriedem@us.ibm.com target=_blank><font size=1 color=#5f5f5f face="Arial"><u>mriedem@us.ibm.com</u></font></a><font size=3>
</font>
<td width=259 style="border-style:solid;border-color:#000000;border-width:0px 0px 0px 0px;padding:1px 1px;">
<div align=right><img src=cid:_1_0A8E06DC0A8E02DC004F54C486257C08 width=83 height=30 alt=IBM><font size=1 color=#5f5f5f face="Arial"><br>
<br>
3605 Hwy 52 N<br>
Rochester, MN 55901-1407<br>
United States</font></div></table>
<br><font size=3><br>
<br>
<br>
<br>
</font><font size=1 color=#5f5f5f face="sans-serif"><br>
From: </font><font size=1 face="sans-serif">Matt
Riedemann/Rochester/IBM@IBMUS</font><font size=3> </font><font size=1 color=#5f5f5f face="sans-serif"><br>
To: </font><font size=1 face="sans-serif">OpenStack
Development Mailing List <openstack-dev@lists.openstack.org>, </font><font size=1 color=#5f5f5f face="sans-serif"><br>
Date: </font><font size=1 face="sans-serif">10/11/2013
10:59 AM</font><font size=3> </font><font size=1 color=#5f5f5f face="sans-serif"><br>
Subject: </font><font size=1 face="sans-serif">Re:
[openstack-dev] [nova][powervm] my notes from the meeting on
powervm CI</font><font size=3> <br>
</font>
<hr noshade><font size=3><br>
<br>
<br>
<br>
<br>
</font><tt><font size=2><br>
<br>
Matthew Treinish <mtreinish@kortar.org> wrote on 10/10/2013 10:31:29
PM:<br>
<br>
> From: Matthew Treinish <mtreinish@kortar.org></font></tt><font size=3>
</font><tt><font size=2><br>
> To: OpenStack Development Mailing List <openstack-dev@lists.openstack.org>,
<br>
> Date: 10/10/2013 11:07 PM</font></tt><font size=3> </font><tt><font size=2><br>
> Subject: Re: [openstack-dev] [nova][powervm] my notes from the <br>
> meeting on powervm CI</font></tt><font size=3> </font><tt><font size=2><br>
> <br>
> On Thu, Oct 10, 2013 at 07:39:37PM -0700, Joe Gordon wrote:<br>
> > On Thu, Oct 10, 2013 at 7:28 PM, Matt Riedemann <mriedem@us.ibm.com>
wrote:<br>
> > > ><br>
> > > > > 4. What is the max amount of time for us to report
test results? Dan<br>
> > > > > didn't seem to think 48 hours would fly. :)<br>
> > > ><br>
> > > > Honestly, I think that 12 hours during peak times is
the upper limit of<br>
> > > > what could be considered useful. If it's longer than
that, many patches<br>
> > > > could go into the tree without a vote, which defeats
the point.<br>
> > ><br>
> > > Yeah, I was just joking about the 48 hour thing, 12 hours
seems excessive<br>
> > > but I guess that has happened when things are super backed
up with gate<br>
> > > issues and rechecks.<br>
> > ><br>
> > > Right now things take about 4 hours, with Tempest being
around 1.5 hours<br>
> > > of that. The rest of the time is setup and install, which
includes heat<br>
> > > and ceilometer. So I guess that raises another question,
if we're really<br>
> > > setting this up right now because of nova, do we need to
have heat and<br>
> > > ceilometer installed and configured in the initial delivery
of this if<br>
> > > we're not going to run tempest tests against them (we don't
right now)?<br>
> > ><br>
> > <br>
> > <br>
> > In general the faster the better, and if things get to slow enough
that we<br>
> > have to wait for powervm CI to report back, I<br>
> > think its reasonable to go ahead and approve things without hearing
back.<br>
> > In reality if you can report back in under 12 hours this
will rarely<br>
> > happen (I think).<br>
> > <br>
> > <br>
> > ><br>
> > > I think some aspect of the slow setup time is related to
DB2 and how<br>
> > > the migrations perform with some of that, but the overall
time is not<br>
> > > considerably different from when we were running this with
MySQL so<br>
> > > I'm reluctant to blame it all on DB2. I think some
of our topology<br>
> > > could have something to do with it too since the IVM hypervisor
is running<br>
> > > on a separate system and we are gated on how it's performing
at any<br>
> > > given time. I think that will be our biggest challenge
for the scale<br>
> > > issues with community CI.<br>
> > ><br>
> > > ><br>
> > > > > 5. What are the minimum tests that need to run
(excluding <br>
> APIs that the<br>
> > > > > powervm driver doesn't currently support)?<br>
> > > > > - smoke/gate/negative/whitebox/scenario/cli?
Right <br>
> now we have<br>
> > > > > 1152 tempest tests running, those are only within
api/scenario/cli and<br>
> > > > > we don't run everything.<br>
> <br>
> Well that's almost a full run right now, the full tempest jobs have
1290 tests<br>
> of which we skip 65 because of bugs or configuration. (don't run neutron
api<br>
> tests without neutron) That number is actually pretty high since you
are<br>
> running with neutron. Right now the neutron gating jobs only have
221 jobs and<br>
> skip 8 of those. Can you share the list of things you've got working
with<br>
> neutron so we can up the number of gating tests?</font></tt><font size=3>
</font><tt><font size=2><br>
<br>
Here is the nose.cfg we run with:</font></tt><font size=3> <br>
<br>
</font><tt><font size=2><br>
<br>
Some of the tests are excluded because of performance issues that still
need to</font></tt><font size=3> </font><tt><font size=2><br>
be worked out (like test_list_image_filters - it works but it takes over
20</font></tt><font size=3> </font><tt><font size=2><br>
minutes sometimes).</font></tt><font size=3> </font><tt><font size=2><br>
<br>
Some of the tests are excluded because of limitations with DB2, e.g. <br>
test_list_servers_filtered_by_name_wildcard</font></tt><font size=3> </font><tt><font size=2><br>
<br>
Some of them are probably old excludes on bugs that are now fixed. We have
to</font></tt><font size=3> </font><tt><font size=2><br>
go back through what's excluded every once in awhile to figure out what's</font></tt><font size=3>
</font><tt><font size=2><br>
still broken and clean things up.</font></tt><font size=3> </font><tt><font size=2><br>
<br>
Here is the tempest.cfg we use on ppc64:</font></tt><font size=3> <br>
<br>
</font><tt><font size=2><br>
<br>
And here are the xunit results from our latest run:</font></tt><font size=3>
<br>
<br>
</font><tt><font size=2><br>
<br>
Note that we have known issues with some cinder and neutron failures</font></tt><font size=3>
</font><tt><font size=2><br>
in there.</font></tt><font size=3> </font><tt><font size=2><br>
<br>
> <br>
> > > ><br>
> > > > I think that "a full run of tempest" should
be required. That said, if<br>
> > > > there are things that the driver legitimately doesn't
support, it makes<br>
> > > > sense to exclude those from the tempest run, otherwise
it's not useful.<br>
> > ><br>
> > <br>
> > ++<br>
> > <br>
> > <br>
> > <br>
> > > ><br>
> > > > I think you should publish the tempest config (or config
script, or<br>
> > > > patch, or whatever) that you're using so that we can
see what it means<br>
> > > > in terms of the coverage you're providing.<br>
> > ><br>
> > > Just to clarify, do you mean publish what we are using now
or publish<br>
> > > once it's all working? I can certainly attach our
nose.cfg and<br>
> > > latest x-unit results xml file.<br>
> > ><br>
> > <br>
> > We should publish all logs, similar to what we do for upstream
(<br>
> > </font></tt><a href="http://logs.openstack.org/96/48196/8/gate/gate-tempest-devstack-"><tt><font size=2 color=blue><u>http://logs.openstack.org/96/48196/8/gate/gate-tempest-devstack-</u></font></tt></a><tt><font size=2><br>
> vm-full/70ae562/<br>
> > ).<br>
> <br>
> Yes, and part of that is the devstack logs which shows all the configuration<br>
> steps for getting an environment up and running. This is sometimes
very useful<br>
> for debugging. So this is probably information that you'll want to
<br>
> replicate in<br>
> whatever the logging output for the powervm jobs ends up being.</font></tt><font size=3>
</font><tt><font size=2><br>
<br>
Agreed. Our Jenkins job already pulls back our install log and all of the
service</font></tt><font size=3> </font><tt><font size=2><br>
logs which would be similar to what we have in community with devstack.</font></tt><font size=3>
</font><tt><font size=2><br>
<br>
> <br>
> > > ><br>
> > > > > 6. Network service? We're running with openvswitch
1.10 today so we<br>
> > > > > probably want to continue with that if possible.<br>
> > > ><br>
> > > > Hmm, so that means neutron? AFAIK, not much of tempest
runs with<br>
> > > > Nova/Neutron.<br>
> > > ><br>
> > > > I kinda think that since nova-network is our default
right now (for<br>
> > > > better or worse) that the run should include that mode,
especially if<br>
> > > > using neutron excludes a large portion of the tests.<br>
> > > ><br>
> > > > I think you said you're actually running a bunch of
tempest right now,<br>
> > > > which conflicts with my understanding of neutron workiness.
Can you<br>
> > > clarify?<br>
> > ><br>
> > > Correct, we're running with neutron using the ovs plugin.
We <br>
> basically have<br>
> > > the same issues that the neutron gate jobs have, which is
related to<br>
> > > concurrency<br>
> > > issues and tenant isolation (we're doing the same as devstack
with neutron<br>
> > > in that we don't run tempest with tenant isolation). We
are running most<br>
> > > of the nova and most of the neutron API tests though (we
don't have all<br>
> > > of the neutron-dependent scenario tests working though,
probably more due<br>
> > > to incompetence in setting up neutron than anything else).<br>
> <br>
> I also agree with Dan here in the short term you should probably at
<br>
> least have a<br>
> run with nova-network since it's the default. It'll also let you run
with<br>
> tenant isolation which should allow you to run these jobs in parallel
which<br>
> might help with your speed issues. (It depends on several things)<br>
> <br>
> There is only one neutron dependent scenario test that gets run in
the neutron<br>
> gating jobs right now. So I'm sure that you're probably at least <br>
> matching that.</font></tt><font size=3> </font><tt><font size=2><br>
<br>
I'd have to give nova-network a shot since we've been running with openvswitch</font></tt><font size=3>
</font><tt><font size=2><br>
since grizzly and then see how things look. I also haven't done a run with</font></tt><font size=3>
</font><tt><font size=2><br>
Fedora 19 yet to see what difference using testr will make.</font></tt><font size=3>
</font><tt><font size=2><br>
<br>
> <br>
> > ><br>
> > > ><br>
> > > > > 7. Cinder backend? We're running with the storwize
driver but we do we<br>
> > > > > do about the remote v7000?<br>
> > > ><br>
> > > > Is there any reason not to just run with a local LVM
setup like we do in<br>
> > > > the real gate? I mean, additional coverage for the
v7000 driver is<br>
> > > > great, but if it breaks and causes you to not have
any coverage at all,<br>
> > > > that seems, like, bad to me :)<br>
> > ><br>
> > > Yeah, I think we'd just run with a local LVM setup, that's
what we do for<br>
> > > x86_64 and s390x tempest runs. For whatever reason we thought
we'd do<br>
> > > storwize for our ppc64 runs, probably just to have a matrix
of coverage.<br>
> > ><br>
> > > ><br>
> > > > > Again, just getting some thoughts out there to
help us figure out our<br>
> > > > > goals for this, especially around 4 and 5.<br>
> > > ><br>
> > > > Yeah, thanks for starting this discussion!<br>
> > > ><br>
> <br>
> <br>
> -Matt Treinish<br>
> <br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> OpenStack-dev@lists.openstack.org<br>
> </font></tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"><tt><font size=2 color=blue><u>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</u></font></tt></a><tt><font size=2><br>
> [attachment "nose-ppc64-havana.cfg" deleted by Matt Riedemann/Rochester/IBM]
[attachment "tempest.conf" deleted by Matt Riedemann/Rochester/IBM]
[attachment "nosetests.xml" deleted by Matt Riedemann/Rochester/IBM]
_______________________________________________<br>
OpenStack-dev mailing list<br>
OpenStack-dev@lists.openstack.org</font></tt><font size=3 color=blue><u><br>
</u></font><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"><tt><font size=2 color=blue><u>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</u></font></tt></a><font size=3><br>
</font><tt><font size=2>_______________________________________________<br>
OpenStack-dev mailing list<br>
OpenStack-dev@lists.openstack.org<br>
</font></tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"><tt><font size=2>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</font></tt></a><tt><font size=2><br>
</font></tt>
<br>