<div dir="ltr"><div>Hello,</div><div>Update for <span style="font-size:13px;color:rgb(80,0,80)">gate-tempest-dsvm-multinode-</span><span style="font-size:13px;color:rgb(80,0,80)">full job.</span></div><div><span style="font-size:13px;color:rgb(80,0,80)">Here is top 12 failing tests in weekly period:</span></div>tempest.api.compute.servers.test_disk_config.ServerDiskConfigTestJSON.test_resize_server_from_manual_to_auto: 14<br>tempest.api.compute.servers.test_disk_config.ServerDiskConfigTestJSON.test_resize_server_from_auto_to_manual: 14<br>tempest.scenario.test_server_advanced_ops.TestServerAdvancedOps.test_resize_server_confirm: 12<br>tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_revert: 12<br>tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_confirm: 12<br>tempest.api.compute.admin.test_live_migration.LiveBlockMigrationTestJSON.test_live_block_migration_paused: 12<br>tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON.test_delete_server_while_in_verify_resize_state: 12<br>tempest.api.compute.admin.test_migrations.MigrationsAdminTest.test_list_migrations_in_flavor_resize_situation: 12<br>tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_confirm_from_stopped: 12<br>tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern: 10<br>tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern: 10<br>tempest.api.compute.admin.test_live_migration.LiveBlockMigrationTestJSON.test_live_block_migration: 10<div><span style="font-size:13px;color:rgb(80,0,80)"><br></span></div><div><span style="font-size:13px;color:rgb(80,0,80)"><br></span>Full list of failing tests: <a href="http://xsnippet.org/360947/">http://xsnippet.org/360947/</a></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Aug 28, 2015 at 12:14 AM, Kraminsky, Arkadiy <span dir="ltr"><<a href="mailto:arkadiy.kraminsky@hp.com" target="_blank">arkadiy.kraminsky@hp.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<br>
I'm a new developer on the Openstack project and am in the process of creating live migration CI for HP's 3PAR and Lefthand backends. I noticed you guys are looking for someone to pick up Joe Gordon's change for volume backed live migration tests and we can sure use something like this. I can take a look into the change, and see what I can do. :)<br>
<br>
Thanks,<br>
<br>
Arkadiy Kraminsky<br>
________________________________<br>
From: Joe Gordon [<a href="mailto:joe.gordon0@gmail.com">joe.gordon0@gmail.com</a>]<br>
Sent: Wednesday, August 26, 2015 9:26 AM<br>
To: OpenStack Development Mailing List (not for usage questions)<br>
Subject: Re: [openstack-dev] [nova] CI for reliable live-migration<br>
<span class=""><br>
<br>
<br>
On Wed, Aug 26, 2015 at 8:18 AM, Matt Riedemann <<a href="mailto:mriedem@linux.vnet.ibm.com">mriedem@linux.vnet.ibm.com</a><mailto:<a href="mailto:mriedem@linux.vnet.ibm.com">mriedem@linux.vnet.ibm.com</a>>> wrote:<br>
<br>
<br>
On 8/26/2015 3:21 AM, Timofei Durakov wrote:<br>
Hello,<br>
<br>
Here is the situation: nova has live-migration feature but doesn't have<br>
ci job to cover it by functional tests, only<br>
gate-tempest-dsvm-multinode-full(non-voting, btw), which covers<br>
block-migration only.<br>
The problem here is, that live-migration could be different, depending<br>
on how instance was booted(volume-backed/ephemeral), how environment is<br>
configured(is shared instance directory(NFS, for example), or RBD used<br>
to store ephemeral disk), or for example user don't have that and is<br>
going to use --block-migrate flag. To claim that we have reliable<br>
live-migration in nova, we should check it at least on envs with rbd or<br>
nfs as more popular than envs without shared storages at all.<br>
Here is the steps for that:<br>
<br>
 1. make  gate-tempest-dsvm-multinode-full voting, as it looks OK for<br>
    block-migration testing purposes;<br>
<br>
When we are ready to make multinode voting we should remove the equivalent single node job.<br>
<br>
<br>
If it's been stable for awhile then I'd be OK with making it voting on nova changes, I agree it's important to have at least *something* that gates on multi-node testing for nova since we seem to break this a few times per release.<br>
<br>
Last I checked it isn't as stable is single node yet: <a href="http://jogo.github.io/gate/multinode" rel="noreferrer" target="_blank">http://jogo.github.io/gate/multinode</a> [0].  The data going into graphite is a bit noisy so this may be a red herring, but at the very least it needs to be investigated. When I was last looking into this there were at least two known bugs:<br>
<br>
<a href="https://bugs.launchpad.net/nova/+bug/1445569" rel="noreferrer" target="_blank">https://bugs.launchpad.net/nova/+bug/1445569</a><br>
<<a href="https://bugs.launchpad.net/nova/+bug/1445569" rel="noreferrer" target="_blank">https://bugs.launchpad.net/nova/+bug/1445569</a>><br>
<a href="https://bugs.launchpad.net/nova/+bug/1462305" rel="noreferrer" target="_blank">https://bugs.launchpad.net/nova/+bug/1462305</a><br>
<br>
<br>
</span>[0] <a href="http://graphite.openstack.org/graph/?from=-36hours&height=500&until=now&width=800&bgcolor=ffffff&fgcolor=000000&yMax=100&yMin=0&target=color(alias(movingAverage(asPercent(stats.zuul.pipeline.check.job.gate-tempest-dsvm-full.FAILURE,sum(stats.zuul.pipeline.check.job.gate-tempest-dsvm-full.{SUCCESS,FAILURE})),%275hours%27),%20%27gate-tempest-dsvm-full%27),%27orange%27)&target=color(alias(movingAverage(asPercent(stats.zuul.pipeline.check.job.gate-tempest-dsvm-multinode-full.FAILURE,sum(stats.zuul.pipeline.check.job.gate-tempest-dsvm-multinode-full.{SUCCESS,FAILURE})),%275hours%27),%20%27gate-tempest-dsvm-multinode-full%27),%27brown%27)&title=Check%20Failure%20Rates%20(36%20hours)&_t=0.48646087432280183" rel="noreferrer" target="_blank">http://graphite.openstack.org/graph/?from=-36hours&height=500&until=now&width=800&bgcolor=ffffff&fgcolor=000000&yMax=100&yMin=0&target=color(alias(movingAverage(asPercent(stats.zuul.pipeline.check.job.gate-tempest-dsvm-full.FAILURE,sum(stats.zuul.pipeline.check.job.gate-tempest-dsvm-full.{SUCCESS,FAILURE})),%275hours%27),%20%27gate-tempest-dsvm-full%27),%27orange%27)&target=color(alias(movingAverage(asPercent(stats.zuul.pipeline.check.job.gate-tempest-dsvm-multinode-full.FAILURE,sum(stats.zuul.pipeline.check.job.gate-tempest-dsvm-multinode-full.{SUCCESS,FAILURE})),%275hours%27),%20%27gate-tempest-dsvm-multinode-full%27),%27brown%27)&title=Check%20Failure%20Rates%20(36%20hours)&_t=0.48646087432280183</a><<a href="http://graphite.openstack.org/graph/?from=-36hours&height=500&until=now&width=800&bgcolor=ffffff&fgcolor=000000&yMax=100&yMin=0&target=color(alias(movingAverage(asPercent(stats.zuul.pipeline.check.job.gate-tempest-dsvm-full.FAILURE,sum(stats.zuul.pipeline.check.job.gate-tempest-dsvm-full.%7BSUCCESS,FAILURE%7D)),%275hours%27),%20%27gate-tempest-dsvm-full%27),%27orange%27)&target=color(alias(movingAverage(asPercent(stats.zuul.pipeline.check.job.gate-tempest-dsvm-multinode-full.FAILURE,sum(stats.zuul.pipeline.check.job.gate-tempest-dsvm-multinode-full.%7BSUCCESS,FAILURE%7D)),%275hours%27),%20%27gate-tempest-dsvm-multinode-full%27),%27brown%27)&title=Check%20Failure%20Rates%20(36%20hours)&_t=0.48646087432280183" rel="noreferrer" target="_blank">http://graphite.openstack.org/graph/?from=-36hours&height=500&until=now&width=800&bgcolor=ffffff&fgcolor=000000&yMax=100&yMin=0&target=color(alias(movingAverage(asPercent(stats.zuul.pipeline.check.job.gate-tempest-dsvm-full.FAILURE,sum(stats.zuul.pipeline.check.job.gate-tempest-dsvm-full.%7BSUCCESS,FAILURE%7D)),%275hours%27),%20%27gate-tempest-dsvm-full%27),%27orange%27)&target=color(alias(movingAverage(asPercent(stats.zuul.pipeline.check.job.gate-tempest-dsvm-multinode-full.FAILURE,sum(stats.zuul.pipeline.check.job.gate-tempest-dsvm-multinode-full.%7BSUCCESS,FAILURE%7D)),%275hours%27),%20%27gate-tempest-dsvm-multinode-full%27),%27brown%27)&title=Check%20Failure%20Rates%20(36%20hours)&_t=0.48646087432280183</a>><br>
<span class=""><br>
<br>
 2. contribute to tempest to cover volume-backed instances live-migration;<br>
<br>
jogo has had a patch up for this for awhile:<br>
<br>
<a href="https://review.openstack.org/#/c/165233/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/165233/</a><br>
<br>
Since it's not full time on openstack anymore I assume some help there in picking up the change would be appreciated.<br>
<br>
yes please<br>
<br>
<br>
 3. make another job with rbd for storing ephemerals, it also requires<br>
    changing tempest config;<br>
<br>
We already have a voting ceph job for nova - can we turn that into a multi-node testing job and run live migration with shared storage using that?<br>
<br>
 4. make job with nfs for ephemerals.<br>
<br>
Can't we use a multi-node ceph job (#3) for this?<br>
<br>
<br>
These steps should help us to improve current situation with<br>
live-migration.<br>
<br>
--<br>
Timofey.<br>
<br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
</span>Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
<span class=""><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
--<br>
<br>
Thanks,<br>
<br>
Matt Riedemann<br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
</span>Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<div class="HOEnZb"><div class="h5"><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>