Open Stack

On 3/4/2016 10:34 AM, Murray, Paul (HP Cloud) wrote:
> Hi All,
>
> Now that we have passed the feature freeze I thought it was worth giving
> a quick update
>
> on where we are with the live migration priority.
>
> The following is a list of work items that have been merged in this
> cycle ( for the live migration
>
> sub-team’s working page see
> https://etherpad.openstack.org/p/mitaka-live-migration ). There
>
> is also a number of merged and on-going bug fixes that are not listed here.
>
> _Progress reporting_
>
> Provide progress reporting information for on-going live migrations.
>
> ·https://blueprints.launchpad.net/nova/+spec/live-migration-progress-report
>
>   * https://review.openstack.org/#/q/topic:bp/live-migration-progress-report
>
> __
>
> _Force complete_
>
> Force an on-going live migration to complete by pausing the virtual
> machine for the
>
> duration of the migration.
>
> ·https://blueprints.launchpad.net/nova/+spec/pause-vm-during-live-migration
>
> ·https://review.openstack.org/#/q/topic:bp/pause-vm-during-live-migration
>
> __
>
> _Cancel_
>
> Cancel an on-going live migration.
>
> ·https://blueprints.launchpad.net/nova/+spec/abort-live-migration
>
>   * https://review.openstack.org/#/q/topic:bp/abort-live-migration
>
> __
>
> _Block live migration with attached volumes_
>
> Enable live migration of VMs with a combination of local and shared storage.
>
> ·https://blueprints.launchpad.net/nova/+spec/block-live-migrate-with-attached-volumes
>
>
> ·https://review.openstack.org/#/c/227278
>
> __
>
> _Split networking_
>
> Send live migration traffic over a specified network.
>
> ·https://blueprints.launchpad.net/nova/+spec/split-network-plane-for-live-migration
>
>
> ·https://review.openstack.org/#/q/topic:bp/split-network-plane-for-live-migration
>
>
> __
>
> _Make live migration api friendly_
>
> Remove –disk_over_commit flag and add –block_migration=auto (let nova
> determine
>
> how to migrate the disks)
>
> ·https://blueprints.launchpad.net/nova/+spec/making-live-migration-api-friendly
>
>
>   * https://review.openstack.org/#/q/topic:bp/making-live-migration-api-friendly
>
> __
>
> _Use request spec_
>
> Add scheduling to live migration and evacuate using original request
> spec (includes all
>
> original scheduling properties)
>
> ·https://blueprints.launchpad.net/nova/+spec/check-destination-on-migrations
>
> ·https://review.openstack.org/#/c/277800/
>
> ·https://review.openstack.org/#/c/273104/
>
> _Deprecate migration flags_
>
> Replace the combination of migration configuration flags with a single
> tunneled flag.
>
> ·(no blueprint)
>
> ·https://review.openstack.org/#/q/project:openstack/nova+branch:master+topic:deprecate-migration-flags-config
>
> __
>
> _Objectify live migrate data_
>
> Use the migrate object instead of a dictionary in migration code.
>
> ·https://blueprints.launchpad.net/nova/+spec/objectify-live-migrate-data
>
> ·https://review.openstack.org/#/q/project:openstack/nova+branch:master+topic:bp/objectify-live-migrate-data
>
>
> Next steps…
>
> Now we have passed the feature freeze we will be turning attention to
> the following
>
> three tasks:
>
> 1.Documenting the new features
>
> 2.Adding expanding the CI coverage
>
> 3.Fixing bugs
>
> The CI job gate-tempest-dsvm-multinode-live-migration was added to the
> experimental
>
> queue earlier In the cycle. We now need to add tests to this job to
> increase coverage. If
>
> you have any suggestions for CI improvements please contribute them on
> this page:
>
> https://etherpad.openstack.org/p/nova-live-migration-CI-ideas
>
> If you can contributed to live migration bug fixing you can look for
> things to do here:
>
> https://bugs.launchpad.net/nova/+bugs?field.tag=live-migration
>
> For priority reviews see the live migration section here:
>
> https://etherpad.openstack.org/p/mitaka-nova-priorities-tracking
>
> The live migration sub-team has an IRC meeting on Tuesdays at 14:00 UTC on
>
> #openstack-meeting-3:
>
> https://wiki.openstack.org/wiki/Meetings/NovaLiveMigration
>
> Best regards,
>
> Paul
>
> Paul Murray
>
> Technical Lead, HPE Cloud
>
> Hewlett Packard Enterprise
>
> +44 117 316 2527
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

The gate-tempest-dsvm-multinode-full job which runs live migration tests 
on nova patches has been non-voting for awhile now. There are at least 
two known tracked bugs so we can keep an eye on failure rates.

1. Volume based live migration aborted unexpectedly:

http://status.openstack.org/elastic-recheck/index.html#1524898

2. Libvirt live block migration migration stalls

http://status.openstack.org/elastic-recheck/index.html#1539271

Those are actually the top two failures in the check queue.

The job is bouncing between 25% and ~80% failure rates:

http://tinyurl.com/gvt5h56

At one point that job was relatively stable, it had to have been because 
it was voting.

So I'm not sure what's going on, but those should probably be the top 
priority bugs for live migration. The problem, unfortunately, could be 
something latent in libvirt 1.2.2 or qemu 2.0.0.

I'm not sure what the difference is in failure rates between the 
volume-backed and block migration tests, mtreinish might be able to tell 
from subunit2sql, but that could be interesting data.

-- 

Thanks,

Matt Riedemann