[Openstack] live_migration only using 8 Mb speed

Satish Patel satish.txt at gmail.com
Fri Aug 24 04:07:13 UTC 2018


Forgive me, by mistake i grab wrong commit and that was the reason i
didn't see any changer after applying patch.

It works after applying correct version :) Thanks
On Thu, Aug 23, 2018 at 6:36 PM Satish Patel <satish.txt at gmail.com> wrote:
>
> I have updated this bug here something is wrong:
> https://bugs.launchpad.net/nova/+bug/1786346
>
> After nova upgrade i have compared these 3 files
> https://review.openstack.org/#/c/591761/  and i am not seeing any
> change here so look like this is not a complete patch.
>
> Are you sure they push this changes in nova repo?
> On Thu, Aug 23, 2018 at 4:36 PM Satish Patel <satish.txt at gmail.com> wrote:
> >
> > I have upgraded my nova and all nova component got upgrade but still
> > my live_migration running on 8Mbps speed.. what else is wrong here?
> >
> > I am using CentOS 7.5
> >
> > On Thu, Aug 23, 2018 at 3:26 PM Satish Patel <satish.txt at gmail.com> wrote:
> > >
> > > Look like it need all 3 line in user_variables.yml file.. after
> > > putting all 3 lines it works!!
> > >
> > > ## Nova service
> > > nova_git_repo: https://git.openstack.org/openstack/nova
> > > nova_git_install_branch: a9c9285a5a68ab89a6543d143c364d90a01cd51c #
> > > HEAD of "stable/queens" as of 06.08.2018
> > > nova_git_project_group: nova_all
> > > On Thu, Aug 23, 2018 at 3:12 PM Satish Patel <satish.txt at gmail.com> wrote:
> > > >
> > > > Matt,
> > > >
> > > > I've added "nova_git_install_branch:
> > > > a9c9285a5a68ab89a6543d143c364d90a01cd51c" in user_variables.yml and
> > > > run repo-build.yml playbook but it didn't change anything
> > > >
> > > > I am inside the repo container and still its showing old timestamp on
> > > > all nova file and i check all file they seems didn't change
> > > >
> > > > at this path in repo container /var/www/repo/openstackgit/nova/nova
> > > >
> > > > repo-build.yml should update that dir right?
> > > > On Thu, Aug 23, 2018 at 2:58 PM Satish Patel <satish.txt at gmail.com> wrote:
> > > > >
> > > > > Thanks Matthew,
> > > > >
> > > > > Going to do that and will update you in few min.
> > > > > On Thu, Aug 23, 2018 at 2:47 PM Matthew Thode <prometheanfire at gentoo.org> wrote:
> > > > > >
> > > > > > On 18-08-23 14:33:44, Satish Patel wrote:
> > > > > > > Matt,
> > > > > > >
> > > > > > > I am going to override following in user_variable.yml file in that
> > > > > > > case do i need to run ./bootstrap-ansible.sh script?
> > > > > > >
> > > > > > > ## Nova service
> > > > > > > nova_git_repo: https://git.openstack.org/openstack/nova
> > > > > > > nova_git_install_branch: a9c9285a5a68ab89a6543d143c364d90a01cd51c #
> > > > > > > HEAD of "stable/queens" as of 06.08.2018
> > > > > > > nova_git_project_group: nova_all
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Aug 23, 2018 at 7:18 AM, Satish Patel <satish.txt at gmail.com> wrote:
> > > > > > > > I'm testing this in lab, no load yet
> > > > > > > >
> > > > > > > > Sent from my iPhone
> > > > > > > >
> > > > > > > >> On Aug 23, 2018, at 2:30 AM, Matthew Thode <prometheanfire at gentoo.org> wrote:
> > > > > > > >>
> > > > > > > >>> On 18-08-22 23:04:57, Satish Patel wrote:
> > > > > > > >>> Mathew,
> > > > > > > >>>
> > > > > > > >>> I haven't applied any patch yet but i am noticing in cluster some host
> > > > > > > >>> migrating VM super fast and some host migrating very slow. Is this
> > > > > > > >>> known behavior?
> > > > > > > >>>
> > > > > > > >>> On Wed, Aug 22, 2018 at 11:28 AM, Matthew Thode
> > > > > > > >>> <prometheanfire at gentoo.org> wrote:
> > > > > > > >>>> On 18-08-22 10:58:48, Satish Patel wrote:
> > > > > > > >>>>> Matthew,
> > > > > > > >>>>>
> > > > > > > >>>>> I have two option looks like, correct me if i am wrong.
> > > > > > > >>>>>
> > > > > > > >>>>> 1. I have two option, upgrade minor release from 17.0.7-6-g9187bb1 to
> > > > > > > >>>>> 17.0.8-23-g0aff517  and upgrade full OSA
> > > > > > > >>>>>
> > > > > > > >>>>> 2. Just do override as you said "nova_git_install_branch:" in my
> > > > > > > >>>>> /etc/openstack_deploy/user_variables.yml  file, and run playbooks.
> > > > > > > >>>>>
> > > > > > > >>>>>
> > > > > > > >>>>> I think option [2] is safe to just touch specific component, also am i
> > > > > > > >>>>> correct about override in /etc/openstack_deploy/user_variables.yml
> > > > > > > >>>>> file?
> > > > > > > >>>>>
> > > > > > > >>>>> You mentioned "nova_git_install_branch:
> > > > > > > >>>>> dee99b1ed03de4b6ded94f3cf6d2ea7214bca93b" but i believe it should be
> > > > > > > >>>>> "a9c9285a5a68ab89a6543d143c364d90a01cd51c" am i correct?
> > > > > > > >>>>>
> > > > > > > >>>>>
> > > > > > > >>>>>
> > > > > > > >>>>> On Wed, Aug 22, 2018 at 10:46 AM, Matthew Thode
> > > > > > > >>>>> <prometheanfire at gentoo.org> wrote:
> > > > > > > >>>>>> On 18-08-22 10:33:11, Satish Patel wrote:
> > > > > > > >>>>>>> Thanks Matthew,
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Can i put that sha in my OSA at
> > > > > > > >>>>>>> playbooks/defaults/repo_packages/openstack_services.yml by hand and
> > > > > > > >>>>>>> run playbooks [repo/nova] ?
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> On Wed, Aug 22, 2018 at 10:24 AM, Matthew Thode
> > > > > > > >>>>>>> <prometheanfire at gentoo.org> wrote:
> > > > > > > >>>>>>>> On 18-08-22 08:35:09, Satish Patel wrote:
> > > > > > > >>>>>>>>> Currently in stable/queens i am seeing this sha
> > > > > > > >>>>>>>>> https://github.com/openstack/openstack-ansible/blob/stable/queens/ansible-role-requirements.yml#L112
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> On Wed, Aug 22, 2018 at 2:02 AM, Matthew Thode
> > > > > > > >>>>>>>>> <prometheanfire at gentoo.org> wrote:
> > > > > > > >>>>>>>>>> On 18-08-22 01:57:17, Satish Patel wrote:
> > > > > > > >>>>>>>>>>> What I need to upgrade, any specific component?
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I have deployed openstack-ansible
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Sent from my iPhone
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On Aug 22, 2018, at 1:06 AM, Matthew Thode <prometheanfire at gentoo.org> wrote:
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On 18-08-22 01:02:53, Satish Patel wrote:
> > > > > > > >>>>>>>>>>>>> Matthew,
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Thanks for reply, Look like i don't have this patch
> > > > > > > >>>>>>>>>>>>> https://review.openstack.org/#/c/591761/
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> So i have to patch following 3 file manually?
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> nova/tests/unit/virt/libvirt/test_driver.py213
> > > > > > > >>>>>>>>>>>>> nova/tests/unit/virt/test_virt_drivers.py2
> > > > > > > >>>>>>>>>>>>> nova/virt/libvirt/driver.py
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On Wed, Aug 22, 2018 at 12:42 AM, Matthew Thode
> > > > > > > >>>>>>>>>>>>> <prometheanfire at gentoo.org> wrote:
> > > > > > > >>>>>>>>>>>>>> On 18-08-22 00:27:08, Satish Patel wrote:
> > > > > > > >>>>>>>>>>>>>>> Folks,
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> I am running openstack queens and hypervisor is kvm, my live migration
> > > > > > > >>>>>>>>>>>>>>> working fine. but somehow it stuck to 8 Mb network speed and taking
> > > > > > > >>>>>>>>>>>>>>> long time to migrate 1G instance.  I have 10Gbps network and i have
> > > > > > > >>>>>>>>>>>>>>> tried to copy 10G file between two compute node and it did copy in 2
> > > > > > > >>>>>>>>>>>>>>> minute, so i am not seeing any network issue also.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> it seem live_migration has some bandwidth limit, I have tried
> > > > > > > >>>>>>>>>>>>>>> following option in nova.conf but it didn't work
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> live_migration_bandwidth = 500
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> My nova.conf look like following:
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> live_migration_uri =
> > > > > > > >>>>>>>>>>>>>>> "qemu+ssh://nova@%s/system?no_verify=1&keyfile=/var/lib/nova/.ssh/id_rsa"
> > > > > > > >>>>>>>>>>>>>>> live_migration_tunnelled = True
> > > > > > > >>>>>>>>>>>>>>> live_migration_bandwidth = 500
> > > > > > > >>>>>>>>>>>>>>> hw_disk_discard = unmap
> > > > > > > >>>>>>>>>>>>>>> disk_cachemodes = network=writeback
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Do you have a this patch (and a couple of patches up to it)?
> > > > > > > >>>>>>>>>>>>>> https://bugs.launchpad.net/nova/+bug/1786346
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> I don't know if that would cleanly apply (there are other patches that
> > > > > > > >>>>>>>>>>>> changed those functions within the last month and a half.  It'd be best
> > > > > > > >>>>>>>>>>>> to upgrade and not do just one patch (which would be an untested
> > > > > > > >>>>>>>>>>>> process).
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> The sha for nova has not been updated yet (next update is 24-48 hours
> > > > > > > >>>>>>>>>> away iirc), once that's done you can use the head of stable/queens from
> > > > > > > >>>>>>>>>> OSA and run a inter-series upgrade (but the minimal thing to do would be
> > > > > > > >>>>>>>>>> to run repo-build and os-nova plays).  I'm not sure when that sha bump
> > > > > > > >>>>>>>>>> will be tagged in a full release if you would rather wait on that.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> it's this sha that needs updating.
> > > > > > > >>>>>>>> https://github.com/openstack/openstack-ansible/blob/stable/queens/playbooks/defaults/repo_packages/openstack_services.yml#L173
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> I'm not sure how you are doing overrides, but set the following as an
> > > > > > > >>>>>> override, then rerun the repo-build playbook (to rebuild the nova venv)
> > > > > > > >>>>>> then rerun the nova playbook to install it.
> > > > > > > >>>>>>
> > > > > > > >>>>>> nova_git_install_branch: dee99b1ed03de4b6ded94f3cf6d2ea7214bca93b
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > > >>>> The sha I gave was head of the queens branch of openstack/nova.  It's
> > > > > > > >>>> also the commit in that branch that containst the fix.
> > > > > > > >>>>
> > > > > > > >>
> > > > > > > >> I don't think that is known behavior, different memory pressure causing
> > > > > > > >> the difference maybe?
> > > > > > > >>
> > > > > >
> > > > > > You just need the following var.
> > > > > >
> > > > > > nova_git_install_branch: a9c9285a5a68ab89a6543d143c364d90a01cd51c
> > > > > >
> > > > > > Once defined you'll need to `cd` into the playbooks directory within
> > > > > > openstack-ansible and run `openstack-ansible repo-build.yml` and
> > > > > > `openstack-ansible os-nova-install.yml`.  That should get you updated.
> > > > > >
> > > > > > --
> > > > > > Matthew Thode (prometheanfire)



More information about the Openstack mailing list