Open Stack

On Wed, 2021-03-03 at 14:03 +0200, Heib, Mohammad (Nokia - IL/Kfar Sava) wrote:
> Hi All,
> 
> we recently moved to openstack Train and according to the version 
> release notes the SR-IOV direct port live-migration support was added to 
> this version.
> 
> so we tried to do some live-migration to vm with two SR-IOV direct ports 
> attached to bond interface on the vm (no vSwitch interface or a indirect 
> only direct ports) but seems that we have an issue with maintaining the 
> network
yes that is expected. live migration with direct mode sriov involvs hot unpluggin
the interface before the migration and hot plugging it after.

nic vendors do not actully support sriov migration so we developed a workaround
for that hardware limiation by using pcie hotplug.
> 
> connectivity during and after the live-migration, according to [1] i see 
> that in order to maintain network connectivity during the migration we 
> have to add one indirect or vswitch port to the bond and this interface 
> will carry the
> traffic during the migration and once the migration completed the direct 
> port will back to be the primary slave on the target VM bond.
yes this is correct
> 
> 
> my question is if we don't add the indirect port we will lose the 
> connectivity during the migration which makes sense to me,
> 
> but why the connectivity does not back after the migration completed 
> successfully, and why the bond master boot up with no slaves on the 
> target VM?
that sound like your networking setup in the guest is not correctly detecting the change.
e.g. you are missing a udev rule or your network manager or systemd-netowrkd configuriton
is only running on first boot but not when an interface is added/removed
> 
> according to some documents and blueprints that I found in google 
> (including [2],[3]) the guest os will receive a virtual hotplug add even 
> and the
> 
> bond master will enslave those devices and connectivity will back which 
> is not the case here.
> so I wondering maybe I need to add some scripts to handle those events 
> (if so where to add them) or some network flags to the ifcfg-* files or 
> i need to use a specific guest os?
you would need to but i do not have files for this unfortunetly.
> 
> 
> [1] : 
> https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/libvirt-neutron-sriov-livemigration.html 
> <https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/libvirt-neutron-sriov-livemigration.html>
> 
> [2] 
> :https://www.researchgate.net/publication/228722278_Live_migration_with_pass-through_device_for_Linux_VM 
> <https://www.researchgate.net/publication/228722278_Live_migration_with_pass-through_device_for_Linux_VM>
> 
> [3] 
> :https://openstack.nimeyo.com/72653/openstack-nova-neutron-live-migration-with-direct-passthru 
> <https://openstack.nimeyo.com/72653/openstack-nova-neutron-live-migration-with-direct-passthru>
> 
> 
> *bond master ifcfg file:*
> 
> DEVICE=bond1
> BONDING_OPTS=mode=active-backup
> HOTPLUG=yes
> TYPE=Bond
> BONDING_MASTER=yes
> BOOTPROTO=none
> NAME=bond1
> ONBOOT=yes
> 
> *slaves ifcfg files:*
> 
> TYPE=Ethernet
> DEVICE=eth0
> HOTPLUG=no
try setting this to yes
i suspect you need to mark all the slave interfaces as hotplugable since they are what will be hotplugged.
you might also want to set onboot but hotplug seams more relevent.
> ONBOOT=no
> MASTER=bond1
> SLAVE=yes
> BOOTPROTO=none
> NM_CONTROLLED=no
> 
> *guest OS:*
> 
> CentOS Linux release 7.7.1908 (Core)
> 
> *
> *
> 
> */Thanks in advance for any help :)/,*
> 
> 
>