Hi,
Is there a documentation that explains how to setup bonding on SR-IOV neutron?
Thank you
Manuel NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
As I remember, Currently there is no mechanism to setup bonding with SR-IOV.
On Wed, Feb 27, 2019 at 4:09 PM Manuel Sopena Ballesteros < manuel.sb@garvan.org.au> wrote:
Hi,
Is there a documentation that explains how to setup bonding on SR-IOV neutron?
Thank you
Manuel NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
Hi,
On Wed, Feb 27, 2019 at 8:00 AM Manuel Sopena Ballesteros manuel.sb@garvan.org.au wrote:
Is there a documentation that explains how to setup bonding on SR-IOV neutron?
Not right now to my knowledge, but I remember seeing effort to design and introduce this feature. I think there may have been multiple rounds of design already, this is maybe the last one that's still ongoing:
Neutron side: https://bugs.launchpad.net/neutron/+bug/1809037
Nova side: https://blueprints.launchpad.net/nova/+spec/schedule-vm-nics-to-different-pf https://blueprints.launchpad.net/nova/+spec/sriov-bond
Hope that helps, Bence
On Wed, 2019-02-27 at 08:38 +0100, Bence Romsics wrote:
Hi,
On Wed, Feb 27, 2019 at 8:00 AM Manuel Sopena Ballesteros manuel.sb@garvan.org.au wrote:
Is there a documentation that explains how to setup bonding on SR-IOV neutron?
Not right now to my knowledge, but I remember seeing effort to design and introduce this feature. I think there may have been multiple rounds of design already, this is maybe the last one that's still ongoing:
Neutron side: https://bugs.launchpad.net/neutron/+bug/1809037
Nova side: https://blueprints.launchpad.net/nova/+spec/schedule-vm-nics-to-different-pf https://blueprints.launchpad.net/nova/+spec/sriov-bond
most of the previous attempts have not proceeded as they have tried to hide the bonding from nova's and neutron's data models vai configs or opaque strings.
for bonding to really be supported at the openstack level we will need to take a similar a approch to trunk ports. e.g. we create a logical bond port and a set of bond peer ports at the neutron api level then we attach the bond port to the vm.
currently im not aware of any proposal that really has tracktion.
you can manually create a bond in the guest but you cannot today guarentee that the vf will come from different pfs.
one of the reason i think we need to go the logical bond port direction is that it will allow us to constuct resouce requests using the fucntionality that is being added for bandwidth based schduleing. that will make expressing affinity and anti affinty simpler useing the request groups syntax. i personally have stopped trying to add bond support untill that bandwith based schduling effort is finished.
Hope that helps, Bence
Ok,
My nic is mellanox connect-4 lx so I was thinking this: Create bonding at the PF level use bond interface as br-ext Do ovs offload
That would mean I don't need to change my switch configuration and keep my LACP.
Does it sounds feasible?
Second question:
OVS offload needs Linux Kernel >= 4.13 can I use CentOS Linux release 7.5.1804 (Core) with kernel 3.10.0-862.el7.x86_64?
Thank you
Manuel
-----Original Message----- From: Sean Mooney [mailto:smooney@redhat.com] Sent: Wednesday, February 27, 2019 9:36 PM To: Bence Romsics; openstack@lists.openstack.org Subject: Re: sriov bonding
On Wed, 2019-02-27 at 08:38 +0100, Bence Romsics wrote:
Hi,
On Wed, Feb 27, 2019 at 8:00 AM Manuel Sopena Ballesteros manuel.sb@garvan.org.au wrote:
Is there a documentation that explains how to setup bonding on SR-IOV neutron?
Not right now to my knowledge, but I remember seeing effort to design and introduce this feature. I think there may have been multiple rounds of design already, this is maybe the last one that's still ongoing:
Neutron side: https://bugs.launchpad.net/neutron/+bug/1809037
Nova side: https://blueprints.launchpad.net/nova/+spec/schedule-vm-nics-to-different-pf https://blueprints.launchpad.net/nova/+spec/sriov-bond
most of the previous attempts have not proceeded as they have tried to hide the bonding from nova's and neutron's data models vai configs or opaque strings.
for bonding to really be supported at the openstack level we will need to take a similar a approch to trunk ports. e.g. we create a logical bond port and a set of bond peer ports at the neutron api level then we attach the bond port to the vm.
currently im not aware of any proposal that really has tracktion.
you can manually create a bond in the guest but you cannot today guarentee that the vf will come from different pfs.
one of the reason i think we need to go the logical bond port direction is that it will allow us to constuct resouce requests using the fucntionality that is being added for bandwidth based schduleing. that will make expressing affinity and anti affinty simpler useing the request groups syntax. i personally have stopped trying to add bond support untill that bandwith based schduling effort is finished.
Hope that helps, Bence
NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
On Fri, 2019-03-01 at 01:59 +0000, Manuel Sopena Ballesteros wrote:
Ok,
My nic is mellanox connect-4 lx so I was thinking this: Create bonding at the PF level use bond interface as br-ext Do ovs offload
That would mean I don't need to change my switch configuration and keep my LACP.
Does it sounds feasible?
i think mellanox would be able to comment better but i dont think that will work for hard ware offloaded ovs the vm data path is carried by an sriov vf with a fall back to ovs via a representer netdev.
the sriov vf that is attached to the vm will be allocated from 1 of the PFs in the bond if that PF losses network conenctivty i dont think the bound will have any effect.
mellanox would have and to specificaly develop ther silocon and drirvers to support this and if you intent was to bound pf form different phyical nic cards that woudl require even more supprot.
some NICs pool vf between all PF on a singel card but even the ones that supprot that general wont allow a VF to float between two different phyical cards.
i have not look at the connectx-4 cards to say what mellanox supprots in this regard but i would be rather surprised if bonding 2 pfs and adding them to an ovs bridge would be suffient to achive your goal
Second question:
OVS offload needs Linux Kernel >= 4.13 can I use CentOS Linux release 7.5.1804 (Core) with kernel 3.10.0- 862.el7.x86_64?
Thank you
Manuel
-----Original Message----- From: Sean Mooney [mailto:smooney@redhat.com] Sent: Wednesday, February 27, 2019 9:36 PM To: Bence Romsics; openstack@lists.openstack.org Subject: Re: sriov bonding
On Wed, 2019-02-27 at 08:38 +0100, Bence Romsics wrote:
Hi,
On Wed, Feb 27, 2019 at 8:00 AM Manuel Sopena Ballesteros manuel.sb@garvan.org.au wrote:
Is there a documentation that explains how to setup bonding on SR-IOV neutron?
Not right now to my knowledge, but I remember seeing effort to design and introduce this feature. I think there may have been multiple rounds of design already, this is maybe the last one that's still ongoing:
Neutron side: https://bugs.launchpad.net/neutron/+bug/1809037
Nova side: https://blueprints.launchpad.net/nova/+spec/schedule-vm-nics-to-different-pf https://blueprints.launchpad.net/nova/+spec/sriov-bond
most of the previous attempts have not proceeded as they have tried to hide the bonding from nova's and neutron's data models vai configs or opaque strings.
for bonding to really be supported at the openstack level we will need to take a similar a approch to trunk ports. e.g. we create a logical bond port and a set of bond peer ports at the neutron api level then we attach the bond port to the vm.
currently im not aware of any proposal that really has tracktion.
you can manually create a bond in the guest but you cannot today guarentee that the vf will come from different pfs.
one of the reason i think we need to go the logical bond port direction is that it will allow us to constuct resouce requests using the fucntionality that is being added for bandwidth based schduleing. that will make expressing affinity and anti affinty simpler useing the request groups syntax. i personally have stopped trying to add bond support untill that bandwith based schduling effort is finished.
Hope that helps, Bence
NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
On Fri, 1 Mar 2019 at 05:24, Sean Mooney smooney@redhat.com wrote:
On Fri, 2019-03-01 at 01:59 +0000, Manuel Sopena Ballesteros wrote:
Ok,
My nic is mellanox connect-4 lx so I was thinking this: Create bonding at the PF level use bond interface as br-ext Do ovs offload
That would mean I don't need to change my switch configuration and keep
my LACP.
Does it sounds feasible?
i think mellanox would be able to comment better but i dont think that will work
The whole thing becomes a problem with L2 interfaces that you get out of SRIOV, and so we need to define what it is that we're actually looking for here.
L2 (typically LACP) bonding is conventionally between sets of ports that are directly connected with a wire. But in the environment we're talking about here, we're looking at your TOR switches (with bonding config) -> a really dumb onboard switch on the NIC -> your VM. You want your VM to run a bonding protocol, because you want it to know when its ports are working and which ports to use. You want the TOR to be the thing that it's talking to, typically, because you're typically talking to something off-host and it's really the TOR uplinks' health you're interested in. But your nearest neighbours are ports on the internal switches of two entirely unrelated SRIOV NICs that can't talk LACP to you. And even if they could LACP wouldn't tell you anything other than the fact that that connection was broken - which it won't be, because the likely source of failure is one of the TORs.
You *could* talk LACP to the TORs and pretend that intervening switch doesn't exist, and that's where Manuel's logic comes in - the PF talks bonding protocols (and pretends it hasn't noticed that intervening switch) and the VFs don't talk bonding protocols (and yet, pretend that they're bonded - which is a weird config). But do you take the VFs down when the PF link drops? If you do, then perfectly well connected local VMs can't talk to each other; and if you don't, then the VM never knows the uplink is broken. Normally you'd have two links, one to both TORs, so one link or TOR going down wouldn't break things - but that dumb switch on the NIC has only one uplink, so no go.
And in all these cases you need, somehow, for the VM to know what it's been given - one half of a bonded pair of SRIOV interfaces is not a great deal of use to anyone, so you have to be careful to both give them what they need (bonded or unbonded) and somehow communicate that fact to them.
There are, indeed, other options for running link bonding. One is that you can run bonds from a vswitch to the TOR, either LACP on a L2 link or ECMP+BFD on an L3 link like VXLAN - the latter is better by far, incidentally, as links come down in milliseconds and the timing for LACP is much slower. That just works - networking-vpp will do it just fine - but if you're using SRIOV and you're using it for reasons of performance you may not want to go anywhere near a vswitch, no matter how fast.
There's also a model for using ECMP directly to the VMs, which doesn't require the SRIOV ports to be bonded at L2 and doesn't necessarily involve Neutron's help to work. That might be another avenue to explore, but you're into grown-up networking at that point.
PSB
-----Original Message----- From: Sean Mooney smooney@redhat.com Sent: Friday, March 1, 2019 3:21 PM To: Manuel Sopena Ballesteros manuel.sb@garvan.org.au; Bence Romsics bence.romsics@gmail.com; openstack@lists.openstack.org Subject: Re: sriov bonding
On Fri, 2019-03-01 at 01:59 +0000, Manuel Sopena Ballesteros wrote:
Ok,
My nic is mellanox connect-4 lx so I was thinking this: Create bonding at the PF level use bond interface as br-ext Do ovs offload
That would mean I don't need to change my switch configuration and keep my LACP.
Does it sounds feasible?
i think mellanox would be able to comment better but i dont think that will work for hard ware offloaded ovs the vm data path is carried by an sriov vf with a fall back to ovs via a representer netdev.
[ML] - Mellanox introduce VF lag feature which will do bond in the e-switch when you will do linux bond on the PF Level. This feature will be upsteam only in rhel7.7. But I think you can also get it if you install the latest Mellanox ofed http://www.mellanox.com/page/products_dyn?product_family=26
the sriov vf that is attached to the vm will be allocated from 1 of the PFs in the bond if that PF losses network conenctivty i dont think the bound will have any effect.
mellanox would have and to specificaly develop ther silocon and drirvers to support this and if you intent was to bound pf form different phyical nic cards that woudl require even more supprot.
some NICs pool vf between all PF on a singel card but even the ones that supprot that general wont allow a VF to float between two different phyical cards.
i have not look at the connectx-4 cards to say what mellanox supprots in this regard but i would be rather surprised if bonding 2 pfs and adding them to an ovs bridge would be suffient to achive your goal
Second question:
OVS offload needs Linux Kernel >= 4.13 can I use CentOS Linux release 7.5.1804 (Core) with kernel 3.10.0- 862.el7.x86_64?
Thank you
Manuel
-----Original Message----- From: Sean Mooney [mailto:smooney@redhat.com] Sent: Wednesday, February 27, 2019 9:36 PM To: Bence Romsics; openstack@lists.openstack.org Subject: Re: sriov bonding
On Wed, 2019-02-27 at 08:38 +0100, Bence Romsics wrote:
Hi,
On Wed, Feb 27, 2019 at 8:00 AM Manuel Sopena Ballesteros manuel.sb@garvan.org.au wrote:
Is there a documentation that explains how to setup bonding on SR-IOV neutron?
Not right now to my knowledge, but I remember seeing effort to design and introduce this feature. I think there may have been multiple rounds of design already, this is maybe the last one that's still ongoing:
Neutron side: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fb ugs.launchpad.net%2Fneutron%2F%2Bbug%2F1809037&data=02%7C01%7Cmo shele%40mellanox.com%7C8979e561cb0041ecd38608d69e49c7ad%7Ca652971c7d 2e4d9ba6a4d149256f461b%7C0%7C0%7C636870437055504756&sdata=a0AFu5 JEKwFkr2PchDzikl9DrclFmAEp95k2juYiPbY%3D&reserved=0
Nova side: https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fb lueprints.launchpad.net%2Fnova%2F%2Bspec%2Fschedule-vm-nics-to-diffe rent-pf&data=02%7C01%7Cmoshele%40mellanox.com%7C8979e561cb0041ec d38608d69e49c7ad%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636870 437055504756&sdata=fKUPr6pwTezvhsOYTv6%2BJqDRFYylg65VKteaXsGlFQk %3D&reserved=0 https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fb lueprints.launchpad.net%2Fnova%2F%2Bspec%2Fsriov-bond&data=02%7C 01%7Cmoshele%40mellanox.com%7C8979e561cb0041ecd38608d69e49c7ad%7Ca65 2971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636870437055504756&sdata =JU10BzjzhuaiARvyXZH8XJIu6N1eNr%2BPrzZvjjRzuHs%3D&reserved=0
most of the previous attempts have not proceeded as they have tried to hide the bonding from nova's and neutron's data models vai configs or opaque strings.
for bonding to really be supported at the openstack level we will need to take a similar a approch to trunk ports. e.g. we create a logical bond port and a set of bond peer ports at the neutron api level then we attach the bond port to the vm.
currently im not aware of any proposal that really has tracktion.
you can manually create a bond in the guest but you cannot today guarentee that the vf will come from different pfs.
one of the reason i think we need to go the logical bond port direction is that it will allow us to constuct resouce requests using the fucntionality that is being added for bandwidth based schduleing. that will make expressing affinity and anti affinty simpler useing the request groups syntax. i personally have stopped trying to add bond support untill that bandwith based schduling effort is finished.
Hope that helps, Bence
NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
participants (6)
-
Bence Romsics
-
Ian Wells
-
Manuel Sopena Ballesteros
-
Moshe Levi
-
Sa Pham
-
Sean Mooney