[Octavia] Can not create LB on SRIOV network
I can not create Octavia LB on SRIOV network in Train. I went to Octavia story board, did a search but was unable to figure out (the story for SRIOV...). I left a comment under this story, I re-post my questions there, hoping someone knows the answer. Thank you so much Jing https://storyboard.openstack.org/#!/story/2006886 Add VM SRIOV Interface Config Guide (Openstack) Hi, In Openstack train release, creating Octavia LB on SRIOV network fails. I come here to search if there is already a plan to add this support, and see this story. This story gives the impression that the capability is already supported, it is a matter of adding user guide. So, my question is, in which Openstack release, creating LB on SRIOV network is supported? Thank you
Hi Jing, To my knowledge no one has done the work to support SRIOV network ports in Octavia load balancers. This is an open roadmap item[1]. It will require some development effort as we hot-plug the tenant traffic ports, which means we need to give nova some hints when booting the instances that the amphora instance will be using SRIOV. You might be able to accomplish it on train using the flavors capability. You would create a special nova flavor with the required "extra_specs"[2] to schedule the instance on the proper SRIOV host with the SRIOV libvirt settings. Then you can create an Octavia flavor[3] that uses this special nova flavor. You could then create a load balancer by passing in the neutron SRIOV port as the VIP port. This would not provide a solution for adding additional SRIOV ports to the load balancer for the member servers, but you can use the VIP port to access members. I have not tried this and would be interested to hear if it works for you. If you are interested in implementing SRIOV support for Octavia, please consider adding it to the PTG agenda[4] and joining us at the virtual PTG. Michael [1] https://wiki.openstack.org/wiki/Octavia/Roadmap [2] https://docs.openstack.org/nova/xena/configuration/extra-specs.html [3] https://docs.openstack.org/octavia/latest/admin/flavors.html [4] https://etherpad.opendev.org/p/yoga-ptg-octavia On Wed, Oct 6, 2021 at 10:24 AM Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang@nokia.com> wrote:
I can not create Octavia LB on SRIOV network in Train. I went to Octavia story board, did a search but was unable to figure out (the story for SRIOV…).
I left a comment under this story, I re-post my questions there, hoping someone knows the answer.
Thank you so much
Jing
https://storyboard.openstack.org/#!/story/2006886 Add VM SRIOV Interface Config Guide (Openstack)
Hi, In Openstack train release, creating Octavia LB on SRIOV network fails. I come here to search if there is already a plan to add this support, and see this story. This story gives the impression that the capability is already supported, it is a matter of adding user guide. So, my question is, in which Openstack release, creating LB on SRIOV network is supported? Thank you
Hi Michael, Thank you so much for the information. I tried the extra-flavor walk-around, I can not use it to create VM in Train release, I suspect this old extra-flavor is too old, but I did not dig further. However, both Train and latest nova spec still shows the above extra-flavor with the old whitelist format: https://docs.openstack.org/nova/train/admin/pci-passthrough.html https://docs.openstack.org/nova/latest/admin/pci-passthrough.html ========================= Here is the detail: Env: NIC is intel 82599, creating VM with SRIOV direct port works well. Nova.conf passthrough_whitelist={"devname":"ens1f0","physical_network":"physnet5"} passthrough_whitelist={"devname":"ens1f1","physical_network":"physnet6"} Sriov_agent.ini [sriov_nic] physical_device_mappings=physnet5:ens1f0,physnet6:ens1f1 (1) Added the alias in nova.conf for nova-compute and nova-api, and restart the two nova components: alias = { "vendor_id":"8086", "product_id":"10ed", "device_type":"type-VF", "name":"vf", "numa_policy": "required" } (2) Used the extra-spec in nova flavor openstack flavor set octavia-flavor --property "pci_passthrough:alias"="vf:1" (3) Failed to create VM with this flavor, sriov agent log does not show port event, for sure also failed to create LB, PortBindingFailed (4) Tried multiple formats to add whitelist for PF and VF in nova.conf for nova-compute, and retried, still failed passthrough_whitelist={"vendor_id":"8086","product_id":"10f8","devname":"ens1f0","physical_network":"physnet5"} #PF passthrough_whitelist={"vendor_id":"8086","product_id":"10ed","physical_network":"physnet5"} #VF The sriov agent log does not show port event for any of them. -----Original Message----- From: Michael Johnson <johnsomor@gmail.com> Sent: Wednesday, October 6, 2021 4:48 PM To: Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang@nokia.com> Cc: openstack-discuss@lists.openstack.org Subject: Re: [Octavia] Can not create LB on SRIOV network Hi Jing, To my knowledge no one has done the work to support SRIOV network ports in Octavia load balancers. This is an open roadmap item[1]. It will require some development effort as we hot-plug the tenant traffic ports, which means we need to give nova some hints when booting the instances that the amphora instance will be using SRIOV. You might be able to accomplish it on train using the flavors capability. You would create a special nova flavor with the required "extra_specs"[2] to schedule the instance on the proper SRIOV host with the SRIOV libvirt settings. Then you can create an Octavia flavor[3] that uses this special nova flavor. You could then create a load balancer by passing in the neutron SRIOV port as the VIP port. This would not provide a solution for adding additional SRIOV ports to the load balancer for the member servers, but you can use the VIP port to access members. I have not tried this and would be interested to hear if it works for you. If you are interested in implementing SRIOV support for Octavia, please consider adding it to the PTG agenda[4] and joining us at the virtual PTG. Michael [1] https://wiki.openstack.org/wiki/Octavia/Roadmap [2] https://docs.openstack.org/nova/xena/configuration/extra-specs.html [3] https://docs.openstack.org/octavia/latest/admin/flavors.html [4] https://etherpad.opendev.org/p/yoga-ptg-octavia On Wed, Oct 6, 2021 at 10:24 AM Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang@nokia.com> wrote:
I can not create Octavia LB on SRIOV network in Train. I went to Octavia story board, did a search but was unable to figure out (the story for SRIOV…).
I left a comment under this story, I re-post my questions there, hoping someone knows the answer.
Thank you so much
Jing
https://storyboard.openstack.org/#!/story/2006886 Add VM SRIOV Interface Config Guide (Openstack)
Hi, In Openstack train release, creating Octavia LB on SRIOV network fails. I come here to search if there is already a plan to add this support, and see this story. This story gives the impression that the capability is already supported, it is a matter of adding user guide. So, my question is, in which Openstack release, creating LB on SRIOV network is supported? Thank you
Hi Michael, I made a mistake when creating VM manually, I should use --nic option not --network option. After correcting that, I can create VM with the extra-flavor: $ openstack server create --flavor octavia-flavor --image Centos7 --nic port-id=test-port --security-group demo-secgroup --key-name demo-key test-vm $ nova list --all --fields name,status,host,networks | grep test-vm | 8548400b-725a-405a-aeeb-ed1d208915e2 | test-vm | ACTIVE | overcloud-sriovperformancecompute-201-1.localdomain | ext-net1=10.5.201.149 A 2nd VF interface is seen inside the VM: [centos@test-vm ~]$ ip a ... 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 0a:b2:d4:85:a2:e6 brd ff:ff:ff:ff:ff:ff This MAC is not seen by neutron though: $ openstack port list | grep 0a:b2:d4:85:a2:e6 [empty] ===================== However when I tried to create LB with the same VM flavor, it failed at the same place as before. Looking at worker.log, it seems the error is similar to use --network option to create the VM manually. But you are the expert. "Error plugging amphora (compute_id: 64be3ced-90b1-4fb2-a13c-dd5e73ba0526) into vip network 7a1fa805-6d21-4da6-9573-1586c2faef52" Here is the full list of command line: $ openstack flavor list | grep octavia-flavor | eb312b9a-d04d-4a88-9db2-7a88ce167cff | octavia-flavor | 4096 | 0 | 0 | 4 | True | openstack loadbalancer flavorprofile create --name ofp1 --provider amphora --flavor-data '{"compute_flavor": "eb312b9a-d04d-4a88-9db2-7a88ce167cff"}' openstack loadbalancer flavor create --name of1 --flavorprofile ofp1 --enable openstack loadbalancer create --name lb1 --flavor of1 --vip-port-id test-port --vip-subnet-id ext-subnet1 |__Flow 'octavia-create-loadbalancer-flow': PlugVIPException: Error plugging amphora (compute_id: 64be3ced-90b1-4fb2-a13c-dd5e73ba0526) into vip network 7a1fa805-6d21-4da6-9573-1586c2faef52. 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker Traceback (most recent call last): 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker result = task.execute(**arguments) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/octavia/controller/worker/v1/tasks/network_tasks.py", line 399, in execute 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker loadbalancer, loadbalancer.vip, amphora, subnet) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 391, in plug_aap_port 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker interface = self._plug_amphora_vip(amphora, subnet) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 123, in _plug_amphora_vip 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker raise base.PlugVIPException(message) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker PlugVIPException: Error plugging amphora (compute_id: 64be3ced-90b1-4fb2-a13c-dd5e73ba0526) into vip network 7a1fa805-6d21-4da6-9573-1586c2faef52. 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker -----Original Message----- From: Zhang, Jing C. (Nokia - CA/Ottawa) Sent: Thursday, October 7, 2021 6:18 PM To: Michael Johnson <johnsomor@gmail.com> Cc: openstack-discuss@lists.openstack.org Subject: RE: [Octavia] Can not create LB on SRIOV network Hi Michael, Thank you so much for the information. I tried the extra-flavor walk-around, I can not use it to create VM in Train release, I suspect this old extra-flavor is too old, but I did not dig further. However, both Train and latest nova spec still shows the above extra-flavor with the old whitelist format: https://docs.openstack.org/nova/train/admin/pci-passthrough.html https://docs.openstack.org/nova/latest/admin/pci-passthrough.html ========================= Here is the detail: Env: NIC is intel 82599, creating VM with SRIOV direct port works well. Nova.conf passthrough_whitelist={"devname":"ens1f0","physical_network":"physnet5"} passthrough_whitelist={"devname":"ens1f1","physical_network":"physnet6"} Sriov_agent.ini [sriov_nic] physical_device_mappings=physnet5:ens1f0,physnet6:ens1f1 (1) Added the alias in nova.conf for nova-compute and nova-api, and restart the two nova components: alias = { "vendor_id":"8086", "product_id":"10ed", "device_type":"type-VF", "name":"vf", "numa_policy": "required" } (2) Used the extra-spec in nova flavor openstack flavor set octavia-flavor --property "pci_passthrough:alias"="vf:1" (3) Failed to create VM with this flavor, sriov agent log does not show port event, for sure also failed to create LB, PortBindingFailed (4) Tried multiple formats to add whitelist for PF and VF in nova.conf for nova-compute, and retried, still failed passthrough_whitelist={"vendor_id":"8086","product_id":"10f8","devname":"ens1f0","physical_network":"physnet5"} #PF passthrough_whitelist={"vendor_id":"8086","product_id":"10ed","physical_network":"physnet5"} #VF The sriov agent log does not show port event for any of them. -----Original Message----- From: Michael Johnson <johnsomor@gmail.com> Sent: Wednesday, October 6, 2021 4:48 PM To: Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang@nokia.com> Cc: openstack-discuss@lists.openstack.org Subject: Re: [Octavia] Can not create LB on SRIOV network Hi Jing, To my knowledge no one has done the work to support SRIOV network ports in Octavia load balancers. This is an open roadmap item[1]. It will require some development effort as we hot-plug the tenant traffic ports, which means we need to give nova some hints when booting the instances that the amphora instance will be using SRIOV. You might be able to accomplish it on train using the flavors capability. You would create a special nova flavor with the required "extra_specs"[2] to schedule the instance on the proper SRIOV host with the SRIOV libvirt settings. Then you can create an Octavia flavor[3] that uses this special nova flavor. You could then create a load balancer by passing in the neutron SRIOV port as the VIP port. This would not provide a solution for adding additional SRIOV ports to the load balancer for the member servers, but you can use the VIP port to access members. I have not tried this and would be interested to hear if it works for you. If you are interested in implementing SRIOV support for Octavia, please consider adding it to the PTG agenda[4] and joining us at the virtual PTG. Michael [1] https://wiki.openstack.org/wiki/Octavia/Roadmap [2] https://docs.openstack.org/nova/xena/configuration/extra-specs.html [3] https://docs.openstack.org/octavia/latest/admin/flavors.html [4] https://etherpad.opendev.org/p/yoga-ptg-octavia On Wed, Oct 6, 2021 at 10:24 AM Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang@nokia.com> wrote:
I can not create Octavia LB on SRIOV network in Train. I went to Octavia story board, did a search but was unable to figure out (the story for SRIOV…).
I left a comment under this story, I re-post my questions there, hoping someone knows the answer.
Thank you so much
Jing
https://storyboard.openstack.org/#!/story/2006886 Add VM SRIOV Interface Config Guide (Openstack)
Hi, In Openstack train release, creating Octavia LB on SRIOV network fails. I come here to search if there is already a plan to add this support, and see this story. This story gives the impression that the capability is already supported, it is a matter of adding user guide. So, my question is, in which Openstack release, creating LB on SRIOV network is supported? Thank you
Interesting, thank you for trying that out. We call the nova "interface_attach" and pass in the port_id you provided on the load balancer create command line. In the worker log, above the "tree" log lines, is there another ERROR log line that includes the exception returned from nova? Also, I would be interested to see what nova logged as to why it was unable to attach the port. That may be in the main nova logs, or possibly on the compute host nova logs. Michael On Thu, Oct 7, 2021 at 5:36 PM Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang@nokia.com> wrote:
Hi Michael,
I made a mistake when creating VM manually, I should use --nic option not --network option. After correcting that, I can create VM with the extra-flavor:
$ openstack server create --flavor octavia-flavor --image Centos7 --nic port-id=test-port --security-group demo-secgroup --key-name demo-key test-vm
$ nova list --all --fields name,status,host,networks | grep test-vm | 8548400b-725a-405a-aeeb-ed1d208915e2 | test-vm | ACTIVE | overcloud-sriovperformancecompute-201-1.localdomain | ext-net1=10.5.201.149
A 2nd VF interface is seen inside the VM:
[centos@test-vm ~]$ ip a ... 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 0a:b2:d4:85:a2:e6 brd ff:ff:ff:ff:ff:ff
This MAC is not seen by neutron though:
$ openstack port list | grep 0a:b2:d4:85:a2:e6
[empty]
===================== However when I tried to create LB with the same VM flavor, it failed at the same place as before.
Looking at worker.log, it seems the error is similar to use --network option to create the VM manually. But you are the expert.
"Error plugging amphora (compute_id: 64be3ced-90b1-4fb2-a13c-dd5e73ba0526) into vip network 7a1fa805-6d21-4da6-9573-1586c2faef52"
Here is the full list of command line:
$ openstack flavor list | grep octavia-flavor | eb312b9a-d04d-4a88-9db2-7a88ce167cff | octavia-flavor | 4096 | 0 | 0 | 4 | True |
openstack loadbalancer flavorprofile create --name ofp1 --provider amphora --flavor-data '{"compute_flavor": "eb312b9a-d04d-4a88-9db2-7a88ce167cff"}' openstack loadbalancer flavor create --name of1 --flavorprofile ofp1 --enable openstack loadbalancer create --name lb1 --flavor of1 --vip-port-id test-port --vip-subnet-id ext-subnet1
|__Flow 'octavia-create-loadbalancer-flow': PlugVIPException: Error plugging amphora (compute_id: 64be3ced-90b1-4fb2-a13c-dd5e73ba0526) into vip network 7a1fa805-6d21-4da6-9573-1586c2faef52. 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker Traceback (most recent call last): 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker result = task.execute(**arguments) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/octavia/controller/worker/v1/tasks/network_tasks.py", line 399, in execute 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker loadbalancer, loadbalancer.vip, amphora, subnet) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 391, in plug_aap_port 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker interface = self._plug_amphora_vip(amphora, subnet) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 123, in _plug_amphora_vip 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker raise base.PlugVIPException(message) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker PlugVIPException: Error plugging amphora (compute_id: 64be3ced-90b1-4fb2-a13c-dd5e73ba0526) into vip network 7a1fa805-6d21-4da6-9573-1586c2faef52. 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker
-----Original Message----- From: Zhang, Jing C. (Nokia - CA/Ottawa) Sent: Thursday, October 7, 2021 6:18 PM To: Michael Johnson <johnsomor@gmail.com> Cc: openstack-discuss@lists.openstack.org Subject: RE: [Octavia] Can not create LB on SRIOV network
Hi Michael,
Thank you so much for the information.
I tried the extra-flavor walk-around, I can not use it to create VM in Train release, I suspect this old extra-flavor is too old, but I did not dig further.
However, both Train and latest nova spec still shows the above extra-flavor with the old whitelist format: https://docs.openstack.org/nova/train/admin/pci-passthrough.html https://docs.openstack.org/nova/latest/admin/pci-passthrough.html
========================= Here is the detail:
Env: NIC is intel 82599, creating VM with SRIOV direct port works well.
Nova.conf
passthrough_whitelist={"devname":"ens1f0","physical_network":"physnet5"} passthrough_whitelist={"devname":"ens1f1","physical_network":"physnet6"}
Sriov_agent.ini
[sriov_nic] physical_device_mappings=physnet5:ens1f0,physnet6:ens1f1
(1) Added the alias in nova.conf for nova-compute and nova-api, and restart the two nova components:
alias = { "vendor_id":"8086", "product_id":"10ed", "device_type":"type-VF", "name":"vf", "numa_policy": "required" }
(2) Used the extra-spec in nova flavor
openstack flavor set octavia-flavor --property "pci_passthrough:alias"="vf:1"
(3) Failed to create VM with this flavor, sriov agent log does not show port event, for sure also failed to create LB, PortBindingFailed
(4) Tried multiple formats to add whitelist for PF and VF in nova.conf for nova-compute, and retried, still failed
passthrough_whitelist={"vendor_id":"8086","product_id":"10f8","devname":"ens1f0","physical_network":"physnet5"} #PF passthrough_whitelist={"vendor_id":"8086","product_id":"10ed","physical_network":"physnet5"} #VF
The sriov agent log does not show port event for any of them.
-----Original Message----- From: Michael Johnson <johnsomor@gmail.com> Sent: Wednesday, October 6, 2021 4:48 PM To: Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang@nokia.com> Cc: openstack-discuss@lists.openstack.org Subject: Re: [Octavia] Can not create LB on SRIOV network
Hi Jing,
To my knowledge no one has done the work to support SRIOV network ports in Octavia load balancers. This is an open roadmap item[1].
It will require some development effort as we hot-plug the tenant traffic ports, which means we need to give nova some hints when booting the instances that the amphora instance will be using SRIOV.
You might be able to accomplish it on train using the flavors capability. You would create a special nova flavor with the required "extra_specs"[2] to schedule the instance on the proper SRIOV host with the SRIOV libvirt settings. Then you can create an Octavia flavor[3] that uses this special nova flavor. You could then create a load balancer by passing in the neutron SRIOV port as the VIP port. This would not provide a solution for adding additional SRIOV ports to the load balancer for the member servers, but you can use the VIP port to access members.
I have not tried this and would be interested to hear if it works for you.
If you are interested in implementing SRIOV support for Octavia, please consider adding it to the PTG agenda[4] and joining us at the virtual PTG.
Michael
[1] https://wiki.openstack.org/wiki/Octavia/Roadmap [2] https://docs.openstack.org/nova/xena/configuration/extra-specs.html [3] https://docs.openstack.org/octavia/latest/admin/flavors.html [4] https://etherpad.opendev.org/p/yoga-ptg-octavia
On Wed, Oct 6, 2021 at 10:24 AM Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang@nokia.com> wrote:
I can not create Octavia LB on SRIOV network in Train. I went to Octavia story board, did a search but was unable to figure out (the story for SRIOV…).
I left a comment under this story, I re-post my questions there, hoping someone knows the answer.
Thank you so much
Jing
https://storyboard.openstack.org/#!/story/2006886 Add VM SRIOV Interface Config Guide (Openstack)
Hi, In Openstack train release, creating Octavia LB on SRIOV network fails. I come here to search if there is already a plan to add this support, and see this story. This story gives the impression that the capability is already supported, it is a matter of adding user guide. So, my question is, in which Openstack release, creating LB on SRIOV network is supported? Thank you
On Mon, Oct 11, 2021 at 6:12 PM Michael Johnson <johnsomor@gmail.com> wrote:
Interesting, thank you for trying that out.
We call the nova "interface_attach" and pass in the port_id you provided on the load balancer create command line.
In the worker log, above the "tree" log lines, is there another ERROR log line that includes the exception returned from nova?
until very recently nova did not support interface attach for sriov interfaces. https://specs.openstack.org/openstack/nova-specs/specs/victoria/implemented/... today we do allow it but we do not guarentee it will work. if there are not enoch pci slots in the vm or there are not enough VF on the host that are attached to the correct phsynet the attach will fail. the most comon reason the attach fails is either numa affintiy cannot be acived or there is an issue in the guest/qemu the guest kernel need to repond to the hotplug event when qemu tries to add the device if it does not it will fail. keeping all of tha tin mind for sriov attach to work octavia will have to create the port with vnic_type=driect or one of the other valid options like macvtap or direct phsyical. you cannot attach sriov device that can be used with octavia using flavor extra specs.
Also, I would be interested to see what nova logged as to why it was unable to attach the port. That may be in the main nova logs, or possibly on the compute host nova logs.
Michael
On Thu, Oct 7, 2021 at 5:36 PM Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang@nokia.com> wrote:
Hi Michael,
I made a mistake when creating VM manually, I should use --nic option not --network option. After correcting that, I can create VM with the extra-flavor:
$ openstack server create --flavor octavia-flavor --image Centos7 --nic port-id=test-port --security-group demo-secgroup --key-name demo-key test-vm
$ nova list --all --fields name,status,host,networks | grep test-vm | 8548400b-725a-405a-aeeb-ed1d208915e2 | test-vm | ACTIVE | overcloud-sriovperformancecompute-201-1.localdomain | ext-net1=10.5.201.149
A 2nd VF interface is seen inside the VM:
[centos@test-vm ~]$ ip a ... 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 0a:b2:d4:85:a2:e6 brd ff:ff:ff:ff:ff:ff
This MAC is not seen by neutron though:
$ openstack port list | grep 0a:b2:d4:85:a2:e6
[empty]
===================== However when I tried to create LB with the same VM flavor, it failed at the same place as before.
Looking at worker.log, it seems the error is similar to use --network option to create the VM manually. But you are the expert.
"Error plugging amphora (compute_id: 64be3ced-90b1-4fb2-a13c-dd5e73ba0526) into vip network 7a1fa805-6d21-4da6-9573-1586c2faef52"
Here is the full list of command line:
$ openstack flavor list | grep octavia-flavor | eb312b9a-d04d-4a88-9db2-7a88ce167cff | octavia-flavor | 4096 | 0 | 0 | 4 | True |
openstack loadbalancer flavorprofile create --name ofp1 --provider amphora --flavor-data '{"compute_flavor": "eb312b9a-d04d-4a88-9db2-7a88ce167cff"}' openstack loadbalancer flavor create --name of1 --flavorprofile ofp1 --enable openstack loadbalancer create --name lb1 --flavor of1 --vip-port-id test-port --vip-subnet-id ext-subnet1
|__Flow 'octavia-create-loadbalancer-flow': PlugVIPException: Error plugging amphora (compute_id: 64be3ced-90b1-4fb2-a13c-dd5e73ba0526) into vip network 7a1fa805-6d21-4da6-9573-1586c2faef52. 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker Traceback (most recent call last): 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker result = task.execute(**arguments) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/octavia/controller/worker/v1/tasks/network_tasks.py", line 399, in execute 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker loadbalancer, loadbalancer.vip, amphora, subnet) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 391, in plug_aap_port 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker interface = self._plug_amphora_vip(amphora, subnet) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 123, in _plug_amphora_vip 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker raise base.PlugVIPException(message) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker PlugVIPException: Error plugging amphora (compute_id: 64be3ced-90b1-4fb2-a13c-dd5e73ba0526) into vip network 7a1fa805-6d21-4da6-9573-1586c2faef52. 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker
-----Original Message----- From: Zhang, Jing C. (Nokia - CA/Ottawa) Sent: Thursday, October 7, 2021 6:18 PM To: Michael Johnson <johnsomor@gmail.com> Cc: openstack-discuss@lists.openstack.org Subject: RE: [Octavia] Can not create LB on SRIOV network
Hi Michael,
Thank you so much for the information.
I tried the extra-flavor walk-around, I can not use it to create VM in Train release, I suspect this old extra-flavor is too old, but I did not dig further.
However, both Train and latest nova spec still shows the above extra-flavor with the old whitelist format: https://docs.openstack.org/nova/train/admin/pci-passthrough.html https://docs.openstack.org/nova/latest/admin/pci-passthrough.html
========================= Here is the detail:
Env: NIC is intel 82599, creating VM with SRIOV direct port works well.
Nova.conf
passthrough_whitelist={"devname":"ens1f0","physical_network":"physnet5"} passthrough_whitelist={"devname":"ens1f1","physical_network":"physnet6"}
Sriov_agent.ini
[sriov_nic] physical_device_mappings=physnet5:ens1f0,physnet6:ens1f1
(1) Added the alias in nova.conf for nova-compute and nova-api, and restart the two nova components:
alias = { "vendor_id":"8086", "product_id":"10ed", "device_type":"type-VF", "name":"vf", "numa_policy": "required" }
(2) Used the extra-spec in nova flavor
openstack flavor set octavia-flavor --property "pci_passthrough:alias"="vf:1"
(3) Failed to create VM with this flavor, sriov agent log does not show port event, for sure also failed to create LB, PortBindingFailed
(4) Tried multiple formats to add whitelist for PF and VF in nova.conf for nova-compute, and retried, still failed
passthrough_whitelist={"vendor_id":"8086","product_id":"10f8","devname":"ens1f0","physical_network":"physnet5"} #PF passthrough_whitelist={"vendor_id":"8086","product_id":"10ed","physical_network":"physnet5"} #VF
The sriov agent log does not show port event for any of them.
-----Original Message----- From: Michael Johnson <johnsomor@gmail.com> Sent: Wednesday, October 6, 2021 4:48 PM To: Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang@nokia.com> Cc: openstack-discuss@lists.openstack.org Subject: Re: [Octavia] Can not create LB on SRIOV network
Hi Jing,
To my knowledge no one has done the work to support SRIOV network ports in Octavia load balancers. This is an open roadmap item[1].
It will require some development effort as we hot-plug the tenant traffic ports, which means we need to give nova some hints when booting the instances that the amphora instance will be using SRIOV.
You might be able to accomplish it on train using the flavors capability. You would create a special nova flavor with the required "extra_specs"[2] to schedule the instance on the proper SRIOV host with the SRIOV libvirt settings. Then you can create an Octavia flavor[3] that uses this special nova flavor. You could then create a load balancer by passing in the neutron SRIOV port as the VIP port. This would not provide a solution for adding additional SRIOV ports to the load balancer for the member servers, but you can use the VIP port to access members.
I have not tried this and would be interested to hear if it works for you.
If you are interested in implementing SRIOV support for Octavia, please consider adding it to the PTG agenda[4] and joining us at the virtual PTG.
Michael
[1] https://wiki.openstack.org/wiki/Octavia/Roadmap [2] https://docs.openstack.org/nova/xena/configuration/extra-specs.html [3] https://docs.openstack.org/octavia/latest/admin/flavors.html [4] https://etherpad.opendev.org/p/yoga-ptg-octavia
On Wed, Oct 6, 2021 at 10:24 AM Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang@nokia.com> wrote:
I can not create Octavia LB on SRIOV network in Train. I went to Octavia story board, did a search but was unable to figure out (the story for SRIOV…).
I left a comment under this story, I re-post my questions there, hoping someone knows the answer.
Thank you so much
Jing
https://storyboard.openstack.org/#!/story/2006886 Add VM SRIOV Interface Config Guide (Openstack)
Hi, In Openstack train release, creating Octavia LB on SRIOV network fails. I come here to search if there is already a plan to add this support, and see this story. This story gives the impression that the capability is already supported, it is a matter of adding user guide. So, my question is, in which Openstack release, creating LB on SRIOV network is supported? Thank you
Ah, so that is probably the issue. Nova doesn't support the interface attach for SRIOV in Train. We do currently require that the port be hot plugged after boot. I would still be interested in seeing the log messages, just to confirm that is the issue or if we have other work to do. The vnic_type=direct should not be an issue as the port is being passed into Octavia pre-created. I think it was already mentioned that the port was successful when used during boot via the --nic option. Thanks for the pointer Sean. Michael On Mon, Oct 11, 2021 at 10:22 AM Sean Mooney <smooney@redhat.com> wrote:
On Mon, Oct 11, 2021 at 6:12 PM Michael Johnson <johnsomor@gmail.com> wrote:
Interesting, thank you for trying that out.
We call the nova "interface_attach" and pass in the port_id you provided on the load balancer create command line.
In the worker log, above the "tree" log lines, is there another ERROR log line that includes the exception returned from nova?
until very recently nova did not support interface attach for sriov interfaces. https://specs.openstack.org/openstack/nova-specs/specs/victoria/implemented/... today we do allow it but we do not guarentee it will work.
if there are not enoch pci slots in the vm or there are not enough VF on the host that are attached to the correct phsynet the attach will fail. the most comon reason the attach fails is either numa affintiy cannot be acived or there is an issue in the guest/qemu the guest kernel need to repond to the hotplug event when qemu tries to add the device if it does not it will fail.
keeping all of tha tin mind for sriov attach to work octavia will have to create the port with vnic_type=driect or one of the other valid options like macvtap or direct phsyical. you cannot attach sriov device that can be used with octavia using flavor extra specs.
Also, I would be interested to see what nova logged as to why it was unable to attach the port. That may be in the main nova logs, or possibly on the compute host nova logs.
Michael
On Thu, Oct 7, 2021 at 5:36 PM Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang@nokia.com> wrote:
Hi Michael,
I made a mistake when creating VM manually, I should use --nic option not --network option. After correcting that, I can create VM with the extra-flavor:
$ openstack server create --flavor octavia-flavor --image Centos7 --nic port-id=test-port --security-group demo-secgroup --key-name demo-key test-vm
$ nova list --all --fields name,status,host,networks | grep test-vm | 8548400b-725a-405a-aeeb-ed1d208915e2 | test-vm | ACTIVE | overcloud-sriovperformancecompute-201-1.localdomain | ext-net1=10.5.201.149
A 2nd VF interface is seen inside the VM:
[centos@test-vm ~]$ ip a ... 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 0a:b2:d4:85:a2:e6 brd ff:ff:ff:ff:ff:ff
This MAC is not seen by neutron though:
$ openstack port list | grep 0a:b2:d4:85:a2:e6
[empty]
===================== However when I tried to create LB with the same VM flavor, it failed at the same place as before.
Looking at worker.log, it seems the error is similar to use --network option to create the VM manually. But you are the expert.
"Error plugging amphora (compute_id: 64be3ced-90b1-4fb2-a13c-dd5e73ba0526) into vip network 7a1fa805-6d21-4da6-9573-1586c2faef52"
Here is the full list of command line:
$ openstack flavor list | grep octavia-flavor | eb312b9a-d04d-4a88-9db2-7a88ce167cff | octavia-flavor | 4096 | 0 | 0 | 4 | True |
openstack loadbalancer flavorprofile create --name ofp1 --provider amphora --flavor-data '{"compute_flavor": "eb312b9a-d04d-4a88-9db2-7a88ce167cff"}' openstack loadbalancer flavor create --name of1 --flavorprofile ofp1 --enable openstack loadbalancer create --name lb1 --flavor of1 --vip-port-id test-port --vip-subnet-id ext-subnet1
|__Flow 'octavia-create-loadbalancer-flow': PlugVIPException: Error plugging amphora (compute_id: 64be3ced-90b1-4fb2-a13c-dd5e73ba0526) into vip network 7a1fa805-6d21-4da6-9573-1586c2faef52. 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker Traceback (most recent call last): 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker result = task.execute(**arguments) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/octavia/controller/worker/v1/tasks/network_tasks.py", line 399, in execute 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker loadbalancer, loadbalancer.vip, amphora, subnet) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 391, in plug_aap_port 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker interface = self._plug_amphora_vip(amphora, subnet) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 123, in _plug_amphora_vip 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker raise base.PlugVIPException(message) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker PlugVIPException: Error plugging amphora (compute_id: 64be3ced-90b1-4fb2-a13c-dd5e73ba0526) into vip network 7a1fa805-6d21-4da6-9573-1586c2faef52. 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker
-----Original Message----- From: Zhang, Jing C. (Nokia - CA/Ottawa) Sent: Thursday, October 7, 2021 6:18 PM To: Michael Johnson <johnsomor@gmail.com> Cc: openstack-discuss@lists.openstack.org Subject: RE: [Octavia] Can not create LB on SRIOV network
Hi Michael,
Thank you so much for the information.
I tried the extra-flavor walk-around, I can not use it to create VM in Train release, I suspect this old extra-flavor is too old, but I did not dig further.
However, both Train and latest nova spec still shows the above extra-flavor with the old whitelist format: https://docs.openstack.org/nova/train/admin/pci-passthrough.html https://docs.openstack.org/nova/latest/admin/pci-passthrough.html
========================= Here is the detail:
Env: NIC is intel 82599, creating VM with SRIOV direct port works well.
Nova.conf
passthrough_whitelist={"devname":"ens1f0","physical_network":"physnet5"} passthrough_whitelist={"devname":"ens1f1","physical_network":"physnet6"}
Sriov_agent.ini
[sriov_nic] physical_device_mappings=physnet5:ens1f0,physnet6:ens1f1
(1) Added the alias in nova.conf for nova-compute and nova-api, and restart the two nova components:
alias = { "vendor_id":"8086", "product_id":"10ed", "device_type":"type-VF", "name":"vf", "numa_policy": "required" }
(2) Used the extra-spec in nova flavor
openstack flavor set octavia-flavor --property "pci_passthrough:alias"="vf:1"
(3) Failed to create VM with this flavor, sriov agent log does not show port event, for sure also failed to create LB, PortBindingFailed
(4) Tried multiple formats to add whitelist for PF and VF in nova.conf for nova-compute, and retried, still failed
passthrough_whitelist={"vendor_id":"8086","product_id":"10f8","devname":"ens1f0","physical_network":"physnet5"} #PF passthrough_whitelist={"vendor_id":"8086","product_id":"10ed","physical_network":"physnet5"} #VF
The sriov agent log does not show port event for any of them.
-----Original Message----- From: Michael Johnson <johnsomor@gmail.com> Sent: Wednesday, October 6, 2021 4:48 PM To: Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang@nokia.com> Cc: openstack-discuss@lists.openstack.org Subject: Re: [Octavia] Can not create LB on SRIOV network
Hi Jing,
To my knowledge no one has done the work to support SRIOV network ports in Octavia load balancers. This is an open roadmap item[1].
It will require some development effort as we hot-plug the tenant traffic ports, which means we need to give nova some hints when booting the instances that the amphora instance will be using SRIOV.
You might be able to accomplish it on train using the flavors capability. You would create a special nova flavor with the required "extra_specs"[2] to schedule the instance on the proper SRIOV host with the SRIOV libvirt settings. Then you can create an Octavia flavor[3] that uses this special nova flavor. You could then create a load balancer by passing in the neutron SRIOV port as the VIP port. This would not provide a solution for adding additional SRIOV ports to the load balancer for the member servers, but you can use the VIP port to access members.
I have not tried this and would be interested to hear if it works for you.
If you are interested in implementing SRIOV support for Octavia, please consider adding it to the PTG agenda[4] and joining us at the virtual PTG.
Michael
[1] https://wiki.openstack.org/wiki/Octavia/Roadmap [2] https://docs.openstack.org/nova/xena/configuration/extra-specs.html [3] https://docs.openstack.org/octavia/latest/admin/flavors.html [4] https://etherpad.opendev.org/p/yoga-ptg-octavia
On Wed, Oct 6, 2021 at 10:24 AM Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang@nokia.com> wrote:
I can not create Octavia LB on SRIOV network in Train. I went to Octavia story board, did a search but was unable to figure out (the story for SRIOV…).
I left a comment under this story, I re-post my questions there, hoping someone knows the answer.
Thank you so much
Jing
https://storyboard.openstack.org/#!/story/2006886 Add VM SRIOV Interface Config Guide (Openstack)
Hi, In Openstack train release, creating Octavia LB on SRIOV network fails. I come here to search if there is already a plan to add this support, and see this story. This story gives the impression that the capability is already supported, it is a matter of adding user guide. So, my question is, in which Openstack release, creating LB on SRIOV network is supported? Thank you
Hi Michael, Nova log does not have other error besides "port-bind-failure...check neutron log"...as if you manually attempts to attach VM to SRIOV provider network not using the direct port type. Jing -----Original Message----- From: Michael Johnson <johnsomor@gmail.com> Sent: Monday, October 11, 2021 2:00 PM To: Sean Mooney <smooney@redhat.com> Cc: Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang@nokia.com>; openstack-discuss@lists.openstack.org Subject: Re: [Octavia] Can not create LB on SRIOV network Ah, so that is probably the issue. Nova doesn't support the interface attach for SRIOV in Train. We do currently require that the port be hot plugged after boot. I would still be interested in seeing the log messages, just to confirm that is the issue or if we have other work to do. The vnic_type=direct should not be an issue as the port is being passed into Octavia pre-created. I think it was already mentioned that the port was successful when used during boot via the --nic option. Thanks for the pointer Sean. Michael On Mon, Oct 11, 2021 at 10:22 AM Sean Mooney <smooney@redhat.com> wrote:
On Mon, Oct 11, 2021 at 6:12 PM Michael Johnson <johnsomor@gmail.com> wrote:
Interesting, thank you for trying that out.
We call the nova "interface_attach" and pass in the port_id you provided on the load balancer create command line.
In the worker log, above the "tree" log lines, is there another ERROR log line that includes the exception returned from nova?
until very recently nova did not support interface attach for sriov interfaces. https://specs.openstack.org/openstack/nova-specs/specs/victoria/implem ented/sriov-interface-attach-detach.html today we do allow it but we do not guarentee it will work.
if there are not enoch pci slots in the vm or there are not enough VF on the host that are attached to the correct phsynet the attach will fail. the most comon reason the attach fails is either numa affintiy cannot be acived or there is an issue in the guest/qemu the guest kernel need to repond to the hotplug event when qemu tries to add the device if it does not it will fail.
keeping all of tha tin mind for sriov attach to work octavia will have to create the port with vnic_type=driect or one of the other valid options like macvtap or direct phsyical. you cannot attach sriov device that can be used with octavia using flavor extra specs.
Also, I would be interested to see what nova logged as to why it was unable to attach the port. That may be in the main nova logs, or possibly on the compute host nova logs.
Michael
On Thu, Oct 7, 2021 at 5:36 PM Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang@nokia.com> wrote:
Hi Michael,
I made a mistake when creating VM manually, I should use --nic option not --network option. After correcting that, I can create VM with the extra-flavor:
$ openstack server create --flavor octavia-flavor --image Centos7 --nic port-id=test-port --security-group demo-secgroup --key-name demo-key test-vm
$ nova list --all --fields name,status,host,networks | grep test-vm | 8548400b-725a-405a-aeeb-ed1d208915e2 | test-vm | ACTIVE | overcloud-sriovperformancecompute-201-1.localdomain | ext-net1=10.5.201.149
A 2nd VF interface is seen inside the VM:
[centos@test-vm ~]$ ip a ... 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 0a:b2:d4:85:a2:e6 brd ff:ff:ff:ff:ff:ff
This MAC is not seen by neutron though:
$ openstack port list | grep 0a:b2:d4:85:a2:e6
[empty]
===================== However when I tried to create LB with the same VM flavor, it failed at the same place as before.
Looking at worker.log, it seems the error is similar to use --network option to create the VM manually. But you are the expert.
"Error plugging amphora (compute_id: 64be3ced-90b1-4fb2-a13c-dd5e73ba0526) into vip network 7a1fa805-6d21-4da6-9573-1586c2faef52"
Here is the full list of command line:
$ openstack flavor list | grep octavia-flavor | eb312b9a-d04d-4a88-9db2-7a88ce167cff | octavia-flavor | 4096 | 0 | 0 | 4 | True |
openstack loadbalancer flavorprofile create --name ofp1 --provider amphora --flavor-data '{"compute_flavor": "eb312b9a-d04d-4a88-9db2-7a88ce167cff"}' openstack loadbalancer flavor create --name of1 --flavorprofile ofp1 --enable openstack loadbalancer create --name lb1 --flavor of1 --vip-port-id test-port --vip-subnet-id ext-subnet1
|__Flow 'octavia-create-loadbalancer-flow': PlugVIPException: Error plugging amphora (compute_id: 64be3ced-90b1-4fb2-a13c-dd5e73ba0526) into vip network 7a1fa805-6d21-4da6-9573-1586c2faef52. 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker Traceback (most recent call last): 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker result = task.execute(**arguments) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/octavia/controller/worker/v1/tasks/network_tasks.py", line 399, in execute 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker loadbalancer, loadbalancer.vip, amphora, subnet) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 391, in plug_aap_port 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker interface = self._plug_amphora_vip(amphora, subnet) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker File "/usr/lib/python2.7/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 123, in _plug_amphora_vip 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker raise base.PlugVIPException(message) 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker PlugVIPException: Error plugging amphora (compute_id: 64be3ced-90b1-4fb2-a13c-dd5e73ba0526) into vip network 7a1fa805-6d21-4da6-9573-1586c2faef52. 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker
-----Original Message----- From: Zhang, Jing C. (Nokia - CA/Ottawa) Sent: Thursday, October 7, 2021 6:18 PM To: Michael Johnson <johnsomor@gmail.com> Cc: openstack-discuss@lists.openstack.org Subject: RE: [Octavia] Can not create LB on SRIOV network
Hi Michael,
Thank you so much for the information.
I tried the extra-flavor walk-around, I can not use it to create VM in Train release, I suspect this old extra-flavor is too old, but I did not dig further.
However, both Train and latest nova spec still shows the above extra-flavor with the old whitelist format: https://docs.openstack.org/nova/train/admin/pci-passthrough.html https://docs.openstack.org/nova/latest/admin/pci-passthrough.html
========================= Here is the detail:
Env: NIC is intel 82599, creating VM with SRIOV direct port works well.
Nova.conf
passthrough_whitelist={"devname":"ens1f0","physical_network":"phys net5"} passthrough_whitelist={"devname":"ens1f1","physical_network":"phys net6"}
Sriov_agent.ini
[sriov_nic] physical_device_mappings=physnet5:ens1f0,physnet6:ens1f1
(1) Added the alias in nova.conf for nova-compute and nova-api, and restart the two nova components:
alias = { "vendor_id":"8086", "product_id":"10ed", "device_type":"type-VF", "name":"vf", "numa_policy": "required" }
(2) Used the extra-spec in nova flavor
openstack flavor set octavia-flavor --property "pci_passthrough:alias"="vf:1"
(3) Failed to create VM with this flavor, sriov agent log does not show port event, for sure also failed to create LB, PortBindingFailed
(4) Tried multiple formats to add whitelist for PF and VF in nova.conf for nova-compute, and retried, still failed
passthrough_whitelist={"vendor_id":"8086","product_id":"10f8","dev name":"ens1f0","physical_network":"physnet5"} #PF passthrough_whitelist={"vendor_id":"8086","product_id":"10ed","phy sical_network":"physnet5"} #VF
The sriov agent log does not show port event for any of them.
-----Original Message----- From: Michael Johnson <johnsomor@gmail.com> Sent: Wednesday, October 6, 2021 4:48 PM To: Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang@nokia.com> Cc: openstack-discuss@lists.openstack.org Subject: Re: [Octavia] Can not create LB on SRIOV network
Hi Jing,
To my knowledge no one has done the work to support SRIOV network ports in Octavia load balancers. This is an open roadmap item[1].
It will require some development effort as we hot-plug the tenant traffic ports, which means we need to give nova some hints when booting the instances that the amphora instance will be using SRIOV.
You might be able to accomplish it on train using the flavors capability. You would create a special nova flavor with the required "extra_specs"[2] to schedule the instance on the proper SRIOV host with the SRIOV libvirt settings. Then you can create an Octavia flavor[3] that uses this special nova flavor. You could then create a load balancer by passing in the neutron SRIOV port as the VIP port. This would not provide a solution for adding additional SRIOV ports to the load balancer for the member servers, but you can use the VIP port to access members.
I have not tried this and would be interested to hear if it works for you.
If you are interested in implementing SRIOV support for Octavia, please consider adding it to the PTG agenda[4] and joining us at the virtual PTG.
Michael
[1] https://wiki.openstack.org/wiki/Octavia/Roadmap [2] https://docs.openstack.org/nova/xena/configuration/extra-specs.htm l [3] https://docs.openstack.org/octavia/latest/admin/flavors.html [4] https://etherpad.opendev.org/p/yoga-ptg-octavia
On Wed, Oct 6, 2021 at 10:24 AM Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang@nokia.com> wrote:
I can not create Octavia LB on SRIOV network in Train. I went to Octavia story board, did a search but was unable to figure out (the story for SRIOV…).
I left a comment under this story, I re-post my questions there, hoping someone knows the answer.
Thank you so much
Jing
https://storyboard.openstack.org/#!/story/2006886 Add VM SRIOV Interface Config Guide (Openstack)
Hi, In Openstack train release, creating Octavia LB on SRIOV network fails. I come here to search if there is already a plan to add this support, and see this story. This story gives the impression that the capability is already supported, it is a matter of adding user guide. So, my question is, in which Openstack release, creating LB on SRIOV network is supported? Thank you
participants (3)
-
Michael Johnson
-
Sean Mooney
-
Zhang, Jing C. (Nokia - CA/Ottawa)