Open Stack

Ah, so that is probably the issue. Nova doesn't support the interface
attach for SRIOV in Train. We do currently require that the port be
hot plugged after boot.

I would still be interested in seeing the log messages, just to
confirm that is the issue or if we have other work to do.

The vnic_type=direct should not be an issue as the port is being
passed into Octavia pre-created. I think it was already mentioned that
the port was successful when used during boot via the --nic option.

Thanks for the pointer Sean.

Michael

On Mon, Oct 11, 2021 at 10:22 AM Sean Mooney <smooney at redhat.com> wrote:
>
> On Mon, Oct 11, 2021 at 6:12 PM Michael Johnson <johnsomor at gmail.com> wrote:
> >
> > Interesting, thank you for trying that out.
> >
> > We call the nova "interface_attach" and pass in the port_id you
> > provided on the load balancer create command line.
> >
> > In the worker log, above the "tree" log lines, is there another ERROR
> > log line that includes the exception returned from nova?
> until very recently nova did not support interface attach for sriov interfaces.
> https://specs.openstack.org/openstack/nova-specs/specs/victoria/implemented/sriov-interface-attach-detach.html
> today we do allow it but we do not guarentee it will work.
>
> if there are not enoch pci slots in the vm or there are not enough VF
> on the host
> that are attached to the correct phsynet the attach will fail.
> the most comon reason the attach fails is either numa affintiy cannot
> be acived or there is an issue in the guest/qemu
> the guest kernel need to repond to the hotplug event when qemu tries
> to add the device if it does not it will fail.
>
> keeping all of tha tin mind for sriov attach to work octavia will have
> to create the port with vnic_type=driect or one of the other valid
> options like macvtap or direct phsyical.
> you cannot attach sriov device that can be used with octavia using
> flavor extra specs.
>
> >
> > Also, I would be interested to see what nova logged as to why it was
> > unable to attach the port. That may be in the main nova logs, or
> > possibly on the compute host nova logs.
> >
> > Michael
> >
> > On Thu, Oct 7, 2021 at 5:36 PM Zhang, Jing C. (Nokia - CA/Ottawa)
> > <jing.c.zhang at nokia.com> wrote:
> > >
> > > Hi Michael,
> > >
> > > I made a mistake when creating VM manually, I should use --nic option not --network option. After correcting that, I can create VM with the extra-flavor:
> > >
> > > $ openstack server create --flavor octavia-flavor --image Centos7 --nic port-id=test-port --security-group demo-secgroup --key-name demo-key test-vm
> > >
> > > $ nova list --all --fields name,status,host,networks | grep test-vm
> > > | 8548400b-725a-405a-aeeb-ed1d208915e2 | test-vm                 | ACTIVE | overcloud-sriovperformancecompute-201-1.localdomain | ext-net1=10.5.201.149
> > >
> > > A 2nd VF interface is seen inside the VM:
> > >
> > > [centos at test-vm ~]$ ip a
> > > ...
> > > 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
> > >     link/ether 0a:b2:d4:85:a2:e6 brd ff:ff:ff:ff:ff:ff
> > >
> > > This MAC is not seen by neutron though:
> > >
> > > $ openstack port list | grep 0a:b2:d4:85:a2:e6
> > >
> > > [empty]
> > >
> > > =====================
> > > However when I tried to create LB with the same VM flavor, it failed at the same place as before.
> > >
> > > Looking at worker.log, it seems the error is similar to use --network option to create the VM manually. But you are the expert.
> > >
> > > "Error plugging amphora (compute_id: 64be3ced-90b1-4fb2-a13c-dd5e73ba0526) into vip network 7a1fa805-6d21-4da6-9573-1586c2faef52"
> > >
> > > Here is the full list of command line:
> > >
> > > $ openstack flavor list | grep octavia-flavor
> > > | eb312b9a-d04d-4a88-9db2-7a88ce167cff | octavia-flavor |  4096 |    0 |         0 |     4 | True      |
> > >
> > > openstack loadbalancer flavorprofile create --name ofp1 --provider amphora --flavor-data '{"compute_flavor": "eb312b9a-d04d-4a88-9db2-7a88ce167cff"}'
> > > openstack loadbalancer flavor create --name of1 --flavorprofile ofp1 --enable
> > > openstack loadbalancer create --name lb1 --flavor of1 --vip-port-id test-port --vip-subnet-id ext-subnet1
> > >
> > >
> > >                                    |__Flow 'octavia-create-loadbalancer-flow': PlugVIPException: Error plugging amphora (compute_id: 64be3ced-90b1-4fb2-a13c-dd5e73ba0526) into vip network 7a1fa805-6d21-4da6-9573-1586c2faef52.
> > > 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker Traceback (most recent call last):
> > > 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
> > > 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker     result = task.execute(**arguments)
> > > 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python2.7/site-packages/octavia/controller/worker/v1/tasks/network_tasks.py", line 399, in execute
> > > 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker     loadbalancer, loadbalancer.vip, amphora, subnet)
> > > 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python2.7/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 391, in plug_aap_port
> > > 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker     interface = self._plug_amphora_vip(amphora, subnet)
> > > 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python2.7/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 123, in _plug_amphora_vip
> > > 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker     raise base.PlugVIPException(message)
> > > 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker PlugVIPException: Error plugging amphora (compute_id: 64be3ced-90b1-4fb2-a13c-dd5e73ba0526) into vip network 7a1fa805-6d21-4da6-9573-1586c2faef52.
> > > 2021-10-08 00:19:26.497 71 ERROR octavia.controller.worker.v1.controller_worker
> > >
> > >
> > > -----Original Message-----
> > > From: Zhang, Jing C. (Nokia - CA/Ottawa)
> > > Sent: Thursday, October 7, 2021 6:18 PM
> > > To: Michael Johnson <johnsomor at gmail.com>
> > > Cc: openstack-discuss at lists.openstack.org
> > > Subject: RE: [Octavia] Can not create LB on SRIOV network
> > >
> > > Hi Michael,
> > >
> > > Thank you so much for the information.
> > >
> > > I tried the extra-flavor walk-around, I can not use it to create VM in Train release, I suspect this old extra-flavor is too old, but I did not dig further.
> > >
> > > However, both Train and  latest nova spec still shows the above extra-flavor with the old whitelist format:
> > > https://docs.openstack.org/nova/train/admin/pci-passthrough.html
> > > https://docs.openstack.org/nova/latest/admin/pci-passthrough.html
> > >
> > > =========================
> > > Here is the detail:
> > >
> > > Env: NIC is intel 82599, creating VM with SRIOV direct port works well.
> > >
> > > Nova.conf
> > >
> > > passthrough_whitelist={"devname":"ens1f0","physical_network":"physnet5"}
> > > passthrough_whitelist={"devname":"ens1f1","physical_network":"physnet6"}
> > >
> > > Sriov_agent.ini
> > >
> > > [sriov_nic]
> > > physical_device_mappings=physnet5:ens1f0,physnet6:ens1f1
> > >
> > > (1) Added the alias in nova.conf for nova-compute and nova-api, and restart the two nova components:
> > >
> > > alias = { "vendor_id":"8086", "product_id":"10ed", "device_type":"type-VF", "name":"vf", "numa_policy": "required" }
> > >
> > > (2) Used the extra-spec in nova flavor
> > >
> > > openstack flavor set octavia-flavor --property "pci_passthrough:alias"="vf:1"
> > >
> > > (3) Failed to create VM with this flavor, sriov agent log does not show port event, for sure also failed to create LB, PortBindingFailed
> > >
> > >
> > > (4) Tried multiple formats to add whitelist for PF and VF in nova.conf for nova-compute, and retried, still failed
> > >
> > > passthrough_whitelist={"vendor_id":"8086","product_id":"10f8","devname":"ens1f0","physical_network":"physnet5"} #PF passthrough_whitelist={"vendor_id":"8086","product_id":"10ed","physical_network":"physnet5"} #VF
> > >
> > > The sriov agent log does not show port event for any of them.
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Michael Johnson <johnsomor at gmail.com>
> > > Sent: Wednesday, October 6, 2021 4:48 PM
> > > To: Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang at nokia.com>
> > > Cc: openstack-discuss at lists.openstack.org
> > > Subject: Re: [Octavia] Can not create LB on SRIOV network
> > >
> > > Hi Jing,
> > >
> > > To my knowledge no one has done the work to support SRIOV network ports in Octavia load balancers. This is an open roadmap item[1].
> > >
> > > It will require some development effort as we hot-plug the tenant traffic ports, which means we need to give nova some hints when booting the instances that the amphora instance will be using SRIOV.
> > >
> > > You might be able to accomplish it on train using the flavors capability. You would create a special nova flavor with the required "extra_specs"[2] to schedule the instance on the proper SRIOV host with the SRIOV libvirt settings. Then you can create an Octavia flavor[3] that uses this special nova flavor. You could then create a load balancer by passing in the neutron SRIOV port as the VIP port.
> > > This would not provide a solution for adding additional SRIOV ports to the load balancer for the member servers, but you can use the VIP port to access members.
> > >
> > > I have not tried this and would be interested to hear if it works for you.
> > >
> > > If you are interested in implementing SRIOV support for Octavia, please consider adding it to the PTG agenda[4] and joining us at the virtual PTG.
> > >
> > > Michael
> > >
> > > [1] https://wiki.openstack.org/wiki/Octavia/Roadmap
> > > [2] https://docs.openstack.org/nova/xena/configuration/extra-specs.html
> > > [3] https://docs.openstack.org/octavia/latest/admin/flavors.html
> > > [4] https://etherpad.opendev.org/p/yoga-ptg-octavia
> > >
> > > On Wed, Oct 6, 2021 at 10:24 AM Zhang, Jing C. (Nokia - CA/Ottawa) <jing.c.zhang at nokia.com> wrote:
> > > >
> > > > I can not create Octavia LB on SRIOV network in Train. I went to Octavia story board, did a search but was unable to figure out (the story for SRIOV…).
> > > >
> > > >
> > > >
> > > > I left a comment under this story, I re-post my questions there, hoping someone knows the answer.
> > > >
> > > >
> > > >
> > > > Thank you so much
> > > >
> > > >
> > > >
> > > > Jing
> > > >
> > > >
> > > >
> > > > https://storyboard.openstack.org/#!/story/2006886 Add VM SRIOV
> > > > Interface Config Guide (Openstack)
> > > >
> > > >
> > > >
> > > > Hi,
> > > > In Openstack train release, creating Octavia LB on SRIOV network fails.
> > > > I come here to search if there is already a plan to add this support, and see this story.
> > > > This story gives the impression that the capability is already supported, it is a matter of adding user guide.
> > > > So, my question is, in which Openstack release, creating LB on SRIOV network is supported?
> > > > Thank you
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> >
>