<div dir="ltr"><div>Hi {Sean, Simon},</div><div><br></div><div>> did you ever give a presentation on the DPU support</div><div><br></div><div>Yes, there were a couple at different stages.</div><div><br></div><div>The following is the one of the older ones that references the SMARTNIC VNIC type but we later switched to REMOTE_MANAGED in the final code: <a href="https://www.openvswitch.org/support/ovscon2021/slides/smartnic_port_binding.pdf">https://www.openvswitch.org/support/ovscon2021/slides/smartnic_port_binding.pdf</a>, however, it has a useful diagram on page 15 which shows the interactions of different components. A lot of other content from it is present in the OpenStack docs now which we added during the feature development.</div><div><br></div><div>There is also a presentation with a demo that we did at the Open Infra summit <a href="https://youtu.be/Amxp-9yEnsU">https://youtu.be/Amxp-9yEnsU</a> (I could not attend but we prepared the material after the features got merged).<br></div><div><br></div><div>Generally, as Sean described, the aim of this feature is to make the interaction between components present at the hypervisor and the DPU side automatic but, in order to make this workflow explicitly different from SR-IOV or offload at the hypervisor side, one has to use the "remote_managed" flag. This flag allows Nova to differentiate between "regular" VFs and the ones that have to be programmed by a remote host (DPU) - hence the name.</div><div><br></div><div>A port needs to be pre-created with the remote-managed type - that way when Nova tries to schedule a VM with that port attached, it will find hosts which actually have PCI devices tagged with the "remote_managed": "true" in the PCI whitelist.</div><div><br></div><div>The important thing to note here is that you must not use PCI passthrough directly for this - Nova will create a PCI device request automatically with the remote_managed flag included. There is currently no way to instruct Nova to choose one vendor/device ID vs the other for this (any remote_managed=true device from a pool will match) but maybe the work that was recently done to store PCI device information in the Placement service will pave the way for such granularity in the future.</div><div><br></div><div dir="ltr"><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Best Regards,<div>Dmitrii Shcherbakov</div><div><span style="color:rgb(136,136,136);font-size:12.8px">LP/MM/oftc: dmitriis</span><br></div></div></div></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 2, 2023 at 1:54 PM Sean Mooney <<a href="mailto:smooney@redhat.com">smooney@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">adding Dmitrii who was the primary developer of the openstack integration so<br>
they can provide more insight.<br>
<br>
Dmitrii did you ever give a presentationon the DPU support and how its configured/integrated<br>
that might help fill in the gaps for simon?<br>
<br>
more inline.<br>
<br>
On Thu, 2023-03-02 at 11:05 +0800, Simon Jones wrote:<br>
> E...<br>
> <br>
> But there are these things:<br>
> <br>
> 1) Show some real happened in my test:<br>
> <br>
> - Let me clear that, I use DPU in compute node:<br>
> The graph in<br>
> <a href="https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html</a> .<br>
> <br>
> - I configure exactly follow<br>
> <a href="https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html</a>,<br>
> which is said bellow in "3) Let me post all what I do follow this link".<br>
> <br>
> - In my test, I found after first three command (which is "openstack<br>
> network create ...", "openstack subnet create", "openstack port create ..."),<br>
> there are network topology exist in DPU side, and there are rules exist in<br>
> OVN north DB, south DB of controller, like this:<br>
> <br>
> > ```<br>
> > root@c1:~# ovn-nbctl show<br>
> > switch 9bdacdd4-ca2a-4e35-82ca-0b5fbd3a5976<br>
> > (neutron-066c8dc2-c98b-4fb8-a541-8b367e8f6e69) (aka selfservice)<br>
> > port 01a68701-0e6a-4c30-bfba-904d1b9813e1<br>
> > addresses: ["unknown"]<br>
> > port 18a44c6f-af50-4830-ba86-54865abb60a1 (aka pf0vf1)<br>
> > addresses: ["fa:16:3e:13:36:e2 172.1.1.228"]<br>
> > <br>
> > gyw@c1:~$ sudo ovn-sbctl list Port_Binding<br>
> > _uuid : 61dc8bc0-ab33-4d67-ac13-0781f89c905a<br>
> > chassis : []<br>
> > datapath : 91d3509c-d794-496a-ba11-3706ebf143c8<br>
> > encap : []<br>
> > external_ids : {name=pf0vf1, "neutron:cidrs"="<a href="http://172.1.1.241/24" rel="noreferrer" target="_blank">172.1.1.241/24</a>",<br>
> > "neutron:device_id"="", "neutron:device_owner"="",<br>
> > "neutron:network_name"=neutron-066c8dc2-c98b-4fb8-a541-8b367e8f6e69,<br>
> > "neutron:port_name"=pf0vf1,<br>
> > "neutron:project_id"="512866f9994f4ad8916d8539a7cdeec9",<br>
> > "neutron:revision_number"="1",<br>
> > "neutron:security_group_ids"="de8883e8-ccac-4be2-9bb2-95e732b0c114"}<br>
> > <br>
> > root@c1c2dpu:~# sudo ovs-vsctl show<br>
> > 62cf78e5-2c02-471e-927e-1d69c2c22195<br>
> > Bridge br-int<br>
> > fail_mode: secure<br>
> > datapath_type: system<br>
> > Port br-int<br>
> > Interface br-int<br>
> > type: internal<br>
> > Port ovn--1<br>
> > Interface ovn--1<br>
> > type: geneve<br>
> > options: {csum="true", key=flow, remote_ip="172.168.2.98"}<br>
> > Port pf0vf1<br>
> > Interface pf0vf1<br>
> > ovs_version: "2.17.2-24a81c8"<br>
> > ```<br>
> > <br>
> That's why I guess "first three command" has already create network<br>
> topology, and "openstack server create" command only need to plug VF into<br>
> VM in HOST SIDE, DO NOT CALL NEUTRON. As network has already done.<br>
no that jsut looks like the standard bridge toplogy that gets created when you provision<br>
the dpu to be used with openstac vai ovn.<br>
<br>
that looks unrelated to the neuton comamnd you ran.<br>
> <br>
> - In my test, then I run "openstack server create" command, I got ERROR<br>
> which said "No valid host...", which is what the email said above.<br>
> The reason has already said, it's nova-scheduler's PCI filter module report<br>
> no valid host. The reason "nova-scheduler's PCI filter module report no<br>
> valid host" is nova-scheduler could NOT see PCI information of compute<br>
> node. The reason "nova-scheduler could NOT see PCI information of compute<br>
> node" is compute node's /etc/nova/nova.conf configure remote_managed tag<br>
> like this:<br>
> <br>
> > ```<br>
> > [pci]<br>
> > passthrough_whitelist = {"vendor_id": "15b3", "product_id": "101e",<br>
> > "physical_network": null, "remote_managed": "true"}<br>
> > alias = { "vendor_id":"15b3", "product_id":"101e",<br>
> > "device_type":"type-VF", "name":"a1" }<br>
> > ```<br>
> > <br>
> <br>
> 2) Discuss some detail design of "remote_managed" tag, I don't know if this<br>
> is right in the design of openstack with DPU:<br>
> <br>
> - In neutron-server side, use remote_managed tag in "openstack port create<br>
> ..." command.<br>
> This command will make neutron-server / OVN / ovn-controller / ovs to make<br>
> the network topology done, like above said.<br>
> I this this is right, because test shows that.<br>
that is not correct<br>
your test do not show what you think it does, they show the baisic bridge<br>
toplogy and flow configuraiton that ovn installs by defualt when it manages<br>
as ovs.<br>
<br>
please read the design docs for this feature for both nova and neutron to understand how the interacction works.<br>
<a href="https://specs.openstack.org/openstack/nova-specs/specs/yoga/implemented/integration-with-off-path-network-backends.html" rel="noreferrer" target="_blank">https://specs.openstack.org/openstack/nova-specs/specs/yoga/implemented/integration-with-off-path-network-backends.html</a><br>
<a href="https://specs.openstack.org/openstack/neutron-specs/specs/yoga/off-path-smartnic-dpu-port-binding-with-ovn.html" rel="noreferrer" target="_blank">https://specs.openstack.org/openstack/neutron-specs/specs/yoga/off-path-smartnic-dpu-port-binding-with-ovn.html</a><br>
> <br>
> - In nova side, there are 2 things should process, first is PCI passthrough<br>
> filter, second is nova-compute to plug VF into VM.<br>
> <br>
> If the link above is right, which remote_managed tag exists in<br>
> /etc/nova/nova.conf of controller node and exists in /etc/nova/nova.conf of<br>
> compute node.<br>
> As above ("- In my test, then I run "openstack server create" command")<br>
> said, got ERROR in this step.<br>
> So what should do in "PCI passthrough filter" ? How to configure ?<br>
> <br>
> Then, if "PCI passthrough filter" stage pass, what will do of nova-compute<br>
> in compute node?<br>
> <br>
> 3) Post all what I do follow this link:<br>
> <a href="https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html</a>.<br>
> - build openstack physical env, link plug DPU into compute mode, use VM as<br>
> controller ... etc.<br>
> - build openstack nova, neutron, ovn, ovn-vif, ovs follow that link.<br>
> - configure DPU side /etc/neutron/neutron.conf<br>
> - configure host side /etc/nova/nova.conf<br>
> - configure host side /etc/nova/nova-compute.conf<br>
> - run first 3 command<br>
> - last, run this command, got ERROR<br>
> <br>
> ----<br>
> Simon Jones<br>
> <br>
> <br>
> Sean Mooney <<a href="mailto:smooney@redhat.com" target="_blank">smooney@redhat.com</a>> 于2023年3月1日周三 18:35写道：<br>
> <br>
> > On Wed, 2023-03-01 at 18:12 +0800, Simon Jones wrote:<br>
> > > Thanks a lot !!!<br>
> > > <br>
> > > As you say, I follow<br>
> > > <a href="https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html</a>.<br>
> > > And I want to use DPU mode. Not "disable DPU mode".<br>
> > > So I think I should follow the link above exactlly, so I use<br>
> > > vnic-type=remote_anaged.<br>
> > > In my opnion, after I run first three command (which is "openstack<br>
> > network<br>
> > > create ...", "openstack subnet create", "openstack port create ..."), the<br>
> > > VF rep port and OVN and OVS rules are all ready.<br>
> > not at that point nothign will have been done on ovn/ovs<br>
> > <br>
> > that will only happen after the port is bound to a vm and host.<br>
> > <br>
> > > What I should do in "openstack server create ..." is to JUST add PCI<br>
> > device<br>
> > > into VM, do NOT call neutron-server in nova-compute of compute node (<br>
> > like<br>
> > > call port_binding or something).<br>
> > this is incorrect.<br>
> > > <br>
> > > But as the log and steps said in the emails above, nova-compute call<br>
> > > port_binding to neutron-server while running the command "openstack<br>
> > server<br>
> > > create ...".<br>
> > > <br>
> > > So I still have questions is:<br>
> > > 1) Is my opinion right? Which is "JUST add PCI device into VM, do NOT<br>
> > call<br>
> > > neutron-server in nova-compute of compute node ( like call port_binding<br>
> > or<br>
> > > something)" .<br>
> > no this is not how its designed.<br>
> > until you attach the logical port to a vm (either at runtime or as part of<br>
> > vm create)<br>
> > the logical port is not assocated with any host or phsical dpu/vf.<br>
> > <br>
> > so its not possibel to instanciate the openflow rules in ovs form the<br>
> > logical switch model<br>
> > in the ovn north db as no chassie info has been populated and we do not<br>
> > have the dpu serial<br>
> > info in the port binding details.<br>
> > > 2) If it's right, how to deal with this? Which is how to JUST add PCI<br>
> > > device into VM, do NOT call neutron-server? By command or by configure?<br>
> > Is<br>
> > > there come document ?<br>
> > no this happens automaticaly when nova does the port binding which cannot<br>
> > happen until after<br>
> > teh vm is schduled to a host.<br>
> > > <br>
> > > ----<br>
> > > Simon Jones<br>
> > > <br>
> > > <br>
> > > Sean Mooney <<a href="mailto:smooney@redhat.com" target="_blank">smooney@redhat.com</a>> 于2023年3月1日周三 16:15写道：<br>
> > > <br>
> > > > On Wed, 2023-03-01 at 15:20 +0800, Simon Jones wrote:<br>
> > > > > BTW, this link (<br>
> > > > > <br>
> > <a href="https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html</a>)<br>
> > > > said<br>
> > > > > I SHOULD add "remote_managed" in /etc/nova/nova.conf, is that WRONG ?<br>
> > > > <br>
> > > > no its not wrong but for dpu smart nics you have to make a choice when<br>
> > you<br>
> > > > deploy<br>
> > > > either they can be used in dpu mode in which case remote_managed<br>
> > shoudl be<br>
> > > > set to true<br>
> > > > and you can only use them via neutron ports with<br>
> > vnic-type=remote_managed<br>
> > > > as descried in that doc<br>
> > > > <br>
> > > > <br>
> > <a href="https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html#launch-an-instance-with-remote-managed-port" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html#launch-an-instance-with-remote-managed-port</a><br>
> > > > <br>
> > > > <br>
> > > > or if you disable dpu mode in the nic frimware then you shoudl remvoe<br>
> > > > remote_managed form the pci device list and<br>
> > > > then it can be used liek a normal vf either for neutron sriov ports<br>
> > > > vnic-type=direct or via flavor based pci passthough.<br>
> > > > <br>
> > > > the issue you were havign is you configured the pci device list to<br>
> > contain<br>
> > > > "remote_managed: ture" which means<br>
> > > > the vf can only be consumed by a neutron port with<br>
> > > > vnic-type=remote_managed, when you have "remote_managed: false" or<br>
> > unset<br>
> > > > you can use it via vnic-type=direct i forgot that slight detail that<br>
> > > > vnic-type=remote_managed is required for "remote_managed: ture".<br>
> > > > <br>
> > > > <br>
> > > > in either case you foudn the correct doc<br>
> > > > <a href="https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html</a><br>
> > > > neutorn sriov port configuration is documented here<br>
> > > > <a href="https://docs.openstack.org/neutron/latest/admin/config-sriov.html" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/admin/config-sriov.html</a><br>
> > > > and nova flavor based pci passthough is documeted here<br>
> > > > <a href="https://docs.openstack.org/nova/latest/admin/pci-passthrough.html" rel="noreferrer" target="_blank">https://docs.openstack.org/nova/latest/admin/pci-passthrough.html</a><br>
> > > > <br>
> > > > all three server slightly differnt uses. both neutron proceedures are<br>
> > > > exclusivly fo network interfaces.<br>
> > > > <a href="https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html</a><br>
> > > > requires the use of ovn deployed on the dpu<br>
> > > > to configure the VF contolplane.<br>
> > > > <a href="https://docs.openstack.org/neutron/latest/admin/config-sriov.html" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/admin/config-sriov.html</a> uses<br>
> > > > the sriov nic agent<br>
> > > > to manage the VF with ip tools.<br>
> > > > <a href="https://docs.openstack.org/nova/latest/admin/pci-passthrough.html" rel="noreferrer" target="_blank">https://docs.openstack.org/nova/latest/admin/pci-passthrough.html</a> is<br>
> > > > intended for pci passthough<br>
> > > > of stateless acclerorators like qat devices. while the nova flavor<br>
> > approch<br>
> > > > cna be used with nics it not how its generally<br>
> > > > ment to be used and when used to passthough a nic expectation is that<br>
> > its<br>
> > > > not related to a neuton network.<br>
> > > > <br>
> > > > <br>
> > <br>
> > <br>
<br>
</blockquote></div></div>