<div dir="ltr"><div dir="ltr"><div dir="ltr"><div>E...</div><div><br></div><div>But there are these things:</div><div><br></div><div>1) Show some real happened in my test:<br></div><div><br></div><div>- Let me clear that, I use DPU in compute node:</div>The graph in <a href="https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html">https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html</a> .</div><div dir="ltr"><br></div><div>- I configure exactly follow <a href="https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html">https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html</a>, which is said bellow in "3) Let me post all what I do follow this link".<br></div><div dir="ltr"><div><br></div><div>- In my test, I found after <span>first three command (which is "openstack network create ...", "openstack subnet create", "openstack port create ...")</span>, there are network topology exist in DPU side, and there are rules exist in OVN north DB, south DB of controller, like this:</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>```</div><div>root@c1:~# ovn-nbctl show<br>switch 9bdacdd4-ca2a-4e35-82ca-0b5fbd3a5976 (neutron-066c8dc2-c98b-4fb8-a541-8b367e8f6e69) (aka selfservice)<br> port 01a68701-0e6a-4c30-bfba-904d1b9813e1<br> addresses: ["unknown"]<br> port 18a44c6f-af50-4830-ba86-54865abb60a1 (aka pf0vf1)<br> addresses: ["fa:16:3e:13:36:e2 172.1.1.228"]<br><br>gyw@c1:~$ sudo ovn-sbctl list Port_Binding<br>_uuid : 61dc8bc0-ab33-4d67-ac13-0781f89c905a<br>chassis : []<br>datapath : 91d3509c-d794-496a-ba11-3706ebf143c8<br>encap : []<br>external_ids : {name=pf0vf1, "neutron:cidrs"="<a href="http://172.1.1.241/24">172.1.1.241/24</a>", "neutron:device_id"="", "neutron:device_owner"="", "neutron:network_name"=neutron-066c8dc2-c98b-4fb8-a541-8b367e8f6e69, "neutron:port_name"=pf0vf1, "neutron:project_id"="512866f9994f4ad8916d8539a7cdeec9", "neutron:revision_number"="1", "neutron:security_group_ids"="de8883e8-ccac-4be2-9bb2-95e732b0c114"}<br><br>root@c1c2dpu:~# sudo ovs-vsctl show<br>62cf78e5-2c02-471e-927e-1d69c2c22195<br> Bridge br-int<br> fail_mode: secure<br> datapath_type: system<br> Port br-int<br> Interface br-int<br> type: internal<br> Port ovn--1<br> Interface ovn--1<br> type: geneve<br> options: {csum="true", key=flow, remote_ip="172.168.2.98"}<br> Port pf0vf1<br> Interface pf0vf1<br> ovs_version: "2.17.2-24a81c8"</div><div>```</div></blockquote><div>That's why I guess "<span>first three command</span>" has already create network topology, and "openstack server create" command only need to plug VF into VM in HOST SIDE, DO NOT CALL NEUTRON. As network has already done.</div><div><br></div><div>- In my test, then I run "openstack server create" command, I got ERROR which said
"No valid host...", which is what the email said above.</div><div>The reason has already said, it's nova-scheduler's PCI filter module report no valid host. The reason "nova-scheduler's PCI filter module report no valid host" is nova-scheduler could NOT see PCI information of compute node. The reason "nova-scheduler could NOT see PCI information of compute node" is compute node's /etc/nova/nova.conf configure remote_managed tag like this:</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>```</div><div>[pci]<br>passthrough_whitelist = {"vendor_id": "15b3", "product_id": "101e", "physical_network": null, "remote_managed": "true"}<br>alias = { "vendor_id":"15b3", "product_id":"101e", "device_type":"type-VF", "name":"a1" }<br></div><div>```<br></div></blockquote><div><br></div><div>2) Discuss some detail design of "remote_managed" tag, I don't know if this is right in the design of openstack with DPU:</div><div><br></div><div>- In neutron-server side, use remote_managed tag in "openstack port create ..." command.</div><div>This command will make neutron-server / OVN / ovn-controller / ovs to make the network topology done, like above said.</div><div>I this this is right, because test shows that.<br></div><div><br></div><div>- In nova side, there are 2 things should process, first is PCI passthrough filter, second is nova-compute to plug VF into VM.</div><div><br></div><div>If the link above is right, which remote_managed tag exists in /etc/nova/nova.conf of controller node and exists in /etc/nova/nova.conf of compute node.</div><div>As above ("- In my test, then I run "openstack server create" command") said, got ERROR in this step.<br></div><div>So what should do in "PCI passthrough filter" ? How to configure ?</div><div><br></div>Then, if "PCI passthrough filter" stage pass, what will do of nova-compute in compute node?<br></div><div dir="ltr"><div><br></div><div>3) Post all what I do follow this link: <span><a href="https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html</a>.</span></div><div>- build openstack physical env, link plug DPU into compute mode, use VM as controller ... etc.</div><div>- build openstack nova, neutron, ovn, ovn-vif, ovs follow that link.</div><div>- configure DPU side /etc/neutron/neutron.conf</div><div>- configure host side /etc/nova/nova.conf</div><div>- configure host side /etc/nova/nova-compute.conf</div><div>- run first 3 command</div><div>- last, run this command, got ERROR<br></div><div><br></div><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div>----</div>Simon Jones</div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Sean Mooney <<a href="mailto:smooney@redhat.com" target="_blank">smooney@redhat.com</a>> 于2023年3月1日周三 18:35写道:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Wed, 2023-03-01 at 18:12 +0800, Simon Jones wrote:<br>
> Thanks a lot !!!<br>
> <br>
> As you say, I follow<br>
> <a href="https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html</a>.<br>
> And I want to use DPU mode. Not "disable DPU mode".<br>
> So I think I should follow the link above exactlly, so I use<br>
> vnic-type=remote_anaged.<br>
> In my opnion, after I run first three command (which is "openstack network<br>
> create ...", "openstack subnet create", "openstack port create ..."), the<br>
> VF rep port and OVN and OVS rules are all ready.<br>
not at that point nothign will have been done on ovn/ovs<br>
<br>
that will only happen after the port is bound to a vm and host.<br>
<br>
> What I should do in "openstack server create ..." is to JUST add PCI device<br>
> into VM, do NOT call neutron-server in nova-compute of compute node ( like<br>
> call port_binding or something).<br>
this is incorrect.<br>
> <br>
> But as the log and steps said in the emails above, nova-compute call<br>
> port_binding to neutron-server while running the command "openstack server<br>
> create ...".<br>
> <br>
> So I still have questions is:<br>
> 1) Is my opinion right? Which is "JUST add PCI device into VM, do NOT call<br>
> neutron-server in nova-compute of compute node ( like call port_binding or<br>
> something)" .<br>
no this is not how its designed.<br>
until you attach the logical port to a vm (either at runtime or as part of vm create)<br>
the logical port is not assocated with any host or phsical dpu/vf.<br>
<br>
so its not possibel to instanciate the openflow rules in ovs form the logical switch model<br>
in the ovn north db as no chassie info has been populated and we do not have the dpu serial<br>
info in the port binding details.<br>
> 2) If it's right, how to deal with this? Which is how to JUST add PCI<br>
> device into VM, do NOT call neutron-server? By command or by configure? Is<br>
> there come document ?<br>
no this happens automaticaly when nova does the port binding which cannot happen until after<br>
teh vm is schduled to a host.<br>
> <br>
> ----<br>
> Simon Jones<br>
> <br>
> <br>
> Sean Mooney <<a href="mailto:smooney@redhat.com" target="_blank">smooney@redhat.com</a>> 于2023年3月1日周三 16:15写道:<br>
> <br>
> > On Wed, 2023-03-01 at 15:20 +0800, Simon Jones wrote:<br>
> > > BTW, this link (<br>
> > > <a href="https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html</a>)<br>
> > said<br>
> > > I SHOULD add "remote_managed" in /etc/nova/nova.conf, is that WRONG ?<br>
> > <br>
> > no its not wrong but for dpu smart nics you have to make a choice when you<br>
> > deploy<br>
> > either they can be used in dpu mode in which case remote_managed shoudl be<br>
> > set to true<br>
> > and you can only use them via neutron ports with vnic-type=remote_managed<br>
> > as descried in that doc<br>
> > <br>
> > <a href="https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html#launch-an-instance-with-remote-managed-port" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html#launch-an-instance-with-remote-managed-port</a><br>
> > <br>
> > <br>
> > or if you disable dpu mode in the nic frimware then you shoudl remvoe<br>
> > remote_managed form the pci device list and<br>
> > then it can be used liek a normal vf either for neutron sriov ports<br>
> > vnic-type=direct or via flavor based pci passthough.<br>
> > <br>
> > the issue you were havign is you configured the pci device list to contain<br>
> > "remote_managed: ture" which means<br>
> > the vf can only be consumed by a neutron port with<br>
> > vnic-type=remote_managed, when you have "remote_managed: false" or unset<br>
> > you can use it via vnic-type=direct i forgot that slight detail that<br>
> > vnic-type=remote_managed is required for "remote_managed: ture".<br>
> > <br>
> > <br>
> > in either case you foudn the correct doc<br>
> > <a href="https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html</a><br>
> > neutorn sriov port configuration is documented here<br>
> > <a href="https://docs.openstack.org/neutron/latest/admin/config-sriov.html" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/admin/config-sriov.html</a><br>
> > and nova flavor based pci passthough is documeted here<br>
> > <a href="https://docs.openstack.org/nova/latest/admin/pci-passthrough.html" rel="noreferrer" target="_blank">https://docs.openstack.org/nova/latest/admin/pci-passthrough.html</a><br>
> > <br>
> > all three server slightly differnt uses. both neutron proceedures are<br>
> > exclusivly fo network interfaces.<br>
> > <a href="https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/admin/ovn/smartnic_dpu.html</a><br>
> > requires the use of ovn deployed on the dpu<br>
> > to configure the VF contolplane.<br>
> > <a href="https://docs.openstack.org/neutron/latest/admin/config-sriov.html" rel="noreferrer" target="_blank">https://docs.openstack.org/neutron/latest/admin/config-sriov.html</a> uses<br>
> > the sriov nic agent<br>
> > to manage the VF with ip tools.<br>
> > <a href="https://docs.openstack.org/nova/latest/admin/pci-passthrough.html" rel="noreferrer" target="_blank">https://docs.openstack.org/nova/latest/admin/pci-passthrough.html</a> is<br>
> > intended for pci passthough<br>
> > of stateless acclerorators like qat devices. while the nova flavor approch<br>
> > cna be used with nics it not how its generally<br>
> > ment to be used and when used to passthough a nic expectation is that its<br>
> > not related to a neuton network.<br>
> > <br>
> > <br>
<br>
</blockquote></div></div>