Open Stack

On Tue, 2018-11-13 at 07:52 -0500, Satish Patel wrote:
> Mike,
> 
> Here is the bug which I reported https://bugs.launchpad.net/bugs/1795920
actully this is a releated but different bug based in the description below.
thanks for highlighting this to me.
> 
> Cc'ing: Sean 
> 
> Sent from my iPhone
> 
> On Nov 12, 2018, at 8:27 AM, Satish Patel <satish.txt at gmail.com> wrote:
> 
> > Mike,
> > 
> > I had same issue month ago when I roll out sriov in my cloud and this is what I did to solve this issue. Set
> > following in flavor 
> > 
> > hw:numa_nodes=2
> > 
> > It will spread out instance vcpu across numa, yes there will be little penalty but if you tune your application
> > according they you are good 
> > 
> > Yes this is bug I have already open ticket and I believe folks are working on it but its not simple fix. They may
> > release new feature in coming oprnstack release. 
> > 
> > Sent from my iPhone
> > 
> > On Nov 11, 2018, at 9:25 PM, Mike Joseph <mj at mode.net> wrote:
> > 
> > > Hi folks,
> > > 
> > > It appears that the numa_policy attribute of a PCI alias is ignored for flavors referencing that alias if the
> > > flavor also has hw:cpu_policy=dedicated set.  The alias config is:
> > > 
> > > alias = { "name": "mlx", "device_type": "type-VF", "vendor_id": "15b3", "product_id": "1004", "numa_policy":
> > > "preferred" }
> > > 
> > > And the flavor config is:
> > > 
> > > {
> > >   "OS-FLV-DISABLED:disabled": false,
> > >   "OS-FLV-EXT-DATA:ephemeral": 0,
> > >   "access_project_ids": null,
> > >   "disk": 10,
> > >   "id": "221e1bcd-2dde-48e6-bd09-820012198908",
> > >   "name": "vm-2",
> > >   "os-flavor-access:is_public": true,
> > >   "properties": "hw:cpu_policy='dedicated', pci_passthrough:alias='mlx:1'",
> > >   "ram": 8192,
> > >   "rxtx_factor": 1.0,
> > >   "swap": "",
> > >   "vcpus": 2
> > > }
Satish in your case you were trying to use neutrons sriov vnic types such that the VF would be connected to a neutron
network. In this case the mellanox connectx 3 virtual funcitons are being passed to the guest using the pci alias via
the flavor which means they cannot be used to connect to neutron networks but they should be able to use affinity
poileices. 
> > > 
> > > In short, our compute nodes have an SR-IOV Mellanox NIC (ConnectX-3) with 16 VFs configured.  We wish to expose
> > > these VFs to VMs that schedule on the host.  However, the NIC is in NUMA region 0 which means that only half of
> > > the compute node's CPU cores would be usable if we required VM affinity to the NIC's NUMA region.  But we don't
> > > need that, since we are okay with cross-region access to the PCI device.
> > > 
> > > However, we do need CPU pinning to work, in order to have efficient cache hits on our VM processes.  Therefore, we
> > > still want to pin our vCPUs to pCPUs, even if the pins end up on on a NUMA region opposite of the NIC.  The spec
> > > for numa_policy seem to indicate that this is exactly the intent of the option:
> > > 
> > > https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/share-pci-between-numa-nodes.html
> > > 
> > > But, with the above config, we still get PCI affinity scheduling errors:
> > > 
> > > 'Insufficient compute resources: Requested instance NUMA topology together with requested PCI devices cannot fit
> > > the given host NUMA topology.'
> > > 
> > > This strikes me as a bug, but perhaps I am missing something here?
yes this does infact seam like a new bug.
can you add myself and stephen to the bug once you file it.
in the bug please include the version of opentack you were deploying.

in the interim setting hw:numa_nodes=2 will allow you to pin the guest without the error
however the flavor and alias you have provided should have been enough.

im hoping that we can fix both the alisa and neutorn based case this cycle but to do so we
will need to reporpose original queens spec for stein and disucss if we can backport any of the
fixes or if this would be only completed in stein+ i would hope we coudl backport fixes for the flavor
based use case but the neutron based sue case would likely be stein+

regards
sean
> > > 
> > > Thanks,
> > > MJ
> > > _______________________________________________
> > > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> > > Post to     : openstack at lists.openstack.org
> > > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack