[openstack-dev] [nova] [neutron] PCI pass-through network support

John Garbutt john at johngarbutt.com
Tue Oct 29 10:46:51 UTC 2013


I would love to see a symmetry between Cinder local volumes and
Neutron PCI passthrough VIFs.

Not entirely sure I have that clear in my head right now, but I just
wanted to share the idea:
* describe resource external to nova that is attached to VM in the API
(block device mapping and/or vif references)
* ideally the nova scheduler needs to be aware of the local capacity,
and how that relates to the above information (relates to the cross
service scheduling issues)
* state of the device should be stored by Neutron/Cinder
(attached/detached, capacity, IP, etc), but still exposed to the
"scheduler"
* connection params get given to Nova from Neutron/Cinder
* nova still has the vif driver or volume driver to make the final connection
* the disk should be formatted/expanded, and network info injected in
the same way as before (cloud-init, config drive, DHCP, etc)

John

On 29 October 2013 10:17, Irena Berezovsky <irenab at mellanox.com> wrote:
> Hi Jiang, Robert,
>
> IRC meeting option works for me.
>
> If I understand your question below, you are looking for a way to tie up
> between requested virtual network(s) and requested PCI device(s). The way we
> did it in our solution  is to map a provider:physical_network to an
> interface that represents the Physical Function. Every virtual network is
> bound to the provider:physical_network, so the PCI device should be
> allocated based on this mapping.  We can  map a PCI alias to the
> provider:physical_network.
>
>
>
> Another topic to discuss is where the mapping between neutron port and PCI
> device should be managed. One way to solve it, is to propagate the allocated
> PCI device details to neutron on port creation.
>
> In case  there is no qbg/qbh support, VF networking configuration should be
> applied locally on the Host.
>
> The question is when and how to apply networking configuration on the PCI
> device?
>
> We see the following options:
>
> ·         it can be done on port creation.
>
> ·         It can be done when nova VIF driver is called for vNIC plugging.
> This will require to  have all networking configuration available to the VIF
> driver or send request to the neutron server to obtain it.
>
> ·         It can be done by  having a dedicated L2 neutron agent on each
> Host that scans for allocated PCI devices  and then retrieves networking
> configuration from the server and configures the device. The agent will be
> also responsible for managing update requests coming from the neutron
> server.
>
>
>
> For macvtap vNIC type assignment, the networking configuration can be
> applied by a dedicated L2 neutron agent.
>
>
>
> BR,
>
> Irena
>
>
>
> From: Jiang, Yunhong [mailto:yunhong.jiang at intel.com]
> Sent: Tuesday, October 29, 2013 9:04 AM
>
>
> To: Robert Li (baoli); Irena Berezovsky; prashant.upadhyaya at aricent.com;
> chris.friesen at windriver.com; He, Yongli; Itzik Brown
>
>
> Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle Mestery
> (kmestery); Sandhya Dasu (sadasu)
> Subject: RE: [openstack-dev] [nova] [neutron] PCI pass-through network
> support
>
>
>
> Robert, is it possible to have a IRC meeting? I’d prefer to IRC meeting
> because it’s more openstack style and also can keep the minutes clearly.
>
>
>
> To your flow, can you give more detailed example. For example, I can
> consider user specify the instance with –nic option specify a network id,
> and then how nova device the requirement to the PCI device? I assume the
> network id should define the switches that the device can connect to , but
> how is that information translated to the PCI property requirement? Will
> this translation happen before the nova scheduler make host decision?
>
>
>
> Thanks
>
> --jyh
>
>
>
> From: Robert Li (baoli) [mailto:baoli at cisco.com]
> Sent: Monday, October 28, 2013 12:22 PM
> To: Irena Berezovsky; prashant.upadhyaya at aricent.com; Jiang, Yunhong;
> chris.friesen at windriver.com; He, Yongli; Itzik Brown
> Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle Mestery
> (kmestery); Sandhya Dasu (sadasu)
> Subject: Re: [openstack-dev] [nova] [neutron] PCI pass-through network
> support
>
>
>
> Hi Irena,
>
>
>
> Thank you very much for your comments. See inline.
>
>
>
> --Robert
>
>
>
> On 10/27/13 3:48 AM, "Irena Berezovsky" <irenab at mellanox.com> wrote:
>
>
>
> Hi Robert,
>
> Thank you very much for sharing the information regarding your efforts. Can
> you please share your idea of the end to end flow? How do you suggest  to
> bind Nova and Neutron?
>
>
>
> The end to end flow is actually encompassed in the blueprints in a nutshell.
> I will reiterate it in below. The binding between Nova and Neutron occurs
> with the neutron v2 API that nova invokes in order to provision the neutron
> services. The vif driver is responsible for plugging in an instance onto the
> networking setup that neutron has created on the host.
>
>
>
> Normally, one will invoke "nova boot" api with the —nic options to specify
> the nic with which the instance will be connected to the network. It
> currently allows net-id, fixed ip and/or port-id to be specified for the
> option. However, it doesn't allow one to specify special networking
> requirements for the instance. Thanks to the nova pci-passthrough work, one
> can specify PCI passthrough device(s) in the nova flavor. But it doesn't
> provide means to tie up these PCI devices in the case of ethernet adpators
> with networking services. Therefore the idea is actually simple as indicated
> by the blueprint titles, to provide means to tie up SRIOV devices with
> neutron services. A work flow would roughly look like this for 'nova boot':
>
>
>
>       -- Specifies networking requirements in the —nic option. Specifically
> for SRIOV, allow the following to be specified in addition to the existing
> required information:
>
>                . PCI alias
>
>                . direct pci-passthrough/macvtap
>
>                . port profileid that is compliant with 802.1Qbh
>
>
>
>         The above information is optional. In the absence of them, the
> existing behavior remains.
>
>
>
>      -- if special networking requirements exist, Nova api creates PCI
> requests in the nova instance type for scheduling purpose
>
>
>
>      -- Nova scheduler schedules the instance based on the requested flavor
> plus the PCI requests that are created for networking.
>
>
>
>      -- Nova compute invokes neutron services with PCI passthrough
> information if any
>
>
>
>      --  Neutron performs its normal operations based on the request, such
> as allocating a port, assigning ip addresses, etc. Specific to SRIOV, it
> should validate the information such as profileid, and stores them in its
> db. It's also possible to associate a port profileid with a neutron network
> so that port profileid becomes optional in the —nic option. Neutron returns
> nova the port information, especially for PCI passthrough related
> information in the port binding object. Currently, the port binding object
> contains the following information:
>
>           binding:vif_type
>
>           binding:host_id
>
>           binding:profile
>
>           binding:capabilities
>
>
>
>     -- nova constructs the domain xml and plug in the instance by calling
> the vif driver. The vif driver can build up the interface xml based on the
> port binding information.
>
>
>
>
>
>
>
>
>
> The blueprints you registered make sense. On Nova side, there is a need to
> bind between requested virtual network and PCI device/interface to be
> allocated as vNIC.
>
> On the Neutron side, there is a need to  support networking configuration of
> the vNIC. Neutron should be able to identify the PCI device/macvtap
> interface in order to apply configuration. I think it makes sense to provide
> neutron integration via dedicated Modular Layer 2 Mechanism Driver to allow
> PCI pass-through vNIC support along with other networking technologies.
>
>
>
> I haven't sorted through this yet. A neutron port could be associated with a
> PCI device or not, which is a common feature, IMHO. However, a ML2 driver
> may be needed specific to a particular SRIOV technology.
>
>
>
>
>
> During the Havana Release, we introduced Mellanox Neutron plugin that
> enables networking via SRIOV pass-through devices or macvtap interfaces.
>
> We want to integrate our solution with PCI pass-through Nova support.  I
> will be glad to share more details if you are interested.
>
>
>
>
>
> Good to know that you already have a SRIOV implementation. I found out some
> information online about the mlnx plugin, but need more time to get to know
> it better. And certainly I'm interested in knowing its details.
>
>
>
> The PCI pass-through networking support is planned to be discussed during
> the summit: http://summit.openstack.org/cfp/details/129. I think it’s worth
> to drill down into more detailed proposal and present it during the summit,
> especially since it impacts both nova and neutron projects.
>
>
>
> I agree. Maybe we can steal some time in that discussion.
>
>
>
> Would you be interested in collaboration on this effort? Would you be
> interested to exchange more emails or set an IRC/WebEx meeting during this
> week before the summit?
>
>
>
> Sure. If folks want to discuss it before the summit, we can schedule a webex
> later this week. Or otherwise, we can continue the discussion with email.
>
>
>
>
>
>
>
> Regards,
>
> Irena
>
>
>
> From: Robert Li (baoli) [mailto:baoli at cisco.com]
> Sent: Friday, October 25, 2013 11:16 PM
> To: prashant.upadhyaya at aricent.com; Irena Berezovsky;
> yunhong.jiang at intel.com; chris.friesen at windriver.com; yongli.he at intel.com
> Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle Mestery
> (kmestery); Sandhya Dasu (sadasu)
> Subject: Re: [openstack-dev] [nova] [neutron] PCI pass-through network
> support
>
>
>
> Hi Irena,
>
>
>
> This is Robert Li from Cisco Systems. Recently, I was tasked to investigate
> such support for Cisco's systems that support VM-FEX, which is a SRIOV
> technology supporting 802-1Qbh. I was able to bring up nova instances with
> SRIOV interfaces, and establish networking in between the instances that
> employes the SRIOV interfaces. Certainly, this was accomplished with hacking
> and some manual intervention. Based on this experience and my study with the
> two existing nova pci-passthrough blueprints that have been implemented and
> committed into Havana
> (https://blueprints.launchpad.net/nova/+spec/pci-passthrough-base and
> https://blueprints.launchpad.net/nova/+spec/pci-passthrough-libvirt),  I
> registered a couple of blueprints (one on Nova side, the other on the
> Neutron side):
>
>
>
> https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov
>
> https://blueprints.launchpad.net/neutron/+spec/pci-passthrough-sriov
>
>
>
> in order to address SRIOV support in openstack.
>
>
>
> Please take a look at them and see if they make sense, and let me know any
> comments and questions. We can also discuss this in the summit, I suppose.
>
>
>
> I noticed that there is another thread on this topic, so copy those folks
> from that thread as well.
>
>
>
> thanks,
>
> Robert
>
>
>
> On 10/16/13 4:32 PM, "Irena Berezovsky" <irenab at mellanox.com> wrote:
>
>
>
> Hi,
>
> As one of the next steps for PCI pass-through I would like to discuss is the
> support for PCI pass-through vNIC.
>
> While nova takes care of PCI pass-through device resources  management and
> VIF settings, neutron should manage their networking configuration.
>
> I would like to register asummit proposal to discuss the support for PCI
> pass-through networking.
>
> I am not sure what would be the right topic to discuss the PCI pass-through
> networking, since it involve both nova and neutron.
>
> There is already a session registered by Yongli on nova topic to discuss the
> PCI pass-through next steps.
>
> I think PCI pass-through networking is quite a big topic and it worth to
> have a separate discussion.
>
> Is there any other people who are interested to discuss it and share their
> thoughts and experience?
>
>
>
> Regards,
>
> Irena
>
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



More information about the OpenStack-dev mailing list