Open Stack

Hi John,

Great to hear from you on Cinder with pcipassthrough. I thought that it
would be coming. I like the idea.

thanks,
Robert

On 10/29/13 6:46 AM, "John Garbutt" <john at johngarbutt.com> wrote:

>I would love to see a symmetry between Cinder local volumes and
>Neutron PCI passthrough VIFs.
>
>Not entirely sure I have that clear in my head right now, but I just
>wanted to share the idea:
>* describe resource external to nova that is attached to VM in the API
>(block device mapping and/or vif references)
>* ideally the nova scheduler needs to be aware of the local capacity,
>and how that relates to the above information (relates to the cross
>service scheduling issues)
>* state of the device should be stored by Neutron/Cinder
>(attached/detached, capacity, IP, etc), but still exposed to the
>"scheduler"
>* connection params get given to Nova from Neutron/Cinder
>* nova still has the vif driver or volume driver to make the final
>connection
>* the disk should be formatted/expanded, and network info injected in
>the same way as before (cloud-init, config drive, DHCP, etc)
>
>John
>
>On 29 October 2013 10:17, Irena Berezovsky <irenab at mellanox.com> wrote:
>> Hi Jiang, Robert,
>>
>> IRC meeting option works for me.
>>
>> If I understand your question below, you are looking for a way to tie up
>> between requested virtual network(s) and requested PCI device(s). The
>>way we
>> did it in our solution  is to map a provider:physical_network to an
>> interface that represents the Physical Function. Every virtual network
>>is
>> bound to the provider:physical_network, so the PCI device should be
>> allocated based on this mapping.  We can  map a PCI alias to the
>> provider:physical_network.
>>
>>
>>
>> Another topic to discuss is where the mapping between neutron port and
>>PCI
>> device should be managed. One way to solve it, is to propagate the
>>allocated
>> PCI device details to neutron on port creation.
>>
>> In case  there is no qbg/qbh support, VF networking configuration
>>should be
>> applied locally on the Host.
>>
>> The question is when and how to apply networking configuration on the
>>PCI
>> device?
>>
>> We see the following options:
>>
>> ·         it can be done on port creation.
>>
>> ·         It can be done when nova VIF driver is called for vNIC
>>plugging.
>> This will require to  have all networking configuration available to
>>the VIF
>> driver or send request to the neutron server to obtain it.
>>
>> ·         It can be done by  having a dedicated L2 neutron agent on each
>> Host that scans for allocated PCI devices  and then retrieves networking
>> configuration from the server and configures the device. The agent will
>>be
>> also responsible for managing update requests coming from the neutron
>> server.
>>
>>
>>
>> For macvtap vNIC type assignment, the networking configuration can be
>> applied by a dedicated L2 neutron agent.
>>
>>
>>
>> BR,
>>
>> Irena
>>
>>
>>
>> From: Jiang, Yunhong [mailto:yunhong.jiang at intel.com]
>> Sent: Tuesday, October 29, 2013 9:04 AM
>>
>>
>> To: Robert Li (baoli); Irena Berezovsky; prashant.upadhyaya at aricent.com;
>> chris.friesen at windriver.com; He, Yongli; Itzik Brown
>>
>>
>> Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle
>>Mestery
>> (kmestery); Sandhya Dasu (sadasu)
>> Subject: RE: [openstack-dev] [nova] [neutron] PCI pass-through network
>> support
>>
>>
>>
>> Robert, is it possible to have a IRC meeting? I¹d prefer to IRC meeting
>> because it¹s more openstack style and also can keep the minutes clearly.
>>
>>
>>
>> To your flow, can you give more detailed example. For example, I can
>> consider user specify the instance with ­nic option specify a network
>>id,
>> and then how nova device the requirement to the PCI device? I assume the
>> network id should define the switches that the device can connect to ,
>>but
>> how is that information translated to the PCI property requirement? Will
>> this translation happen before the nova scheduler make host decision?
>>
>>
>>
>> Thanks
>>
>> --jyh
>>
>>
>>
>> From: Robert Li (baoli) [mailto:baoli at cisco.com]
>> Sent: Monday, October 28, 2013 12:22 PM
>> To: Irena Berezovsky; prashant.upadhyaya at aricent.com; Jiang, Yunhong;
>> chris.friesen at windriver.com; He, Yongli; Itzik Brown
>> Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle
>>Mestery
>> (kmestery); Sandhya Dasu (sadasu)
>> Subject: Re: [openstack-dev] [nova] [neutron] PCI pass-through network
>> support
>>
>>
>>
>> Hi Irena,
>>
>>
>>
>> Thank you very much for your comments. See inline.
>>
>>
>>
>> --Robert
>>
>>
>>
>> On 10/27/13 3:48 AM, "Irena Berezovsky" <irenab at mellanox.com> wrote:
>>
>>
>>
>> Hi Robert,
>>
>> Thank you very much for sharing the information regarding your efforts.
>>Can
>> you please share your idea of the end to end flow? How do you suggest
>>to
>> bind Nova and Neutron?
>>
>>
>>
>> The end to end flow is actually encompassed in the blueprints in a
>>nutshell.
>> I will reiterate it in below. The binding between Nova and Neutron
>>occurs
>> with the neutron v2 API that nova invokes in order to provision the
>>neutron
>> services. The vif driver is responsible for plugging in an instance
>>onto the
>> networking setup that neutron has created on the host.
>>
>>
>>
>> Normally, one will invoke "nova boot" api with the ‹nic options to
>>specify
>> the nic with which the instance will be connected to the network. It
>> currently allows net-id, fixed ip and/or port-id to be specified for the
>> option. However, it doesn't allow one to specify special networking
>> requirements for the instance. Thanks to the nova pci-passthrough work,
>>one
>> can specify PCI passthrough device(s) in the nova flavor. But it doesn't
>> provide means to tie up these PCI devices in the case of ethernet
>>adpators
>> with networking services. Therefore the idea is actually simple as
>>indicated
>> by the blueprint titles, to provide means to tie up SRIOV devices with
>> neutron services. A work flow would roughly look like this for 'nova
>>boot':
>>
>>
>>
>>       -- Specifies networking requirements in the ‹nic option.
>>Specifically
>> for SRIOV, allow the following to be specified in addition to the
>>existing
>> required information:
>>
>>                . PCI alias
>>
>>                . direct pci-passthrough/macvtap
>>
>>                . port profileid that is compliant with 802.1Qbh
>>
>>
>>
>>         The above information is optional. In the absence of them, the
>> existing behavior remains.
>>
>>
>>
>>      -- if special networking requirements exist, Nova api creates PCI
>> requests in the nova instance type for scheduling purpose
>>
>>
>>
>>      -- Nova scheduler schedules the instance based on the requested
>>flavor
>> plus the PCI requests that are created for networking.
>>
>>
>>
>>      -- Nova compute invokes neutron services with PCI passthrough
>> information if any
>>
>>
>>
>>      --  Neutron performs its normal operations based on the request,
>>such
>> as allocating a port, assigning ip addresses, etc. Specific to SRIOV, it
>> should validate the information such as profileid, and stores them in
>>its
>> db. It's also possible to associate a port profileid with a neutron
>>network
>> so that port profileid becomes optional in the ‹nic option. Neutron
>>returns
>> nova the port information, especially for PCI passthrough related
>> information in the port binding object. Currently, the port binding
>>object
>> contains the following information:
>>
>>           binding:vif_type
>>
>>           binding:host_id
>>
>>           binding:profile
>>
>>           binding:capabilities
>>
>>
>>
>>     -- nova constructs the domain xml and plug in the instance by
>>calling
>> the vif driver. The vif driver can build up the interface xml based on
>>the
>> port binding information.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> The blueprints you registered make sense. On Nova side, there is a need
>>to
>> bind between requested virtual network and PCI device/interface to be
>> allocated as vNIC.
>>
>> On the Neutron side, there is a need to  support networking
>>configuration of
>> the vNIC. Neutron should be able to identify the PCI device/macvtap
>> interface in order to apply configuration. I think it makes sense to
>>provide
>> neutron integration via dedicated Modular Layer 2 Mechanism Driver to
>>allow
>> PCI pass-through vNIC support along with other networking technologies.
>>
>>
>>
>> I haven't sorted through this yet. A neutron port could be associated
>>with a
>> PCI device or not, which is a common feature, IMHO. However, a ML2
>>driver
>> may be needed specific to a particular SRIOV technology.
>>
>>
>>
>>
>>
>> During the Havana Release, we introduced Mellanox Neutron plugin that
>> enables networking via SRIOV pass-through devices or macvtap interfaces.
>>
>> We want to integrate our solution with PCI pass-through Nova support.  I
>> will be glad to share more details if you are interested.
>>
>>
>>
>>
>>
>> Good to know that you already have a SRIOV implementation. I found out
>>some
>> information online about the mlnx plugin, but need more time to get to
>>know
>> it better. And certainly I'm interested in knowing its details.
>>
>>
>>
>> The PCI pass-through networking support is planned to be discussed
>>during
>> the summit: http://summit.openstack.org/cfp/details/129. I think it¹s
>>worth
>> to drill down into more detailed proposal and present it during the
>>summit,
>> especially since it impacts both nova and neutron projects.
>>
>>
>>
>> I agree. Maybe we can steal some time in that discussion.
>>
>>
>>
>> Would you be interested in collaboration on this effort? Would you be
>> interested to exchange more emails or set an IRC/WebEx meeting during
>>this
>> week before the summit?
>>
>>
>>
>> Sure. If folks want to discuss it before the summit, we can schedule a
>>webex
>> later this week. Or otherwise, we can continue the discussion with
>>email.
>>
>>
>>
>>
>>
>>
>>
>> Regards,
>>
>> Irena
>>
>>
>>
>> From: Robert Li (baoli) [mailto:baoli at cisco.com]
>> Sent: Friday, October 25, 2013 11:16 PM
>> To: prashant.upadhyaya at aricent.com; Irena Berezovsky;
>> yunhong.jiang at intel.com; chris.friesen at windriver.com;
>>yongli.he at intel.com
>> Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle
>>Mestery
>> (kmestery); Sandhya Dasu (sadasu)
>> Subject: Re: [openstack-dev] [nova] [neutron] PCI pass-through network
>> support
>>
>>
>>
>> Hi Irena,
>>
>>
>>
>> This is Robert Li from Cisco Systems. Recently, I was tasked to
>>investigate
>> such support for Cisco's systems that support VM-FEX, which is a SRIOV
>> technology supporting 802-1Qbh. I was able to bring up nova instances
>>with
>> SRIOV interfaces, and establish networking in between the instances that
>> employes the SRIOV interfaces. Certainly, this was accomplished with
>>hacking
>> and some manual intervention. Based on this experience and my study
>>with the
>> two existing nova pci-passthrough blueprints that have been implemented
>>and
>> committed into Havana
>> (https://blueprints.launchpad.net/nova/+spec/pci-passthrough-base and
>> https://blueprints.launchpad.net/nova/+spec/pci-passthrough-libvirt),  I
>> registered a couple of blueprints (one on Nova side, the other on the
>> Neutron side):
>>
>>
>>
>> https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov
>>
>> https://blueprints.launchpad.net/neutron/+spec/pci-passthrough-sriov
>>
>>
>>
>> in order to address SRIOV support in openstack.
>>
>>
>>
>> Please take a look at them and see if they make sense, and let me know
>>any
>> comments and questions. We can also discuss this in the summit, I
>>suppose.
>>
>>
>>
>> I noticed that there is another thread on this topic, so copy those
>>folks
>> from that thread as well.
>>
>>
>>
>> thanks,
>>
>> Robert
>>
>>
>>
>> On 10/16/13 4:32 PM, "Irena Berezovsky" <irenab at mellanox.com> wrote:
>>
>>
>>
>> Hi,
>>
>> As one of the next steps for PCI pass-through I would like to discuss
>>is the
>> support for PCI pass-through vNIC.
>>
>> While nova takes care of PCI pass-through device resources  management
>>and
>> VIF settings, neutron should manage their networking configuration.
>>
>> I would like to register asummit proposal to discuss the support for PCI
>> pass-through networking.
>>
>> I am not sure what would be the right topic to discuss the PCI
>>pass-through
>> networking, since it involve both nova and neutron.
>>
>> There is already a session registered by Yongli on nova topic to
>>discuss the
>> PCI pass-through next steps.
>>
>> I think PCI pass-through networking is quite a big topic and it worth to
>> have a separate discussion.
>>
>> Is there any other people who are interested to discuss it and share
>>their
>> thoughts and experience?
>>
>>
>>
>> Regards,
>>
>> Irena
>>
>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>