Open Stack

> * describe resource external to nova that is attached to VM in the API
> (block device mapping and/or vif references)
> * ideally the nova scheduler needs to be aware of the local capacity,
> and how that relates to the above information (relates to the cross
> service scheduling issues)

I think this possibly a bit different. For volume, it's sure managed by Cinder, but for PCI devices, currently
It ;s managed by nova. So we possibly need nova to translate the information (possibly before nova scheduler).

> * state of the device should be stored by Neutron/Cinder
> (attached/detached, capacity, IP, etc), but still exposed to the
> "scheduler"

I'm not sure if we can keep the state of the device in Neutron. Currently nova manage all PCI devices.

Thanks
--jyh


> * connection params get given to Nova from Neutron/Cinder
> * nova still has the vif driver or volume driver to make the final connection
> * the disk should be formatted/expanded, and network info injected in
> the same way as before (cloud-init, config drive, DHCP, etc)
> 
> John
> 
> On 29 October 2013 10:17, Irena Berezovsky <irenab at mellanox.com>
> wrote:
> > Hi Jiang, Robert,
> >
> > IRC meeting option works for me.
> >
> > If I understand your question below, you are looking for a way to tie up
> > between requested virtual network(s) and requested PCI device(s). The
> way we
> > did it in our solution  is to map a provider:physical_network to an
> > interface that represents the Physical Function. Every virtual network is
> > bound to the provider:physical_network, so the PCI device should be
> > allocated based on this mapping.  We can  map a PCI alias to the
> > provider:physical_network.
> >
> >
> >
> > Another topic to discuss is where the mapping between neutron port
> and PCI
> > device should be managed. One way to solve it, is to propagate the
> allocated
> > PCI device details to neutron on port creation.
> >
> > In case  there is no qbg/qbh support, VF networking configuration
> should be
> > applied locally on the Host.
> >
> > The question is when and how to apply networking configuration on the
> PCI
> > device?
> >
> > We see the following options:
> >
> > *         it can be done on port creation.
> >
> > *         It can be done when nova VIF driver is called for vNIC
> plugging.
> > This will require to  have all networking configuration available to the
> VIF
> > driver or send request to the neutron server to obtain it.
> >
> > *         It can be done by  having a dedicated L2 neutron agent on
> each
> > Host that scans for allocated PCI devices  and then retrieves networking
> > configuration from the server and configures the device. The agent will
> be
> > also responsible for managing update requests coming from the neutron
> > server.
> >
> >
> >
> > For macvtap vNIC type assignment, the networking configuration can be
> > applied by a dedicated L2 neutron agent.
> >
> >
> >
> > BR,
> >
> > Irena
> >
> >
> >
> > From: Jiang, Yunhong [mailto:yunhong.jiang at intel.com]
> > Sent: Tuesday, October 29, 2013 9:04 AM
> >
> >
> > To: Robert Li (baoli); Irena Berezovsky;
> prashant.upadhyaya at aricent.com;
> > chris.friesen at windriver.com; He, Yongli; Itzik Brown
> >
> >
> > Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle
> Mestery
> > (kmestery); Sandhya Dasu (sadasu)
> > Subject: RE: [openstack-dev] [nova] [neutron] PCI pass-through network
> > support
> >
> >
> >
> > Robert, is it possible to have a IRC meeting? I'd prefer to IRC meeting
> > because it's more openstack style and also can keep the minutes
> clearly.
> >
> >
> >
> > To your flow, can you give more detailed example. For example, I can
> > consider user specify the instance with -nic option specify a network id,
> > and then how nova device the requirement to the PCI device? I assume
> the
> > network id should define the switches that the device can connect to ,
> but
> > how is that information translated to the PCI property requirement? Will
> > this translation happen before the nova scheduler make host decision?
> >
> >
> >
> > Thanks
> >
> > --jyh
> >
> >
> >
> > From: Robert Li (baoli) [mailto:baoli at cisco.com]
> > Sent: Monday, October 28, 2013 12:22 PM
> > To: Irena Berezovsky; prashant.upadhyaya at aricent.com; Jiang, Yunhong;
> > chris.friesen at windriver.com; He, Yongli; Itzik Brown
> > Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle
> Mestery
> > (kmestery); Sandhya Dasu (sadasu)
> > Subject: Re: [openstack-dev] [nova] [neutron] PCI pass-through network
> > support
> >
> >
> >
> > Hi Irena,
> >
> >
> >
> > Thank you very much for your comments. See inline.
> >
> >
> >
> > --Robert
> >
> >
> >
> > On 10/27/13 3:48 AM, "Irena Berezovsky" <irenab at mellanox.com>
> wrote:
> >
> >
> >
> > Hi Robert,
> >
> > Thank you very much for sharing the information regarding your efforts.
> Can
> > you please share your idea of the end to end flow? How do you suggest
> to
> > bind Nova and Neutron?
> >
> >
> >
> > The end to end flow is actually encompassed in the blueprints in a
> nutshell.
> > I will reiterate it in below. The binding between Nova and Neutron
> occurs
> > with the neutron v2 API that nova invokes in order to provision the
> neutron
> > services. The vif driver is responsible for plugging in an instance onto the
> > networking setup that neutron has created on the host.
> >
> >
> >
> > Normally, one will invoke "nova boot" api with the -nic options to specify
> > the nic with which the instance will be connected to the network. It
> > currently allows net-id, fixed ip and/or port-id to be specified for the
> > option. However, it doesn't allow one to specify special networking
> > requirements for the instance. Thanks to the nova pci-passthrough work,
> one
> > can specify PCI passthrough device(s) in the nova flavor. But it doesn't
> > provide means to tie up these PCI devices in the case of ethernet
> adpators
> > with networking services. Therefore the idea is actually simple as
> indicated
> > by the blueprint titles, to provide means to tie up SRIOV devices with
> > neutron services. A work flow would roughly look like this for 'nova
> boot':
> >
> >
> >
> >       -- Specifies networking requirements in the -nic option.
> Specifically
> > for SRIOV, allow the following to be specified in addition to the existing
> > required information:
> >
> >                . PCI alias
> >
> >                . direct pci-passthrough/macvtap
> >
> >                . port profileid that is compliant with 802.1Qbh
> >
> >
> >
> >         The above information is optional. In the absence of them, the
> > existing behavior remains.
> >
> >
> >
> >      -- if special networking requirements exist, Nova api creates PCI
> > requests in the nova instance type for scheduling purpose
> >
> >
> >
> >      -- Nova scheduler schedules the instance based on the requested
> flavor
> > plus the PCI requests that are created for networking.
> >
> >
> >
> >      -- Nova compute invokes neutron services with PCI passthrough
> > information if any
> >
> >
> >
> >      --  Neutron performs its normal operations based on the request,
> such
> > as allocating a port, assigning ip addresses, etc. Specific to SRIOV, it
> > should validate the information such as profileid, and stores them in its
> > db. It's also possible to associate a port profileid with a neutron network
> > so that port profileid becomes optional in the -nic option. Neutron
> returns
> > nova the port information, especially for PCI passthrough related
> > information in the port binding object. Currently, the port binding object
> > contains the following information:
> >
> >           binding:vif_type
> >
> >           binding:host_id
> >
> >           binding:profile
> >
> >           binding:capabilities
> >
> >
> >
> >     -- nova constructs the domain xml and plug in the instance by
> calling
> > the vif driver. The vif driver can build up the interface xml based on the
> > port binding information.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > The blueprints you registered make sense. On Nova side, there is a need
> to
> > bind between requested virtual network and PCI device/interface to be
> > allocated as vNIC.
> >
> > On the Neutron side, there is a need to  support networking
> configuration of
> > the vNIC. Neutron should be able to identify the PCI device/macvtap
> > interface in order to apply configuration. I think it makes sense to
> provide
> > neutron integration via dedicated Modular Layer 2 Mechanism Driver to
> allow
> > PCI pass-through vNIC support along with other networking technologies.
> >
> >
> >
> > I haven't sorted through this yet. A neutron port could be associated
> with a
> > PCI device or not, which is a common feature, IMHO. However, a ML2
> driver
> > may be needed specific to a particular SRIOV technology.
> >
> >
> >
> >
> >
> > During the Havana Release, we introduced Mellanox Neutron plugin that
> > enables networking via SRIOV pass-through devices or macvtap
> interfaces.
> >
> > We want to integrate our solution with PCI pass-through Nova support.
> I
> > will be glad to share more details if you are interested.
> >
> >
> >
> >
> >
> > Good to know that you already have a SRIOV implementation. I found out
> some
> > information online about the mlnx plugin, but need more time to get to
> know
> > it better. And certainly I'm interested in knowing its details.
> >
> >
> >
> > The PCI pass-through networking support is planned to be discussed
> during
> > the summit: http://summit.openstack.org/cfp/details/129. I think it's
> worth
> > to drill down into more detailed proposal and present it during the
> summit,
> > especially since it impacts both nova and neutron projects.
> >
> >
> >
> > I agree. Maybe we can steal some time in that discussion.
> >
> >
> >
> > Would you be interested in collaboration on this effort? Would you be
> > interested to exchange more emails or set an IRC/WebEx meeting during
> this
> > week before the summit?
> >
> >
> >
> > Sure. If folks want to discuss it before the summit, we can schedule a
> webex
> > later this week. Or otherwise, we can continue the discussion with email.
> >
> >
> >
> >
> >
> >
> >
> > Regards,
> >
> > Irena
> >
> >
> >
> > From: Robert Li (baoli) [mailto:baoli at cisco.com]
> > Sent: Friday, October 25, 2013 11:16 PM
> > To: prashant.upadhyaya at aricent.com; Irena Berezovsky;
> > yunhong.jiang at intel.com; chris.friesen at windriver.com;
> yongli.he at intel.com
> > Cc: OpenStack Development Mailing List; Brian Bowen (brbowen); Kyle
> Mestery
> > (kmestery); Sandhya Dasu (sadasu)
> > Subject: Re: [openstack-dev] [nova] [neutron] PCI pass-through network
> > support
> >
> >
> >
> > Hi Irena,
> >
> >
> >
> > This is Robert Li from Cisco Systems. Recently, I was tasked to investigate
> > such support for Cisco's systems that support VM-FEX, which is a SRIOV
> > technology supporting 802-1Qbh. I was able to bring up nova instances
> with
> > SRIOV interfaces, and establish networking in between the instances that
> > employes the SRIOV interfaces. Certainly, this was accomplished with
> hacking
> > and some manual intervention. Based on this experience and my study
> with the
> > two existing nova pci-passthrough blueprints that have been
> implemented and
> > committed into Havana
> > (https://blueprints.launchpad.net/nova/+spec/pci-passthrough-base and
> > https://blueprints.launchpad.net/nova/+spec/pci-passthrough-libvirt),  I
> > registered a couple of blueprints (one on Nova side, the other on the
> > Neutron side):
> >
> >
> >
> > https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov
> >
> > https://blueprints.launchpad.net/neutron/+spec/pci-passthrough-sriov
> >
> >
> >
> > in order to address SRIOV support in openstack.
> >
> >
> >
> > Please take a look at them and see if they make sense, and let me know
> any
> > comments and questions. We can also discuss this in the summit, I
> suppose.
> >
> >
> >
> > I noticed that there is another thread on this topic, so copy those folks
> > from that thread as well.
> >
> >
> >
> > thanks,
> >
> > Robert
> >
> >
> >
> > On 10/16/13 4:32 PM, "Irena Berezovsky" <irenab at mellanox.com>
> wrote:
> >
> >
> >
> > Hi,
> >
> > As one of the next steps for PCI pass-through I would like to discuss is
> the
> > support for PCI pass-through vNIC.
> >
> > While nova takes care of PCI pass-through device resources
> management and
> > VIF settings, neutron should manage their networking configuration.
> >
> > I would like to register asummit proposal to discuss the support for PCI
> > pass-through networking.
> >
> > I am not sure what would be the right topic to discuss the PCI
> pass-through
> > networking, since it involve both nova and neutron.
> >
> > There is already a session registered by Yongli on nova topic to discuss
> the
> > PCI pass-through next steps.
> >
> > I think PCI pass-through networking is quite a big topic and it worth to
> > have a separate discussion.
> >
> > Is there any other people who are interested to discuss it and share
> their
> > thoughts and experience?
> >
> >
> >
> > Regards,
> >
> > Irena
> >
> >
> >
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >