Open Stack

Hi Dan, and All,

 

I wanted to follow up on this conversation where we left off last week.
The discussion has progressed to the point that we understand the
semantics of create/attach port a little better (or at least how those
semantics could be realized in certain cases). 

 

I believe where we stand now is with the realization that for
nova-compute to craft the correct network interface configuration for a
VM (prior to spawning the VM), in certain cases, some added information
might be needed from Quantum (apart from what nova-compute already
posses on the compute side). In effect, we need a generic approach so as
to facilitate getting information from Quantum before a VM's network
interface configuration is crafted.

 

We started off by thinking that an additional core API could be added to
achieve this. However, this can probably be done in a cleaner way by
adding an attribute to the "Port" resource as shown below. It's called
"configuration" here (but could be any other name) and could have zero
or more key-value pairs.

 

<Port>

<port-id> Universally Unique ID </id>

<VIF-id> Universally Unique ID or None </id>

<state> Attached | Unattached </state>

<configuration> 

<conf key="key-1">value-1</conf>

<conf key="key-2">value-2</conf>

...

</configuration>

</Port>

 

 

Again, as we discussed, this information is required prior to the
interface configuration for a VM being created. As such, for
nova-compute to be able to generically process all interface types, I am
suggesting that this "configuration" attribute be present as a core
attribute (and not as an extension). If present, nova-compute will
always make the core API call to "list port details", after making a
call to "create port", and before creating the configuration file for a
VM.

 

Also note that nova-compute uses a template file to construct the VM
configuration and it is possible to refer to different templates by way
of configuring a flag in nova.conf. Using that option, one could craft
different template files, and point to the relevant one depending on
which network interface configuration needs to be used for a VM on that
host (I am not suggesting anything new here, this is the way it works
today). The attribute values which go into that interface configuration
would either be already available to nova-compute, or could come by way
of the "configuration" key-value pairs as described above.

 

Let me take the example of an 802.1Qbh interface configuration to make
this clearer. Let's assume that "create network" and "create port" calls
have already been made. Now, nova-compute makes a call to "list port
details" and gets back the following response:

 

<Port>

<port-id>xyz</id>

<VIF-id>abc</id>

<state>Unattached</state>

<configuration> 

<conf key="source_dev">eth4</conf>

<conf key="profile_id">enterprise-profile</conf>

</configuration>

</Port>

 

nova-compute can iterate through the key-value pairs of the
configuration element and store them in a list (without having to
understand them). If the keys used here correspond to the placeholders
used in the template file, the corresponding values for those keys can
be easily substituted in the template file to construct the VM
configuration (just as it is done today with all other place holders in
this template file, and again, without nova-compute understanding any of
it). The example template file shown below, maps the keys "source_dev"
and "profile_id" to placeholders, such that the values "eth4" and
"enterprise-profile" will be substituted when nova-compute generates the
VM configuration file.

 

<domain type='${type}'>

    <name>${name}</name>

    <memory>${memory_kb}</memory>

   ...

   ...

   <devices>

   ...

   ...

    <interface type='direct'>

      <mac address='${mac_address}'/>

      <source dev='${source_dev}' mode='private'/>

      <virtualport type='802.1Qbh'>

        <parameters profileid='${profile_id}'/>

      </virtualport>

      <model type='virtio'/>

    </interface>

 

    ...

    ...

    </devices>

</domain>

 

In summary (and apologies for the length of this email), the proposal
here is to

 

(a) add the "configuration" element to the "Port" resource, and

(b) establish the convention that nova-compute makes a call to obtain
the port details after it creates the port, and before it creates the VM
configuration.

 

I have also tried to capture this proposal (and the earlier discussion)
in a document which might be a better read for someone new to this
discussion -

http://wiki.openstack.org/QuantumAPIBase?action=AttachFile&do=view&targe
t=quantum-vnic-configuration-SumitNaiksatam-v1.pdf

 

Thanks,

~Sumit.

 

 

From: openstack-bounces+snaiksat=cisco.com at lists.launchpad.net
[mailto:openstack-bounces+snaiksat=cisco.com at lists.launchpad.net] On
Behalf Of Sumit Naiksatam (snaiksat)
Sent: Tuesday, May 31, 2011 11:15 AM
To: Dan Wendlandt
Cc: openstack at lists.launchpad.net
Subject: Re: [Openstack] [NetStack] "Attach resource to port" semantics

 

Hi Dan,

 

Thanks. By using the type "ethernet" the decision about which bridge to
use is made by the script which is referenced by the "script" element in
the interface, right?

 

   <interface type='ethernet'>

      <target dev='vnet7'/>

      <script path='/etc/qemu-ifup-mynet'/>

    </interface

 

In that case, how would Nova know which script to point to? Is it always
the same script?

 

In the context of 802.1Qbh (VN-Tag), I have worked with it before, so
here is an example interface configuration:

 

    <interface type='direct'>

      <mac address='02:16:3e:65:a9:e7'/>

      <source dev='eth4' mode='private'/>

      <virtualport type='802.1Qbh'>

        <parameters profileid='enterprise-xyz'/>

      </virtualport>

      <model type='virtio'/>

    </interface>

 

To construct the above configuration, the source dev, i.e. "eth4" and
the profileid, i.e. "enterprise-xyz" has to be known. The profile
"enterprise-xyz" itself has the VLAN Id (among other pieces of network
information) encapsulated. Again, the question is, how will Nova know
the "source dev" and "profiled"?

 

Thanks,

~Sumit.

 

From: Dan Wendlandt [mailto:dan at nicira.com] 
Sent: Tuesday, May 31, 2011 9:53 AM
To: Sumit Naiksatam (snaiksat)
Cc: openstack at lists.launchpad.net
Subject: Re: [Openstack][NetStack] "Attach resource to port" semantics

 

Hi Sumit, responses inline. Thanks,

 

Dan

On Tue, May 31, 2011 at 9:12 AM, Sumit Naiksatam (snaiksat)
<snaiksat at cisco.com> wrote:

Hi Dan,

Thanks for your prompt response on this.

As I understand from your explanation, the knowledge of the edge binding
resides with Nova, and in the case of my example that edge binding is
br100. Is that correct? If so, how does Nova acquire this knowledge? I
ask because, my expectation was that br100 (i.e. the Linux bridge) is
created and managed by Quantum (and/or the plugin underneath).

 

The way I see it, the "edge binding" is determined by Nova, then
communicated to Quantum. It is the one bit of information that must be
shared across both systems. Nova acquires this information as part of
the VM creation process. For example, in the case of libvirt with type
ethernet on a linux system, nova would be the one creating the linux
device that represents the VM NIC. It must then pass this edge binding
of (device-name, linux-system, interface-id) to Quantum, which must be
running a plugin that knows how to manage a vswitch on that linux host
that can get/receive packets from the linux device. 

	Also, if we move away from using the interface type bridge (in
the context of libvirt) to ethernet as you suggest, would that be the
only type that would be supported? If so, that will not work with
VN-Tag/VEPA since that requires a type direct. If we are indeed not
limiting to just the ethernet type, then how does Nova know which type
to use (assuming that the knowledge of the underlying network
connectivity resides with Quantum)?

 

Our goal is to add more flexibility to the "vif plugging" portion of the
libvirt code to support multiple use cases, including both "ethernet"
and "direct". In fact, VN-Tag/VEPA was one of the use cases we explored
when deciding on the model. Based on my understanding of how libvirt
handles 802.1Qbg, the libvirt XML includes a <virtualport> object that
corresponds to such a VM NIC and this virtual port includes parameters
that identify the VM, including a an "instanceid" parameter that
globally represents the virtual port. I would expect that the
"edge-binding" in this case would be (instanceid, interface-id). As a
note, I have not actually used libvirt like this on an actual
VN-Tag/VEPA system. If you are able to explore this use case in more
detail it would be great feedback for the Quantum design. 

 

 

	Its good to hear that the work on Quantum and Nova refactoring
is going well. While that is in progress, it might be helpful to get
some skeletal flowcharts (or even textual descriptions) describing the
flows (with the interaction between Nova and Quantum fleshed out).

 

I agree that would be valuable. We'll work on this.

 

	Thanks,

	~Sumit. 

	From: Dan Wendlandt [mailto:dan at nicira.com] 
	Sent: Monday, May 30, 2011 11:15 PM
	To: Sumit Naiksatam (snaiksat)

	
	Cc: openstack at lists.launchpad.net

	Subject: Re: [Openstack] [NetStack] "Attach resource to port"
semantics

	Hi Sumit,

	This is a great question to be asking, as it gets to the heart
of how Quantum will work with Nova (or any any other OpenStack service
that plugs into the network). As you suggest, our goal here is to try
and come up with a clean separation between the responsibilities of Nova
and theresponsibilitiesof Quantum.

	The way I think about it, the act of "attaching a VM NIC to the
network" actually happens at two distinct layers: logical (managed by
Quantum) and, for lack of a better name, non-logical (managed by Nova). 

	Non-Logical: This occurs when the Nova spins up a VM and plugs
the VM NIC into the vswitch. For example, with Nova using libvirt, this
amounts to creating a linux device representing the VM NIC (e.g., tap0)
and associating it with a vswitch (e.g., br0). With Quantum, this act
alone should not actually cause the VM to be able to send and receive
packets, unless the logical connectivity of the VM NIC has already been
setup (see below). Instead, it is just establishing a channel by which
vswitch controlled by Quantum can pass packets to and from the VM NIC
once the proper logical connectivity has been specified. With Quantum,
Nova must have an "interface-id" for each VM NIC. When Nova spins up a
VM and creates a new port on a vswitch, it communicates to Quantum the
interface-id associated with that port on the vswitch. We refer to as
the "edge binding" (see slides #7 and #14
inhttp://www.slideshare.net/danwent/quantum-diablo-summary). 

	Logical: This type of "plugging" is done by the tenant via the
Quantum API. The Quantum API exposes the concept of networks and ports
on those networks to model this logical connectivity. Interface-ids can
be "attached" to a port, indicating the network connectivity that should
be seen by the VM NIC with the associated interface-id (e.g., two VMs
attached to ports on the same logical network can connect directly at
L2, others cannot). A simple implementation of a logical network could
be to put implement each logical network as a different VLAN within a
single physical L2 domain, with a Quantum plugin managing the vswitch to
put each port on the appropriate VLAN (see slide #15). 

	The motivation for this design is that Nova does not need to be
involved when the logical connectivity of a VM changes. Nova just needs
to worry about making sure the vswitch (or physical switch if using
VNTag/VEPA) managed by the Quantum plugin has a channel to pass bits to
and from each VM NIC, and knows the identify of each of those channels
(the edge binding). It is up to the Quantum plugin managing the vswitch
to make sure the right logical connectivity (or lack thereof) is
implemented. 

	Along with folks from the Nova team, we've been discussing the
details of how this model applies to different Nova "virt"
layerimplementations, including libvirt, XenServer, VMware, etc. As you
suggest, the libvirt model where Nova specifies only a bridge to connect
to is less flexible, so we're looking at using an <interface> elements
of type "ethernet" instead of type "bridge" as an alternative (Ryu did
some work on this in a branch). 

	Implementation work on an experimental version of Quantum has
been progressing well, so in another week or two I would expect that we
can have a full stack of code that people can play with using a slightly
tweaked version of Nova. Having thisavailablewill hopefully make the
above description of a logical + non-logical plugging step more concrete
and will serve as a good vehicle for feedback from the community. Happy
to chat more about this tomorrow at the Networking meeting on IRC.

	dan

	On Mon, May 30, 2011 at 9:01 PM, Sumit Naiksatam (snaiksat)
<snaiksat at cisco.com> wrote:

	Hi,

	I would like to request clarity on the semantics of the "attach
resource to port" operation proposed in Quantum in the context of
spawning a VM. Let's discuss this with respect to a specific example. As
of today, nova-compute generates the domain configuration (libvirt.xml),
including the network interface configuration as below -

	<interface type='bridge'>

	<source bridge='br100'/>

	<mac address='02:16:3e:78:4d:84'/>

	<!-- <model type='virtio'/> CANT RUN virtio network right now
-->

	<filterref filter="nova-instance-instance-00000001">

	<parameter name="IP" value="10.1.0.2" />

	<parameter name="DHCPSERVER" value="10.1.0.1" />

	<parameter name="PROJNET" value="10.1.0.0" />

	<parameter name="PROJMASK" value="255.255.255.128" />

	</filterref>

	</interface>

	By virtue of specifying "br100" as the bridge to connect to,
nova-compute is configuring the attachment from the virtual NIC (on the
VM) to the network, and the "attachment" operation is (implicitly)
realized when the VM is spawned (and running).

	With the introduction of the "attach resource to port"
operation, we are introducing an explicit step in the process of the VM
spawning. It implies that prior to invoking the "attach resource to
port" operation, nova-compute would not know which bridge to connect to
(i.e. it would not know the identity of "br100"). So does nova-compute
first spawn the VM without the network interface configuration? If that
is the case, it brings us to the next question, which is, what does the
"attach resource to port" operation actually do? There are at least two
possibilities here -

	(1) The operation merely results in the identification (and, if
required, the configuration) of "br100" as the bridge to connect the
VM's network interface to. If that is the case, we need a way to
propagate this information (i.e. "br100") back to the entity which
invoked the "attach resource to port" operation. Subsequently, that same
entity will have to perform another action to craft the VM's network
interface configuration (with the acquired knowledge of "br100") and
also add this network interface to the configuration of the VM (by
making appropriate libvirt calls). Adding this network interface
configuration could be done, both, prior to starting the VM, or even
while the VM is running.

	OR 

	(2) The operation not only identifies/configuration of "br100",
but also crafts the VM's network interface configuration, and adds that
interface to the VM (by making appropriate libvirt calls). Note that
this second option amounts to Quantum controlling a resource on the VM
(which ideally should be outside the scope of Quantum).

	OR, is it something else?

	Kindly let me know what you thoughts are on this. As projected
above, these semantics have implications on how nova-compute will deal
with spawning VMs (and it might be different from what it is today), and
hence it might be important to address this issue sooner.

	Thanks,

	~Sumit.

	PS: As I write this, it occurs to me that it's not very clear as
to what the "create port" operation would do in the above case as well,
since no actual port is created on the bridge. Any thoughts?

	
	_______________________________________________
	Mailing list: https://launchpad.net/~openstack
	Post to : openstack at lists.launchpad.net
	Unsubscribe : https://launchpad.net/~openstack
	More help : https://help.launchpad.net/ListHelp

	
	
	
	-- 
	~~~~~~~~~~~~~~~~~~~~~~~~~~~
	Dan Wendlandt 
	Nicira Networks, Inc. 
	www.nicira.com | www.openvswitch.org
	Sr. Product Manager 
	cell: 650-906-2650
	~~~~~~~~~~~~~~~~~~~~~~~~~~~




-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt 
Nicira Networks, Inc. 
www.nicira.com | www.openvswitch.org
Sr. Product Manager 
cell: 650-906-2650
~~~~~~~~~~~~~~~~~~~~~~~~~~~

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20110606/90beb503/attachment.html>