Thanks for the reply Daniel. From your description, I think we've both been pushing in the same general direction, but are primarily looking at the question from two subtly different angles: you focusing on internal nova APIs + data structures representing VIF configuration data, and me focusing on the Quantum API used by Nova to fetch that data from the Quantum plugin. Some responses inline to hopefully clarify some of my thinking in light of this. Thanks,<div>
<br></div><div>Dan<br><div><br><div class="gmail_quote">On Mon, Jan 14, 2013 at 1:07 PM, Daniel P. Berrange <span dir="ltr"><<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On Mon, Jan 14, 2013 at 12:43:07PM -0800, Dan Wendlandt wrote:<br>
><br>
> I think we agree on both goals and mechanism at a high-level. The point I<br>
> was trying to make above is whether we have a FORMAL Quantum API definition<br>
> of what keys are included in this dictionary, and how this set of values<br>
> changes over time (in the "bad old days", many of the interfaces within<br>
> nova where just python dictionaries, where the only "definition" of what<br>
> they contained was the code that shoved k-v pairs into them... that is what<br>
> I want to avoid).<br>
<br>
</div>I don't strong opinion about the specific format of the Quantum -> Nova data.<br>
What is ultimately important is the data model defined for the Nova virt drivers<br>
in nova/network/model.py, the VIF & Network classes. These are the integration<br>
point between the Nova network API driver and the virt drivers. The important<br>
thing is that the per-hypervisor VIF driver impls must be fully isolated from<br>
the implementation details of the network API driver.<br>
<br>
As an example, when configuring OpenVSwitch one of the pieces of information<br>
require is an 'interfaceid'. Previously the libvirt VIF driver set the interfaceid<br>
based on the vif['uuid'] field, since that is what quantum expected. This is not<br>
portable though. The correct approach is for nova.network.model to have an<br>
explicit 'ovs-interfaceid' field and the nova.network.quantumv2.api driver<br>
sets this based on the on vif['uuid']. The libvirt VIF driver can now configure<br>
OVS with any network driver, not simply Quantum. Similarly for things like the<br>
OVS bridge name, of the TAP device names, all of which previously had to be<br>
hardcoded to Quantum specific data. This extends to bridging, 801.qbh, etc,etc<br></blockquote><div><br></div><div>I have no real concerns here with respect to these representations within Nova. Since you mentioned that you view this as not being libvirt specific, would the VIF/Network model in nova/network/model.py be extended with additional fields not currently represented in the spec (<a href="http://wiki.openstack.org/LibvirtVIFDrivers">http://wiki.openstack.org/LibvirtVIFDrivers</a>) for platforms like XenServer (plug into XenServer network UUIDs), vmware (plug into port group-id), a hyper-v virtual network, etc? </div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
> Another interesting issue that I'd like to see discussed in the spec is<br>
> versioning as new network capabilities are added to a particular platform<br>
> like libvirt. It seems like Quantum may need to know more about the<br>
> capabilities of the virt-layer itself in order to know what to pass back.<br>
> An existing example of this is the fact that the libvirt XML constructs<br>
> for OVS are only available in libvirt 0.9.11. If someone was using a old<br>
> version, presumably the port creation operation should either fall back to<br>
> using something that was available (e.g., plain bridge) or fail.<br>
<br>
</div>Whether libvirt has support for this or not has no bearing on Quantum. It<br>
is purely an impl detail for the libvirt VIF driver - the following change<br>
demonstrates this<br>
<br>
<a href="https://review.openstack.org/#/c/19125/" target="_blank">https://review.openstack.org/#/c/19125/</a></blockquote><div><br></div><div>It looks like my example was not relevant given that your proposal masks both types of OVS plugging behind a single type, but it seems like the broader question still applies. Surely over time nova will have to add entirely new vif_types. Since all Nova code for a given deployment will be in sync, this is not a problem within Nova, but what if you deploy a newer Quantum with an older Nova? Let's say in "H" Nova introduces vif_type "qbx" (presumably some "improved" for of VEPA :P) with requires a different set of parameters to configure. If a tenant is running an "H" version of Quantum, with a Grizzly version of Nova, how does it know that it can't specify vif-type qbx? </div>
<div><br></div><div>To me this points to an even larger question of how we handle heterogeneous hosts. A couple possible examples for discussion: </div><div>- Only the newer servers in your cloud have the fancy new NICs to support the type of vif-plugging most desired by your quantum plugin. for older servers, the quantum plugin needs to use a legacy platform. </div>
<div>- Your identifier for indicating how a vif-should be plugged (e.g., a xenserver network UUID) is specific to each cluster of servers, and your quantum deployment spans many clusters. How does the quantum server know what value to provide? </div>
<div>- A quantum deployment spans hypervisors of multiple types (e.g., kvm, xenserver, and esx) and the vif-type and values returned by the plugin need to vary for different hypervisor platforms.</div><div><br></div><div>
The above items are possible with the existing vif-plugging, since different config flags can be passed to different nova-compute instances. </div><div><br></div><div>It seems like in the new model, we would need Nova to pass some information to Quantum so that Quantum can make the right decision about what vif-type and data to send. </div>
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
> There's also the opposite direction, which is how Nova knows whether the<br>
> corresponding Quantum server is running a updated enough version to have<br>
> all of the key-value pairs it expects (for example, it looks like a recent<br>
> commit just extended garyk's original extension:<br>
> <a href="https://review.openstack.org/#/c/19542/3/quantum/extensions/portbindings.py" target="_blank">https://review.openstack.org/#/c/19542/3/quantum/extensions/portbindings.py</a>).<br>
> In this case, Nova should probably be checking the version of this<br>
> extension that is running to know what k-v pairs to expect.<br>
<br>
</div>The patches I've written so far will allow a Grizzly based Nova to talk<br>
to a Folsom based Quantum - you'll simply have to configure the old VIF<br>
driver classes as Nova have in Folsom. Meanwhile a Grizzly release of<br>
Nova will be able to talk to a Grizzly release of Quantum. What won't<br>
neccessarily work is random development snapshots of Quantum & Nova.<br>
<br>
In general, for the future, Nova should treat any new fields as optional,<br>
so if Quantum does not provide them, Nova should fallback to some sensible<br>
back-compatible behaviour.<br></blockquote><div><br></div><div>Yeah, mandating such a defensive posture when writing the Nova side of things is one option, and seems reasonable. Also, since this will be an official API extension in Quantum, it will need to be properly versioned, as it changes, so in theory Nova should know exactly what fields to expect based on the version of the extension Quantum is running. </div>
<div><br></div><div>One other minor comment and question: </div><div><br></div><div>I noticed the "filtered" value in the spec. The current description seems to be a bit of a deviation from the rest of the spec, as it explicitly describes the behavior of the quantum plugin, not the configuration value for the virt layer (which I'm guessing is implicitly the inverse, such that if this value is true, we do NOT configure filtering on the VIF). </div>
<div> </div><div><font face="arial, helvetica, sans-serif">And one last item that I was curious about: I noticed that one part of VIF-configuration that is not part of the spec currently is information about the Vif-driver mechanism (e.g., in libivrt, choosing model=virtio and specifying a <driver> element, or on esx, choosing e1000 vs. vmxnet). Is your thinking on this would be that it is even though it is vif-config from a compute perspective, its value is unlikely to depend on quantum and thus can still just be configured as a Nova config value for the virt layer (e.g., <span style="background-color:rgb(255,255,255);line-height:17.77777862548828px;white-space:pre">libvirt_use_virtio_for_bridges')? </span></font></div>
<div><br></div><div><span style="background-color:rgb(255,255,255);color:rgb(221,17,68);font-family:Consolas,'Liberation Mono',Courier,monospace;font-size:12.222222328186035px;line-height:17.77777862548828px;white-space:pre"><br>
</span></div><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5"><br>
Regards,<br>
Daniel<br>
--<br>
|: <a href="http://berrange.com" target="_blank">http://berrange.com</a> -o- <a href="http://www.flickr.com/photos/dberrange/" target="_blank">http://www.flickr.com/photos/dberrange/</a> :|<br>
|: <a href="http://libvirt.org" target="_blank">http://libvirt.org</a> -o- <a href="http://virt-manager.org" target="_blank">http://virt-manager.org</a> :|<br>
|: <a href="http://autobuild.org" target="_blank">http://autobuild.org</a> -o- <a href="http://search.cpan.org/~danberr/" target="_blank">http://search.cpan.org/~danberr/</a> :|<br>
|: <a href="http://entangle-photo.org" target="_blank">http://entangle-photo.org</a> -o- <a href="http://live.gnome.org/gtk-vnc" target="_blank">http://live.gnome.org/gtk-vnc</a> :|<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>Dan Wendlandt <div>Nicira, Inc: <a href="http://www.nicira.com" target="_blank">www.nicira.com</a><br><div>twitter: danwendlandt<br>
~~~~~~~~~~~~~~~~~~~~~~~~~~~<br></div></div>
</div></div>