Hi Daniel,<div><br></div><div>I actually think (hope?) that we're converging here. I'll try one more response below, as I think the context of the thread is helpful, but if you see this before you leave today and want to chat directly, you can ping me on IRC (i'm danwent)<br>
<br><div class="gmail_quote">On Thu, Jan 24, 2013 at 3:46 AM, Daniel P. Berrange <span dir="ltr"><<a href="mailto:berrange@redhat.com" target="_blank">berrange@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On Wed, Jan 23, 2013 at 09:46:46PM -0800, Dan Wendlandt wrote:<br>
> Hi Daniel,<br>
><br>
</div><div class="im">> Ok, I think we're zeroing in on the key differences in our points of view.<br>
> Let me know if you think it would be more efficient to discuss this via<br>
> IRC, as email lag when we're on different sides of the globe is tough.<br>
<br>
</div>Yeah, it could well be. I'm 'danpb' in #openstack-dev IRC and usually<br>
online in some fashion between 9-7pm GMT<br>
<div class="im"><br>
> On Wed, Jan 23, 2013 at 3:02 AM, Daniel P. Berrange <<a href="mailto:berrange@redhat.com">berrange@redhat.com</a>>wrote:<br>
><br>
</div><div><div class="h5">> > > I'm confused, as my understanding of your proposal requires Quantum know<br>
> > > this type of information. Perhaps I'm missing something? From the code<br>
> > > review, you have comments like:<br>
> > ><br>
> > > # TODO(berrange) temporary hack until Quantum can pass over the<br>
> > > # name of the OVS bridge it is configured with<br>
> > ><br>
> > > and<br>
> > ><br>
> > > # TODO(berrange) Quantum should pass the bridge name<br>
> > > # in another binding metadata field<br>
> > ><br>
> > > In this case, the name of the bridge is a concept specific to certain<br>
> > linux<br>
> > > hypervisors. If I was using XenServer, the equivalent might be a Network<br>
> > > UUID, or with ESX a port group uuid.<br>
> > ><br>
> > > My current thinking is that Quantum shouldn't have to know such<br>
> > information<br>
> > > either, but based on your comments, I was assuming this was a point of<br>
> > > disagreement. Can you clarify?<br>
> ><br>
> > Actually the Xen OVS VIF driver does require a bridge name. The bridge<br>
> > name is then used by the VIF driver to lookup the Xen network UUID.<br>
><br>
> So this is a good example of the same information being required by<br>
> > multiple hypervisor drivers. Similarly the Xen OVS VIF driver also<br>
> > requires the ovs interfaceid - again it currently hardcodes the<br>
> > assumption the the interfaceid is based on the vif['id'] field.<br>
> > So again my chage to the VIF model to include an explicit ovs_interfaceid<br>
> > parameter makes sense for Xen too.<br>
> ><br>
><br>
> I completely agree that some fields make sense to multiple hypervisors... I<br>
> certainly did not intend to say anything to the contrary. The point I was<br>
> making was that there is no single set of information is relevant to all<br>
> hypervisors. Do you agree with that statement, or are you advocating that<br>
> there is a single set of such information?<br>
><br>
> Also, I'm still trying to get confirmation to my question above, namely<br>
> that you do intend that Quantum would provide all such data needed to plug<br>
> a VIF, for example, providing a bridge name to a hypervisor running KVM, or<br>
> a port-group id for a hypervisor running ESX.<br>
<br>
</div></div>In essence yes. It is hard for me to answer your question about bridge<br>
name vs port-group id for ESX because AFAICK there's no plugin that<br>
exists for ESX + Quantum today - nova.virt.vmwareapi.vif certainly<br>
doesn't appear to have any such code. I'm not overly concerned though.<br></blockquote><div><br></div><div>I agree that if you look at the simple linux bridge or OVS plugins, they follow a very basic model where a vif_type and even bridge name would be uniform for an all KVM deployment. </div>
<div><br></div><div>But, for example, the NVP plugin can control KVM, XenServer, and soon ESX (waiting on a code change to add some more logic to ESX vif-plugging, which is one of the reasons I'm mentioning it as a specific example). With KVM vs. ESX, the data returned is different in kind (i.e., one is a linux bridge name, another is a port-group). And with KVM and XenServer, even though they are same same in kind (both bridge names), they are very likely to be different in form, since XenServer generates bridge names using a standard format (e.g., xapi0, or xenbr1). Below you propose something that with a very minor tweak would solve this concern, I believe. </div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I'm of the opinion that in general it should be possible to provide a<br>
set of information that is usable by all hypervisors. It may be that<br>
some hypervisors don't use all the pieces of information, but that's<br>
OK, as long as it doesn't deteriorate to the point where we have<br>
completely different data for every hypervisor. Given that the Quantum<br>
plugins I've looked at don't change their behaviour based on the hypervisor<br>
in use, I don't think we're going to have that problem in general.<br>
<div class="im"><br>
> If so, I do not see how the existing proposal can handle the situation<br>
> where two different sets of hypervisors that need different information are<br>
> deployed simultaneously. This could happen either because the two sets of<br>
> hypervisors identify vswitches differently (e.g., linux bridge names vs.<br>
> port-group ids) or because deployment constraints make it impossible to use<br>
> the same vswitch identifier across all hypervisors (e.g., a vswitch is<br>
> identified by a uuid, but that UUID is a per-cluster ID and the deplyoment<br>
> has multiple clusters). Helping me understand how you see this working<br>
> would help me out a lot.<br>
<br>
</div>The key is that in the Quantum plugin code, we're not doing any different<br>
work based on hypervisor in use. eg whether using Xen or KVM, the Linux<br>
Bridge plugin in Quantum is doing the same work to setup the bridge and<br>
has same requirements for port mapping. There's no if xen...else ...<br>
code there.<br></blockquote><div><br></div><div>I think this is the key point that we need to agree on. This is certainly true for very simple plugins, but not for more advanced ones (e.g., NVP today) or what others are trying to build (e.g., Bob's efforts for a module L2 plugin). I think the proposal below can handle this though. </div>
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
> > > I think we both agree that using vif_type qbx would not work (and that<br>
> > this<br>
> > > is reasonable). That wasn't my question though. My question was: If<br>
> > > Quantum returns qbx, presumably the older Nova would error-out when<br>
> > > provisioning the VM, so how does Quantum from the "H" series know that it<br>
> > > is or is not OK to return vif-type "qbx"? If we have to have an admin<br>
> > > configure quantum with the set of vif_types the Nova install supports,<br>
> > > where back to what we wanted to avoid: having to have an admin sync<br>
> > config<br>
> > > between Nova + Quantum.<br>
> ><br>
> > You seem to be considering the case of a new vif_type, which replaces a<br>
> > previously used vif_type when upgrading Quantum. This is something which<br>
> > should never be done, unless Quantum wants to mandate use of a newer Nova.<br>
><br>
> > If you want to retain full version compatibility in all directions, then<br>
> > when upgrading Quantum, the existing information should never change,<br>
> > unless the admin has explicitly chosen to reconfigure the plugin in some<br>
> > way.<br>
> ><br>
><br>
><br>
> I don't think that's the use case I'm trying to describe. If a Quantum<br>
> plugin only supports one vif_type for a given release, its trivial for it<br>
> to know which one to respond with. I'm talking about the case of a Quantum<br>
> plugin that supports two different vif_types simultaneousl, but needs to<br>
> know information about what vif_types the corresponding nova-compute<br>
> supports in order to know how to respond. I believe some other Red Hat<br>
> folks are keen on being able to realize this.<br>
<br>
</div>There's possible answers, depending on how I interpret what you describe<br>
above. If you considering that one Quantum plugin can create mutiple<br>
networks and each network can require a different vif_type, then I don't<br>
think there's any problem. Upon upgrading Quantum, any existing network<br>
that exists should retain use of its previous 'vif_type' value for<br>
compatibility with all existing Nova deployments using it. If the<br>
admin decides to create a new network with the new 'vif_type', then I<br>
think it is perfectly OK for Nova to report an error if the admin tries<br>
to start a VM on a Nova instance that doesn't support this new vif_type.<br>
<br>
If on the other hand you are considering one Quantum plugin with one<br>
network, where each port on the network can have a different vif-type<br>
then we have a more complicated issue. In such a case, I think we would<br>
have to make sure that when Nova issues the "create port" API call to<br>
Quantum, it passes across a list of all the vif_types it is able to<br>
handle.<br>
<br>
Maybe we should just make Nova pass across its list of supported<br>
vif types during 'create port' regardless of whether we need it<br>
now and be done with it.<br></blockquote><div><br></div><div>Yes, this is what I was thinking as well. Somewhat of a "negotiation" where Nova sends a certain amount of information over (e.g., its supported vif_types, its node-id) and then Quantum can determine what vif_type to respond with. I'm thinking that node-id may be needed to handle the case where Quantum needs to respond with different data even for the same vif_type (e.g., two different esx clusters that have different port-group-ids). </div>
<div><br></div><div>This adds some more complexity to Quantum, as the centralized quantum-server must know the mapping from a node-id to bridge-id + vif_type, but some part of the Quantum plugin must know this information already (e.g., an agent), so it would really just be a matter of shifting bits around within Quantum, which seems reasonable given time to implement this. </div>
<div><br></div><div>Dan</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div class="h5"><br>
<br>
> > > This isn't just about old vs. new versions. See the example above about<br>
> > a<br>
> > > deployment that spans multiple cluster for a hypervisor like XenServer or<br>
> > > ESX, thereby requiring that a different identifier is passed back based<br>
> > on<br>
> > > which cluster. Or an even more extreme example (but one I've already<br>
> > seen<br>
> > > in production with Quantum) is that there are multiple hypervisor types,<br>
> > so<br>
> > > even the vif_type that would need to be passed back may well be different<br>
> > > on different hypervisors.<br>
> ><br>
> > I don't think the vif_type should ever need to be different for different<br>
> > hypervisors. The vif_type is describing how Quantum has setup the network.<br>
> > Based off this, the hypervisor VIF drivers then decide how to configure<br>
> > the hypervisor. You seem to be thinking of the vif_type as a description<br>
> > of how the hypervisors must configure the network, which is backwards to<br>
> > how it should be.<br>
> ><br>
><br>
> Yes, I think this is where a lot of the difference of opinion lies. To me<br>
> the key question is whether you would use the same vif_type to plug a VM<br>
> into an OVS bridge and into an ESX port group (assuming you had a quantum<br>
> plugin that could both manage OVS and the vmware vswitch)? And even if you<br>
> did, I feel that the identifiers that quantum would need to return would be<br>
> different. Or are you suggesting that something like a port-group would be<br>
> modeled as a "bridge"? It seems pretty clear from this commit (<br>
> <a href="https://review.openstack.org/#/c/19117/5/nova/network/model.py" target="_blank">https://review.openstack.org/#/c/19117/5/nova/network/model.py</a>) that bridge<br>
> means a linux bridge, as there's a variable that describes the length of<br>
> the bridge name as the length of a linux device name.<br>
><br>
> I think if we can get to the bottom of this use case, we'll be well on our<br>
> way to being on the same page.<br>
<br>
</div></div>Ok, so you're describing a Quantum plugin that has two completely separate<br>
modes of operation, depending on the hypervisor in use. For that to work,<br>
Nova is going to have to be telling Quantum what hypervisor is needs the<br>
vport created for, and Quantum will do different work based on that. The<br>
'vif_type' reflects the type of setup that Quantum did, so if the quantum<br>
plugin has two different setups it can do, this implies two different<br>
possible 'vif_type' values to be returned - one for each type of setup -<br>
and each vif type will have its own appropriate data associated with it.<br>
Yes, this leads to more possible 'vif_type' definitions, but I think<br>
that is OK, as long as we consider their design carefully. The preference<br>
should be to minimise the number of vif types and only introduce new ones<br>
if there truely is a significant difference in needs. In other words we<br>
want the same vif type to apply across multiple hypervisors wherever<br>
it is reasonable todo so, and not needlessly create hypervisor specific<br>
vif types.<br>
<div class="HOEnZb"><div class="h5"><br>
Daniel<br>
--<br>
|: <a href="http://berrange.com" target="_blank">http://berrange.com</a> -o- <a href="http://www.flickr.com/photos/dberrange/" target="_blank">http://www.flickr.com/photos/dberrange/</a> :|<br>
|: <a href="http://libvirt.org" target="_blank">http://libvirt.org</a> -o- <a href="http://virt-manager.org" target="_blank">http://virt-manager.org</a> :|<br>
|: <a href="http://autobuild.org" target="_blank">http://autobuild.org</a> -o- <a href="http://search.cpan.org/~danberr/" target="_blank">http://search.cpan.org/~danberr/</a> :|<br>
|: <a href="http://entangle-photo.org" target="_blank">http://entangle-photo.org</a> -o- <a href="http://live.gnome.org/gtk-vnc" target="_blank">http://live.gnome.org/gtk-vnc</a> :|<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>Dan Wendlandt <div>Nicira, Inc: <a href="http://www.nicira.com" target="_blank">www.nicira.com</a><br><div>twitter: danwendlandt<br>
~~~~~~~~~~~~~~~~~~~~~~~~~~~<br></div></div>
</div>