<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif;">
<span id="OLK_SRC_BODY_SECTION">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<span id="OLK_SRC_BODY_SECTION">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<span id="OLK_SRC_BODY_SECTION">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>Hi all,</div>
<div><br>
</div>
<div>Wanted to share details of an issue we just discovered around hybrid ml2/ovs configuration under Icehouse.</div>
<div><br>
</div>
<div>We run ml2 on the API nodes, but the openvswitch plugin/ovs-agent on the compute/network nodes. We ran this split setup because under Havana this was the only way we could get ml2 working correctly, and this setup was recommend by an ml2 dev. We kept
this design because it continued to work under Icehouse, seemingly without issue. We upgraded from havana to icehouse without too much trouble a couple months ago.</div>
<div><br>
</div>
<div>However, we had not rebooted any compute nodes since then until this week. When the compute nodes came back up, instances that had been created before moving to icehouse did not start up because the vif for them was not being created.</div>
</div>
</span></div>
</span></div>
</span>
<div><br>
</div>
<div>Exact error is: <a href="https://gist.githubusercontent.com/krislindgren/c1f4f79dc12403c4815d/raw/386ef0607f32088ad372a27e06e3606f6c1ac220/gistfile1.txt">https://gist.githubusercontent.com/krislindgren/c1f4f79dc12403c4815d/raw/386ef0607f32088ad372a27e06e3606f6c1ac220/gistfile1.txt</a></div>
<span id="OLK_SRC_BODY_SECTION">
<div><br>
</div>
<font face="Calibri,sans-serif">Turns out this is because ports created under havana were missing the ‘hybrid’ property. And this </font>was preventing the vif from being recreated on the compute host. The ports for instances created after the icehouse upgrade
did have this property, and those instances started back up without a problem.</span><span id="OLK_SRC_BODY_SECTION">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<span id="OLK_SRC_BODY_SECTION">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div><br>
</div>
<div>
<div>Specifically, the problem is that in the neutron.ml2_port_bindings table, instances created before the upgrade had this for vif_details:</div>
<blockquote style="margin: 0px 0px 0px 40px; border: none; padding: 0px;">{"port_filter": true}</blockquote>
<div>Instances created after the upgrade had this for vif_details:</div>
<blockquote style="margin: 0px 0px 0px 40px; border: none; padding: 0px;">{"port_filter": true, "ovs_hybrid_plug": true}</blockquote>
<div>Missing this flag caused instances’ vifs to never get plug. The cause is this method:</div>
<div><a href="https://github.com/openstack/nova/blob/2014.1.2/nova/virt/libvirt/vif.py#L464-L470">https://github.com/openstack/nova/blob/2014.1.2/nova/virt/libvirt/vif.py#L464-L470</a></div>
<div>Specifically, because the ovs_hybrid_plug flag isn’t in the vif_details, vif.is_hybrid_plug_enabled() returns False and instead of calling plug_ovs_hybrid(), the driver calles plug_ovs_bridge(). plug_ovs_bridge() only calls its super implementation, which
is a no-op method, so the vif never actually gets plugged.</div>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div><br>
</div>
<div>We ended up solving this by manually assigning the hybrid property on the ports that were missing it via MySQL <i>(maybe paste the mysql query we used, or at least an example.)</i> Then starting all the havana instances worked normally.</div>
</div>
</span>
<div>Here’s the sql update we used:</div>
<div>update ml2_port_bindings set vif_details = '{"port_filter": true, "ovs_hybrid_plug": true}' where vif_details not like '%ovs_hybrid_plug%';</div>
</div>
</span>
<div>Note: that update statement will overwrite ALL entries that don’t contain the <font face="Calibri,sans-serif">ovs_hybrid_plug property. This was fine for us, but you should verify that it won’t munge any of your data.</font></div>
<span id="OLK_SRC_BODY_SECTION">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div><br>
</div>
<div>Not sure if we missed a step in the icehouse upgrade, and/or if this is just a function of our particular configuration. It might be possible that running the ml2 pluging with the openvswitch mechansim driver and the ovs-agent is now the correct solution
because that solution has a hardcoded ovs_hybrid_plug =true value. <a href="https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/mech_openvswitch.py#L40">https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/mech_openvswitch.py#L40</a></div>
</div>
</span></div>
</span>
<div><br>
</div>
<div>Hope this may be useful info for somebody.</div>
<div><br>
</div>
<div>Mike (et. al.)</div>
<div style="color: rgb(0, 0, 0);"><br>
</div>
<span id="OLK_SRC_BODY_SECTION" style="color: rgb(0, 0, 0);">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
</div>
</span>
</body>
</html>