<div dir="ltr"><div>Hi all, </div><div><br></div><div>I'd like to talk about the delete instance device of nova.</div><div><br></div><div>Here is the libvirt doc string to describe it underly function <span style="color:rgb(6,40,126);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">detachDeviceFlags</span>. </div><div><a href="http://paste.openstack.org/show/480330/">http://paste.openstack.org/show/480330/</a><br></div><div><br></div><div>It says: </div><div><pre style="margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><span class="" style="color:rgb(64,112,160);font-style:italic">detaching a device</span><span class=""><font color="#000000"> </font></span><span class="" style="color:rgb(64,112,160);font-style:italic">from a running domain may be asynchronous.</span></pre><pre style="margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><font color="#4070a0"><i>and it suggests:</i></font></pre><pre style="margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><span class="" style="color:rgb(64,112,160);font-style:italic">To check whether the device was</span><span class=""><font color="#000000"> </font></span><span class="" style="color:rgb(64,112,160);font-style:italic">successfully removed, either recheck domain</span></pre><pre style="margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><span class="" style="color:rgb(64,112,160);font-style:italic">configuration using </span><span class="" style="color:rgb(64,112,160);font-style:italic">virDomainGetXMLDesc() or add handler for</span></pre><pre style="margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><span class="" style="color:rgb(64,112,160);font-style:italic">VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED</span><font color="#000000">
</font></pre><pre style="margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><br></pre></div><div>Also Daniel elaborated it, and gave us more some scenarios about it.</div><div><br></div><div><span style="color:rgb(0,0,0);font-family:sans-serif;background-color:rgb(229,236,249)">it is not guaranteed to succeed. What happens is that the hypervisor injects an ACPI request to unplug the device. The guest OS must co-operate by releasing the device, before the hypervisor will complete the action of physically removing it. So you require a guest OS that supports ACPI unplug of course, and if the guest is crashed or being malicious there is no guarantee the unplug will succeed. Libvirt will wait a short while for success, but you must monitor for libvirt events to see if/when it finally completes. This delayed release has implications for when Nova can mark the PCI device as unused and available for other guests to assign.</span><br></div><div><span style="color:rgb(0,0,0);font-family:sans-serif;background-color:rgb(229,236,249)"><br></span></div><div><span style="color:rgb(0,0,0);font-family:sans-serif;background-color:rgb(229,236,249)"><br></span></div><div><div>Now I have checked the code, both detach volume or detach interface call </div></div><div><div> 5 1220 nova/virt/libvirt/driver.py <<detach_volume>></div><div> guest . detach_device ( conf , persistent = True , live = live )</div><div> 6 1280 nova/virt/libvirt/driver.py <<detach_interface>></div><div> guest . detach_device ( cfg , persistent = True , live = live )</div><div> 7 3016 nova/virt/libvirt/driver.py <<_detach_pci_devices>></div><div> guest . detach_device ( self . _get_guest_pci_device ( dev ) , live = True )</div><div> 8 3105 nova/virt/libvirt/driver.py <<_detach_sriov_ports>></div><div> guest . detach_device ( cfg , live = True )</div></div><div><br></div><div>And for detach_interface in nova/compute/manager.py:</div><div><br></div><div><pre style="margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px;color:rgb(0,0,0)"> <span class="" style="color:rgb(85,85,85);font-weight:bold">@wrap_exception</span><span class="">()</span>
<span class="" style="color:rgb(85,85,85);font-weight:bold">@wrap_instance_fault</span>
<span class="" style="color:rgb(0,112,32);font-weight:bold">def</span> <span class="" style="color:rgb(6,40,126)">detach_interface</span><span class="">(</span><span class="" style="color:rgb(0,112,32)">self</span><span class="">,</span> <span class="">context</span><span class="">,</span> <span class="">instance</span><span class="">,</span> <span class="">port_id</span><span class="">):</span>
<span class="" style="color:rgb(64,112,160);font-style:italic">"""Detach an network adapter from an instance."""</span>
<span class="">network_info</span> <span class="" style="color:rgb(102,102,102)">=</span> <span class="">instance</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">info_cache</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">network_info</span>
<span class="">condemned</span> <span class="" style="color:rgb(102,102,102)">=</span> <span class="" style="color:rgb(0,112,32)">None</span>
<span class="" style="color:rgb(0,112,32);font-weight:bold">for</span> <span class="">vif</span> <span class="" style="color:rgb(0,112,32);font-weight:bold">in</span> <span class="">network_info</span><span class="">:</span>
<span class="" style="color:rgb(0,112,32);font-weight:bold">if</span> <span class="">vif</span><span class="">[</span><span class="" style="color:rgb(64,112,160)">'id'</span><span class="">]</span> <span class="" style="color:rgb(102,102,102)">==</span> <span class="">port_id</span><span class="">:</span>
<span class="">condemned</span> <span class="" style="color:rgb(102,102,102)">=</span> <span class="">vif</span>
<span class="" style="color:rgb(0,112,32);font-weight:bold">break</span>
<span class="" style="color:rgb(0,112,32);font-weight:bold">if</span> <span class="">condemned</span> <span class="" style="color:rgb(0,112,32);font-weight:bold">is</span> <span class="" style="color:rgb(0,112,32)">None</span><span class="">:</span>
<span class="" style="color:rgb(0,112,32);font-weight:bold">raise</span> <span class="">exception</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">PortNotFound</span><span class="">(</span><span class="">_</span><span class="">(</span><span class="" style="color:rgb(64,112,160)">"Port </span><span class="" style="color:rgb(112,160,208);font-style:italic">%s</span><span class="" style="color:rgb(64,112,160)"> is not "</span>
<span class="" style="color:rgb(64,112,160)">"attached"</span><span class="">)</span> <span class="" style="color:rgb(102,102,102)">%</span> <span class="">port_id</span><span class="">)</span>
<span class="" style="color:rgb(0,112,32);font-weight:bold">try</span><span class="">:</span>
<span class="" style="color:rgb(0,112,32)">self</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">driver</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">detach_interface</span><span class="">(</span><span class="">instance</span><span class="">,</span> <span class="">condemned</span><span class="">)</span>
<span class="" style="color:rgb(0,112,32);font-weight:bold">except</span> <span class="">exception</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">NovaException</span> <span class="" style="color:rgb(0,112,32);font-weight:bold">as</span> <span class="">ex</span><span class="">:</span>
<span class="">LOG</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">warning</span><span class="">(</span><span class="">_LW</span><span class="">(</span><span class="" style="color:rgb(64,112,160)">"Detach interface failed, port_id=</span><span class="" style="color:rgb(112,160,208);font-style:italic">%(port_id)s</span><span class="" style="color:rgb(64,112,160)">,"</span>
<span class="" style="color:rgb(64,112,160)">" reason: </span><span class="" style="color:rgb(112,160,208);font-style:italic">%(msg)s</span><span class="" style="color:rgb(64,112,160)">"</span><span class="">),</span>
<span class="">{</span><span class="" style="color:rgb(64,112,160)">'port_id'</span><span class="">:</span> <span class="">port_id</span><span class="">,</span> <span class="" style="color:rgb(64,112,160)">'msg'</span><span class="">:</span> <span class="">ex</span><span class="">},</span> <span class="">instance</span><span class="" style="color:rgb(102,102,102)">=</span><span class="">instance</span><span class="">)</span>
<span class="" style="color:rgb(0,112,32);font-weight:bold">raise</span> <span class="">exception</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">InterfaceDetachFailed</span><span class="">(</span><span class="">instance_uuid</span><span class="" style="color:rgb(102,102,102)">=</span><span class="">instance</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">uuid</span><span class="">)</span>
<span class="" style="color:rgb(0,112,32);font-weight:bold">else</span><span class="">:</span>
<span class="" style="color:rgb(0,112,32);font-weight:bold">try</span><span class="">:</span>
<span class="" style="color:rgb(0,112,32)">self</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">network_api</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">deallocate_port_for_instance</span><span class="">(</span>
<span class="">context</span><span class="">,</span> <span class="">instance</span><span class="">,</span> <span class="">port_id</span><span class="">)</span>
<span class="" style="color:rgb(0,112,32);font-weight:bold">except</span> <span class="" style="color:rgb(0,112,32)">Exception</span> <span class="" style="color:rgb(0,112,32);font-weight:bold">as</span> <span class="">ex</span><span class="">:</span>
<span class="" style="color:rgb(0,112,32);font-weight:bold">with</span> <span class="">excutils</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">save_and_reraise_exception</span><span class="">():</span>
<span class="" style="color:rgb(96,160,176);font-style:italic"># Since this is a cast operation, log the failure for</span>
<span class="" style="color:rgb(96,160,176);font-style:italic"># triage.</span>
<span class="">LOG</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">warning</span><span class="">(</span><span class="">_LW</span><span class="">(</span><span class="" style="color:rgb(64,112,160)">'Failed to deallocate port </span><span class="" style="color:rgb(112,160,208);font-style:italic">%(port_id)s</span><span class="" style="color:rgb(64,112,160)"> '</span>
<span class="" style="color:rgb(64,112,160)">'for instance. Error: </span><span class="" style="color:rgb(112,160,208);font-style:italic">%(error)s</span><span class="" style="color:rgb(64,112,160)">'</span><span class="">),</span>
<span class="">{</span><span class="" style="color:rgb(64,112,160)">'port_id'</span><span class="">:</span> <span class="">port_id</span><span class="">,</span> <span class="" style="color:rgb(64,112,160)">'error'</span><span class="">:</span> <span class="">ex</span><span class="">},</span>
<span class="">instance</span><span class="" style="color:rgb(102,102,102)">=</span><span class="">instance</span><span class="">)</span></pre></div><div><br></div><div><span style="color:rgb(0,0,0);font-family:sans-serif;background-color:rgb(229,236,249)"><br></span></div><div>It just <span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">detach_interface, no double check the device is detached finally. </span></div><div><br></div><div><font color="#000000" face="Bitstream Vera Sans Mono, monospace">Now I will support the detach SRIOV code. </font></div><div><font color="#000000" face="Bitstream Vera Sans Mono, monospace"><a href="https://review.openstack.org/#/c/139910/">https://review.openstack.org/#/c/139910/</a></font></div><div><font color="#000000" face="Bitstream Vera Sans Mono, monospace">I'm not sure should I need to double check the device is </font><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">finally </span><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">detached.</span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><br></span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">If yes.</span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><br></span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">What should I support? </span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><br></span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">3 options:</span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">1. Just ignored it. key the nova code.</span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><br></span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">2. sync check. </span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">as the libvirt suggests: use </span><span style="color:rgb(64,112,160);font-style:italic;font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">virDomainGetXMLDesc()</span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><br></span></div><div><pre style="font-size:13px;margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace;color:rgb(0,0,0)"><span class="" style="color:rgb(0,112,32);font-weight:bold">def</span> <span class="" style="color:rgb(6,40,126)">detach_interface</span><span class="">(</span><span class="" style="color:rgb(0,112,32)">self</span><span class="">,</span> <span class="">context</span><span class="">,</span> <span class="">instance</span><span class="">,</span> <span class="">port_id</span><span class="">):</span></pre></div><div><pre style="font-size:13px;margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace;color:rgb(0,0,0)"><span class="" style="color:rgb(0,112,32)"><span style="color:rgb(0,0,0)"> </span>self</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">driver</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">detach_interface</span><span class="">(</span><span class="">instance</span><span class="">,</span> <span class="">condemned</span><span class="">)</span></pre><pre style="margin-top:0px;margin-bottom:0px;padding:5px 0px"><span class="" style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><span style="color:rgb(64,112,160);font-style:italic;white-space:normal"> # just </span></span><font color="#4070a0" face="Bitstream Vera Sans Mono, monospace"><span style="font-size:13px;white-space:normal"><i>pseudo-code</i></span></font></pre><pre style="margin-top:0px;margin-bottom:0px;padding:5px 0px"><font color="#4070a0" face="Bitstream Vera Sans Mono, monospace"><span style="font-size:13px;white-space:normal"><i> for i in range(1, 51):</i></span></font></pre><pre style="font-size:13px;margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace;color:rgb(0,0,0)"><span class=""><span style="color:rgb(64,112,160);font-style:italic;white-space:normal"> if not(virDomainGetXMLDesc()):</span><br></span></pre><pre style="font-size:13px;margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace;color:rgb(0,0,0)"><span class=""><span style="color:rgb(64,112,160);font-style:italic;white-space:normal"> sleep(1)</span></span></pre><pre style="font-size:13px;margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace;color:rgb(0,0,0)"><span class=""><span style="color:rgb(64,112,160);font-style:italic;white-space:normal"> else if i == 51：</span></span></pre><pre style="font-size:13px;margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace;color:rgb(0,0,0)"><span class=""><span style="color:rgb(64,112,160);font-style:italic;white-space:normal"> raise exception</span></span></pre><pre style="font-size:13px;margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace;color:rgb(0,0,0)"><span class=""><span style="color:rgb(64,112,160);font-style:italic;white-space:normal"> else:</span></span></pre><pre style="font-size:13px;margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace;color:rgb(0,0,0)"><span class=""><span style="color:rgb(64,112,160);font-style:italic;white-space:normal"> break</span></span></pre></div><div><pre style="font-size:13px;margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace;color:rgb(0,0,0)"> <span class="" style="color:rgb(0,112,32)">self</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">network_api</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">deallocate_port_for_instance</span><span class="">(</span>
<span class="">context</span><span class="">,</span> <span class="">instance</span><span class="">,</span> <span class="">port_id</span><span class="">)</span></pre></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><br></span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><br></span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">3. async notification. </span><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><br></span></div><div> <span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">as the libvirt suggests: </span></div><div><pre style="font-size:13px;margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace"><span class="" style="color:rgb(64,112,160);font-style:italic">add event handler for </span><span style="color:rgb(64,112,160);font-style:italic">VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED.</span></pre></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><br></span></div><div><span class="" style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">call network_api</span><span class="" style="font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px;color:rgb(102,102,102)">.</span><span class="" style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">deallocate_port_for_instance in a backend task.</span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">The backend receives the event result from </span><span style="color:rgb(64,112,160);font-style:italic;font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">event handler by AMPQ </span><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">and filter the</span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">device is the the expected </span><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">interface</span><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"> device, not the volume device. </span></div><div><br></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">Then backend call </span><span class="" style="font-size:13px;color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace">network_api</span><span class="" style="font-size:13px;font-family:'Bitstream Vera Sans Mono',monospace;color:rgb(102,102,102)">.</span><span class="" style="font-size:13px;color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace">deallocate_port_for_instance to deallocate the port.</span></div><div><span class="" style="font-size:13px;color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace"><br></span></div><div><span class="" style="font-size:13px;color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace"><br></span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">I have not check the volume detach, not sure it has the same issues. </span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><br></span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">Beside this issue: </span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px">But from the libvirt Doc string, </span></div><div><pre style="margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px;color:rgb(0,0,0)"><span class="" style="color:rgb(64,112,160);font-style:italic">hypervisors may prevent this operation if there is a current</span>
<span class="" style="color:rgb(64,112,160);font-style:italic">block copy operation on the device being detached;</span></pre></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"> </span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><br></span></div><div><span style="color:rgb(0,0,0);font-family:'Bitstream Vera Sans Mono',monospace;font-size:13px"><br></span></div></div>