<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi Openstack Team,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We were trying to boot VMs via Openstack Nova (off Liberty release) into DPDK-enabled OVS instances.<o:p></o:p></p>
<p class="MsoNormal">Our OVS is version 2.5 + DPDK 2.2 + Qemu 2.3 + Libvirt 1.2.16.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We followed the guidelines here for installation of OVS Datapath and enable them for DPDK:<o:p></o:p></p>
<p class="MsoNormal"><a href="https://github.com/openvswitch/ovs/blob/master/INSTALL.DPDK.md">https://github.com/openvswitch/ovs/blob/master/INSTALL.DPDK.md</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Howeve,r we were not able to get the VMs booted successfully via Nova. When we attempted to boot the VMs , they went into error state.<o:p></o:p></p>
<p class="MsoNormal">The vhostdpdkuser ports got created in the OVS-DPDK instance, but Libvirt Driver invoked by Nova, could not successfully attach this port to the VM.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>We got these set of errors in libvirtd.log whenever we booted the VMs:<o:p></o:p></b></p>
<p class="MsoNormal">2016-03-26 13:57:20.535+0000: 7354: error : virSecurityDriverLookup:80 : unsupported configuration: Security driver apparmor not enabled<o:p></o:p></p>
<p class="MsoNormal">2016-03-26 13:57:51.007+0000: 7339: error : virNetSocketReadWire:1520 : End of file while reading data: Input/output error<o:p></o:p></p>
<p class="MsoNormal">2016-03-26 13:58:36.293+0000: 7341: error : qemuMonitorOpenUnix:358 : failed to connect to monitor socket: No such process<o:p></o:p></p>
<p class="MsoNormal">2016-03-26 13:58:36.293+0000: 7341: error : qemuProcessWaitForMonitor:2113 : internal error: process exited while connecting to monitor: 2016-03-<b>26T13:58:36.235874Z qemu-system-x86_64: -chardev socket,id=charnet0,path=/var/lib/libvirt/qemu/vhu189ade8e-3b:
Failed to connect socket: Permission denied<o:p></o:p></b></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b><o:p> </o:p></b></p>
<p class="MsoNormal"><b>On the QEMU log:<o:p></o:p></b></p>
<p class="MsoNormal">2016-03-26 13:58:35.479+0000: starting up libvirt version: 1.2.16, package: 1.2.16-2ubuntu11.15.10.4~cloud0, qemu version: 2.3.0 (Debian 1:2.3+dfsg-5ubuntu9.2~cloud0)<o:p></o:p></p>
<p class="MsoNormal">LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name instance-0000000c -S -machine pc-i440fx-vivid,accel=kvm,usb=off -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1
-uuid dbe34ace-960e-4915-884a-5a30b8ec896d -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=12.0.3,serial=00000000-0000-0000-0000-002590883f20,uuid=dbe34ace-960e-4915-884a-5a30b8ec896d,family=Virtual Machine -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-0000000c.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -kernel
/opt/stack/data/nova/instances/dbe34ace-960e-4915-884a-5a30b8ec896d/kernel -initrd /opt/stack/data/nova/instances/dbe34ace-960e-4915-884a-5a30b8ec896d/ramdisk -append root=/dev/vda console=tty0 console=ttyS0 -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
-drive file=/opt/stack/data/nova/instances/dbe34ace-960e-4915-884a-5a30b8ec896d/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/opt/stack/data/nova/instances/dbe34ace-960e-4915-884a-5a30b8ec896d/disk.config,if=none,id=drive-ide0-1-1,readonly=on,format=raw,cache=none
-device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -chardev socket,id=charnet0,path=/var/lib/libvirt/qemu/vhu189ade8e-3b -netdev type=vhost-user,id=hostnet0,chardev=charnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:bd:cb:d6,bus=pci.0,addr=0x3
-chardev file,id=charserial0,path=/opt/stack/data/nova/instances/dbe34ace-960e-4915-884a-5a30b8ec896d/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -vnc 10.183.254.64:0
-k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on<o:p></o:p></p>
<p class="MsoNormal"><b>2016-03-26T13:58:36.235874Z qemu-system-x86_64: -chardev socket,id=charnet0,path=/var/lib/libvirt/qemu/vhu189ade8e-3b: Failed to connect socket: Permission denied<o:p></o:p></b></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We verified in our setup that we have given user/group as ‘root/root’ in etc/libvirt/qemu.conf and also
<o:p></o:p></p>
<p class="MsoNormal">affirmed that our ovs-vswitchd runs as ‘root’ with DPDK enabled.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The ‘vhuxxx’ ports are appearing in /var/lib/libvirt/qemu which is the vsock_host_dir folder that we have configured
<o:p></o:p></p>
<p class="MsoNormal">for ovs-vswitchd as well as in Nova.<o:p></o:p></p>
<p class="MsoNormal">srwxr-xr-x 1 root root 0 Mar 26 20:12 vhu98c8f690-41<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">But QEMU is not able to complete attaching the VM to the vhostdpdkuser port.<o:p></o:p></p>
<p class="MsoNormal">We included "/var/lib/libvirt/qemu" which is the vhost_sock_dir into cgroup_device_acl as well.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">And we also tried to put ‘security_driver’ to ‘None’ in qemu.conf to see if that helps (just to remove any bearing on SELinux (or) apparmor), but that
<o:p></o:p></p>
<p class="MsoNormal">didn’t help either.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Highly Appreciate any tips.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">--<o:p></o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Vivek<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>