[openstack-dev] [networking-ovs-dpdk]
Mooney, Sean K
sean.k.mooney at intel.com
Tue Nov 24 13:49:00 UTC 2015
Out of interest
Have you removed apparmor or placed all Libvirt apparmor profies into complain mode?
If not you will get permission denied errors.
You can confirm by checking dmesg to see if you have any permission denied messages from apparmor
Or run aa-status and see if the the Libvirt profie is in enforce/complain mode.
The /tmp/qemu.orig file is just a file we write the original qemu command to for debugging. It is not needed
But all uses should be able to read/write to /tmp.
We wrap the qemu/kvm binary with a script that on Ubuntu can be found here /usr/bin/kvm
If you comment out echo "qemu ${args[@]}" > /tmp/qemu.orig in this script it will silence that warning.
https://github.com/openstack/networking-ovs-dpdk/blob/master/devstack/libs/ovs-dpdk#L104
I may remove this from our wrapper script as we most never use it for debugging anymore however in the past it was
Useful to compare the original qemu command line and the update qemu command line.
I don’t know if I have mentioned this before but we also have a Ubuntu version of our getting start guide that should merge shortly
https://review.openstack.org/#/c/243190/6/doc/source/getstarted/ubuntu.rst
Regards
Sean.
From: Prathyusha Guduri [mailto:prathyushaconnects at gmail.com]
Sent: Tuesday, November 24, 2015 12:42 PM
To: Mooney, Sean K
Cc: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [networking-ovs-dpdk]
Hi All,
I also found another error while launching an instance.
libvirtError: internal error: process exited while connecting to monitor: /usr/bin/kvm-spice: line 42: /tmp/qemu.orig: Permission denied
I dont want to change any permissions manually and again face the dependency issues. So kindly help
Thanks,
Prathyusha
On Tue, Nov 24, 2015 at 4:02 PM, Prathyusha Guduri <prathyushaconnects at gmail.com<mailto:prathyushaconnects at gmail.com>> wrote:
Hi Sean,
Thanks for you kind help.
I did the following.
# apt-get install ubuntu-cloud-keyring
# echo "deb http://ubuntu-cloud.archive.canonical.com/ubuntu<http://www.google.com/url?q=http%3A%2F%2Fubuntu-cloud.archive.canonical.com%2Fubuntu&sa=D&sntz=1&usg=AFQjCNGlCfoplf1zSbILNxCSVK23zRxa2A>" \
"trusty-updates/kilo main" > /etc/apt/sources.list.d/cloudarchive-kilo.list
# apt-get update && apt-get dist-upgrade
and then uninstalled the libvirt and qemu that were installed manually and then ran stack.sh after cleaning and unstacking.
Now fortunately libvirt and qemu satisfy minimum requirements.
$ virsh --version
1.2.12
$ kvm --version
/usr/bin/kvm: line 42: /tmp/qemu.orig: Permission denied
QEMU emulator version 2.2.0 (Debian 1:2.2+dfsg-5expubuntu9.3~cloud0), Copyright (c) 2003-2008 Fabrice Bellard
Am using an ubuntu 14.04 system
$ uname -a
Linux ubuntu-Precision-Tower-5810 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
After stack.sh which was successful, tried creating a new instance - which gave an ERROR again.
$ nova list
+--------------------------------------+----------------+--------+------------+-------------+-------------------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+----------------+--------+------------+-------------+-------------------------------------------------------+
| 31a7e160-d04c-4216-91cf-30ce86c2b1fa | demo-instance1 | ERROR | - | NOSTATE | private=10.0.0.3, fd34:f4c5:412:0:f816:3eff:fea4:b9fe |
$ sudo service ovs-dpdk status
sourcing config
ovs alive
VHOST_CONFIG: bind to /var/run/openvswitch/vhufb8052e5-d3
2015-11-24T10:23:25Z|00126|dpdk|INFO|Socket /var/run/openvswitch/vhufb8052e5-d3 created for vhost-user port vhufb8052e5-d3
2015-11-24T10:23:25Z|00004|dpif_netdev(pmd18)|INFO|Core 2 processing port 'vhufb8052e5-d3'
2015-11-24T10:23:25Z|00002|dpif_netdev(pmd19)|INFO|Core 8 processing port 'dpdk0'
2015-11-24T10:23:25Z|00127|bridge|INFO|bridge br-int: added interface vhufb8052e5-d3 on port 6
2015-11-24T10:23:25Z|00005|dpif_netdev(pmd18)|INFO|Core 2 processing port 'dpdk0'
2015-11-24T10:23:26Z|00128|connmgr|INFO|br-int<->unix: 1 flow_mods in the last 0 s (1 deletes)
2015-11-24T10:23:26Z|00129|ofp_util|INFO|normalization changed ofp_match, details:
2015-11-24T10:23:26Z|00130|ofp_util|INFO| pre: in_port=5,nw_proto=58,tp_src=136
2015-11-24T10:23:26Z|00131|ofp_util|INFO|post: in_port=5
2015-11-24T10:23:26Z|00132|connmgr|INFO|br-int<->unix: 1 flow_mods in the last 0 s (1 deletes)
2015-11-24T10:23:26Z|00133|connmgr|INFO|br-int<->unix: 1 flow_mods in the last 0 s (1 deletes)
2015-11-24T10:23:29Z|00134|bridge|WARN|could not open network device vhufb8052e5-d3 (No such device)
VHOST_CONFIG: socket created, fd:52
VHOST_CONFIG: bind to /var/run/openvswitch/vhufb8052e5-d3
2015-11-24T10:23:29Z|00135|dpdk|INFO|Socket /var/run/openvswitch/vhufb8052e5-d3 created for vhost-user port vhufb8052e5-d3
2015-11-24T10:23:29Z|00006|dpif_netdev(pmd18)|INFO|Core 2 processing port 'vhufb8052e5-d3'
2015-11-24T10:23:29Z|00003|dpif_netdev(pmd19)|INFO|Core 8 processing port 'dpdk0'
2015-11-24T10:23:29Z|00136|bridge|INFO|bridge br-int: added interface vhufb8052e5-d3 on port 7
2015-11-24T10:23:30Z|00007|dpif_netdev(pmd18)|INFO|Core 2 processing port 'dpdk0'
0
I understand that ovs-dpdk is running. The error log of n-cpu.log is
2015-11-24 15:47:27.957 ^[[01;31mERROR nova.compute.manager [^[[01;36mreq-24fc3f16-ccd5-4e2d-b583-60ade23bc1ed ^[[00;36mNone None^[[01;31m] ^[[01;35m^[[01;31mNo compute node record for host ubuntu-Precision-Tower-5810^[[00m
2015-11-24 15:47:27.964 ^[[01;33mWARNING nova.compute.monitors [^[[01;36mreq-24fc3f16-ccd5-4e2d-b583-60ade23bc1ed ^[[00;36mNone None^[[01;33m] ^[[01;35m^[[01;33mExcluding nova.compute.monitors.cpu monitor virt_driver. Not in the list of enabled monitors (CONF.compute_monitors).^[[00m
2015-11-24 15:47:27.964 ^[[00;36mINFO nova.compute.resource_tracker [^[[01;36mreq-24fc3f16-ccd5-4e2d-b583-60ade23bc1ed ^[[00;36mNone None^[[00;36m] ^[[01;35m^[[00;36mAuditing locally available compute resources for node ubuntu-Precision-Tower-5810^[[00m
2015-11-24 15:47:28.087 ^[[00;32mDEBUG nova.compute.resource_tracker [^[[01;36mreq-24fc3f16-ccd5-4e2d-b583-60ade23bc1ed ^[[00;36mNone None^[[00;32m] ^[[01;35m^[[00;32mHypervisor: free VCPUs: 9^[[00m ^[[00;33mfrom (pid=7880) _report_hypervisor_resource_view /opt/stack/nova/nova/compute/resource_tracker.py:588^[[00m
I dont understand where am I missing out now.
On Mon, Nov 23, 2015 at 7:50 PM, Mooney, Sean K <sean.k.mooney at intel.com<mailto:sean.k.mooney at intel.com>> wrote:
Hi
qemu version was 2.0.0 does not support mapping hugepage as shared.
as a result the dpdk implementation of vhost-user cannot function with this version.
Similarly libvirt 1.2.2 has no knowledge of vhost-user.
If you are on fedora 21 then the virt-preview repo packages the required Libvirt and qemu.
On Ubuntu the kilo cloud archive also packages version that meet the minimum versions.
Looking at the log I would agree that this error is probably related to manually installing Libvirt and
not adding the appropriate libvirtd configuration options.
Devstack does not have native support for installing a vhost-user compatible Libvirt or qemu.
when using our networking-ovs-dpdk plugin we ask you to enable the virt-prevew or cloud archive
Before stacking so that when devestack install Libvirt/qemu it get compatible versions.
for centos as we have been unable to find an equivalent we explicitly install Libvirt/qemu.
In general we don’t provide support for installing the required version Libvirt/qemu in
the networking-ovs-dpdk devstack plugin.
What os are you currently using? ubuntu?
in your case enabling the kilo cloud archive, uinstalling Libvirt/qemu and then restacking
should provide the appropriate packages.
In liberty we use that standard neutron openvswich agent binary and set the agent type to DPDK OVS Agent to enable
the ovsdpdk ml2 driver to manage the node. In mitaka the standard ovs neutron agent and standard openvswitch ml2 drivers will be used instead.
To confirm if ovs-dpdk is running you can use the following command
sudo service ovs-dpdk status
this will check if ovs-dpdk is running by reading the pid files and checking if the ovsdb and ovs-vswitchd
process are running and tail the last 20ish lines of the vswitchd log file.
note this is not a systemd service so the systemd equivalent command will not work.
Alternative you can user ps and grep
ps aux | grep ovs
regards
sean.
From: Prathyusha Guduri [mailto:prathyushaconnects at gmail.com<mailto:prathyushaconnects at gmail.com>]
Sent: Monday, November 23, 2015 9:16 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [networking-ovs-dpdk]
Hi Sean,
Thanks for your help before. It would be great if you look into another issue too.
Am able to run stack.sh successfully and all services are up. But,
libvirt version was 1.2.2 and qemu version was 2.0.0
To satisfy the minimum requirement of qemu- version >=2.1 and libvirt-version >= 1.2.10
I manually installed qemu and libvirt from respective sources.
Now
$ kvm --version
/usr/bin/kvm: line 42: /tmp/qemu.orig: Permission denied
QEMU emulator version 2.1.3, Copyright (c) 2003-2008 Fabrice Bellard
$ virsh --version
1.2.10
So basic requirement is satisfied.
Before creating an instance ran the below command,
$ nova flavor-key m1.tiny set "hw:mem_page_size=large"
Now created an instance
$ nova boot --flavor m1.tiny --image cirros-0.3.4-x86_64-uec --nic net-id=445e2dc5-221b-48ea-aea4-d04dee12fc7f --security-group default demo-instance1
It gives the ERROR :
2015-11-23 13:19:59.654 ERROR nova.virt.libvirt.host [req-2d9d060d-1934-4e9e-af1c-010e177bea11 None None] Connection to libvirt failed: error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host Traceback (most recent call last):
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File "/opt/stack/nova/nova/virt/libvirt/host.py", line 527, in get_connection
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host conn = self._get_connection()
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File "/opt/stack/nova/nova/virt/libvirt/host.py", line 514, in _get_connection
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host wrapped_conn = self._get_new_connection()
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File "/opt/stack/nova/nova/virt/libvirt/host.py", line 466, in _get_new_connection
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host wrapped_conn = self._connect(self._uri, self._read_only)
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File "/opt/stack/nova/nova/virt/libvirt/host.py", line 320, in _connect
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host libvirt.openAuth, uri, auth, flags)
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host rv = execute(f, *args, **kwargs)
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host six.reraise(c, e, tb)
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host rv = meth(*args, **kwargs)
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 105, in openAuth
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host if ret is None:raise libvirtError('virConnectOpenAuth() failed')
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host libvirtError: error from service: CheckAuthorization: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host
Traceback (most recent call last):
I suspect this is because, I manually installed libvirt and qemu. My doubt is why devstack is not installing a correct version when it is supposed to. why a version less than min requirement is being installed???
Now that because am installing manually, there might be a problem with groups - devstack creates some group and installs but manual installation doesn't bother about that groups.
Can you please suggest a way on how do avoid that???
Also, I just want to make sure that the agent running is neutron-openvswitch only. No ovsdpdk agent running.
$ ps -Al | grep neutron
0 S 1000 8882 8859 3 80 0 - 49946 ep_pol pts/34 00:02:24 neutron-openvsw
But
$ neutron agent-list
3385a430-5738-43cb-b853-059add5ab602 | DPDK OVS Agent | ubuntu-Precision-Tower-5810 | :-) | True | neutron-openvswitch-agent
So this implies that dpdk agent is running right??? I remember reading in launchpad bugs that ovsdpdk agent is removed and that now openvswitch takes care of everything. Just wanted to confirm that my setup has ovs-dpdk running.
Regards,
Prathyusha
On Wed, Nov 18, 2015 at 7:23 PM, James Page <james.page at ubuntu.com<mailto:james.page at ubuntu.com>> wrote:
Hi Sean
On Wed, Nov 18, 2015 at 12:30 PM, Mooney, Sean K <sean.k.mooney at intel.com<mailto:sean.k.mooney at intel.com>> wrote:
Hi james
Yes we are planning on testing the packaged release to see if it is compatible with our ml2 driver and the
Changes we are submitting upstream. If it is we will add a use binary flag to our devstack plugin to skip the
Compilation step and use that instead on 15.10 or 14.04 cloud-archive:liberty
Excellent.
As part of your packaging did ye fix pciutils to correctly report the unused drivers when an interface is bound
The dpdk driver? Also does it support both igb_uio and/or vfio-pci drivers for dpdk interface?
Re pcituils, we've not done any work in that area - can you give an example of what you would expect?
The dpdk package supports both driver types in /etc/dpdk/interfaces - when you declare an adapter for use, you get to specify the module you want to use as well; we're relying the in-tree kernel drivers (uio-pci-generic and vfio-pci) right now.
Anyway yes I hope to check it out and seeing what ye have done. When ovs-dpdk starts getting packaged in more operating systems
We will probably swap our default to the binary install though we will keep the source install option as it allows us to work on new features
Before they are packaged and to have better performance.
That sounds sensible; re 'better performance' - yeah we do have to baseline the optimizations at compile time right now (ssse3 only right now) , but I really hope that does change so that we can move to a runtime CPU feature detection model, allowing the best possible performance through the packages we have in Ubuntu (or any other distribution for that matter).
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151124/72eed010/attachment.html>
More information about the OpenStack-dev
mailing list