[openstack-dev] [networking-ovs-dpdk]
Prathyusha Guduri
prathyushaconnects at gmail.com
Wed Nov 25 06:05:28 UTC 2015
Thank you so much Sean,
It helped a lot and that I have VMs active running :) That's so kind of you
to spend your time. It would be nice if this apparmor thing is mentioned
somewhere in the documentation, will help newbies like me.
@Hui - Thank you for the info and the config file :)
Thanks,
Prathyusha
On Wed, Nov 25, 2015 at 7:28 AM, Hui Xiang <hui.xiang at canonical.com> wrote:
> FYI, bug 1513367 is opened for this apparmor problem to track when
> booting vms failed with ovs-dpdk enabled.
>
> https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1513367
>
> On Tue, Nov 24, 2015 at 9:49 PM, Mooney, Sean K <sean.k.mooney at intel.com>
> wrote:
>
>> Out of interest
>>
>> Have you removed apparmor or placed all Libvirt apparmor profies into
>> complain mode?
>>
>>
>>
>> If not you will get permission denied errors.
>>
>>
>>
>> You can confirm by checking dmesg to see if you have any permission
>> denied messages from apparmor
>>
>> Or run aa-status and see if the the Libvirt profie is in enforce/complain
>> mode.
>>
>>
>>
>> The /tmp/qemu.orig file is just a file we write the original qemu
>> command to for debugging. It is not needed
>>
>> But all uses should be able to read/write to /tmp.
>>
>>
>>
>> We wrap the qemu/kvm binary with a script that on Ubuntu can be found
>> here /usr/bin/kvm
>>
>>
>>
>> If you comment out echo "qemu ${args[@]}" > /tmp/qemu.orig in this script
>> it will silence that warning.
>>
>>
>>
>>
>> https://github.com/openstack/networking-ovs-dpdk/blob/master/devstack/libs/ovs-dpdk#L104
>>
>>
>>
>> I may remove this from our wrapper script as we most never use it for
>> debugging anymore however in the past it was
>>
>> Useful to compare the original qemu command line and the update qemu
>> command line.
>>
>>
>>
>> I don’t know if I have mentioned this before but we also have a Ubuntu
>> version of our getting start guide that should merge shortly
>>
>>
>>
>> https://review.openstack.org/#/c/243190/6/doc/source/getstarted/ubuntu.rst
>>
>>
>>
>> Regards
>>
>> Sean.
>>
>>
>>
>> *From:* Prathyusha Guduri [mailto:prathyushaconnects at gmail.com]
>> *Sent:* Tuesday, November 24, 2015 12:42 PM
>> *To:* Mooney, Sean K
>> *Cc:* OpenStack Development Mailing List (not for usage questions)
>> *Subject:* Re: [openstack-dev] [networking-ovs-dpdk]
>>
>>
>>
>> Hi All,
>>
>> I also found another error while launching an instance.
>>
>> libvirtError: internal error: process exited while connecting to monitor:
>> /usr/bin/kvm-spice: line 42: /tmp/qemu.orig: Permission denied
>>
>> I dont want to change any permissions manually and again face the
>> dependency issues. So kindly help
>>
>> Thanks,
>>
>> Prathyusha
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Nov 24, 2015 at 4:02 PM, Prathyusha Guduri <
>> prathyushaconnects at gmail.com> wrote:
>>
>> Hi Sean,
>>
>> Thanks for you kind help.
>>
>> I did the following.
>>
>> # apt-get install ubuntu-cloud-keyring
>> # echo "deb http://ubuntu-cloud.archive.canonical.com/ubuntu
>> <http://www.google.com/url?q=http%3A%2F%2Fubuntu-cloud.archive.canonical.com%2Fubuntu&sa=D&sntz=1&usg=AFQjCNGlCfoplf1zSbILNxCSVK23zRxa2A>"
>> \
>> "trusty-updates/kilo main" >
>> /etc/apt/sources.list.d/cloudarchive-kilo.list
>> # apt-get update && apt-get dist-upgrade
>>
>> and then uninstalled the libvirt and qemu that were installed manually
>> and then ran stack.sh after cleaning and unstacking.
>>
>> Now fortunately libvirt and qemu satisfy minimum requirements.
>>
>> $ virsh --version
>> 1.2.12
>>
>> $ kvm --version
>> /usr/bin/kvm: line 42: /tmp/qemu.orig: Permission denied
>> QEMU emulator version 2.2.0 (Debian 1:2.2+dfsg-5expubuntu9.3~cloud0),
>> Copyright (c) 2003-2008 Fabrice Bellard
>>
>> Am using an ubuntu 14.04 system
>> $ uname -a
>> Linux ubuntu-Precision-Tower-5810 3.13.0-24-generic #46-Ubuntu SMP Thu
>> Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>>
>> After stack.sh which was successful, tried creating a new instance -
>> which gave an ERROR again.
>>
>> $ nova list
>>
>> +--------------------------------------+----------------+--------+------------+-------------+-------------------------------------------------------+
>> | ID | Name | Status | Task
>> State | Power State | Networks
>> |
>>
>> +--------------------------------------+----------------+--------+------------+-------------+-------------------------------------------------------+
>> | 31a7e160-d04c-4216-91cf-30ce86c2b1fa | demo-instance1 | ERROR |
>> - | NOSTATE | private=10.0.0.3,
>> fd34:f4c5:412:0:f816:3eff:fea4:b9fe |
>>
>> $ sudo service ovs-dpdk status
>> sourcing config
>> ovs alive
>> VHOST_CONFIG: bind to /var/run/openvswitch/vhufb8052e5-d3
>> 2015-11-24T10:23:25Z|00126|dpdk|INFO|Socket
>> /var/run/openvswitch/vhufb8052e5-d3 created for vhost-user port
>> vhufb8052e5-d3
>> 2015-11-24T10:23:25Z|00004|dpif_netdev(pmd18)|INFO|Core 2 processing port
>> 'vhufb8052e5-d3'
>> 2015-11-24T10:23:25Z|00002|dpif_netdev(pmd19)|INFO|Core 8 processing port
>> 'dpdk0'
>> 2015-11-24T10:23:25Z|00127|bridge|INFO|bridge br-int: added interface
>> vhufb8052e5-d3 on port 6
>> 2015-11-24T10:23:25Z|00005|dpif_netdev(pmd18)|INFO|Core 2 processing port
>> 'dpdk0'
>> 2015-11-24T10:23:26Z|00128|connmgr|INFO|br-int<->unix: 1 flow_mods in the
>> last 0 s (1 deletes)
>> 2015-11-24T10:23:26Z|00129|ofp_util|INFO|normalization changed ofp_match,
>> details:
>> 2015-11-24T10:23:26Z|00130|ofp_util|INFO| pre:
>> in_port=5,nw_proto=58,tp_src=136
>> 2015-11-24T10:23:26Z|00131|ofp_util|INFO|post: in_port=5
>> 2015-11-24T10:23:26Z|00132|connmgr|INFO|br-int<->unix: 1 flow_mods in the
>> last 0 s (1 deletes)
>> 2015-11-24T10:23:26Z|00133|connmgr|INFO|br-int<->unix: 1 flow_mods in the
>> last 0 s (1 deletes)
>> 2015-11-24T10:23:29Z|00134|bridge|WARN|could not open network device
>> vhufb8052e5-d3 (No such device)
>> VHOST_CONFIG: socket created, fd:52
>> VHOST_CONFIG: bind to /var/run/openvswitch/vhufb8052e5-d3
>> 2015-11-24T10:23:29Z|00135|dpdk|INFO|Socket
>> /var/run/openvswitch/vhufb8052e5-d3 created for vhost-user port
>> vhufb8052e5-d3
>> 2015-11-24T10:23:29Z|00006|dpif_netdev(pmd18)|INFO|Core 2 processing port
>> 'vhufb8052e5-d3'
>> 2015-11-24T10:23:29Z|00003|dpif_netdev(pmd19)|INFO|Core 8 processing port
>> 'dpdk0'
>> 2015-11-24T10:23:29Z|00136|bridge|INFO|bridge br-int: added interface
>> vhufb8052e5-d3 on port 7
>> 2015-11-24T10:23:30Z|00007|dpif_netdev(pmd18)|INFO|Core 2 processing port
>> 'dpdk0'
>> 0
>>
>> I understand that ovs-dpdk is running. The error log of n-cpu.log is
>>
>> 2015-11-24 15:47:27.957 ^[[01;31mERROR nova.compute.manager
>> [^[[01;36mreq-24fc3f16-ccd5-4e2d-b583-60ade23bc1ed ^[[00;36mNone
>> None^[[01;31m] ^[[01;35m^[[01;31mNo compute node record for host
>> ubuntu-Precision-Tower-5810^[[00m
>> 2015-11-24 15:47:27.964 ^[[01;33mWARNING nova.compute.monitors
>> [^[[01;36mreq-24fc3f16-ccd5-4e2d-b583-60ade23bc1ed ^[[00;36mNone
>> None^[[01;33m] ^[[01;35m^[[01;33mExcluding nova.compute.monitors.cpu
>> monitor virt_driver. Not in the list of enabled monitors
>> (CONF.compute_monitors).^[[00m
>> 2015-11-24 15:47:27.964 ^[[00;36mINFO nova.compute.resource_tracker
>> [^[[01;36mreq-24fc3f16-ccd5-4e2d-b583-60ade23bc1ed ^[[00;36mNone
>> None^[[00;36m] ^[[01;35m^[[00;36mAuditing locally available compute
>> resources for node ubuntu-Precision-Tower-5810^[[00m
>> 2015-11-24 15:47:28.087 ^[[00;32mDEBUG nova.compute.resource_tracker
>> [^[[01;36mreq-24fc3f16-ccd5-4e2d-b583-60ade23bc1ed ^[[00;36mNone
>> None^[[00;32m] ^[[01;35m^[[00;32mHypervisor: free VCPUs: 9^[[00m
>> ^[[00;33mfrom (pid=7880) _report_hypervisor_resource_view
>> /opt/stack/nova/nova/compute/resource_tracker.py:588^[[00m
>>
>> I dont understand where am I missing out now.
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Nov 23, 2015 at 7:50 PM, Mooney, Sean K <sean.k.mooney at intel.com>
>> wrote:
>>
>> Hi
>>
>> qemu version was 2.0.0 does not support mapping hugepage as shared.
>> as a result the dpdk implementation of vhost-user cannot function with
>> this version.
>> Similarly libvirt 1.2.2 has no knowledge of vhost-user.
>>
>> If you are on fedora 21 then the virt-preview repo packages the required
>> Libvirt and qemu.
>>
>> On Ubuntu the kilo cloud archive also packages version that meet the
>> minimum versions.
>>
>> Looking at the log I would agree that this error is probably related to
>> manually installing Libvirt and
>> not adding the appropriate libvirtd configuration options.
>>
>> Devstack does not have native support for installing a vhost-user
>> compatible Libvirt or qemu.
>> when using our networking-ovs-dpdk plugin we ask you to enable the
>> virt-prevew or cloud archive
>> Before stacking so that when devestack install Libvirt/qemu it get
>> compatible versions.
>> for centos as we have been unable to find an equivalent we explicitly
>> install Libvirt/qemu.
>>
>> In general we don’t provide support for installing the required version
>> Libvirt/qemu in
>> the networking-ovs-dpdk devstack plugin.
>>
>> What os are you currently using? ubuntu?
>> in your case enabling the kilo cloud archive, uinstalling Libvirt/qemu
>> and then restacking
>> should provide the appropriate packages.
>>
>> In liberty we use that standard neutron openvswich agent binary and set
>> the agent type to DPDK OVS Agent to enable
>> the ovsdpdk ml2 driver to manage the node. In mitaka the standard ovs
>> neutron agent and standard openvswitch ml2 drivers will be used instead.
>>
>>
>>
>> To confirm if ovs-dpdk is running you can use the following command
>>
>> sudo service ovs-dpdk status
>>
>> this will check if ovs-dpdk is running by reading the pid files and
>> checking if the ovsdb and ovs-vswitchd
>> process are running and tail the last 20ish lines of the vswitchd log
>> file.
>>
>> note this is not a systemd service so the systemd equivalent command will
>> not work.
>>
>>
>>
>> Alternative you can user ps and grep
>>
>> ps aux | grep ovs
>>
>>
>>
>> regards
>>
>> sean.
>>
>>
>>
>>
>>
>>
>>
>> *From:* Prathyusha Guduri [mailto:prathyushaconnects at gmail.com]
>> *Sent:* Monday, November 23, 2015 9:16 AM
>> *To:* OpenStack Development Mailing List (not for usage questions)
>> *Subject:* Re: [openstack-dev] [networking-ovs-dpdk]
>>
>>
>>
>> Hi Sean,
>>
>> Thanks for your help before. It would be great if you look into another
>> issue too.
>>
>> Am able to run stack.sh successfully and all services are up. But,
>>
>> libvirt version was 1.2.2 and qemu version was 2.0.0
>>
>> To satisfy the minimum requirement of qemu- version >=2.1 and
>> libvirt-version >= 1.2.10
>>
>> I manually installed qemu and libvirt from respective sources.
>>
>> Now
>> $ kvm --version
>> /usr/bin/kvm: line 42: /tmp/qemu.orig: Permission denied
>> QEMU emulator version 2.1.3, Copyright (c) 2003-2008 Fabrice Bellard
>>
>> $ virsh --version
>> 1.2.10
>>
>> So basic requirement is satisfied.
>>
>> Before creating an instance ran the below command,
>> $ nova flavor-key m1.tiny set "hw:mem_page_size=large"
>>
>> Now created an instance
>> $ nova boot --flavor m1.tiny --image cirros-0.3.4-x86_64-uec --nic
>> net-id=445e2dc5-221b-48ea-aea4-d04dee12fc7f --security-group default
>> demo-instance1
>>
>> It gives the ERROR :
>>
>> 2015-11-23 13:19:59.654 ERROR nova.virt.libvirt.host
>> [req-2d9d060d-1934-4e9e-af1c-010e177bea11 None None] Connection to
>> libvirt failed: error from service: CheckAuthorization: Did not receive
>> a reply. Possible causes include: the remote application did not send a
>> reply, the message bus security policy blocked the reply, the reply timeout
>> expired, or the network connection was broken.
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host Traceback (most
>> recent call last):
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File
>> "/opt/stack/nova/nova/virt/libvirt/host.py", line 527, in get_connection
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host conn =
>> self._get_connection()
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File
>> "/opt/stack/nova/nova/virt/libvirt/host.py", line 514, in _get_connection
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host wrapped_conn =
>> self._get_new_connection()
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File
>> "/opt/stack/nova/nova/virt/libvirt/host.py", line 466, in
>> _get_new_connection
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host wrapped_conn =
>> self._connect(self._uri, self._read_only)
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File
>> "/opt/stack/nova/nova/virt/libvirt/host.py", line 320, in _connect
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host
>> libvirt.openAuth, uri, auth, flags)
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File
>> "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in
>> proxy_call
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host rv = execute(f,
>> *args, **kwargs)
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File
>> "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in
>> execute
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host six.reraise(c,
>> e, tb)
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File
>> "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in
>> tworker
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host rv = meth(*args,
>> **kwargs)
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File
>> "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 105, in openAuth
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host if ret is
>> None:raise libvirtError('virConnectOpenAuth() failed')
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host libvirtError: error
>> from service: CheckAuthorization: Did not receive a reply. Possible causes
>> include: the remote application did not send a reply, the message bus
>> security policy blocked the reply, the reply timeout expired, or the
>> network connection was broken.
>> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host
>> Traceback (most recent call last):
>>
>> I suspect this is because, I manually installed libvirt and qemu. My
>> doubt is why devstack is not installing a correct version when it is
>> supposed to. why a version less than min requirement is being installed???
>>
>> Now that because am installing manually, there might be a problem with
>> groups - devstack creates some group and installs but manual installation
>> doesn't bother about that groups.
>> Can you please suggest a way on how do avoid that???
>>
>>
>>
>> Also, I just want to make sure that the agent running is
>> neutron-openvswitch only. No ovsdpdk agent running.
>>
>> $ ps -Al | grep neutron
>> 0 S 1000 8882 8859 3 80 0 - 49946 ep_pol pts/34 00:02:24
>> neutron-openvsw
>>
>> But
>>
>> $ neutron agent-list
>> 3385a430-5738-43cb-b853-059add5ab602 | DPDK OVS Agent |
>> ubuntu-Precision-Tower-5810 | :-) | True |
>> neutron-openvswitch-agent
>>
>> So this implies that dpdk agent is running right??? I remember reading in
>> launchpad bugs that ovsdpdk agent is removed and that now openvswitch takes
>> care of everything. Just wanted to confirm that my setup has ovs-dpdk
>> running.
>>
>> Regards,
>>
>> Prathyusha
>>
>>
>>
>>
>>
>> On Wed, Nov 18, 2015 at 7:23 PM, James Page <james.page at ubuntu.com>
>> wrote:
>>
>> Hi Sean
>>
>>
>>
>> On Wed, Nov 18, 2015 at 12:30 PM, Mooney, Sean K <sean.k.mooney at intel.com>
>> wrote:
>>
>> Hi james
>>
>> Yes we are planning on testing the packaged release to see if it is
>> compatible with our ml2 driver and the
>>
>> Changes we are submitting upstream. If it is we will add a use binary
>> flag to our devstack plugin to skip the
>>
>> Compilation step and use that instead on 15.10 or 14.04
>> cloud-archive:liberty
>>
>>
>>
>> Excellent.
>>
>>
>>
>> As part of your packaging did ye fix pciutils to correctly report the
>> unused drivers when an interface is bound
>>
>> The dpdk driver? Also does it support both igb_uio and/or vfio-pci
>> drivers for dpdk interface?
>>
>>
>>
>> Re pcituils, we've not done any work in that area - can you give an
>> example of what you would expect?
>>
>>
>>
>> The dpdk package supports both driver types in /etc/dpdk/interfaces -
>> when you declare an adapter for use, you get to specify the module you want
>> to use as well; we're relying the in-tree kernel drivers (uio-pci-generic
>> and vfio-pci) right now.
>>
>>
>>
>>
>>
>> Anyway yes I hope to check it out and seeing what ye have done. When
>> ovs-dpdk starts getting packaged in more operating systems
>>
>> We will probably swap our default to the binary install though we will
>> keep the source install option as it allows us to work on new features
>>
>> Before they are packaged and to have better performance.
>>
>>
>>
>> That sounds sensible; re 'better performance' - yeah we do have to
>> baseline the optimizations at compile time right now (ssse3 only right now)
>> , but I really hope that does change so that we can move to a runtime CPU
>> feature detection model, allowing the best possible performance through the
>> packages we have in Ubuntu (or any other distribution for that matter).
>>
>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>
>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Best Regards.
> Hui.
>
> OpenStack Engineer
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151125/3e359145/attachment.html>
More information about the OpenStack-dev
mailing list