[openstack-dev] [networking-ovs-dpdk]
Hui Xiang
hui.xiang at canonical.com
Wed Nov 25 01:58:35 UTC 2015
FYI, bug 1513367 is opened for this apparmor problem to track when booting
vms failed with ovs-dpdk enabled.
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1513367
On Tue, Nov 24, 2015 at 9:49 PM, Mooney, Sean K <sean.k.mooney at intel.com>
wrote:
> Out of interest
>
> Have you removed apparmor or placed all Libvirt apparmor profies into
> complain mode?
>
>
>
> If not you will get permission denied errors.
>
>
>
> You can confirm by checking dmesg to see if you have any permission denied
> messages from apparmor
>
> Or run aa-status and see if the the Libvirt profie is in enforce/complain
> mode.
>
>
>
> The /tmp/qemu.orig file is just a file we write the original qemu command
> to for debugging. It is not needed
>
> But all uses should be able to read/write to /tmp.
>
>
>
> We wrap the qemu/kvm binary with a script that on Ubuntu can be found here
> /usr/bin/kvm
>
>
>
> If you comment out echo "qemu ${args[@]}" > /tmp/qemu.orig in this script
> it will silence that warning.
>
>
>
>
> https://github.com/openstack/networking-ovs-dpdk/blob/master/devstack/libs/ovs-dpdk#L104
>
>
>
> I may remove this from our wrapper script as we most never use it for
> debugging anymore however in the past it was
>
> Useful to compare the original qemu command line and the update qemu
> command line.
>
>
>
> I don’t know if I have mentioned this before but we also have a Ubuntu
> version of our getting start guide that should merge shortly
>
>
>
> https://review.openstack.org/#/c/243190/6/doc/source/getstarted/ubuntu.rst
>
>
>
> Regards
>
> Sean.
>
>
>
> *From:* Prathyusha Guduri [mailto:prathyushaconnects at gmail.com]
> *Sent:* Tuesday, November 24, 2015 12:42 PM
> *To:* Mooney, Sean K
> *Cc:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [networking-ovs-dpdk]
>
>
>
> Hi All,
>
> I also found another error while launching an instance.
>
> libvirtError: internal error: process exited while connecting to monitor:
> /usr/bin/kvm-spice: line 42: /tmp/qemu.orig: Permission denied
>
> I dont want to change any permissions manually and again face the
> dependency issues. So kindly help
>
> Thanks,
>
> Prathyusha
>
>
>
>
>
>
>
> On Tue, Nov 24, 2015 at 4:02 PM, Prathyusha Guduri <
> prathyushaconnects at gmail.com> wrote:
>
> Hi Sean,
>
> Thanks for you kind help.
>
> I did the following.
>
> # apt-get install ubuntu-cloud-keyring
> # echo "deb http://ubuntu-cloud.archive.canonical.com/ubuntu
> <http://www.google.com/url?q=http%3A%2F%2Fubuntu-cloud.archive.canonical.com%2Fubuntu&sa=D&sntz=1&usg=AFQjCNGlCfoplf1zSbILNxCSVK23zRxa2A>"
> \
> "trusty-updates/kilo main" > /etc/apt/sources.list.d/cloudarchive-kilo.list
> # apt-get update && apt-get dist-upgrade
>
> and then uninstalled the libvirt and qemu that were installed manually and
> then ran stack.sh after cleaning and unstacking.
>
> Now fortunately libvirt and qemu satisfy minimum requirements.
>
> $ virsh --version
> 1.2.12
>
> $ kvm --version
> /usr/bin/kvm: line 42: /tmp/qemu.orig: Permission denied
> QEMU emulator version 2.2.0 (Debian 1:2.2+dfsg-5expubuntu9.3~cloud0),
> Copyright (c) 2003-2008 Fabrice Bellard
>
> Am using an ubuntu 14.04 system
> $ uname -a
> Linux ubuntu-Precision-Tower-5810 3.13.0-24-generic #46-Ubuntu SMP Thu Apr
> 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>
> After stack.sh which was successful, tried creating a new instance - which
> gave an ERROR again.
>
> $ nova list
>
> +--------------------------------------+----------------+--------+------------+-------------+-------------------------------------------------------+
> | ID | Name | Status | Task
> State | Power State | Networks
> |
>
> +--------------------------------------+----------------+--------+------------+-------------+-------------------------------------------------------+
> | 31a7e160-d04c-4216-91cf-30ce86c2b1fa | demo-instance1 | ERROR |
> - | NOSTATE | private=10.0.0.3,
> fd34:f4c5:412:0:f816:3eff:fea4:b9fe |
>
> $ sudo service ovs-dpdk status
> sourcing config
> ovs alive
> VHOST_CONFIG: bind to /var/run/openvswitch/vhufb8052e5-d3
> 2015-11-24T10:23:25Z|00126|dpdk|INFO|Socket
> /var/run/openvswitch/vhufb8052e5-d3 created for vhost-user port
> vhufb8052e5-d3
> 2015-11-24T10:23:25Z|00004|dpif_netdev(pmd18)|INFO|Core 2 processing port
> 'vhufb8052e5-d3'
> 2015-11-24T10:23:25Z|00002|dpif_netdev(pmd19)|INFO|Core 8 processing port
> 'dpdk0'
> 2015-11-24T10:23:25Z|00127|bridge|INFO|bridge br-int: added interface
> vhufb8052e5-d3 on port 6
> 2015-11-24T10:23:25Z|00005|dpif_netdev(pmd18)|INFO|Core 2 processing port
> 'dpdk0'
> 2015-11-24T10:23:26Z|00128|connmgr|INFO|br-int<->unix: 1 flow_mods in the
> last 0 s (1 deletes)
> 2015-11-24T10:23:26Z|00129|ofp_util|INFO|normalization changed ofp_match,
> details:
> 2015-11-24T10:23:26Z|00130|ofp_util|INFO| pre:
> in_port=5,nw_proto=58,tp_src=136
> 2015-11-24T10:23:26Z|00131|ofp_util|INFO|post: in_port=5
> 2015-11-24T10:23:26Z|00132|connmgr|INFO|br-int<->unix: 1 flow_mods in the
> last 0 s (1 deletes)
> 2015-11-24T10:23:26Z|00133|connmgr|INFO|br-int<->unix: 1 flow_mods in the
> last 0 s (1 deletes)
> 2015-11-24T10:23:29Z|00134|bridge|WARN|could not open network device
> vhufb8052e5-d3 (No such device)
> VHOST_CONFIG: socket created, fd:52
> VHOST_CONFIG: bind to /var/run/openvswitch/vhufb8052e5-d3
> 2015-11-24T10:23:29Z|00135|dpdk|INFO|Socket
> /var/run/openvswitch/vhufb8052e5-d3 created for vhost-user port
> vhufb8052e5-d3
> 2015-11-24T10:23:29Z|00006|dpif_netdev(pmd18)|INFO|Core 2 processing port
> 'vhufb8052e5-d3'
> 2015-11-24T10:23:29Z|00003|dpif_netdev(pmd19)|INFO|Core 8 processing port
> 'dpdk0'
> 2015-11-24T10:23:29Z|00136|bridge|INFO|bridge br-int: added interface
> vhufb8052e5-d3 on port 7
> 2015-11-24T10:23:30Z|00007|dpif_netdev(pmd18)|INFO|Core 2 processing port
> 'dpdk0'
> 0
>
> I understand that ovs-dpdk is running. The error log of n-cpu.log is
>
> 2015-11-24 15:47:27.957 ^[[01;31mERROR nova.compute.manager
> [^[[01;36mreq-24fc3f16-ccd5-4e2d-b583-60ade23bc1ed ^[[00;36mNone
> None^[[01;31m] ^[[01;35m^[[01;31mNo compute node record for host
> ubuntu-Precision-Tower-5810^[[00m
> 2015-11-24 15:47:27.964 ^[[01;33mWARNING nova.compute.monitors
> [^[[01;36mreq-24fc3f16-ccd5-4e2d-b583-60ade23bc1ed ^[[00;36mNone
> None^[[01;33m] ^[[01;35m^[[01;33mExcluding nova.compute.monitors.cpu
> monitor virt_driver. Not in the list of enabled monitors
> (CONF.compute_monitors).^[[00m
> 2015-11-24 15:47:27.964 ^[[00;36mINFO nova.compute.resource_tracker
> [^[[01;36mreq-24fc3f16-ccd5-4e2d-b583-60ade23bc1ed ^[[00;36mNone
> None^[[00;36m] ^[[01;35m^[[00;36mAuditing locally available compute
> resources for node ubuntu-Precision-Tower-5810^[[00m
> 2015-11-24 15:47:28.087 ^[[00;32mDEBUG nova.compute.resource_tracker
> [^[[01;36mreq-24fc3f16-ccd5-4e2d-b583-60ade23bc1ed ^[[00;36mNone
> None^[[00;32m] ^[[01;35m^[[00;32mHypervisor: free VCPUs: 9^[[00m
> ^[[00;33mfrom (pid=7880) _report_hypervisor_resource_view
> /opt/stack/nova/nova/compute/resource_tracker.py:588^[[00m
>
> I dont understand where am I missing out now.
>
>
>
>
>
>
>
> On Mon, Nov 23, 2015 at 7:50 PM, Mooney, Sean K <sean.k.mooney at intel.com>
> wrote:
>
> Hi
>
> qemu version was 2.0.0 does not support mapping hugepage as shared.
> as a result the dpdk implementation of vhost-user cannot function with
> this version.
> Similarly libvirt 1.2.2 has no knowledge of vhost-user.
>
> If you are on fedora 21 then the virt-preview repo packages the required
> Libvirt and qemu.
>
> On Ubuntu the kilo cloud archive also packages version that meet the
> minimum versions.
>
> Looking at the log I would agree that this error is probably related to
> manually installing Libvirt and
> not adding the appropriate libvirtd configuration options.
>
> Devstack does not have native support for installing a vhost-user
> compatible Libvirt or qemu.
> when using our networking-ovs-dpdk plugin we ask you to enable the
> virt-prevew or cloud archive
> Before stacking so that when devestack install Libvirt/qemu it get
> compatible versions.
> for centos as we have been unable to find an equivalent we explicitly
> install Libvirt/qemu.
>
> In general we don’t provide support for installing the required version
> Libvirt/qemu in
> the networking-ovs-dpdk devstack plugin.
>
> What os are you currently using? ubuntu?
> in your case enabling the kilo cloud archive, uinstalling Libvirt/qemu and
> then restacking
> should provide the appropriate packages.
>
> In liberty we use that standard neutron openvswich agent binary and set
> the agent type to DPDK OVS Agent to enable
> the ovsdpdk ml2 driver to manage the node. In mitaka the standard ovs
> neutron agent and standard openvswitch ml2 drivers will be used instead.
>
>
>
> To confirm if ovs-dpdk is running you can use the following command
>
> sudo service ovs-dpdk status
>
> this will check if ovs-dpdk is running by reading the pid files and
> checking if the ovsdb and ovs-vswitchd
> process are running and tail the last 20ish lines of the vswitchd log file.
>
> note this is not a systemd service so the systemd equivalent command will
> not work.
>
>
>
> Alternative you can user ps and grep
>
> ps aux | grep ovs
>
>
>
> regards
>
> sean.
>
>
>
>
>
>
>
> *From:* Prathyusha Guduri [mailto:prathyushaconnects at gmail.com]
> *Sent:* Monday, November 23, 2015 9:16 AM
> *To:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [networking-ovs-dpdk]
>
>
>
> Hi Sean,
>
> Thanks for your help before. It would be great if you look into another
> issue too.
>
> Am able to run stack.sh successfully and all services are up. But,
>
> libvirt version was 1.2.2 and qemu version was 2.0.0
>
> To satisfy the minimum requirement of qemu- version >=2.1 and
> libvirt-version >= 1.2.10
>
> I manually installed qemu and libvirt from respective sources.
>
> Now
> $ kvm --version
> /usr/bin/kvm: line 42: /tmp/qemu.orig: Permission denied
> QEMU emulator version 2.1.3, Copyright (c) 2003-2008 Fabrice Bellard
>
> $ virsh --version
> 1.2.10
>
> So basic requirement is satisfied.
>
> Before creating an instance ran the below command,
> $ nova flavor-key m1.tiny set "hw:mem_page_size=large"
>
> Now created an instance
> $ nova boot --flavor m1.tiny --image cirros-0.3.4-x86_64-uec --nic
> net-id=445e2dc5-221b-48ea-aea4-d04dee12fc7f --security-group default
> demo-instance1
>
> It gives the ERROR :
>
> 2015-11-23 13:19:59.654 ERROR nova.virt.libvirt.host
> [req-2d9d060d-1934-4e9e-af1c-010e177bea11 None None] Connection to
> libvirt failed: error from service: CheckAuthorization: Did not receive a
> reply. Possible causes include: the remote application did not send a
> reply, the message bus security policy blocked the reply, the reply timeout
> expired, or the network connection was broken.
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host Traceback (most
> recent call last):
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File
> "/opt/stack/nova/nova/virt/libvirt/host.py", line 527, in get_connection
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host conn =
> self._get_connection()
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File
> "/opt/stack/nova/nova/virt/libvirt/host.py", line 514, in _get_connection
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host wrapped_conn =
> self._get_new_connection()
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File
> "/opt/stack/nova/nova/virt/libvirt/host.py", line 466, in
> _get_new_connection
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host wrapped_conn =
> self._connect(self._uri, self._read_only)
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File
> "/opt/stack/nova/nova/virt/libvirt/host.py", line 320, in _connect
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host libvirt.openAuth,
> uri, auth, flags)
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File
> "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in
> proxy_call
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host rv = execute(f,
> *args, **kwargs)
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File
> "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in
> execute
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host six.reraise(c, e,
> tb)
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File
> "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in
> tworker
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host rv = meth(*args,
> **kwargs)
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File
> "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 105, in openAuth
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host if ret is
> None:raise libvirtError('virConnectOpenAuth() failed')
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host libvirtError: error
> from service: CheckAuthorization: Did not receive a reply. Possible causes
> include: the remote application did not send a reply, the message bus
> security policy blocked the reply, the reply timeout expired, or the
> network connection was broken.
> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host
> Traceback (most recent call last):
>
> I suspect this is because, I manually installed libvirt and qemu. My doubt
> is why devstack is not installing a correct version when it is supposed to.
> why a version less than min requirement is being installed???
>
> Now that because am installing manually, there might be a problem with
> groups - devstack creates some group and installs but manual installation
> doesn't bother about that groups.
> Can you please suggest a way on how do avoid that???
>
>
>
> Also, I just want to make sure that the agent running is
> neutron-openvswitch only. No ovsdpdk agent running.
>
> $ ps -Al | grep neutron
> 0 S 1000 8882 8859 3 80 0 - 49946 ep_pol pts/34 00:02:24
> neutron-openvsw
>
> But
>
> $ neutron agent-list
> 3385a430-5738-43cb-b853-059add5ab602 | DPDK OVS Agent |
> ubuntu-Precision-Tower-5810 | :-) | True |
> neutron-openvswitch-agent
>
> So this implies that dpdk agent is running right??? I remember reading in
> launchpad bugs that ovsdpdk agent is removed and that now openvswitch takes
> care of everything. Just wanted to confirm that my setup has ovs-dpdk
> running.
>
> Regards,
>
> Prathyusha
>
>
>
>
>
> On Wed, Nov 18, 2015 at 7:23 PM, James Page <james.page at ubuntu.com> wrote:
>
> Hi Sean
>
>
>
> On Wed, Nov 18, 2015 at 12:30 PM, Mooney, Sean K <sean.k.mooney at intel.com>
> wrote:
>
> Hi james
>
> Yes we are planning on testing the packaged release to see if it is
> compatible with our ml2 driver and the
>
> Changes we are submitting upstream. If it is we will add a use binary flag
> to our devstack plugin to skip the
>
> Compilation step and use that instead on 15.10 or 14.04
> cloud-archive:liberty
>
>
>
> Excellent.
>
>
>
> As part of your packaging did ye fix pciutils to correctly report the
> unused drivers when an interface is bound
>
> The dpdk driver? Also does it support both igb_uio and/or vfio-pci drivers
> for dpdk interface?
>
>
>
> Re pcituils, we've not done any work in that area - can you give an
> example of what you would expect?
>
>
>
> The dpdk package supports both driver types in /etc/dpdk/interfaces - when
> you declare an adapter for use, you get to specify the module you want to
> use as well; we're relying the in-tree kernel drivers (uio-pci-generic and
> vfio-pci) right now.
>
>
>
>
>
> Anyway yes I hope to check it out and seeing what ye have done. When
> ovs-dpdk starts getting packaged in more operating systems
>
> We will probably swap our default to the binary install though we will
> keep the source install option as it allows us to work on new features
>
> Before they are packaged and to have better performance.
>
>
>
> That sounds sensible; re 'better performance' - yeah we do have to
> baseline the optimizations at compile time right now (ssse3 only right now)
> , but I really hope that does change so that we can move to a runtime CPU
> feature detection model, allowing the best possible performance through the
> packages we have in Ubuntu (or any other distribution for that matter).
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
--
Best Regards.
Hui.
OpenStack Engineer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151125/4c0935cb/attachment-0001.html>
More information about the OpenStack-dev
mailing list