[openstack-dev] [networking-ovs-dpdk]
Prathyusha Guduri
prathyushaconnects at gmail.com
Wed Nov 18 12:57:59 UTC 2015
Hi James,
Thanks for the information. Will experiment on it and share my
observations.
Sean - I guess this timing issue is fixed in DPDK 2.1. Will try with that
and let you know. Anyway, thanks for the help
Regards,
Prathyusha
On Wed, Nov 18, 2015 at 6:00 PM, Mooney, Sean K <sean.k.mooney at intel.com>
wrote:
> Hi james
>
> Yes we are planning on testing the packaged release to see if it is
> compatible with our ml2 driver and the
>
> Changes we are submitting upstream. If it is we will add a use binary flag
> to our devstack plugin to skip the
>
> Compilation step and use that instead on 15.10 or 14.04
> cloud-archive:liberty
>
>
>
> As part of your packaging did ye fix pciutils to correctly report the
> unused drivers when an interface is bound
>
> The dpdk driver? Also does it support both igb_uio and/or vfio-pci drivers
> for dpdk interface?
>
>
>
> Anyway yes I hope to check it out and seeing what ye have done. When
> ovs-dpdk starts getting packaged in more operating systems
>
> We will probably swap our default to the binary install though we will
> keep the source install option as it allows us to work on new features
>
> Before they are packaged and to have better performance.
>
>
>
> Regards
>
> sean
>
>
>
>
>
> *From:* James Page [mailto:james.page at ubuntu.com]
> *Sent:* Wednesday, November 18, 2015 11:55 AM
>
> *To:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [networking-ovs-dpdk]
>
>
>
> Hi All
>
>
>
> Reading through this thread I thought participants might be interested to
> know that Ubuntu now provides a DPDK enabled Open vSwitch package in Ubuntu
> 15.10 and for Ubuntu 14.04 using the Liberty UCA; you can use it as follows:
>
>
>
> sudo add-apt-repository cloud-archive:liberty # not required for 15.10
>
> sudo apt-get update
>
> sudo apt-get install openvswitch-switch-dpdk
>
> sudo update-alternatives --set
> ovs-vswitchd /usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk
>
> sudo service openvswitch-switch restart
>
>
>
> DPDK configuration is passed through OVS using
> /etc/default/openvswitch-switch (read for details). The dpdk package also
> provides some basic configuration mechanisms for hugepages and binding
> networking adapters to userspace in /etc/dpdk.
>
>
>
> This is a first packaging release and is experimental - please do provide
> feedback on what you do and don't like about it...
>
>
>
> On Wed, Nov 18, 2015 at 6:12 AM, Prathyusha Guduri <
> prathyushaconnects at gmail.com> wrote:
>
> Thanks a lot Sean, that was helpful.
>
> Changing the target from ivshmem to native-linuxapp removed the error and
> it doesn't hang at creating external bridge anymore.
>
> All processes(nova-api, neutron, ovs-vswitchd, etc) did start.
>
> Thanks,
>
> Prathyusha
>
>
>
> On Tue, Nov 17, 2015 at 7:57 PM, Mooney, Sean K <sean.k.mooney at intel.com>
> wrote:
>
> We mainly test with 2M hugepages not 1G however our ci does use 1G pages.
>
> We recently noticed a different but unrelated related issue with using the
> ivshmem target when building dpdk.
>
> (https://bugs.launchpad.net/networking-ovs-dpdk/+bug/1517032)
>
>
>
> Instead of modifying dpdk can you try
>
> Changing the default dpdk build target to x86_64-native-linuxapp-gcc.
>
>
>
> This can be done by adding
>
>
>
> RTE_TARGET=x86_64-native-linuxapp-gcc to the local.conf
>
> And removing the following file to force a rebuild
> “/opt/stack/ovs/BUILD_COMPLETE”
>
>
>
> I agree with your assessment though this appears to be a timing issue in
> dpdk 2.0
>
>
>
>
>
>
>
> *From:* Prathyusha Guduri [mailto:prathyushaconnects at gmail.com]
> *Sent:* Tuesday, November 17, 2015 1:42 PM
> *To:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [networking-ovs-dpdk]
>
>
>
> Here is stack.sh log -
>
>
> 2015-11-17 13:38:50.010 | Loading uio module
> 2015-11-17 13:38:50.028 | Loading DPDK UIO module
> 2015-11-17 13:38:50.038 | starting ovs db
> 2015-11-17 13:38:50.038 | binding nics
> 2015-11-17 13:38:50.039 | starting vswitchd
> 2015-11-17 13:38:50.190 | sudo RTE_SDK=/opt/stack/DPDK-v2.0.0
> RTE_TARGET=build /opt/stack/DPDK-v2.0.0/tools/dpdk_nic_bind.py -b igb_uio
> 0000:07:00.0
> 2015-11-17 13:38:50.527 | sudo ovs-vsctl --no-wait --may-exist add-port
> br-eth1 dpdk0 -- set Interface dpdk0 type=dpdk
> 2015-11-17 13:38:51.671 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:38:52.685 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:38:53.702 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:38:54.720 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:38:55.733 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:38:56.749 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:38:57.768 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:38:58.787 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:38:59.802 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:00.818 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:01.836 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:02.849 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:03.866 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:04.884 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:05.905 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:06.923 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:07.937 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:08.956 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:09.973 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:10.988 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:12.004 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:13.022 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:14.040 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:15.060 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:16.073 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:17.089 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:18.108 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:19.121 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:20.138 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:21.156 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:22.169 | Waiting for ovs-vswitchd to start...
> 2015-11-17 13:39:23.185 | Waiting for ovs-vswitchd to start...
>
>
>
> On Tue, Nov 17, 2015 at 6:50 PM, Prathyusha Guduri <
> prathyushaconnects at gmail.com> wrote:
>
> Hi Sean,
>
> Here is ovs-vswitchd.log
>
>
> 2015-11-13T12:48:01Z|00001|dpdk|INFO|User-provided -vhost_sock_dir in use:
> /var/run/openvswitch
> EAL: Detected lcore 0 as core 0 on socket 0
> EAL: Detected lcore 1 as core 1 on socket 0
> EAL: Detected lcore 2 as core 2 on socket 0
> EAL: Detected lcore 3 as core 3 on socket 0
> EAL: Detected lcore 4 as core 4 on socket 0
> EAL: Detected lcore 5 as core 5 on socket 0
> EAL: Detected lcore 6 as core 0 on socket 0
> EAL: Detected lcore 7 as core 1 on socket 0
> EAL: Detected lcore 8 as core 2 on socket 0
> EAL: Detected lcore 9 as core 3 on socket 0
> EAL: Detected lcore 10 as core 4 on socket 0
> EAL: Detected lcore 11 as core 5 on socket 0
> EAL: Support maximum 128 logical core(s) by configuration.
> EAL: Detected 12 lcore(s)
> EAL: VFIO modules not all loaded, skip VFIO support...
> EAL: Searching for IVSHMEM devices...
> EAL: No IVSHMEM configuration found!
> EAL: Setting up memory...
> EAL: Ask a virtual area of 0x180000000 bytes
> EAL: Virtual area found at 0x7f1e00000000 (size = 0x180000000)
> EAL: remap_all_hugepages(): mmap failed: Cannot allocate memory
> EAL: Failed to remap 1024 MB pages
> PANIC in rte_eal_init():
> Cannot init memory
> 7: [/usr/sbin/ovs-vswitchd() [0x40b803]]
> 6: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)
> [0x7f1fb52d3ec5]]
> 5: [/usr/sbin/ovs-vswitchd() [0x40a822]]
> 4: [/usr/sbin/ovs-vswitchd() [0x675432]]
> 3: [/usr/sbin/ovs-vswitchd() [0x442155]]
> 2: [/usr/sbin/ovs-vswitchd() [0x407c9f]]
> 1: [/usr/sbin/ovs-vswitchd() [0x447828]]
>
> Before this hugepages were free and port binding was also done. So I
> suspected that this is a DPDK specific issue and found that in
> remap_all_hugepages( ) of
> /opt/stack/DPDK-v2.0.0/lib/librte_eal/linuxapp/eal/eal_memory.c which first
> unmaps and then mmaps, there is an issue here and so mmap here fails. In
> DPDK mailing list I found that the unmap is taking longer time because of
> which mmap fails, so putting a sleep(1) between unmap and map is supposed
> to solve the issue. Please check the below link :
> https://lists.01.org/pipermail/dpdk-ovs/2014-April/000864.html
>
> After changing so, the ovs-vswitchd command hangs at this place
>
> 2015-11-17T10:52:38Z|00001|dpdk|INFO|User-provided -vhost_sock_dir in use:
> /var/run/openvswitch
> 2015-11-17 10:52:38.680 | EAL: Detected lcore 0 as core 0 on socket 0
> 2015-11-17 10:52:38.680 | EAL: Detected lcore 1 as core 1 on socket 0
> 2015-11-17 10:52:38.680 | EAL: Detected lcore 2 as core 2 on socket 0
> 2015-11-17 10:52:38.680 | EAL: Detected lcore 3 as core 3 on socket 0
> 2015-11-17 10:52:38.680 | EAL: Detected lcore 4 as core 4 on socket 0
> 2015-11-17 10:52:38.680 | EAL: Detected lcore 5 as core 5 on socket 0
> 2015-11-17 10:52:38.680 | EAL: Detected lcore 6 as core 0 on socket 0
> 2015-11-17 10:52:38.680 | EAL: Detected lcore 7 as core 1 on socket 0
> 2015-11-17 10:52:38.680 | EAL: Detected lcore 8 as core 2 on socket 0
> 2015-11-17 10:52:38.680 | EAL: Detected lcore 9 as core 3 on socket 0
> 2015-11-17 10:52:38.680 | EAL: Detected lcore 10 as core 4 on socket 0
> 2015-11-17 10:52:38.680 | EAL: Detected lcore 11 as core 5 on socket 0
> 2015-11-17 10:52:38.680 | EAL: Support maximum 128 logical core(s) by
> configuration.
> 2015-11-17 10:52:38.680 | EAL: Detected 12 lcore(s)
> 2015-11-17 10:52:38.687 | EAL: VFIO modules not all loaded, skip VFIO
> support...
> 2015-11-17 10:52:38.687 | EAL: Searching for IVSHMEM devices...
> 2015-11-17 10:52:38.687 | EAL: No IVSHMEM configuration found!
> 2015-11-17 10:52:38.687 | EAL: Setting up memory...
> 2015-11-17 10:52:39.252 | EAL: Ask a virtual area of 0x1c00000 bytes
> 2015-11-17 10:52:39.252 | EAL: Virtual area found at 0x7fcab3a00000 (size
> = 0x1c00000)
> 2015-11-17 10:52:53.265 | EAL: Ask a virtual area of 0x200000 bytes
> 2015-11-17 10:52:53.266 | EAL: Virtual area found at 0x7fcab3600000 (size
> = 0x200000)
> 2015-11-17 10:52:54.266 | EAL: Ask a virtual area of 0x200000 bytes
> 2015-11-17 10:52:54.266 | EAL: Virtual area found at 0x7fcab3200000 (size
> = 0x200000)
> 2015-11-17 10:52:55.267 | EAL: Ask a virtual area of 0x22c00000 bytes
> 2015-11-17 10:52:55.267 | EAL: Virtual area found at 0x7fca90400000 (size
> = 0x22c00000)
> 2015-11-17 10:57:33.574 | EAL: Ask a virtual area of 0x1800000 bytes
> 2015-11-17 10:57:33.574 | EAL: Virtual area found at 0x7fca8ea00000 (size
> = 0x1800000)
> 2015-11-17 10:57:45.585 | EAL: Ask a virtual area of 0xd9800000 bytes
> 2015-11-17 10:57:45.585 | EAL: Virtual area found at 0x7fc9b5000000 (size
> = 0xd9800000)
> 2015-11-17 11:26:50.605 | EAL: Ask a virtual area of 0x200000 bytes
> 2015-11-17 11:26:50.605 | EAL: Virtual area found at 0x7fc9b4c00000 (size
> = 0x200000)
> 2015-11-17 11:26:51.606 | EAL: Ask a virtual area of 0x200000 bytes
> 2015-11-17 11:26:51.606 | EAL: Virtual area found at 0x7fc9b4800000 (size
> = 0x200000)
> 2015-11-17 11:26:52.608 | EAL: Requesting 1024 pages of size 2MB from
> socket 0
> 2015-11-17 11:26:53.111 | EAL: TSC frequency is ~3491914 KHz
> 2015-11-17 11:26:53.111 | EAL: Master lcore 1 is ready
> (tid=b73cd700;cpuset=[1])
> 2015-11-17 11:26:53.111 | PMD: ENICPMD trace: rte_enic_pmd_init
> 2015-11-17 11:26:53.111 | EAL: PCI device 0000:07:00.0 on NUMA socket 0
> 2015-11-17 11:26:53.111 | EAL: probe driver: 8086:10d3 rte_em_pmd
> 2015-11-17 11:26:53.111 | EAL: PCI memory mapped at 0x7fcab5600000
> 2015-11-17 11:26:53.111 | EAL: PCI memory mapped at 0x7fcab730f000
> 2015-11-17 11:26:53.111 | EAL: PCI memory mapped at 0x7fcab73d6000
> 2015-11-17 11:26:53.189 | PMD: eth_em_dev_init(): port_id 0
> vendorID=0x8086 deviceID=0x10d3
> 2015-11-17 11:26:53.190 |
> 2015-11-17T11:26:53Z|00002|ovs_numa|INFO|Discovered 12 CPU cores on NUMA
> node 0
> 2015-11-17 11:26:53.190 |
> 2015-11-17T11:26:53Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 12 CPU
> cores
> 2015-11-17 11:26:53.190 | 2015-11-17T11:26:53Z|00004|memory|INFO|10680 kB
> peak resident set size after 2054.5 seconds
> 2015-11-17 11:26:53.190 |
> 2015-11-17T11:26:53Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
> connecting...
> 2015-11-17 11:26:53.190 |
> 2015-11-17T11:26:53Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock:
> connected
> 2015-11-17 11:26:53.194 |
> 2015-11-17T11:26:53Z|00007|ofproto_dpif|INFO|netdev at ovs-netdev: Datapath
> supports recirculation
> 2015-11-17 11:26:53.194 |
> 2015-11-17T11:26:53Z|00008|ofproto_dpif|INFO|netdev at ovs-netdev: MPLS
> label stack length probed as 3
> 2015-11-17 11:26:53.194 |
> 2015-11-17T11:26:53Z|00009|ofproto_dpif|INFO|netdev at ovs-netdev: Datapath
> supports unique flow ids
> 2015-11-17 11:26:53.195 | 2015-11-17T11:26:53Z|00010|bridge|INFO|bridge
> br-eth1: added interface br-eth1 on port 65534
> 2015-11-17 11:26:53.197 |
> 2015-11-17T11:26:53Z|00011|dpif_netlink|ERR|Generic Netlink family
> 'ovs_datapath' does not exist. The Open vSwitch kernel module is probably
> not loaded.
> 2015-11-17 11:26:53.287 | Zone 0: name:<MALLOC_S0_HEAP_0>,
> phys:0x9b600000, len:0xb00000, virt:0x7fca8ea00000, socket_id:0, flags:0
> 2015-11-17 11:26:53.287 | Zone 1: name:<RG_MP_log_history>,
> phys:0x36600000, len:0x2080, virt:0x7fcab3600000, socket_id:0, flags:0
> 2015-11-17 11:26:53.287 | Zone 2: name:<MP_log_history>, phys:0x9c100000,
> len:0x28a0c0, virt:0x7fca8f500000, socket_id:0, flags:0
> 2015-11-17 11:26:53.287 | Zone 3: name:<rte_eth_dev_data>,
> phys:0x36602080, len:0x1f400, virt:0x7fcab3602080, socket_id:0, flags:0
> 2015-11-17 11:26:53.287 | PMD: eth_em_tx_queue_setup():
> sw_ring=0x7fca8f4efd40 hw_ring=0x7fcab3621480 dma_addr=0x36621480
> 2015-11-17 11:26:53.287 | PMD: eth_em_rx_queue_setup():
> sw_ring=0x7fca8f4ebc40 hw_ring=0x7fcab3631480 dma_addr=0x36631480
> 2015-11-17 11:26:53.368 | PMD: eth_em_start(): <<
> 2015-11-17 11:26:53.368 | 2015-11-17T11:26:53Z|00012|dpdk|INFO|Port 0:
> 68:05:ca:1b:ca:c9
> 2015-11-17 11:26:53.405 | PMD: eth_em_tx_queue_setup():
> sw_ring=0x7fca8f4efe00 hw_ring=0x7fcab3621480 dma_addr=0x36621480
> 2015-11-17 11:26:53.405 | PMD: eth_em_rx_queue_setup():
> sw_ring=0x7fca8f4ebdc0 hw_ring=0x7fcab3631480 dma_addr=0x36631480
> 2015-11-17 11:26:53.486 | PMD: eth_em_start(): <<
> 2015-11-17 11:26:53.486 | 2015-11-17T11:26:53Z|00013|dpdk|INFO|Port 0:
> 68:05:ca:1b:ca:c9
> 2015-11-17 11:26:53.487 |
> 2015-11-17T11:26:53Z|00014|dpif_netdev|INFO|Created 1 pmd threads on numa
> node 0
> 2015-11-17 11:26:53.487 |
> 2015-11-17T11:26:53Z|00001|dpif_netdev(pmd10)|INFO|Core 0 processing port
> 'dpdk0'
> 2015-11-17 11:26:53.488 |
> 2015-11-17T11:26:53Z|00002|dpif_netdev(pmd10)|INFO|Core 0 processing port
> 'dpdk0'
> 2015-11-17 11:26:53.488 | 2015-11-17T11:26:53Z|00015|bridge|INFO|bridge
> br-eth1: added interface dpdk0 on port 1
> 2015-11-17 11:26:53.488 | 2015-11-17T11:26:53Z|00016|bridge|INFO|bridge
> br-int: added interface br-int on port 65534
> 2015-11-17 11:26:53.488 | 2015-11-17T11:26:53Z|00017|bridge|INFO|bridge
> br-eth1: using datapath ID 00006805ca1bcac9
> 2015-11-17 11:26:53.488 | 2015-11-17T11:26:53Z|00018|connmgr|INFO|br-eth1:
> added service controller "punix:/var/run/openvswitch/br-eth1.mgmt"
> 2015-11-17 11:26:53.489 | 2015-11-17T11:26:53Z|00019|bridge|INFO|bridge
> br-int: using datapath ID 00002ef7b66a8742
> 2015-11-17 11:26:53.489 | 2015-11-17T11:26:53Z|00020|connmgr|INFO|br-int:
> added service controller "punix:/var/run/openvswitch/br-int.mgmt"
> 2015-11-17 11:26:53.490 |
> 2015-11-17T11:26:53Z|00021|dpif_netdev|INFO|Created 2 pmd threads on numa
> node 0
> 2015-11-17 11:26:53.492 |
> 2015-11-17T11:26:53Z|00022|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.4.90
> 2015-11-17 11:26:53.493 |
> 2015-11-17T11:26:53Z|00001|dpif_netdev(pmd23)|INFO|Core 2 processing port
> 'dpdk0'
> 2015-11-17 11:27:03.494 | 2015-11-17T11:27:03Z|00023|memory|INFO|peak
> resident set size grew 93% in last 10.3 seconds, from 10680 kB to 20572 kB
> 2015-11-17 11:27:03.494 |
> 2015-11-17T11:27:03Z|00024|memory|INFO|handlers:4 ports:3 revalidators:2
> rules:10
>
>
> ubuntu at ubuntu-Precision-Tower-5810:/opt/stack/DPDK-v2.0.0/lib/librte_eal/linuxapp/eal$
> ps -Al | grep ovs
> 5 S 0 1681 2595 0 80 0 - 4433 poll_s ? 00:00:00
> ovsdb-server
> 4 S 0 1716 1715 0 80 0 - 4636 wait pts/3 00:00:00 ovs-dpdk
> 4 S 0 2124 1716 99 80 0 - 870841 poll_s pts/3 03:42:31
> ovs-vswitchd
>
> So now ovs-vswitchd runs unlike the last time.
> I really dont understand where am missing out....
>
>
>
> On Tue, Nov 17, 2015 at 5:14 PM, Mooney, Sean K <sean.k.mooney at intel.com>
> wrote:
>
> Can you provide the ovs-vswitchd log form ${OVS_LOG_DIR}/ovs-vswitchd.log
>
> /tmp/ovs-vswitchd.log in your case.
>
>
>
> If the vswitch fails to start we clean up by unmounting the hugepages.
>
>
>
> *From:* Prathyusha Guduri [mailto:prathyushaconnects at gmail.com]
> *Sent:* Tuesday, November 17, 2015 7:37 AM
>
>
> *To:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [networking-ovs-dpdk]
>
>
>
> Hi Sean,
>
> I realised on debugging ovs-dpdk-init script that the main issue is with
> the following command
>
> $ screen -dms ovs-vswitchd sudo sg $qemu_group -c "umask 002;
> ${OVS_INSTALL_DIR}/sbin/ovs-vswitchd --dpdk -vhost_sock_dir
> $OVS_DB_SOCKET_DIR -c $OVS_CORE_MASK -n $OVS_MEM_CHANNELS --proc-type
> primary --huge-dir $OVS_HUGEPAGE_MOUNT --socket-mem $OVS_SOCKET_MEM
> $pciAddressWhitelist -- unix:$OVS_DB_SOCKET 2>&1 | tee
> ${OVS_LOG_DIR}/ovs-vswitchd.log"
>
> which I guess is starting the ovs-vswitchd application. Before this
> command, huge pages is mounted and port binding is also done but still the
> screen command fails.
> I verified the db.sock and conf.db files.
>
> Any help is highly appreciated.
>
> Thanks,
>
> Prathyusha
>
>
>
>
>
> On Mon, Nov 16, 2015 at 5:12 PM, Prathyusha Guduri <
> prathyushaconnects at gmail.com> wrote:
>
> Hi Sean,
>
> Thanks for your response.
>
> in your case though you are using 1GB hugepages so I don’t think this is
> related to memory fragmentation
> or a lack of free hugepages.
>
>
>
> to use preallocated 1GB page with ovs you should instead set the following
> in your local.conf
>
>
>
> OVS_HUGEPAGE_MOUNT_PAGESIZE=1G
>
> OVS_ALLOCATE_HUGEPAGES=False
>
> Added the above two parameters to the local.conf. The same problem again.
>
> Basically it throws this error -
> 2015-11-16 11:31:44.741 | starting vswitchd
> 2015-11-16 11:31:44.863 | sudo RTE_SDK=/opt/stack/DPDK-v2.0.0
> RTE_TARGET=build /opt/stack/DPDK-v2.0.0/tools/dpdk_nic_bind.py -b igb_uio
> 0000:07:00.0
> 2015-11-16 11:31:45.169 | sudo ovs-vsctl --no-wait --may-exist add-port
> br-eth1 dpdk0 -- set Interface dpdk0 type=dpdk
> 2015-11-16 11:31:46.314 | Waiting for ovs-vswitchd to start...
> 2015-11-16 11:31:47.442 | libvirt-bin stop/waiting
> 2015-11-16 11:31:49.473 | libvirt-bin start/running, process 2255
> 2015-11-16 11:31:49.477 | [ERROR] /etc/init.d/ovs-dpdk:563 ovs-vswitchd
> application failed to start
>
> manually mounting /mnt/huge and then commenting that part from the
> /etc/init.d/ovs-dpdk script also throws the same error.
>
> Using 1G hugepagesize should not give any memory related problem. I dont
> understand why it is not mounting then.
>
> Here is the /opt/stack/networking-ovs-dpdk/devstack/ovs-dpdk/ovs-dpdk.conf
>
> RTE_SDK=${RTE_SDK:-/opt/stack/DPDK}
> RTE_TARGET=${RTE_TARGET:-x86_64-ivshmem-linuxapp-gcc}
>
> OVS_INSTALL_DIR=/usr
> OVS_DB_CONF_DIR=/etc/openvswitch
> OVS_DB_SOCKET_DIR=/var/run/openvswitch
> OVS_DB_CONF=$OVS_DB_CONF_DIR/conf.db
> OVS_DB_SOCKET=OVS_DB_SOCKET_DIR/db.sock
>
> OVS_SOCKET_MEM=2048,2048
> OVS_MEM_CHANNELS=4
> OVS_CORE_MASK=${OVS_CORE_MASK:-2}
> OVS_PMD_CORE_MASK=${OVS_PMD_CORE_MASK:-4}
> OVS_LOG_DIR=/tmp
> OVS_LOCK_DIR=''
> OVS_SRC_DIR=/opt/stack/ovs
> OVS_DIR=${OVS_DIR:-${OVS_SRC_DIR}}
> OVS_UTILS=${OVS_DIR}/utilities/
> OVS_DB_UTILS=${OVS_DIR}/ovsdb/
> OVS_DPDK_DIR=$RTE_SDK
> OVS_NUM_HUGEPAGES=${OVS_NUM_HUGEPAGES:-5}
> OVS_HUGEPAGE_MOUNT=${OVS_HUGEPAGE_MOUNT:-/mnt/huge}
> OVS_HUGEPAGE_MOUNT_PAGESIZE=''
> OVS_BOND_MODE=$OVS_BOND_MODE
> OVS_BOND_PORTS=$OVS_BOND_PORTS
> OVS_BRIDGE_MAPPINGS=eth1
> OVS_PCI_MAPPINGS=0000:07:00.0#eth1
> OVS_DPDK_PORT_MAPPINGS=''
> OVS_TUNNEL_CIDR_MAPPING=''
> OVS_ALLOCATE_HUGEPAGES=True
> OVS_INTERFACE_DRIVER='igb_uio'
>
> Verified the OVS_DB_SOCKET_DIR and all others. conf.db and db.sock exist.
> So why ovs-vswitchd is failing to start??? Am I missing something???
>
>
>
>
>
> Thanks,
>
> Prathyusha
>
>
>
>
>
> On Mon, Nov 16, 2015 at 4:39 PM, Mooney, Sean K <sean.k.mooney at intel.com>
> wrote:
>
>
>
> Hi
>
>
>
> Yes sorry for the delay in responding to you and samta.
>
>
>
> In your case assuming you are using 2mb hugepages it is easy to hit dpdks
> default max memory segments
>
>
>
> This can be changed by setting OVS_DPDK_MEM_SEGMENTS=<arbitrary large
> number that you will never hit>
>
> In the local.conf and recompiling. To do this simply remove the build
> complete file in /opt/stack/ovs
>
> rm –f /opt/stack/BUILD_COMPLETE
>
>
>
> in your case though you are using 1GB hugepages so I don’t think this is
> related to memory fragmentation
> or a lack of free hugepages.
>
>
>
> to use preallocated 1GB page with ovs you should instead set the following
> in your local.conf
>
>
>
> OVS_HUGEPAGE_MOUNT_PAGESIZE=1G
>
> OVS_ALLOCATE_HUGEPAGES=False
>
>
>
> Regards
>
> sean
>
>
>
> *From:* Prathyusha Guduri [mailto:prathyushaconnects at gmail.com]
> *Sent:* Monday, November 16, 2015 6:20 AM
> *To:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [networking-ovs-dpdk]
>
>
>
> Hi all,
>
> I have a similar problem as Samta. Am also stuck at the same place. The
> following command
>
> $sudo ovs-vsctl br-set-external-id br-ex bridge-id br-ex
>
> hangs forever. As Sean said, it might be because of ovs-vswitchd proces.
>
> > The vswitchd process may exit if it failed to allocate memory (due to
> memory fragmentation or lack of free hugepages)
> > if the ovs-vswitchd.log is not available can you check the the hugepage
> mount point was created in
> > /mnt/huge And that Iis mounted
> > Run
> > ls -al /mnt/huge
> > and
> > mount
> >
>
> $mount
>
> /dev/sda6 on / type ext4 (rw,errors=remount-ro)
> proc on /proc type proc (rw,noexec,nosuid,nodev)
> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
> none on /sys/fs/cgroup type tmpfs (rw)
> none on /sys/fs/fuse/connections type fusectl (rw)
> none on /sys/kernel/debug type debugfs (rw)
> none on /sys/kernel/security type securityfs (rw)
> udev on /dev type devtmpfs (rw,mode=0755)
> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
> none on /run/shm type tmpfs (rw,nosuid,nodev)
> none on /run/user type tmpfs
> (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
> none on /sys/fs/pstore type pstore (rw)
> cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,relatime,cpuset)
> cgroup on /sys/fs/cgroup/cpu type cgroup (rw,relatime,cpu)
> cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,relatime,cpuacct)
> cgroup on /sys/fs/cgroup/memory type cgroup (rw,relatime,memory)
> cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)
> cgroup on /sys/fs/cgroup/freezer type cgroup (rw,relatime,freezer)
> cgroup on /sys/fs/cgroup/blkio type cgroup (rw,relatime,blkio)
> cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,relatime,perf_event)
> cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,relatime,hugetlb)
> systemd on /sys/fs/cgroup/systemd type cgroup
> (rw,noexec,nosuid,nodev,none,name=systemd)
> gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse
> (rw,nosuid,nodev,user=ubuntu)
>
> /mnt/huge is my mount point. So no mounting happening.
>
> ovs-vswitchd.log says
>
> 2015-11-13T12:48:01Z|00001|dpdk|INFO|User-provided -vhost_sock_dir in use:
> /var/run/openvswitch
> EAL: Detected lcore 0 as core 0 on socket 0
> EAL: Detected lcore 1 as core 1 on socket 0
> EAL: Detected lcore 2 as core 2 on socket 0
> EAL: Detected lcore 3 as core 3 on socket 0
> EAL: Detected lcore 4 as core 4 on socket 0
> EAL: Detected lcore 5 as core 5 on socket 0
> EAL: Detected lcore 6 as core 0 on socket 0
> EAL: Detected lcore 7 as core 1 on socket 0
> EAL: Detected lcore 8 as core 2 on socket 0
> EAL: Detected lcore 9 as core 3 on socket 0
> EAL: Detected lcore 10 as core 4 on socket 0
> EAL: Detected lcore 11 as core 5 on socket 0
> EAL: Support maximum 128 logical core(s) by configuration.
> EAL: Detected 12 lcore(s)
> EAL: VFIO modules not all loaded, skip VFIO support...
> EAL: Searching for IVSHMEM devices...
> EAL: No IVSHMEM configuration found!
> EAL: Setting up memory...
> EAL: Ask a virtual area of 0x180000000 bytes
> EAL: Virtual area found at 0x7f1e00000000 (size = 0x180000000)
> EAL: remap_all_hugepages(): mmap failed: Cannot allocate memory
> EAL: Failed to remap 1024 MB pages
> PANIC in rte_eal_init():
> Cannot init memory
> 7: [/usr/sbin/ovs-vswitchd() [0x40b803]]
> 6: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)
> [0x7f1fb52d3ec5]]
> 5: [/usr/sbin/ovs-vswitchd() [0x40a822]]
> 4: [/usr/sbin/ovs-vswitchd() [0x675432]]
> 3: [/usr/sbin/ovs-vswitchd() [0x442155]]
> 2: [/usr/sbin/ovs-vswitchd() [0x407c9f]]
> 1: [/usr/sbin/ovs-vswitchd() [0x447828]]
>
> I have given hugepages in /boot/grub/grub.cfg file. So there are free
> hugepages.
>
>
> AnonHugePages: 378880 kB
> HugePages_Total: 6
> HugePages_Free: 6
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 1048576 kB
>
> It failed to allocate memory because mounting was not done. Did not
> understand why mounting is not done when there are free hugepages.
>
> And also dpdk binding did happen.
>
> $../DPDK-v2.0.0/tools/dpdk_nic_bind.py --status
>
> Network devices using DPDK-compatible driver
> ============================================
> 0000:07:00.0 '82574L Gigabit Network Connection' unused=igb_uio
>
> Network devices using kernel driver
> ===================================
> 0000:00:19.0 'Ethernet Connection I217-LM' if=eth0 drv=e1000e
> unused=igb_uio *Active*
> 0000:06:02.0 '82540EM Gigabit Ethernet Controller' if=eth2 drv=e1000
> unused=igb_uio
>
> Other network devices
> =====================
>
> None
>
> Am using a 1G NIC card for the port (eth1) binds dpdk. Is that a
> problem??? Should dpdk binding port necessarily have a 10G NIC???? I dont
> think its a problem anyway because binding is done. Please correct me if am
> going wrong...
>
> Thanks,
>
> Prathyusha
>
>
>
>
>
>
>
> On Wed, Nov 11, 2015 at 3:52 PM, Samta Rangare <samtarangare at gmail.com>
> wrote:
>
> Hi Sean,
>
> Thanks for replying back, response inline.
>
> On Mon, Nov 9, 2015 at 8:24 PM, Mooney, Sean K <sean.k.mooney at intel.com>
> wrote:
> > Hi
> > Can you provide some more information regarding your deployment?
> >
> > Can you check which kernel you are using.
> >
> > uname -a
>
> Linux ubuntu 3.16.0-50-generic #67~14.04.1-Ubuntu SMP Fri Oct 2 22:07:51
> UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>
> >
> > If you are using a 3.19 kernel changes to some locking code in the
> kennel broke synchronization dpdk2.0 and requires dpdk 2.1 to be used
> instead.
> > In general it is not advisable to use a 3.19 kernel with dpdk as it can
> lead to non-deterministic behavior.
> >
> > When devstack hangs can you connect with a second ssh session and run
> > sudo service ovs-dpdk status
> > and
> > ps aux | grep ovs
> >
>
> sudo service ovs-dpdk status
>
> sourcing config
>
> /opt/stack/logs/ovs-vswitchd.pid is not running
>
> Not all processes are running restart!!!
>
> 1
>
> ubuntu at ubuntu:~/samta/devstack$ ps -ef | grep ovs
>
> root 13385 1 0 15:17 ? 00:00:00 /usr/sbin/ovsdb-server
> --detach --pidfile=/opt/stack/logs/ovsdb-server.pid
> --remote=punix:/usr/local/var/run/openvswitch/db.sock
> --remote=db:Open_vSwitch,Open_vSwitch,manager_options
>
> ubuntu 24451 12855 0 15:45 pts/0 00:00:00 grep --color=auto ovs
>
>
>
> >
> > When the deployment hangs at sudo ovs-vsctl br-set-external-id br-ex
> bridge-id br-ex
> > It usually means that the ovs-vswitchd process has exited.
> >
>
> The above result shows that ovs-vswitchd is not running.
>
> > This can happen for a number of reasons.
> > The vswitchd process may exit if it failed to allocate memory (due to
> memory fragmentation or lack of free hugepages)
> > if the ovs-vswitchd.log is not available can you check the the hugepage
> mount point was created in
> > /mnt/huge And that Iis mounted
> > Run
> > ls -al /mnt/huge
> > and
> > mount
> >
>
> ls -al /mnt/huge
>
> total 4
>
> drwxr-xr-x 2 libvirt-qemu kvm 0 Nov 11 15:18 .
>
> drwxr-xr-x 3 root root 4096 May 15 00:09 ..
>
>
>
> ubuntu at ubuntu:~/samta/devstack$ mount
>
> /dev/mapper/ubuntu--vg-root on / type ext4 (rw,errors=remount-ro)
>
> proc on /proc type proc (rw,noexec,nosuid,nodev)
>
> sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
>
> none on /sys/fs/cgroup type tmpfs (rw)
>
> none on /sys/fs/fuse/connections type fusectl (rw)
>
> none on /sys/kernel/debug type debugfs (rw)
>
> none on /sys/kernel/security type securityfs (rw)
>
> udev on /dev type devtmpfs (rw,mode=0755)
>
> devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
>
> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
>
> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
>
> none on /run/shm type tmpfs (rw,nosuid,nodev)
>
> none on /run/user type tmpfs
> (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
>
> none on /sys/fs/pstore type pstore (rw)
>
> cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,relatime,cpuset)
>
> cgroup on /sys/fs/cgroup/cpu type cgroup (rw,relatime,cpu)
>
> cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,relatime,cpuacct)
>
> cgroup on /sys/fs/cgroup/memory type cgroup (rw,relatime,memory)
>
> cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)
>
> cgroup on /sys/fs/cgroup/freezer type cgroup (rw,relatime,freezer)
>
> cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,relatime,net_cls)
>
> cgroup on /sys/fs/cgroup/blkio type cgroup (rw,relatime,blkio)
>
> cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,relatime,perf_event)
>
> cgroup on /sys/fs/cgroup/net_prio type cgroup (rw,relatime,net_prio)
>
> cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,relatime,hugetlb)
>
> /dev/sda1 on /boot type ext2 (rw)
>
> systemd on /sys/fs/cgroup/systemd type cgroup
> (rw,noexec,nosuid,nodev,none,name=systemd)
>
> hugetlbfs-kvm on /run/hugepages/kvm type hugetlbfs (rw,mode=775,gid=106)
>
> nodev on /mnt/huge type hugetlbfs (rw,uid=106,gid=106)
>
> nodev on /mnt/huge type hugetlbfs (rw,uid=106,gid=106)
>
>
>
> > then checkout how many hugepages are mounted
> >
> > cat /proc/meminfo | grep huge
> >
>
>
>
> cat /proc/meminfo | grep Huge
>
> AnonHugePages: 292864 kB
>
> HugePages_Total: 5
>
> HugePages_Free: 5
>
> HugePages_Rsvd: 0
>
> HugePages_Surp: 0
>
> Hugepagesize: 1048576 kB
>
>
> >
> > the vswitchd process may also exit if it failed to initializes dpdk
> interfaces.
> > This can happen if no interface is compatible with the igb-uio or
> vfio-pci drivers
> > (note in the vfio-pci case all interface in the same iommu group must be
> bound to the vfio-pci driver and
> > The iommu must be enabled in the kernel command line with VT-d enabled
> in the bios)
> >
> > Can you check which interface are bound to the dpdk driver by running
> the following command
> >
> > /opt/stack/DPDK-v2.0.0/tools/dpdk_nic_bind.py --status
> >
>
> /opt/stack/DPDK-v2.0.0/tools/dpdk_nic_bind.py --status
>
>
>
> Network devices using DPDK-compatible driver
>
> ============================================
>
> <none>
>
>
>
> Network devices using kernel driver
>
> ===================================
>
> 0000:01:00.0 'Ethernet Controller 10-Gigabit X540-AT2' if=p1p1 drv=ixgbe
> unused=igb_uio
>
> 0000:02:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=p4p1 drv=i40e
> unused=igb_uio
>
> 0000:03:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=p2p1 drv=i40e
> unused=igb_uio
>
> 0000:06:00.0 'I350 Gigabit Network Connection' if=em1 drv=igb
> unused=igb_uio *Active*
>
> 0000:06:00.1 'I350 Gigabit Network Connection' if=em2 drv=igb
> unused=igb_uio
>
>
>
> Other network devices
>
> =====================
>
> 0000:01:00.1 'Ethernet Controller 10-Gigabit X540-AT2' unused=igb_uio
>
>
> >
> > Finally can you confim that ovs-dpdk compiled successfully by either
> check the xstack.log or
> > Checking for the BUILD_COMPLETE file in /opt/stack/ovs
>
> BUILD_COMPLETE exist in /opt/stack/ovs though its empty.
>
> >
> > Regards
> > sean
> >
> >
> >
> >
> > -----Original Message-----
> > From: Samta Rangare [mailto:samtarangare at gmail.com]
> > Sent: Monday, November 9, 2015 2:31 PM
> > To: Czesnowicz, Przemyslaw
> > Cc: OpenStack Development Mailing List (not for usage questions)
> > Subject: Re: [openstack-dev] [networking-ovs-dpdk]
> >
> > Thanks for replying Przemyslaw, there is no ovs-vswitchd.log in
> /opt/stack/logs/. This is all contains inside (ovsdb-server.pid, screen).
> >
> > When I cancel stack .sh (ctr c), and try to rerun this $sudo ovs-vsctl
> br-set-external-id br-ex bridge-id br-ex it didnt hang, that means vSwitch
> was running isn't it ?
> >
> > But rerunning stack.sh after unstack hangs again.
> >
> > Thanks,
> > Samta
> >
> > On Mon, Nov 9, 2015 at 7:50 PM, Czesnowicz, Przemyslaw <
> przemyslaw.czesnowicz at intel.com> wrote:
> >> Hi Samta,
> >>
> >> This usually means that the vSwitch is not running/has crashed.
> >> Can you check in /opt/stack/logs/ovs-vswitchd.log ? There should be an
> error msg there.
> >>
> >> Regards
> >> Przemek
> >>
> >>> -----Original Message-----
> >>> From: Samta Rangare [mailto:samtarangare at gmail.com]
> >>> Sent: Monday, November 9, 2015 1:51 PM
> >>> To: OpenStack Development Mailing List (not for usage questions)
> >>> Subject: [openstack-dev] [networking-ovs-dpdk]
> >>>
> >>> Hello Everyone,
> >>>
> >>> I am installing devstack with networking-ovs-dpdk. The local.conf
> >>> exactly looks like the one is available in /opt/stack/networking-ovs-
> >>> dpdk/doc/source/_downloads/local.conf.single_node.
> >>> So I believe all the necessary configuration will be taken care.
> >>>
> >>> However I am stuck at place where devstack is trying to set
> >>> external-id ($ sudo ovs-vsctl br-set-external-id br-ex bridge-id
> >>> br-ex). As soon as it hits at this place it's just hangs forever. I
> >>> tried commenting this line from
> >>> lib/neutron_plugin/ml2 (I know this is wrong) and then all services
> >>> came up except ovs-dpdk agent and ovs agent.
> >>>
> >>> BTW I am deploying it in ubuntu 14.04. Any pointer will be really
> helpful.
> >>>
> >>> Thanks,
> >>> Samta
> >>>
> >>> __________________________________________________________
> >>> ________________
> >>> OpenStack Development Mailing List (not for usage questions)
> >>> Unsubscribe: OpenStack-dev-
> >>> request at lists.openstack.org?subject:unsubscribe
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151118/14f3981c/attachment.html>
More information about the OpenStack-dev
mailing list