<div dir="ltr"><div><div><div><div><div>Hi all,<br><br>I have a similar problem as Samta. Am also stuck at the same place. The following command<br><br>$sudo ovs-vsctl br-set-external-id br-ex bridge-id br-ex<br><br>hangs forever. As Sean said, it might be because of ovs-vswitchd proces.<br><br><span style="color:rgb(0,0,255)">> The vswitchd process may exit if it failed to allocate memory (due to memory fragmentation or lack of free hugepages)<br>> if the ovs-vswitchd.log is not available can you check the the hugepage mount point was created in<br>> /mnt/huge And that Iis mounted<br>> Run<br>> ls -al /mnt/huge<br>> and<br>> mount<br>><br><br></span></div><div><span style="color:rgb(0,0,255)"><font color="#000000">$mount</font><br></span></div><span style="color:rgb(0,0,255)"><font color="#000000">/dev/sda6 on / type ext4 (rw,errors=remount-ro)<br>proc on /proc type proc (rw,noexec,nosuid,nodev)<br>sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)<br>none on /sys/fs/cgroup type tmpfs (rw)<br>none on /sys/fs/fuse/connections type fusectl (rw)<br>none on /sys/kernel/debug type debugfs (rw)<br>none on /sys/kernel/security type securityfs (rw)<br>udev on /dev type devtmpfs (rw,mode=0755)<br>devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)<br>tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)<br>none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)<br>none on /run/shm type tmpfs (rw,nosuid,nodev)<br>none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)<br>none on /sys/fs/pstore type pstore (rw)<br>cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,relatime,cpuset)<br>cgroup on /sys/fs/cgroup/cpu type cgroup (rw,relatime,cpu)<br>cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,relatime,cpuacct)<br>cgroup on /sys/fs/cgroup/memory type cgroup (rw,relatime,memory)<br>cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)<br>cgroup on /sys/fs/cgroup/freezer type cgroup (rw,relatime,freezer)<br>cgroup on /sys/fs/cgroup/blkio type cgroup (rw,relatime,blkio)<br>cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,relatime,perf_event)<br>cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,relatime,hugetlb)<br>systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)<br>gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,user=ubuntu)<br><br></font></span></div><span style="color:rgb(0,0,255)"><font color="#000000">/mnt/huge is my mount point. So no mounting happening. <br><br></font></span></div><span style="color:rgb(0,0,255)"><font color="#000000">ovs-vswitchd.log says<br><br>2015-11-13T12:48:01Z|00001|dpdk|INFO|User-provided -vhost_sock_dir in use: /var/run/openvswitch<br>EAL: Detected lcore 0 as core 0 on socket 0<br>EAL: Detected lcore 1 as core 1 on socket 0<br>EAL: Detected lcore 2 as core 2 on socket 0<br>EAL: Detected lcore 3 as core 3 on socket 0<br>EAL: Detected lcore 4 as core 4 on socket 0<br>EAL: Detected lcore 5 as core 5 on socket 0<br>EAL: Detected lcore 6 as core 0 on socket 0<br>EAL: Detected lcore 7 as core 1 on socket 0<br>EAL: Detected lcore 8 as core 2 on socket 0<br>EAL: Detected lcore 9 as core 3 on socket 0<br>EAL: Detected lcore 10 as core 4 on socket 0<br>EAL: Detected lcore 11 as core 5 on socket 0<br>EAL: Support maximum 128 logical core(s) by configuration.<br>EAL: Detected 12 lcore(s)<br>EAL: VFIO modules not all loaded, skip VFIO support...<br>EAL: Searching for IVSHMEM devices...<br>EAL: No IVSHMEM configuration found!<br>EAL: Setting up memory...<br>EAL: Ask a virtual area of 0x180000000 bytes<br>EAL: Virtual area found at 0x7f1e00000000 (size = 0x180000000)<br>EAL: remap_all_hugepages(): mmap failed: Cannot allocate memory<br>EAL: Failed to remap 1024 MB pages<br>PANIC in rte_eal_init():<br>Cannot init memory<br>7: [/usr/sbin/ovs-vswitchd() [0x40b803]]<br>6: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f1fb52d3ec5]]<br>5: [/usr/sbin/ovs-vswitchd() [0x40a822]]<br>4: [/usr/sbin/ovs-vswitchd() [0x675432]]<br>3: [/usr/sbin/ovs-vswitchd() [0x442155]]<br>2: [/usr/sbin/ovs-vswitchd() [0x407c9f]]<br>1: [/usr/sbin/ovs-vswitchd() [0x447828]]<br><br></font></span></div><span style="color:rgb(0,0,255)"><font color="#000000">I have given hugepages in /boot/grub/grub.cfg file. So there are free hugepages.<br><br><br>AnonHugePages: 378880 kB<br>HugePages_Total: 6<br>HugePages_Free: 6<br>HugePages_Rsvd: 0<br>HugePages_Surp: 0<br>Hugepagesize: 1048576 kB<br><br></font></span></div><div><span style="color:rgb(0,0,255)"><font color="#000000">It failed to allocate memory because mounting was not done. Did not understand why mounting is not done when there are free hugepages. <br></font></span></div><div><span style="color:rgb(0,0,255)"><font color="#000000">And also dpdk binding did happen.<br><br>$../DPDK-v2.0.0/tools/dpdk_nic_bind.py --status<br><br>Network devices using DPDK-compatible driver<br>============================================<br></font></span><span style="color:rgb(0,0,255)"><font color="#000000"><span style="color:rgb(0,0,255)"><font color="#000000">0000:07:00.0 '82574L Gigabit Network Connection' unused=igb_uio</font></span><br><br>Network devices using kernel driver<br>===================================<br>0000:00:19.0 'Ethernet Connection I217-LM' if=eth0 drv=e1000e unused=igb_uio *Active*<br>0000:06:02.0 '82540EM Gigabit Ethernet Controller' if=eth2 drv=e1000 unused=igb_uio <br><br>Other network devices<br>=====================<br></font></span></div><div><span style="color:rgb(0,0,255)"><font color="#000000">None<br><br></font></span></div><div><span style="color:rgb(0,0,255)"><font color="#000000">Am using a 1G NIC card for the port (eth1) binds dpdk. Is that a problem??? Should dpdk binding port necessarily have a 10G NIC???? I dont think its a problem anyway because binding is done. Please correct me if am going wrong...<br><br></font></span></div><div><span style="color:rgb(0,0,255)"><font color="#000000">Thanks,<br></font></span></div><div><span style="color:rgb(0,0,255)"><font color="#000000">Prathyusha<br></font></span></div><span style="color:rgb(0,0,255)"><font color="#000000"></font></span><div><div><span style="color:rgb(0,0,255)"><font color="#000000"><br></font></span><div><div><div><div><span style="color:rgb(0,0,255)"><br></span></div></div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 11, 2015 at 3:52 PM, Samta Rangare <span dir="ltr"><<a href="mailto:samtarangare@gmail.com" target="_blank">samtarangare@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Sean,<br><br>Thanks for replying back, response inline.<span class=""><br><br>On Mon, Nov 9, 2015 at 8:24 PM, Mooney, Sean K <<a href="mailto:sean.k.mooney@intel.com" target="_blank">sean.k.mooney@intel.com</a>> wrote:<br>> Hi<br>> Can you provide some more information regarding your deployment?<br>><br>> Can you check which kernel you are using.<br>><br>> uname -a<br><br></span><font color="#0000ff">Linux ubuntu 3.16.0-50-generic #67~14.04.1-Ubuntu SMP Fri Oct 2 22:07:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux</font><span class=""><br><br>><br>> If you are using a 3.19 kernel changes to some locking code in the kennel broke synchronization dpdk2.0 and requires dpdk 2.1 to be used instead.<br>> In general it is not advisable to use a 3.19 kernel with dpdk as it can lead to non-deterministic behavior.<br>><br>> When devstack hangs can you connect with a second ssh session and run<br>> sudo service ovs-dpdk status<br>> and<br>> ps aux | grep ovs<br>></span><div><div><font color="#0000ff">sudo service ovs-dpdk status</font></div><div><font color="#0000ff">sourcing config</font></div><div><font color="#0000ff">/opt/stack/logs/ovs-vswitchd.pid is not running</font></div><div><font color="#0000ff">Not all processes are running restart!!!</font></div><div><font color="#0000ff">1</font></div><div><font color="#0000ff">ubuntu@ubuntu:~/samta/devstack$ ps -ef | grep ovs</font></div><div><font color="#0000ff">root 13385 1 0 15:17 ? 00:00:00 /usr/sbin/ovsdb-server --detach --pidfile=/opt/stack/logs/ovsdb-server.pid --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options</font></div><div><font color="#0000ff">ubuntu 24451 12855 0 15:45 pts/0 00:00:00 grep --color=auto ovs</font></div></div><span class=""><div><br></div><div>><br>> When the deployment hangs at sudo ovs-vsctl br-set-external-id br-ex bridge-id br-ex<br>> It usually means that the ovs-vswitchd process has exited.<br>></div></span><div><font color="#0000ff">The above result shows that ovs-vswitchd is not running.</font></div><span class=""><div>> This can happen for a number of reasons.<br>> The vswitchd process may exit if it failed to allocate memory (due to memory fragmentation or lack of free hugepages)<br>> if the ovs-vswitchd.log is not available can you check the the hugepage mount point was created in<br>> /mnt/huge And that Iis mounted<br>> Run<br>> ls -al /mnt/huge<br>> and<br>> mount<br>></div></span><div><div><font color="#0000ff">ls -al /mnt/huge</font></div><div><font color="#0000ff">total 4</font></div><div><font color="#0000ff">drwxr-xr-x 2 libvirt-qemu kvm 0 Nov 11 15:18 .</font></div><div><font color="#0000ff">drwxr-xr-x 3 root root 4096 May 15 00:09 ..</font></div></div><div><font color="#0000ff"><br></font></div><div><font color="#0000ff"><div>ubuntu@ubuntu:~/samta/devstack$ mount</div><div>/dev/mapper/ubuntu--vg-root on / type ext4 (rw,errors=remount-ro)</div><div>proc on /proc type proc (rw,noexec,nosuid,nodev)</div><div>sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)</div><div>none on /sys/fs/cgroup type tmpfs (rw)</div><div>none on /sys/fs/fuse/connections type fusectl (rw)</div><div>none on /sys/kernel/debug type debugfs (rw)</div><div>none on /sys/kernel/security type securityfs (rw)</div><div>udev on /dev type devtmpfs (rw,mode=0755)</div><div>devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)</div><div>tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)</div><div>none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)</div><div>none on /run/shm type tmpfs (rw,nosuid,nodev)</div><div>none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)</div><div>none on /sys/fs/pstore type pstore (rw)</div><div>cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,relatime,cpuset)</div><div>cgroup on /sys/fs/cgroup/cpu type cgroup (rw,relatime,cpu)</div><div>cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,relatime,cpuacct)</div><div>cgroup on /sys/fs/cgroup/memory type cgroup (rw,relatime,memory)</div><div>cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)</div><div>cgroup on /sys/fs/cgroup/freezer type cgroup (rw,relatime,freezer)</div><div>cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,relatime,net_cls)</div><div>cgroup on /sys/fs/cgroup/blkio type cgroup (rw,relatime,blkio)</div><div>cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,relatime,perf_event)</div><div>cgroup on /sys/fs/cgroup/net_prio type cgroup (rw,relatime,net_prio)</div><div>cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,relatime,hugetlb)</div><div>/dev/sda1 on /boot type ext2 (rw)</div><div>systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)</div><div>hugetlbfs-kvm on /run/hugepages/kvm type hugetlbfs (rw,mode=775,gid=106)</div><div>nodev on /mnt/huge type hugetlbfs (rw,uid=106,gid=106)</div><div>nodev on /mnt/huge type hugetlbfs (rw,uid=106,gid=106)</div></font></div><span class=""><div><br></div><div>> then checkout how many hugepages are mounted<br>><br>> cat /proc/meminfo | grep huge<br>></div><div><br></div></span><div><div><font color="#0000ff">cat /proc/meminfo | grep Huge</font></div><div><font color="#0000ff">AnonHugePages: 292864 kB</font></div><div><font color="#0000ff">HugePages_Total: 5</font></div><div><font color="#0000ff">HugePages_Free: 5</font></div><div><font color="#0000ff">HugePages_Rsvd: 0</font></div><div><font color="#0000ff">HugePages_Surp: 0</font></div><div><font color="#0000ff">Hugepagesize: 1048576 kB</font></div></div><span class=""><div><br>><br>> the vswitchd process may also exit if it failed to initializes dpdk interfaces.<br>> This can happen if no interface is compatible with the igb-uio or vfio-pci drivers<br>> (note in the vfio-pci case all interface in the same iommu group must be bound to the vfio-pci driver and<br>> The iommu must be enabled in the kernel command line with VT-d enabled in the bios)<br>><br>> Can you check which interface are bound to the dpdk driver by running the following command<br>><br>> /opt/stack/DPDK-v2.0.0/tools/dpdk_nic_bind.py --status<br>></div></span><div><div><font color="#0000ff">/opt/stack/DPDK-v2.0.0/tools/dpdk_nic_bind.py --status</font></div><div><font color="#0000ff"><br></font></div><div><font color="#0000ff">Network devices using DPDK-compatible driver</font></div><div><font color="#0000ff">============================================</font></div><div><font color="#0000ff"><none></font></div><div><font color="#0000ff"><br></font></div><div><font color="#0000ff">Network devices using kernel driver</font></div><div><font color="#0000ff">===================================</font></div><div><font color="#0000ff">0000:01:00.0 'Ethernet Controller 10-Gigabit X540-AT2' if=p1p1 drv=ixgbe unused=igb_uio </font></div><div><font color="#0000ff">0000:02:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=p4p1 drv=i40e unused=igb_uio </font></div><div><font color="#0000ff">0000:03:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=p2p1 drv=i40e unused=igb_uio </font></div><div><font color="#0000ff">0000:06:00.0 'I350 Gigabit Network Connection' if=em1 drv=igb unused=igb_uio *Active*</font></div><div><font color="#0000ff">0000:06:00.1 'I350 Gigabit Network Connection' if=em2 drv=igb unused=igb_uio </font></div><div><font color="#0000ff"><br></font></div><div><font color="#0000ff">Other network devices</font></div><div><font color="#0000ff">=====================</font></div><div><font color="#0000ff">0000:01:00.1 'Ethernet Controller 10-Gigabit X540-AT2' unused=igb_uio</font></div></div><span class=""><div><br>><br>> Finally can you confim that ovs-dpdk compiled successfully by either check the xstack.log or<br>> Checking for the BUILD_COMPLETE file in /opt/stack/ovs</div></span><div><font color="#0000ff">BUILD_COMPLETE exist in /opt/stack/ovs though its empty.<br></font><div><div class="h5">><br>> Regards<br>> sean<br>><br>><br>><br>><br>> -----Original Message-----<br>> From: Samta Rangare [mailto:<a href="mailto:samtarangare@gmail.com" target="_blank">samtarangare@gmail.com</a>]<br>> Sent: Monday, November 9, 2015 2:31 PM<br>> To: Czesnowicz, Przemyslaw<br>> Cc: OpenStack Development Mailing List (not for usage questions)<br>> Subject: Re: [openstack-dev] [networking-ovs-dpdk]<br>><br>> Thanks for replying Przemyslaw, there is no ovs-vswitchd.log in /opt/stack/logs/. This is all contains inside (ovsdb-server.pid, screen).<br>><br>> When I cancel stack .sh (ctr c), and try to rerun this $sudo ovs-vsctl br-set-external-id br-ex bridge-id br-ex it didnt hang, that means vSwitch was running isn't it ?<br>><br>> But rerunning stack.sh after unstack hangs again.<br>><br>> Thanks,<br>> Samta<br>><br>> On Mon, Nov 9, 2015 at 7:50 PM, Czesnowicz, Przemyslaw <<a href="mailto:przemyslaw.czesnowicz@intel.com" target="_blank">przemyslaw.czesnowicz@intel.com</a>> wrote:<br>>> Hi Samta,<br>>><br>>> This usually means that the vSwitch is not running/has crashed.<br>>> Can you check in /opt/stack/logs/ovs-vswitchd.log ? There should be an error msg there.<br>>><br>>> Regards<br>>> Przemek<br>>><br>>>> -----Original Message-----<br>>>> From: Samta Rangare [mailto:<a href="mailto:samtarangare@gmail.com" target="_blank">samtarangare@gmail.com</a>]<br>>>> Sent: Monday, November 9, 2015 1:51 PM<br>>>> To: OpenStack Development Mailing List (not for usage questions)<br>>>> Subject: [openstack-dev] [networking-ovs-dpdk]<br>>>><br>>>> Hello Everyone,<br>>>><br>>>> I am installing devstack with networking-ovs-dpdk. The local.conf<br>>>> exactly looks like the one is available in /opt/stack/networking-ovs-<br>>>> dpdk/doc/source/_downloads/local.conf.single_node.<br>>>> So I believe all the necessary configuration will be taken care.<br>>>><br>>>> However I am stuck at place where devstack is trying to set<br>>>> external-id ($ sudo ovs-vsctl br-set-external-id br-ex bridge-id<br>>>> br-ex). As soon as it hits at this place it's just hangs forever. I<br>>>> tried commenting this line from<br>>>> lib/neutron_plugin/ml2 (I know this is wrong) and then all services<br>>>> came up except ovs-dpdk agent and ovs agent.<br>>>><br>>>> BTW I am deploying it in ubuntu 14.04. Any pointer will be really helpful.<br>>>><br>>>> Thanks,<br>>>> Samta<br>>>><br>>>> __________________________________________________________<br>>>> ________________<br>>>> OpenStack Development Mailing List (not for usage questions)<br>>>> Unsubscribe: OpenStack-dev-<br>>>> <a href="http://request@lists.openstack.org?subject:unsubscribe" target="_blank">request@lists.openstack.org?subject:unsubscribe</a><br>>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>><br>> __________________________________________________________________________<br>> OpenStack Development Mailing List (not for usage questions)<br>> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>><br>> __________________________________________________________________________<br>> OpenStack Development Mailing List (not for usage questions)<br>> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br></div></div></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>