<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi that is great to know.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I will internally report this behavior to our dpdk team<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">But I have already got a patch to change our default target to
</span>native-linuxapp<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><a href="https://review.openstack.org/#/c/246375/">https://review.openstack.org/#/c/246375/</a> which should merge shortly.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Im glad it is now working for you.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Prathyusha Guduri [mailto:prathyushaconnects@gmail.com]
<br>
<b>Sent:</b> Wednesday, November 18, 2015 6:13 AM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [networking-ovs-dpdk]<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Thanks a lot Sean, that was helpful.<o:p></o:p></p>
</div>
<p class="MsoNormal">Changing the target from ivshmem to native-linuxapp removed the error and it doesn't hang at creating external bridge anymore.
<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">All processes(nova-api, neutron, ovs-vswitchd, etc) did start.<o:p></o:p></p>
</div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<p class="MsoNormal">Prathyusha<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Tue, Nov 17, 2015 at 7:57 PM, Mooney, Sean K <<a href="mailto:sean.k.mooney@intel.com" target="_blank">sean.k.mooney@intel.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">We mainly test with 2M hugepages not 1G however our ci does use 1G pages.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">We recently noticed a different but unrelated related issue with using the ivshmem target when building
dpdk.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">(<a href="https://bugs.launchpad.net/networking-ovs-dpdk/+bug/1517032" target="_blank">https://bugs.launchpad.net/networking-ovs-dpdk/+bug/1517032</a>)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Instead of modifying dpdk can you try</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Changing the default dpdk build target to x86_64-native-linuxapp-gcc.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">This can be done by adding</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">RTE_TARGET=x86_64-native-linuxapp-gcc to the local.conf</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">And removing the following file to force a rebuild “/opt/stack/ovs/BUILD_COMPLETE”</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I agree with your assessment though this appears to be a timing issue in dpdk 2.0</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Prathyusha Guduri [mailto:<a href="mailto:prathyushaconnects@gmail.com" target="_blank">prathyushaconnects@gmail.com</a>]
<br>
<b>Sent:</b> Tuesday, November 17, 2015 1:42 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [networking-ovs-dpdk]</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Here is stack.sh log -<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><br>
2015-11-17 13:38:50.010 | Loading uio module<br>
2015-11-17 13:38:50.028 | Loading DPDK UIO module<br>
2015-11-17 13:38:50.038 | starting ovs db<br>
2015-11-17 13:38:50.038 | binding nics<br>
2015-11-17 13:38:50.039 | starting vswitchd<br>
2015-11-17 13:38:50.190 | sudo RTE_SDK=/opt/stack/DPDK-v2.0.0 RTE_TARGET=build /opt/stack/DPDK-v2.0.0/tools/dpdk_nic_bind.py -b igb_uio 0000:07:00.0<br>
2015-11-17 13:38:50.527 | sudo ovs-vsctl --no-wait --may-exist add-port br-eth1 dpdk0 -- set Interface dpdk0 type=dpdk<br>
2015-11-17 13:38:51.671 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:38:52.685 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:38:53.702 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:38:54.720 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:38:55.733 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:38:56.749 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:38:57.768 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:38:58.787 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:38:59.802 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:00.818 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:01.836 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:02.849 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:03.866 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:04.884 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:05.905 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:06.923 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:07.937 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:08.956 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:09.973 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:10.988 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:12.004 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:13.022 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:14.040 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:15.060 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:16.073 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:17.089 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:18.108 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:19.121 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:20.138 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:21.156 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:22.169 | Waiting for ovs-vswitchd to start...<br>
2015-11-17 13:39:23.185 | Waiting for ovs-vswitchd to start...<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Tue, Nov 17, 2015 at 6:50 PM, Prathyusha Guduri <<a href="mailto:prathyushaconnects@gmail.com" target="_blank">prathyushaconnects@gmail.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Hi Sean,<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Here is ovs-vswitchd.log
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><br>
<span style="color:black">2015-11-13T12:48:01Z|00001|dpdk|INFO|User-provided -vhost_sock_dir in use: /var/run/openvswitch<br>
EAL: Detected lcore 0 as core 0 on socket 0<br>
EAL: Detected lcore 1 as core 1 on socket 0<br>
EAL: Detected lcore 2 as core 2 on socket 0<br>
EAL: Detected lcore 3 as core 3 on socket 0<br>
EAL: Detected lcore 4 as core 4 on socket 0<br>
EAL: Detected lcore 5 as core 5 on socket 0<br>
EAL: Detected lcore 6 as core 0 on socket 0<br>
EAL: Detected lcore 7 as core 1 on socket 0<br>
EAL: Detected lcore 8 as core 2 on socket 0<br>
EAL: Detected lcore 9 as core 3 on socket 0<br>
EAL: Detected lcore 10 as core 4 on socket 0<br>
EAL: Detected lcore 11 as core 5 on socket 0<br>
EAL: Support maximum 128 logical core(s) by configuration.<br>
EAL: Detected 12 lcore(s)<br>
EAL: VFIO modules not all loaded, skip VFIO support...<br>
EAL: Searching for IVSHMEM devices...<br>
EAL: No IVSHMEM configuration found!<br>
EAL: Setting up memory...<br>
EAL: Ask a virtual area of 0x180000000 bytes<br>
EAL: Virtual area found at 0x7f1e00000000 (size = 0x180000000)<br>
EAL: remap_all_hugepages(): mmap failed: Cannot allocate memory<br>
EAL: Failed to remap 1024 MB pages<br>
PANIC in rte_eal_init():<br>
Cannot init memory<br>
7: [/usr/sbin/ovs-vswitchd() [0x40b803]]<br>
6: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f1fb52d3ec5]]<br>
5: [/usr/sbin/ovs-vswitchd() [0x40a822]]<br>
4: [/usr/sbin/ovs-vswitchd() [0x675432]]<br>
3: [/usr/sbin/ovs-vswitchd() [0x442155]]<br>
2: [/usr/sbin/ovs-vswitchd() [0x407c9f]]<br>
1: [/usr/sbin/ovs-vswitchd() [0x447828]]</span><o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="color:black">Before this hugepages were free and port binding was also done. So I suspected that this is a DPDK specific issue and found that in remap_all_hugepages( ) of
/opt/stack/DPDK-v2.0.0/lib/librte_eal/linuxapp/eal/eal_memory.c which first unmaps and then mmaps, there is an issue here and so mmap here fails. In DPDK mailing list I found that the unmap is taking longer time because of which mmap fails, so putting a sleep(1)
between unmap and map is supposed to solve the issue. Please check the below link :
<br>
<a href="https://lists.01.org/pipermail/dpdk-ovs/2014-April/000864.html" target="_blank">https://lists.01.org/pipermail/dpdk-ovs/2014-April/000864.html</a></span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="color:black">After changing so, the ovs-vswitchd command hangs at this place<br>
<br>
2015-11-17T10:52:38Z|00001|dpdk|INFO|User-provided -vhost_sock_dir in use: /var/run/openvswitch<br>
2015-11-17 10:52:38.680 | EAL: Detected lcore 0 as core 0 on socket 0<br>
2015-11-17 10:52:38.680 | EAL: Detected lcore 1 as core 1 on socket 0<br>
2015-11-17 10:52:38.680 | EAL: Detected lcore 2 as core 2 on socket 0<br>
2015-11-17 10:52:38.680 | EAL: Detected lcore 3 as core 3 on socket 0<br>
2015-11-17 10:52:38.680 | EAL: Detected lcore 4 as core 4 on socket 0<br>
2015-11-17 10:52:38.680 | EAL: Detected lcore 5 as core 5 on socket 0<br>
2015-11-17 10:52:38.680 | EAL: Detected lcore 6 as core 0 on socket 0<br>
2015-11-17 10:52:38.680 | EAL: Detected lcore 7 as core 1 on socket 0<br>
2015-11-17 10:52:38.680 | EAL: Detected lcore 8 as core 2 on socket 0<br>
2015-11-17 10:52:38.680 | EAL: Detected lcore 9 as core 3 on socket 0<br>
2015-11-17 10:52:38.680 | EAL: Detected lcore 10 as core 4 on socket 0<br>
2015-11-17 10:52:38.680 | EAL: Detected lcore 11 as core 5 on socket 0<br>
2015-11-17 10:52:38.680 | EAL: Support maximum 128 logical core(s) by configuration.<br>
2015-11-17 10:52:38.680 | EAL: Detected 12 lcore(s)<br>
2015-11-17 10:52:38.687 | EAL: VFIO modules not all loaded, skip VFIO support...<br>
2015-11-17 10:52:38.687 | EAL: Searching for IVSHMEM devices...<br>
2015-11-17 10:52:38.687 | EAL: No IVSHMEM configuration found! <br>
2015-11-17 10:52:38.687 | EAL: Setting up memory...<br>
2015-11-17 10:52:39.252 | EAL: Ask a virtual area of 0x1c00000 bytes<br>
2015-11-17 10:52:39.252 | EAL: Virtual area found at 0x7fcab3a00000 (size = 0x1c00000)<br>
2015-11-17 10:52:53.265 | EAL: Ask a virtual area of 0x200000 bytes<br>
2015-11-17 10:52:53.266 | EAL: Virtual area found at 0x7fcab3600000 (size = 0x200000)<br>
2015-11-17 10:52:54.266 | EAL: Ask a virtual area of 0x200000 bytes<br>
2015-11-17 10:52:54.266 | EAL: Virtual area found at 0x7fcab3200000 (size = 0x200000)<br>
2015-11-17 10:52:55.267 | EAL: Ask a virtual area of 0x22c00000 bytes<br>
2015-11-17 10:52:55.267 | EAL: Virtual area found at 0x7fca90400000 (size = 0x22c00000)<br>
2015-11-17 10:57:33.574 | EAL: Ask a virtual area of 0x1800000 bytes<br>
2015-11-17 10:57:33.574 | EAL: Virtual area found at 0x7fca8ea00000 (size = 0x1800000)<br>
2015-11-17 10:57:45.585 | EAL: Ask a virtual area of 0xd9800000 bytes<br>
2015-11-17 10:57:45.585 | EAL: Virtual area found at 0x7fc9b5000000 (size = 0xd9800000)<br>
2015-11-17 11:26:50.605 | EAL: Ask a virtual area of 0x200000 bytes<br>
2015-11-17 11:26:50.605 | EAL: Virtual area found at 0x7fc9b4c00000 (size = 0x200000)<br>
2015-11-17 11:26:51.606 | EAL: Ask a virtual area of 0x200000 bytes<br>
2015-11-17 11:26:51.606 | EAL: Virtual area found at 0x7fc9b4800000 (size = 0x200000)<br>
2015-11-17 11:26:52.608 | EAL: Requesting 1024 pages of size 2MB from socket 0<br>
2015-11-17 11:26:53.111 | EAL: TSC frequency is ~3491914 KHz<br>
2015-11-17 11:26:53.111 | EAL: Master lcore 1 is ready (tid=b73cd700;cpuset=[1])<br>
2015-11-17 11:26:53.111 | PMD: ENICPMD trace: rte_enic_pmd_init<br>
2015-11-17 11:26:53.111 | EAL: PCI device 0000:07:00.0 on NUMA socket 0<br>
2015-11-17 11:26:53.111 | EAL: probe driver: 8086:10d3 rte_em_pmd<br>
2015-11-17 11:26:53.111 | EAL: PCI memory mapped at 0x7fcab5600000<br>
2015-11-17 11:26:53.111 | EAL: PCI memory mapped at 0x7fcab730f000<br>
2015-11-17 11:26:53.111 | EAL: PCI memory mapped at 0x7fcab73d6000<br>
2015-11-17 11:26:53.189 | PMD: eth_em_dev_init(): port_id 0 vendorID=0x8086 deviceID=0x10d3<br>
2015-11-17 11:26:53.190 | 2015-11-17T11:26:53Z|00002|ovs_numa|INFO|Discovered 12 CPU cores on NUMA node 0<br>
2015-11-17 11:26:53.190 | 2015-11-17T11:26:53Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 12 CPU cores<br>
2015-11-17 11:26:53.190 | 2015-11-17T11:26:53Z|00004|memory|INFO|10680 kB peak resident set size after 2054.5 seconds<br>
2015-11-17 11:26:53.190 | 2015-11-17T11:26:53Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...<br>
2015-11-17 11:26:53.190 | 2015-11-17T11:26:53Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected<br>
2015-11-17 11:26:53.194 | 2015-11-17T11:26:53Z|00007|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports recirculation<br>
2015-11-17 11:26:53.194 | 2015-11-17T11:26:53Z|00008|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS label stack length probed as 3<br>
2015-11-17 11:26:53.194 | 2015-11-17T11:26:53Z|00009|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports unique flow ids<br>
2015-11-17 11:26:53.195 | 2015-11-17T11:26:53Z|00010|bridge|INFO|bridge br-eth1: added interface br-eth1 on port 65534<br>
2015-11-17 11:26:53.197 | 2015-11-17T11:26:53Z|00011|dpif_netlink|ERR|Generic Netlink family 'ovs_datapath' does not exist. The Open vSwitch kernel module is probably not loaded.<br>
2015-11-17 11:26:53.287 | Zone 0: name:<MALLOC_S0_HEAP_0>, phys:0x9b600000, len:0xb00000, virt:0x7fca8ea00000, socket_id:0, flags:0<br>
2015-11-17 11:26:53.287 | Zone 1: name:<RG_MP_log_history>, phys:0x36600000, len:0x2080, virt:0x7fcab3600000, socket_id:0, flags:0<br>
2015-11-17 11:26:53.287 | Zone 2: name:<MP_log_history>, phys:0x9c100000, len:0x28a0c0, virt:0x7fca8f500000, socket_id:0, flags:0<br>
2015-11-17 11:26:53.287 | Zone 3: name:<rte_eth_dev_data>, phys:0x36602080, len:0x1f400, virt:0x7fcab3602080, socket_id:0, flags:0<br>
2015-11-17 11:26:53.287 | PMD: eth_em_tx_queue_setup(): sw_ring=0x7fca8f4efd40 hw_ring=0x7fcab3621480 dma_addr=0x36621480<br>
2015-11-17 11:26:53.287 | PMD: eth_em_rx_queue_setup(): sw_ring=0x7fca8f4ebc40 hw_ring=0x7fcab3631480 dma_addr=0x36631480<br>
2015-11-17 11:26:53.368 | PMD: eth_em_start(): <<<br>
2015-11-17 11:26:53.368 | 2015-11-17T11:26:53Z|00012|dpdk|INFO|Port 0: 68:05:ca:1b:ca:c9<br>
2015-11-17 11:26:53.405 | PMD: eth_em_tx_queue_setup(): sw_ring=0x7fca8f4efe00 hw_ring=0x7fcab3621480 dma_addr=0x36621480<br>
2015-11-17 11:26:53.405 | PMD: eth_em_rx_queue_setup(): sw_ring=0x7fca8f4ebdc0 hw_ring=0x7fcab3631480 dma_addr=0x36631480<br>
2015-11-17 11:26:53.486 | PMD: eth_em_start(): <<<br>
2015-11-17 11:26:53.486 | 2015-11-17T11:26:53Z|00013|dpdk|INFO|Port 0: 68:05:ca:1b:ca:c9<br>
2015-11-17 11:26:53.487 | 2015-11-17T11:26:53Z|00014|dpif_netdev|INFO|Created 1 pmd threads on numa node 0<br>
2015-11-17 11:26:53.487 | 2015-11-17T11:26:53Z|00001|dpif_netdev(pmd10)|INFO|Core 0 processing port 'dpdk0'<br>
2015-11-17 11:26:53.488 | 2015-11-17T11:26:53Z|00002|dpif_netdev(pmd10)|INFO|Core 0 processing port 'dpdk0'<br>
2015-11-17 11:26:53.488 | 2015-11-17T11:26:53Z|00015|bridge|INFO|bridge br-eth1: added interface dpdk0 on port 1<br>
2015-11-17 11:26:53.488 | 2015-11-17T11:26:53Z|00016|bridge|INFO|bridge br-int: added interface br-int on port 65534<br>
2015-11-17 11:26:53.488 | 2015-11-17T11:26:53Z|00017|bridge|INFO|bridge br-eth1: using datapath ID 00006805ca1bcac9<br>
2015-11-17 11:26:53.488 | 2015-11-17T11:26:53Z|00018|connmgr|INFO|br-eth1: added service controller "punix:/var/run/openvswitch/br-eth1.mgmt"<br>
2015-11-17 11:26:53.489 | 2015-11-17T11:26:53Z|00019|bridge|INFO|bridge br-int: using datapath ID 00002ef7b66a8742<br>
2015-11-17 11:26:53.489 | 2015-11-17T11:26:53Z|00020|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt"<br>
2015-11-17 11:26:53.490 | 2015-11-17T11:26:53Z|00021|dpif_netdev|INFO|Created 2 pmd threads on numa node 0<br>
2015-11-17 11:26:53.492 | 2015-11-17T11:26:53Z|00022|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.4.90<br>
2015-11-17 11:26:53.493 | 2015-11-17T11:26:53Z|00001|dpif_netdev(pmd23)|INFO|Core 2 processing port 'dpdk0'<br>
2015-11-17 11:27:03.494 | 2015-11-17T11:27:03Z|00023|memory|INFO|peak resident set size grew 93% in last 10.3 seconds, from 10680 kB to 20572 kB<br>
2015-11-17 11:27:03.494 | 2015-11-17T11:27:03Z|00024|memory|INFO|handlers:4 ports:3 revalidators:2 rules:10</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="color:black"><a href="mailto:ubuntu@ubuntu-Precision-Tower-5810:/opt/stack/DPDK-v2.0.0/lib/librte_eal/linuxapp/eal$" target="_blank">ubuntu@ubuntu-Precision-Tower-5810:/opt/stack/DPDK-v2.0.0/lib/librte_eal/linuxapp/eal$</a>
ps -Al | grep ovs<br>
5 S 0 1681 2595 0 80 0 - 4433 poll_s ? 00:00:00 ovsdb-server<br>
4 S 0 1716 1715 0 80 0 - 4636 wait pts/3 00:00:00 ovs-dpdk<br>
4 S 0 2124 1716 99 80 0 - 870841 poll_s pts/3 03:42:31 ovs-vswitchd</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">So now ovs-vswitchd runs unlike the last time.
<br>
I really dont understand where am missing out.... </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Tue, Nov 17, 2015 at 5:14 PM, Mooney, Sean K <<a href="mailto:sean.k.mooney@intel.com" target="_blank">sean.k.mooney@intel.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Can you provide the ovs-vswitchd log form
</span>${OVS_LOG_DIR}/ovs-vswitchd.log<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/tmp/ovs-vswitchd.log in your case.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">If the vswitch fails to start we clean up by unmounting the hugepages.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Prathyusha Guduri [mailto:<a href="mailto:prathyushaconnects@gmail.com" target="_blank">prathyushaconnects@gmail.com</a>]
<br>
<b>Sent:</b> Tuesday, November 17, 2015 7:37 AM</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [networking-ovs-dpdk]<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Hi Sean,<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">I realised on debugging ovs-dpdk-init script that the main issue is with the following command<br>
<br>
$ screen -dms ovs-vswitchd sudo sg $qemu_group -c "umask 002; ${OVS_INSTALL_DIR}/sbin/ovs-vswitchd --dpdk -vhost_sock_dir $OVS_DB_SOCKET_DIR -c $OVS_CORE_MASK -n $OVS_MEM_CHANNELS --proc-type primary --huge-dir $OVS_HUGEPAGE_MOUNT --socket-mem $OVS_SOCKET_MEM
$pciAddressWhitelist -- unix:$OVS_DB_SOCKET 2>&1 | tee ${OVS_LOG_DIR}/ovs-vswitchd.log"<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">which I guess is starting the ovs-vswitchd application. Before this command, huge pages is mounted and port binding is also done but still the screen command fails.
<br>
I verified the db.sock and conf.db files. <o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Any help is highly appreciated.<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks,<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Prathyusha<o:p></o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Mon, Nov 16, 2015 at 5:12 PM, Prathyusha Guduri <<a href="mailto:prathyushaconnects@gmail.com" target="_blank">prathyushaconnects@gmail.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Hi Sean,<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Thanks for your response.
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">in your case though you are using 1GB hugepages so I don’t think this is related to memory fragmentation<br>
or a lack of free hugepages.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">to use preallocated 1GB page with ovs you should instead set the following in your local.conf</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">OVS_HUGEPAGE_MOUNT_PAGESIZE=1G</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">OVS_ALLOCATE_HUGEPAGES=False</span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Added the above two parameters to the local.conf. The same problem again.
<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Basically it throws this error -<br>
2015-11-16 11:31:44.741 | starting vswitchd<br>
2015-11-16 11:31:44.863 | sudo RTE_SDK=/opt/stack/DPDK-v2.0.0 RTE_TARGET=build /opt/stack/DPDK-v2.0.0/tools/dpdk_nic_bind.py -b igb_uio 0000:07:00.0<br>
2015-11-16 11:31:45.169 | sudo ovs-vsctl --no-wait --may-exist add-port br-eth1 dpdk0 -- set Interface dpdk0 type=dpdk<br>
2015-11-16 11:31:46.314 | Waiting for ovs-vswitchd to start...<br>
2015-11-16 11:31:47.442 | libvirt-bin stop/waiting<br>
2015-11-16 11:31:49.473 | libvirt-bin start/running, process 2255<br>
2015-11-16 11:31:49.477 |<span style="color:red"> [ERROR] /etc/init.d/ovs-dpdk:563 ovs-vswitchd application failed to start</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">manually mounting /mnt/huge and then commenting that part from the /etc/init.d/ovs-dpdk script also throws the same error.
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Using 1G hugepagesize should not give any memory related problem. I dont understand why it is not mounting then.
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Here is the /opt/stack/networking-ovs-dpdk/devstack/ovs-dpdk/ovs-dpdk.conf
<br>
<br>
RTE_SDK=${RTE_SDK:-/opt/stack/DPDK}<br>
RTE_TARGET=${RTE_TARGET:-x86_64-ivshmem-linuxapp-gcc}<br>
<br>
OVS_INSTALL_DIR=/usr<br>
OVS_DB_CONF_DIR=/etc/openvswitch<br>
OVS_DB_SOCKET_DIR=/var/run/openvswitch<br>
OVS_DB_CONF=$OVS_DB_CONF_DIR/conf.db<br>
OVS_DB_SOCKET=OVS_DB_SOCKET_DIR/db.sock<br>
<br>
OVS_SOCKET_MEM=2048,2048<br>
OVS_MEM_CHANNELS=4<br>
OVS_CORE_MASK=${OVS_CORE_MASK:-2}<br>
OVS_PMD_CORE_MASK=${OVS_PMD_CORE_MASK:-4}<br>
OVS_LOG_DIR=/tmp<br>
OVS_LOCK_DIR=''<br>
OVS_SRC_DIR=/opt/stack/ovs<br>
OVS_DIR=${OVS_DIR:-${OVS_SRC_DIR}}<br>
OVS_UTILS=${OVS_DIR}/utilities/<br>
OVS_DB_UTILS=${OVS_DIR}/ovsdb/<br>
OVS_DPDK_DIR=$RTE_SDK<br>
OVS_NUM_HUGEPAGES=${OVS_NUM_HUGEPAGES:-5}<br>
OVS_HUGEPAGE_MOUNT=${OVS_HUGEPAGE_MOUNT:-/mnt/huge}<br>
OVS_HUGEPAGE_MOUNT_PAGESIZE=''<br>
OVS_BOND_MODE=$OVS_BOND_MODE<br>
OVS_BOND_PORTS=$OVS_BOND_PORTS<br>
OVS_BRIDGE_MAPPINGS=eth1<br>
OVS_PCI_MAPPINGS=0000:07:00.0#eth1<br>
OVS_DPDK_PORT_MAPPINGS=''<br>
OVS_TUNNEL_CIDR_MAPPING=''<br>
OVS_ALLOCATE_HUGEPAGES=True<br>
OVS_INTERFACE_DRIVER='igb_uio'<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Verified the OVS_DB_SOCKET_DIR and all others. conf.db and db.sock exist. So why ovs-vswitchd is failing to start??? Am I missing something???
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Prathyusha<o:p></o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Mon, Nov 16, 2015 at 4:39 PM, Mooney, Sean K <<a href="mailto:sean.k.mooney@intel.com" target="_blank">sean.k.mooney@intel.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Yes sorry for the delay in responding to you and samta.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">In your case assuming you are using 2mb hugepages it is easy to hit dpdks default max memory segments</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">This can be changed by setting OVS_DPDK_MEM_SEGMENTS=<arbitrary large number that you will never hit></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">In the local.conf and recompiling. To do this simply remove the build complete file in /opt/stack/ovs</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">rm –f /opt/stack/BUILD_COMPLETE</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">in your case though you are using 1GB hugepages so I don’t think this is related to memory fragmentation<br>
or a lack of free hugepages.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">to use preallocated 1GB page with ovs you should instead set the following in your local.conf</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">OVS_HUGEPAGE_MOUNT_PAGESIZE=1G</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">OVS_ALLOCATE_HUGEPAGES=False</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Regards</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">sean</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Prathyusha Guduri [mailto:<a href="mailto:prathyushaconnects@gmail.com" target="_blank">prathyushaconnects@gmail.com</a>]
<br>
<b>Sent:</b> Monday, November 16, 2015 6:20 AM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [networking-ovs-dpdk]</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Hi all,<br>
<br>
I have a similar problem as Samta. Am also stuck at the same place. The following command<br>
<br>
$sudo ovs-vsctl br-set-external-id br-ex bridge-id br-ex<br>
<br>
hangs forever. As Sean said, it might be because of ovs-vswitchd proces.<br>
<br>
<span style="color:blue">> The vswitchd process may exit if it failed to allocate memory (due to memory fragmentation or lack of free hugepages)<br>
> if the ovs-vswitchd.log is not available can you check the the hugepage mount point was created in<br>
> /mnt/huge And that Iis mounted<br>
> Run<br>
> ls -al /mnt/huge<br>
> and<br>
> mount<br>
></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">$mount</span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="color:black">/dev/sda6 on / type ext4 (rw,errors=remount-ro)<br>
proc on /proc type proc (rw,noexec,nosuid,nodev)<br>
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)<br>
none on /sys/fs/cgroup type tmpfs (rw)<br>
none on /sys/fs/fuse/connections type fusectl (rw)<br>
none on /sys/kernel/debug type debugfs (rw)<br>
none on /sys/kernel/security type securityfs (rw)<br>
udev on /dev type devtmpfs (rw,mode=0755)<br>
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)<br>
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)<br>
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)<br>
none on /run/shm type tmpfs (rw,nosuid,nodev)<br>
none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)<br>
none on /sys/fs/pstore type pstore (rw)<br>
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,relatime,cpuset)<br>
cgroup on /sys/fs/cgroup/cpu type cgroup (rw,relatime,cpu)<br>
cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,relatime,cpuacct)<br>
cgroup on /sys/fs/cgroup/memory type cgroup (rw,relatime,memory)<br>
cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)<br>
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,relatime,freezer)<br>
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,relatime,blkio)<br>
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,relatime,perf_event)<br>
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,relatime,hugetlb)<br>
systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)<br>
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,user=ubuntu)</span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="color:black">/mnt/huge is my mount point. So no mounting happening.
</span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="color:black">ovs-vswitchd.log says<br>
<br>
2015-11-13T12:48:01Z|00001|dpdk|INFO|User-provided -vhost_sock_dir in use: /var/run/openvswitch<br>
EAL: Detected lcore 0 as core 0 on socket 0<br>
EAL: Detected lcore 1 as core 1 on socket 0<br>
EAL: Detected lcore 2 as core 2 on socket 0<br>
EAL: Detected lcore 3 as core 3 on socket 0<br>
EAL: Detected lcore 4 as core 4 on socket 0<br>
EAL: Detected lcore 5 as core 5 on socket 0<br>
EAL: Detected lcore 6 as core 0 on socket 0<br>
EAL: Detected lcore 7 as core 1 on socket 0<br>
EAL: Detected lcore 8 as core 2 on socket 0<br>
EAL: Detected lcore 9 as core 3 on socket 0<br>
EAL: Detected lcore 10 as core 4 on socket 0<br>
EAL: Detected lcore 11 as core 5 on socket 0<br>
EAL: Support maximum 128 logical core(s) by configuration.<br>
EAL: Detected 12 lcore(s)<br>
EAL: VFIO modules not all loaded, skip VFIO support...<br>
EAL: Searching for IVSHMEM devices...<br>
EAL: No IVSHMEM configuration found!<br>
EAL: Setting up memory...<br>
EAL: Ask a virtual area of 0x180000000 bytes<br>
EAL: Virtual area found at 0x7f1e00000000 (size = 0x180000000)<br>
EAL: remap_all_hugepages(): mmap failed: Cannot allocate memory<br>
EAL: Failed to remap 1024 MB pages<br>
PANIC in rte_eal_init():<br>
Cannot init memory<br>
7: [/usr/sbin/ovs-vswitchd() [0x40b803]]<br>
6: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f1fb52d3ec5]]<br>
5: [/usr/sbin/ovs-vswitchd() [0x40a822]]<br>
4: [/usr/sbin/ovs-vswitchd() [0x675432]]<br>
3: [/usr/sbin/ovs-vswitchd() [0x442155]]<br>
2: [/usr/sbin/ovs-vswitchd() [0x407c9f]]<br>
1: [/usr/sbin/ovs-vswitchd() [0x447828]]</span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="color:black">I have given hugepages in /boot/grub/grub.cfg file. So there are free hugepages.<br>
<br>
<br>
AnonHugePages: 378880 kB<br>
HugePages_Total: 6<br>
HugePages_Free: 6<br>
HugePages_Rsvd: 0<br>
HugePages_Surp: 0<br>
Hugepagesize: 1048576 kB</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">It failed to allocate memory because mounting was not done. Did not understand why mounting is not done when there are free hugepages.
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">And also dpdk binding did happen.<br>
<br>
$../DPDK-v2.0.0/tools/dpdk_nic_bind.py --status<br>
<br>
Network devices using DPDK-compatible driver<br>
============================================<br>
0000:07:00.0 '82574L Gigabit Network Connection' unused=igb_uio<br>
<br>
Network devices using kernel driver<br>
===================================<br>
0000:00:19.0 'Ethernet Connection I217-LM' if=eth0 drv=e1000e unused=igb_uio *Active*<br>
0000:06:02.0 '82540EM Gigabit Ethernet Controller' if=eth2 drv=e1000 unused=igb_uio
<br>
<br>
Other network devices<br>
=====================</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="color:black">None</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="color:black">Am using a 1G NIC card for the port (eth1) binds dpdk. Is that a problem??? Should dpdk binding port necessarily have a 10G NIC???? I dont think its a problem
anyway because binding is done. Please correct me if am going wrong...</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">Thanks,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:black">Prathyusha</span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Wed, Nov 11, 2015 at 3:52 PM, Samta Rangare <<a href="mailto:samtarangare@gmail.com" target="_blank">samtarangare@gmail.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi Sean,<br>
<br>
Thanks for replying back, response inline.<br>
<br>
On Mon, Nov 9, 2015 at 8:24 PM, Mooney, Sean K <<a href="mailto:sean.k.mooney@intel.com" target="_blank">sean.k.mooney@intel.com</a>> wrote:<br>
> Hi<br>
> Can you provide some more information regarding your deployment?<br>
><br>
> Can you check which kernel you are using.<br>
><br>
> uname -a<br>
<br>
<span style="color:blue">Linux ubuntu 3.16.0-50-generic #67~14.04.1-Ubuntu SMP Fri Oct 2 22:07:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux</span><br>
<br>
><br>
> If you are using a 3.19 kernel changes to some locking code in the kennel broke synchronization dpdk2.0 and requires dpdk 2.1 to be used instead.<br>
> In general it is not advisable to use a 3.19 kernel with dpdk as it can lead to non-deterministic behavior.<br>
><br>
> When devstack hangs can you connect with a second ssh session and run<br>
> sudo service ovs-dpdk status<br>
> and<br>
> ps aux | grep ovs<br>
><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">sudo service ovs-dpdk status</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">sourcing config</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">/opt/stack/logs/ovs-vswitchd.pid is not running</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">Not all processes are running restart!!!</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">ubuntu@ubuntu:~/samta/devstack$ ps -ef | grep ovs</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">root 13385 1 0 15:17 ? 00:00:00 /usr/sbin/ovsdb-server --detach --pidfile=/opt/stack/logs/ovsdb-server.pid --remote=punix:/usr/local/var/run/openvswitch/db.sock
--remote=db:Open_vSwitch,Open_vSwitch,manager_options</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">ubuntu 24451 12855 0 15:45 pts/0 00:00:00 grep --color=auto ovs</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">><br>
> When the deployment hangs at sudo ovs-vsctl br-set-external-id br-ex bridge-id br-ex<br>
> It usually means that the ovs-vswitchd process has exited.<br>
><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">The above result shows that ovs-vswitchd is not running.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">> This can happen for a number of reasons.<br>
> The vswitchd process may exit if it failed to allocate memory (due to memory fragmentation or lack of free hugepages)<br>
> if the ovs-vswitchd.log is not available can you check the the hugepage mount point was created in<br>
> /mnt/huge And that Iis mounted<br>
> Run<br>
> ls -al /mnt/huge<br>
> and<br>
> mount<br>
><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">ls -al /mnt/huge</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">total 4</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">drwxr-xr-x 2 libvirt-qemu kvm 0 Nov 11 15:18 .</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">drwxr-xr-x 3 root root 4096 May 15 00:09 ..</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">ubuntu@ubuntu:~/samta/devstack$ mount</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">/dev/mapper/ubuntu--vg-root on / type ext4 (rw,errors=remount-ro)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">proc on /proc type proc (rw,noexec,nosuid,nodev)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">none on /sys/fs/cgroup type tmpfs (rw)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">none on /sys/fs/fuse/connections type fusectl (rw)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">none on /sys/kernel/debug type debugfs (rw)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">none on /sys/kernel/security type securityfs (rw)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">udev on /dev type devtmpfs (rw,mode=0755)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">none on /run/shm type tmpfs (rw,nosuid,nodev)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">none on /sys/fs/pstore type pstore (rw)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,relatime,cpuset)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">cgroup on /sys/fs/cgroup/cpu type cgroup (rw,relatime,cpu)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,relatime,cpuacct)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">cgroup on /sys/fs/cgroup/memory type cgroup (rw,relatime,memory)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">cgroup on /sys/fs/cgroup/freezer type cgroup (rw,relatime,freezer)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,relatime,net_cls)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">cgroup on /sys/fs/cgroup/blkio type cgroup (rw,relatime,blkio)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,relatime,perf_event)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">cgroup on /sys/fs/cgroup/net_prio type cgroup (rw,relatime,net_prio)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,relatime,hugetlb)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">/dev/sda1 on /boot type ext2 (rw)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">hugetlbfs-kvm on /run/hugepages/kvm type hugetlbfs (rw,mode=775,gid=106)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">nodev on /mnt/huge type hugetlbfs (rw,uid=106,gid=106)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">nodev on /mnt/huge type hugetlbfs (rw,uid=106,gid=106)</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">> then checkout how many hugepages are mounted<br>
><br>
> cat /proc/meminfo | grep huge<br>
><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">cat /proc/meminfo | grep Huge</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">AnonHugePages: 292864 kB</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">HugePages_Total: 5</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">HugePages_Free: 5</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">HugePages_Rsvd: 0</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">HugePages_Surp: 0</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">Hugepagesize: 1048576 kB</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
><br>
> the vswitchd process may also exit if it failed to initializes dpdk interfaces.<br>
> This can happen if no interface is compatible with the igb-uio or vfio-pci drivers<br>
> (note in the vfio-pci case all interface in the same iommu group must be bound to the vfio-pci driver and<br>
> The iommu must be enabled in the kernel command line with VT-d enabled in the bios)<br>
><br>
> Can you check which interface are bound to the dpdk driver by running the following command<br>
><br>
> /opt/stack/DPDK-v2.0.0/tools/dpdk_nic_bind.py --status<br>
><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">/opt/stack/DPDK-v2.0.0/tools/dpdk_nic_bind.py --status</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">Network devices using DPDK-compatible driver</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">============================================</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue"><none></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">Network devices using kernel driver</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">===================================</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">0000:01:00.0 'Ethernet Controller 10-Gigabit X540-AT2' if=p1p1 drv=ixgbe unused=igb_uio </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">0000:02:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=p4p1 drv=i40e unused=igb_uio </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">0000:03:00.0 'Ethernet Controller XL710 for 40GbE QSFP+' if=p2p1 drv=i40e unused=igb_uio </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">0000:06:00.0 'I350 Gigabit Network Connection' if=em1 drv=igb unused=igb_uio *Active*</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">0000:06:00.1 'I350 Gigabit Network Connection' if=em2 drv=igb unused=igb_uio </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">Other network devices</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">=====================</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">0000:01:00.1 'Ethernet Controller 10-Gigabit X540-AT2' unused=igb_uio</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
><br>
> Finally can you confim that ovs-dpdk compiled successfully by either check the xstack.log or<br>
> Checking for the BUILD_COMPLETE file in /opt/stack/ovs<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:blue">BUILD_COMPLETE exist in /opt/stack/ovs though its empty.</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">><br>
> Regards<br>
> sean<br>
><br>
><br>
><br>
><br>
> -----Original Message-----<br>
> From: Samta Rangare [mailto:<a href="mailto:samtarangare@gmail.com" target="_blank">samtarangare@gmail.com</a>]<br>
> Sent: Monday, November 9, 2015 2:31 PM<br>
> To: Czesnowicz, Przemyslaw<br>
> Cc: OpenStack Development Mailing List (not for usage questions)<br>
> Subject: Re: [openstack-dev] [networking-ovs-dpdk]<br>
><br>
> Thanks for replying Przemyslaw, there is no ovs-vswitchd.log in /opt/stack/logs/. This is all contains inside (ovsdb-server.pid, screen).<br>
><br>
> When I cancel stack .sh (ctr c), and try to rerun this $sudo ovs-vsctl br-set-external-id br-ex bridge-id br-ex it didnt hang, that means vSwitch was running isn't it ?<br>
><br>
> But rerunning stack.sh after unstack hangs again.<br>
><br>
> Thanks,<br>
> Samta<br>
><br>
> On Mon, Nov 9, 2015 at 7:50 PM, Czesnowicz, Przemyslaw <<a href="mailto:przemyslaw.czesnowicz@intel.com" target="_blank">przemyslaw.czesnowicz@intel.com</a>> wrote:<br>
>> Hi Samta,<br>
>><br>
>> This usually means that the vSwitch is not running/has crashed.<br>
>> Can you check in /opt/stack/logs/ovs-vswitchd.log ? There should be an error msg there.<br>
>><br>
>> Regards<br>
>> Przemek<br>
>><br>
>>> -----Original Message-----<br>
>>> From: Samta Rangare [mailto:<a href="mailto:samtarangare@gmail.com" target="_blank">samtarangare@gmail.com</a>]<br>
>>> Sent: Monday, November 9, 2015 1:51 PM<br>
>>> To: OpenStack Development Mailing List (not for usage questions)<br>
>>> Subject: [openstack-dev] [networking-ovs-dpdk]<br>
>>><br>
>>> Hello Everyone,<br>
>>><br>
>>> I am installing devstack with networking-ovs-dpdk. The local.conf<br>
>>> exactly looks like the one is available in /opt/stack/networking-ovs-<br>
>>> dpdk/doc/source/_downloads/local.conf.single_node.<br>
>>> So I believe all the necessary configuration will be taken care.<br>
>>><br>
>>> However I am stuck at place where devstack is trying to set<br>
>>> external-id ($ sudo ovs-vsctl br-set-external-id br-ex bridge-id<br>
>>> br-ex). As soon as it hits at this place it's just hangs forever. I<br>
>>> tried commenting this line from<br>
>>> lib/neutron_plugin/ml2 (I know this is wrong) and then all services<br>
>>> came up except ovs-dpdk agent and ovs agent.<br>
>>><br>
>>> BTW I am deploying it in ubuntu 14.04. Any pointer will be really helpful.<br>
>>><br>
>>> Thanks,<br>
>>> Samta<br>
>>><br>
>>> __________________________________________________________<br>
>>> ________________<br>
>>> OpenStack Development Mailing List (not for usage questions)<br>
>>> Unsubscribe: OpenStack-dev-<br>
>>> <a href="http://request@lists.openstack.org?subject:unsubscribe" target="_blank">
request@lists.openstack.org?subject:unsubscribe</a><br>
>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>