<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mv="http://macVmlSchemaUri" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="标题" content="">
<meta name="关键词" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
        {font-family:"Courier New";
        panose-1:2 7 3 9 2 2 5 2 4 4;}
@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
        {font-family:DengXian;
        panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
        {font-family:"\@DengXian";}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
        {font-family:SimSun;
        panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
        {font-family:"\@SimSun";}
@font-face
        {font-family:"MS Mincho";
        panose-1:2 2 6 9 4 2 5 8 3 4;}
@font-face
        {font-family:"\@MS Mincho";}
@font-face
        {font-family:PMingLiU;
        panose-1:2 2 5 0 0 0 0 0 0 0;}
@font-face
        {font-family:"\@PMingLiU";}
@font-face
        {font-family:Consolas;
        panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0cm;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman";}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:purple;
        text-decoration:underline;}
p
        {mso-style-priority:99;
        mso-margin-top-alt:auto;
        margin-right:0cm;
        mso-margin-bottom-alt:auto;
        margin-left:0cm;
        font-size:12.0pt;
        font-family:"Times New Roman";}
code
        {mso-style-priority:99;
        font-family:"Courier New";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
        {mso-style-priority:34;
        margin:0cm;
        margin-bottom:.0001pt;
        text-indent:21.0pt;
        font-size:12.0pt;
        font-family:"Times New Roman";}
span.EmailStyle20
        {mso-style-type:personal;
        font-family:DengXian;
        color:windowtext;}
span.EmailStyle21
        {mso-style-type:personal-reply;
        font-family:DengXian;
        color:windowtext;}
span.msoIns
        {mso-style-type:export-only;
        mso-style-name:"";
        text-decoration:underline;
        color:teal;}
.MsoChpDefault
        {mso-style-type:export-only;
        font-size:10.0pt;}
@page WordSection1
        {size:595.0pt 842.0pt;
        margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
        {page:WordSection1;}
/* List Definitions */
@list l0
        {mso-list-id:1332686438;
        mso-list-type:hybrid;
        mso-list-template-ids:-1352624874 -1753724354 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
        {mso-level-tab-stop:none;
        mso-level-number-position:left;
        margin-left:18.0pt;
        text-indent:-18.0pt;}
@list l0:level2
        {mso-level-number-format:alpha-lower;
        mso-level-text:"%2\)";
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        margin-left:48.0pt;
        text-indent:-24.0pt;}
@list l0:level3
        {mso-level-number-format:roman-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:right;
        margin-left:72.0pt;
        text-indent:-24.0pt;}
@list l0:level4
        {mso-level-tab-stop:none;
        mso-level-number-position:left;
        margin-left:96.0pt;
        text-indent:-24.0pt;}
@list l0:level5
        {mso-level-number-format:alpha-lower;
        mso-level-text:"%5\)";
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        margin-left:120.0pt;
        text-indent:-24.0pt;}
@list l0:level6
        {mso-level-number-format:roman-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:right;
        margin-left:144.0pt;
        text-indent:-24.0pt;}
@list l0:level7
        {mso-level-tab-stop:none;
        mso-level-number-position:left;
        margin-left:168.0pt;
        text-indent:-24.0pt;}
@list l0:level8
        {mso-level-number-format:alpha-lower;
        mso-level-text:"%8\)";
        mso-level-tab-stop:none;
        mso-level-number-position:left;
        margin-left:192.0pt;
        text-indent:-24.0pt;}
@list l0:level9
        {mso-level-number-format:roman-lower;
        mso-level-tab-stop:none;
        mso-level-number-position:right;
        margin-left:216.0pt;
        text-indent:-24.0pt;}
ol
        {margin-bottom:0cm;}
ul
        {margin-bottom:0cm;}
--></style>
</head>
<body bgcolor="white" lang="ZH-CN" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">I get my message has been automatically rejected by the
</span><span lang="EN-US" style="font-family:Consolas"><a href="mailto:openstack-operators-owner@lists.openstack.org"><span style="font-size:11.0pt;color:#0000E9">openstack-operators-owner@lists.openstack.org</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:Consolas">.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:Consolas">Resend.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Hi John,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Did you know where the packet dropped? On physical interface / tap device / ovs port or in the vm.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">We hit udp packet loss when the large pps. The following things you may double check:</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2">
<![if !supportLists]><span lang="EN-US"><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">     
</span></span></span><![endif]><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Double check if your physical interface is dropping packet.  Usually if you rx queue ring size or rx queue number is default value ,it will drop udp packet . It
 will start to drop packet if it reach to about 200kpps in one cpu core(rss will distribute traffic to different core, for one single core, it will drop packet about 200kpps in my exp).</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Usually you can get the statics from ethtool -S interface to check if there is packet loss because of rx queue full. And
 use ethtool to increase your ring size. I tested in my environment that if ring size increase from 512 to 4096, it can double the throughput from 200kpps to 400kpps in one cpu core. This may help in some case.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2">
<![if !supportLists]><span lang="EN-US"><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">     
</span></span></span><![endif]><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Double check if your TAP device dropped packet, the default tx_queue length is 500 or 1000, increase it to 10000 may help in some case.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2">
<![if !supportLists]><span lang="EN-US"><span style="mso-list:Ignore">3.<span style="font:7.0pt "Times New Roman"">     
</span></span></span><![endif]><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Double check your nf_conntrack_max in compute node and network node, the default value is 65535, in our case it usually reach to 500k-1m . we change it as following:</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">net.netfilter.nf_conntrack_max=10240000</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">net.nf_conntrack_max=10240000</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">if you see , something like
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">nf_conntrack: table full, dropping packet</span>”<span lang="EN-US"> in your /var/log/message log, that means you hit this one.</span></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2">
<![if !supportLists]><span lang="EN-US"><span style="mso-list:Ignore">4.<span style="font:7.0pt "Times New Roman"">     
</span></span></span><![endif]><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">You could check if drop happened inside your vm, increase the following param maybe help in some case:</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">net.core.rmem_max / net.core.rmem_default / net.core.wmem_max / net.core.rmem_default</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2">
<![if !supportLists]><span lang="EN-US"><span style="mso-list:Ignore">5.<span style="font:7.0pt "Times New Roman"">     
</span></span></span><![endif]><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">If you are using default network driver(virtio-net), you can double check if your vhost of your vm is full with CPU soft irq. You can find it by the process name
 is vhost-$PID_OF_YOUR_VM . In this case, if you can try the following feature in
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">L</span>”<span lang="EN-US">:</span></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><a href="https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-virtiomq.html">https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-virtiomq.html</a></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">      multi-queue may help you some case, but it will use more vhost and more cpu in your host.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2">
<![if !supportLists]><span lang="EN-US"><span style="mso-list:Ignore">6.<span style="font:7.0pt "Times New Roman"">     
</span></span></span><![endif]><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Sometimes cpu numa pin can also help, but you need to reserve them and static plan you cpu.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">I think we should figure out the packet lost in where and which is the bottleneck. Hope this help,  John.
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Thanks.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Regards,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Liping Mao</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><o:p></o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-family:SimSun;color:black">发件人</span></b><b><span lang="EN-US" style="font-family:Calibri;color:black">:
</span></b><span lang="EN-US" style="font-family:Calibri;color:black">John Petrini <jpetrini@coredial.com><br>
</span><b><span style="font-family:"MS Mincho";color:black">日期</span></b><b><span lang="EN-US" style="font-family:Calibri;color:black">:
</span></b><span lang="EN-US" style="font-family:Calibri;color:black">2017</span><span style="font-family:"MS Mincho";color:black">年</span><span lang="EN-US" style="font-family:Calibri;color:black">7</span><span style="font-family:"MS Mincho";color:black">月</span><span lang="EN-US" style="font-family:Calibri;color:black">28</span><span style="font-family:"MS Mincho";color:black">日</span><span style="font-family:Calibri;color:black">
</span><span style="font-family:"MS Mincho";color:black">星期五</span><span lang="EN-US" style="font-family:Calibri;color:black"> 03:35</span><span lang="EN-US" style="font-family:PMingLiU;color:black"><br>
</span><b><span style="font-family:"MS Mincho";color:black">至</span></b><b><span lang="EN-US" style="font-family:Calibri;color:black">:
</span></b><span lang="EN-US" style="font-family:Calibri;color:black">Pedro Sousa <pgsousa@gmail.com>, OpenStack Mailing List <openstack@lists.openstack.org>, "openstack-operators@lists.openstack.org" <openstack-operators@lists.openstack.org><br>
</span><b><span style="font-family:"MS Mincho";color:black">主</span></b><b><span style="font-family:SimSun;color:black">题</span></b><b><span lang="EN-US" style="font-family:Calibri;color:black">:
</span></b><span lang="EN-US" style="font-family:Calibri;color:black">Re: [Openstack] [Openstack-operators] UDP Buffer Filling</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi Pedro, <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thank you for the suggestion. I will look into this.<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><br clear="all">
<o:p></o:p></span></p>
<div>
<div>
<div>
<div>
<p style="mso-margin-top-alt:6.0pt;margin-right:0cm;margin-bottom:6.0pt;margin-left:0cm;background:white">
<span lang="EN-US" style="font-size:14.0pt;color:#333333">John Petrini</span><span lang="EN-US"><o:p></o:p></span></p>
<p style="mso-margin-top-alt:6.0pt;margin-right:0cm;margin-bottom:6.0pt;margin-left:0cm;background:white">
<span lang="EN-US" style="font-size:9.0pt;font-family:Helvetica;color:#333333">Platforms Engineer   //   <b>CoreDial, LLC</b>   //   <a href="http://coredial.com/" target="_blank" title="CoreDial"><span style="color:#2368B3;text-decoration:none">coredial.com</span></a>   //   <a href="https://twitter.com/coredial" target="_blank"><span style="color:#1155CC;text-decoration:none"><img border="0" width="19" height="18" id="_x0000_i1025" src="http://www.stratusinteractive.com/coredial/email/twitter.gif" alt="itter"></span></a>   <a href="http://www.linkedin.com/company/99631" target="_blank"><span style="color:#1155CC;text-decoration:none"><img border="0" width="18" height="18" id="_x0000_i1026" src="http://www.stratusinteractive.com/coredial/email/linkedin.gif" alt="nkedIn"></span></a>   <a href="https://plus.google.com/104062177220750809525/posts" target="_blank"><span style="color:#1155CC;text-decoration:none"><img border="0" width="19" height="18" id="_x0000_i1027" src="http://www.stratusinteractive.com/coredial/email/googleplus.gif" alt="ogle Plus"></span></a>   <a href="http://success.coredial.com/blog" target="_blank"><span style="color:#1155CC;text-decoration:none"><img border="0" width="18" height="18" id="_x0000_i1028" src="http://www.stratusinteractive.com/coredial/email/rss.gif" alt="og"></span></a> <br>
751 Arbor Way, Hillcrest I, Suite 150, Blue Bell, PA 19422<br>
<b>P:</b> 215.297.4400 x232   //   <b>F: </b>215.297.4401   //   <b>E: </b><a href="mailto:jpetrini@coredial.com" target="_blank">jpetrini@coredial.com</a></span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Thu, Jul 27, 2017 at 12:25 PM, Pedro Sousa <<a href="mailto:pgsousa@gmail.com" target="_blank">pgsousa@gmail.com</a>> wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span lang="EN-US">Hi, <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">have you considered to implement some network acceleration technique like to OVS-DPDK or SR-IOV?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">In these kind of workloads (voice, video) that have low latency requirements you might need to use something like DPDK to avoid these issues.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Regards<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">On Thu, Jul 27, 2017 at 4:49 PM, John Petrini <<a href="mailto:jpetrini@coredial.com" target="_blank">jpetrini@coredial.com</a>> wrote:<o:p></o:p></span></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi List, <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">We are running Mitaka with VLAN provider networking. We've recently encountered a problem where the UDP receive queue on instances is filling up and we begin dropping packets. Moving instances out of OpenStack onto bare
 metal resolves the issue completely.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">These instances are running asterisk which should be pulling these packets off the queue but it appears to be falling behind no matter the resources we give it.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">We can't seem to pin down a reason why we would see this behavior in KVM but not on metal. I'm hoping someone on the list might have some insight or ideas.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thank You,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">John<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US">_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><o:p></o:p></span></p>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
</body>
</html>