<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:646979960;
mso-list-template-ids:1111799702;}
@list l0:level1
{mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Rahul – it seems your issue is similar to the one reported here, probably due to hostname resolution issue.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><a href="https://bugs.launchpad.net/charms/+source/quantum-gateway/+bug/1405588">https://bugs.launchpad.net/charms/+source/quantum-gateway/+bug/1405588</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Regards~hrushi<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Rahul Sharma [<a href="mailto:rahulsharmaait@gmail.com">mailto:rahulsharmaait@gmail.com</a>]
<br>
<b>Sent:</b> Monday, March 14, 2016 3:32 PM<br>
<b>To:</b> openstack <<a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a>>; OpenStack Development Mailing List <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>>;
<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a><br>
<b>Subject:</b> [Openstack-operators] [neutron] openvswitch-agent spins up too many /bin/ovsdb-client processes<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi All,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We are trying to debug an issue with our production environment. We are seeing neutron-openvswitch-agent starts failing after some time (1-2 days). After debugging, we found that there are large number of entries for the ovsdb-client. On
some nodes, it crosses more than 330 processes and then ovsdb process starts failing.<o:p></o:p></p>
</div>
<div>
<div style="border:none;border-left:solid #DDDDDD 1.0pt;padding:0in 0in 0in 6.0pt;background-image:initial;background-repeat:initial">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:0in;text-indent:-.25in;line-height:18.0pt;mso-list:l0 level1 lfo2;background:white;vertical-align:top">
<![if !supportLists]><span style="font-size:9.0pt;font-family:Consolas;color:#333333"><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:9.0pt;font-family:Consolas;color:#333333">root 30689 1 0 00:37 ? 00:00:00 /bin/ovsdb-client monitor Interface name,ofport --format=json<o:p></o:p></span></p>
</div>
<div style="border:none;border-left:solid #DDDDDD 1.0pt;padding:0in 0in 0in 6.0pt;background-image:initial;background-repeat:initial">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:0in;text-indent:-.25in;line-height:18.0pt;mso-list:l0 level1 lfo2;background:white;vertical-align:top">
<![if !supportLists]><span style="font-size:9.0pt;font-family:Consolas;color:#333333"><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:9.0pt;font-family:Consolas;color:#333333">root 30804 1 0 00:38 ? 00:00:00 /bin/ovsdb-client monitor Interface name,ofport --format=json<o:p></o:p></span></p>
</div>
<div style="border:none;border-left:solid #DDDDDD 1.0pt;padding:0in 0in 0in 6.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:0in;text-indent:-.25in;line-height:18.0pt;mso-list:l0 level1 lfo2;background:#F8F8F8;vertical-align:top">
<![if !supportLists]><span style="font-size:9.0pt;font-family:Consolas;color:#333333"><span style="mso-list:Ignore">3.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:9.0pt;font-family:Consolas;color:#333333">root 30909 1 0 00:38 ? 00:00:00 /bin/ovsdb-client monitor Interface name,ofport --format=json<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Pastebin link for the processes: <a href="http://pastebin.com/QGQC0Jrt">http://pastebin.com/QGQC0Jrt</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Pastebin link with openvswitch starting all of them: <a href="http://pastebin.com/repHMkHu">http://pastebin.com/repHMkHu</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">In logs, we start getting errors as:-<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">Mar 14 05:41:29 node2 ovs-vsctl: ovs|00001|fatal_signal|WARN|terminating with signal 14 (Alarm clock)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Mar 14 05:41:39 node2 ovs-vsctl: ovs|00001|fatal_signal|WARN|terminating with signal 14 (Alarm clock)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Mar 14 05:41:49 node2 ovs-vsctl: ovs|00001|fatal_signal|WARN|terminating with signal 14 (Alarm clock)<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Mar 14 05:49:30 node2 ovs-vsctl: ovs|00001|vsctl|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (Protocol error)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Mar 14 05:49:32 node2 ovs-vsctl: ovs|00001|vsctl|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (Protocol error)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Mar 14 05:49:34 node2 ovs-vsctl: ovs|00001|vsctl|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (Protocol error)<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Openvswitch version:-<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">[root@node2 ~(openstack_admin)]# ovs-vsctl --version<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">ovs-vsctl (Open vSwitch) 2.4.0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Compiled Sep 4 2015 09:49:34<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">DB Schema 7.12.1<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We have to restart openvswitch service everytime and that clears up all the processes. We are trying to figure out why so many processes are getting started by neutron-agent? Also, we found that if we restart the host's networking, one
new process for the /bin/ovsdb-client starts. We checked and found that we don't have any network fluctuations or any nic-flappings. Are there any pointers where we should be looking into? It occurs on both controller and compute nodes.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;color:#888888">Rahul Sharma</span></b><span style="font-size:9.5pt;color:#888888"><br>
</span><i><span style="font-size:7.5pt;color:#888888">MS in Computer Science, 2016</span></i><span style="font-size:7.5pt;color:#888888"><br>
College of Computer and Information Science, Northeastern University<br>
Mobile: 801-706-7860<br>
Email: <a href="mailto:rahulsharmaait@gmail.com" target="_blank">rahulsharmaait@gmail.com</a></span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>