<div dir="ltr">Thanks Hrushi. <div><br></div><div>After further troubleshooting, we found that somehow openvswitch-agent was reading a file named "sensu" from /etc/sudoers.d directory and was failing in reading it, and we haven't configured anything like that in neutron configs. Removing that helped in bringing everything back to normal, but still its not clear why it was reading that file. We are trying to figure that out.<br><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><b style="font-size:12.8000001907349px;color:rgb(136,136,136)"><font size="2">Rahul Sharma</font></b><br style="font-size:12.8000001907349px;color:rgb(136,136,136)"><font size="1" style="color:rgb(136,136,136)"><i>MS in Computer Science, 2016</i><br>College of Computer and Information Science, Northeastern University<br>Mobile: 801-706-7860<br>Email: <a href="mailto:rahulsharmaait@gmail.com" target="_blank">rahulsharmaait@gmail.com</a></font><br></div></div></div>
<br><div class="gmail_quote">On Mon, Mar 14, 2016 at 9:10 PM, Gangur, Hrushikesh <span dir="ltr"><<a href="mailto:hrushikesh.gangur@hpe.com" target="_blank">hrushikesh.gangur@hpe.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Rahul – it seems your issue is similar to the one reported here, probably due to hostname resolution issue.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><a href="https://bugs.launchpad.net/charms/+source/quantum-gateway/+bug/1405588" target="_blank">https://bugs.launchpad.net/charms/+source/quantum-gateway/+bug/1405588</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Regards~hrushi<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Rahul Sharma [<a href="mailto:rahulsharmaait@gmail.com" target="_blank">mailto:rahulsharmaait@gmail.com</a>]
<br>
<b>Sent:</b> Monday, March 14, 2016 3:32 PM<br>
<b>To:</b> openstack <<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a>>; OpenStack Development Mailing List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>>;
<a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.openstack.org</a><br>
<b>Subject:</b> [Openstack-operators] [neutron] openvswitch-agent spins up too many /bin/ovsdb-client processes<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hi All,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We are trying to debug an issue with our production environment. We are seeing neutron-openvswitch-agent starts failing after some time (1-2 days). After debugging, we found that there are large number of entries for the ovsdb-client. On
some nodes, it crosses more than 330 processes and then ovsdb process starts failing.<u></u><u></u></p>
</div>
<div>
<div style="border:none;border-left:solid #dddddd 1.0pt;padding:0in 0in 0in 6.0pt;background-image:initial;background-repeat:initial">
<p class="MsoNormal" style="margin-left:0in;line-height:18.0pt;background:white;vertical-align:top">
<u></u><span style="font-size:9.0pt;font-family:Consolas;color:#333333"><span>1.<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:9.0pt;font-family:Consolas;color:#333333">root 30689 1 0 00:37 ? 00:00:00 /bin/ovsdb-client monitor Interface name,ofport --format=json<u></u><u></u></span></p>
</div>
<div style="border:none;border-left:solid #dddddd 1.0pt;padding:0in 0in 0in 6.0pt;background-image:initial;background-repeat:initial">
<p class="MsoNormal" style="margin-left:0in;line-height:18.0pt;background:white;vertical-align:top">
<u></u><span style="font-size:9.0pt;font-family:Consolas;color:#333333"><span>2.<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:9.0pt;font-family:Consolas;color:#333333">root 30804 1 0 00:38 ? 00:00:00 /bin/ovsdb-client monitor Interface name,ofport --format=json<u></u><u></u></span></p>
</div>
<div style="border:none;border-left:solid #dddddd 1.0pt;padding:0in 0in 0in 6.0pt">
<p class="MsoNormal" style="margin-left:0in;line-height:18.0pt;background:#f8f8f8;vertical-align:top">
<u></u><span style="font-size:9.0pt;font-family:Consolas;color:#333333"><span>3.<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:9.0pt;font-family:Consolas;color:#333333">root 30909 1 0 00:38 ? 00:00:00 /bin/ovsdb-client monitor Interface name,ofport --format=json<u></u><u></u></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Pastebin link for the processes: <a href="http://pastebin.com/QGQC0Jrt" target="_blank">http://pastebin.com/QGQC0Jrt</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Pastebin link with openvswitch starting all of them: <a href="http://pastebin.com/repHMkHu" target="_blank">http://pastebin.com/repHMkHu</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">In logs, we start getting errors as:-<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">Mar 14 05:41:29 node2 ovs-vsctl: ovs|00001|fatal_signal|WARN|terminating with signal 14 (Alarm clock)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Mar 14 05:41:39 node2 ovs-vsctl: ovs|00001|fatal_signal|WARN|terminating with signal 14 (Alarm clock)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Mar 14 05:41:49 node2 ovs-vsctl: ovs|00001|fatal_signal|WARN|terminating with signal 14 (Alarm clock)<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Mar 14 05:49:30 node2 ovs-vsctl: ovs|00001|vsctl|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (Protocol error)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Mar 14 05:49:32 node2 ovs-vsctl: ovs|00001|vsctl|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (Protocol error)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Mar 14 05:49:34 node2 ovs-vsctl: ovs|00001|vsctl|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (Protocol error)<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Openvswitch version:-<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">[root@node2 ~(openstack_admin)]# ovs-vsctl --version<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">ovs-vsctl (Open vSwitch) 2.4.0<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Compiled Sep 4 2015 09:49:34<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">DB Schema 7.12.1<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We have to restart openvswitch service everytime and that clears up all the processes. We are trying to figure out why so many processes are getting started by neutron-agent? Also, we found that if we restart the host's networking, one
new process for the /bin/ovsdb-client starts. We checked and found that we don't have any network fluctuations or any nic-flappings. Are there any pointers where we should be looking into? It occurs on both controller and compute nodes.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;color:#888888">Rahul Sharma</span></b><span style="font-size:9.5pt;color:#888888"><br>
</span><i><span style="font-size:7.5pt;color:#888888">MS in Computer Science, 2016</span></i><span style="font-size:7.5pt;color:#888888"><br>
College of Computer and Information Science, Northeastern University<br>
Mobile: <a href="tel:801-706-7860" value="+18017067860" target="_blank">801-706-7860</a><br>
Email: <a href="mailto:rahulsharmaait@gmail.com" target="_blank">rahulsharmaait@gmail.com</a></span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div></div></blockquote></div><br></div></div></div>