<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"HP Simplified";
panose-1:2 11 6 4 2 2 4 2 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"HP Simplified","sans-serif";
color:windowtext;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hello Marun,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Please find the details about my setup and tests which i have done so far<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><u>Setup<o:p></o:p></u></p>
<p class="MsoNormal"> - One Physical Box with 16c, 256G memory. 2 VMs created on this Box - One for Controller and One for Network Node<o:p></o:p></p>
<p class="MsoNormal"> - 16x compute nodes (each has 16c, 256G memory)<o:p></o:p></p>
<p class="MsoNormal"> - All the systems are installed with Ubuntu Precise + Havana Bits from Ubuntu Cloud Archive<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Steps to simulate the issue<o:p></o:p></p>
<p class="MsoNormal"> 1) Concurrently create 30 Instances (m1.small) using REST API with mincount=30<o:p></o:p></p>
<p class="MsoNormal"> 2) sleep for 20min and repeat the step (1)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><u>Issue 1<o:p></o:p></u></p>
<p class="MsoNormal">In Havana, once we cross 150 instances (5 batches x 30) during 6th batch some instances are going into ERROR state<o:p></o:p></p>
<p class="MsoNormal">due to network port not able to create and some instances are getting duplicate IP address<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Per Maru Newby this issue might related to this bug<o:p></o:p></p>
<p class="MsoNormal">https://bugs.launchpad.net/bugs/1192381<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have done the similar with Grizzly on the same environment 2 months back, where I could able to deploy close to 240 instances without any errors<o:p></o:p></p>
<p class="MsoNormal">Initially on Grizzly also seen the same behavior but with these tunings based on this bug
<o:p></o:p></p>
<p class="MsoNormal">https://bugs.launchpad.net/neutron/+bug/1160442, never had issues (tested more than 10 times)<o:p></o:p></p>
<p class="MsoNormal"> sqlalchemy_pool_size = 60 <o:p></o:p></p>
<p class="MsoNormal"> sqlalchemy_max_overflow = 120 <o:p></o:p></p>
<p class="MsoNormal"> sqlalchemy_pool_timeout = 2 <o:p></o:p></p>
<p class="MsoNormal"> agent_down_time = 60<o:p></o:p></p>
<p class="MsoNormal"> report_internval = 20<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">In Havana, I have tuned the same tunables but I could never get past 150+ instances. Without the tunables I could not able to get past<o:p></o:p></p>
<p class="MsoNormal">100 instances. We are getting many timeout errors from the DHCP agent and neutron clients<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">NOTE: After tuning the agent_down_time to 60 and report_interval to 20, we no longer getting these error messages<o:p></o:p></p>
<p class="MsoNormal"> 2013-12-02 11:44:43.421 28201 WARNING neutron.scheduler.dhcp_agent_scheduler [-] No more DHCP agents<o:p></o:p></p>
<p class="MsoNormal"> 2013-12-02 11:44:43.439 28201 WARNING neutron.scheduler.dhcp_agent_scheduler [-] No more DHCP agents<o:p></o:p></p>
<p class="MsoNormal"> 2013-12-02 11:44:43.452 28201 WARNING neutron.scheduler.dhcp_agent_scheduler [-] No more DHCP agents<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In the compute node openvswitch agent logs, we see these errors repeating continuously<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">2013-12-04 06:46:02.081 3546 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Timeout: Timeout while waiting on RPC response - topic: "q-plugin", RPC method: "security_group_rules_for_devices" info: "<unknown>"<o:p></o:p></p>
<p class="MsoNormal">and WARNING neutron.openstack.common.rpc.amqp [-] No calling threads waiting for msg_id<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">DHCP agent has below errors<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">2013-12-02 15:35:19.557 22125 ERROR neutron.agent.dhcp_agent [-] Unable to reload_allocations dhcp.<o:p></o:p></p>
<p class="MsoNormal">2013-12-02 15:35:19.557 22125 TRACE neutron.agent.dhcp_agent Timeout: Timeout while waiting on RPC response - topic: "q-plugin", RPC method: "get_dhcp_port" info: "<unknown>"<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">2013-12-02 15:35:34.266 22125 ERROR neutron.agent.dhcp_agent [-] Unable to sync network state.<o:p></o:p></p>
<p class="MsoNormal">2013-12-02 15:35:34.266 22125 TRACE neutron.agent.dhcp_agent Timeout: Timeout while waiting on RPC response - topic: "q-plugin", RPC method: "get_active_networks_info" info: "<unknown>"<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In Havana, I have merged the code from this patch and set api_workers to 8 (My Controller VM has 8cores/16Hyperthreads)<o:p></o:p></p>
<p class="MsoNormal">https://review.openstack.org/#/c/37131/<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">After this patch and starting 8 neutron-server worker threads, during the batch creation of 240 instances with 30 concurrent requests during each batch,<o:p></o:p></p>
<p class="MsoNormal">238 instances became active and 2 instances went into error. Interesting these 2 instances which went into error state are from the same compute node.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Unlike earlier this time, the errors are due to 'Too Many Connections' to the MySQL database.
<o:p></o:p></p>
<p class="MsoNormal">2013-12-04 17:07:59.877 21286 AUDIT nova.compute.manager [req-26d64693-d1ef-40f3-8350-659e34d5b1d7 c4d609870d4447c684858216da2f8041 9b073211dd5c4988993341cc955e200b] [instance: c14596fd-13d5-482b-85af-e87077d4ed9b] Terminating instance<o:p></o:p></p>
<p class="MsoNormal">2013-12-04 17:08:00.578 21286 ERROR nova.compute.manager [req-26d64693-d1ef-40f3-8350-659e34d5b1d7 c4d609870d4447c684858216da2f8041 9b073211dd5c4988993341cc955e200b] [instance: c14596fd-13d5-482b-85af-e87077d4ed9b] Error: Remote error:
OperationalError (OperationalError) (1040, 'Too many connections') None None<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Need to back port the patch 'https://review.openstack.org/#/c/37131/' to address the Neutron Scaling issues in Havana.<o:p></o:p></p>
<p class="MsoNormal">Carl already back porting this patch into Havana <span class="apple-converted-space">
<span style="font-size:10.0pt;font-family:Consolas;color:black;background:#F0F7FF"> </span></span><a href="https://review.openstack.org/#/c/60082/" target="_blank"><span style="font-size:10.0pt;font-family:Consolas;background:#F0F7FF">https://review.openstack.org/#/c/60082/</span></a>
which is good. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><u>Issue 2<o:p></o:p></u></p>
<p class="MsoNormal">Grizzly :<o:p></o:p></p>
<p class="MsoNormal">During the concurrent instance creation in Grizzly, once we cross 210 instances, during subsequent 30 instance creation some of<o:p></o:p></p>
<p class="MsoNormal">the instances could not get their IP address during the first boot with in first few min. Instance MAC and IP Address details<o:p></o:p></p>
<p class="MsoNormal">were updated in the dnsmasq host file but with a delay. Instances are able to get their IP address with a delay eventually.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">If we reboot the instance using 'nova reboot' instance used to get IP Address.
<o:p></o:p></p>
<p class="MsoNormal">* Amount of delay is depending on number of network ports and delay is in the range of 8seconds to 2min<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Havana :<o:p></o:p></p>
<p class="MsoNormal">But in Havana only 81 instances could get the IP during the first boot. Port is getting created and IP address are getting allocated<o:p></o:p></p>
<p class="MsoNormal">very fast, but by the time port is UP its taking quite lot of time. Once the port is UP, Instances are able to send the DHCP Request<o:p></o:p></p>
<p class="MsoNormal">and get the IP address. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">During the network port create and network port update, there are lot of 'security_group_rules_for_devices' messages. OVS Agents in the<o:p></o:p></p>
<p class="MsoNormal">compute nodes are getting Timeouts during "security_group_rules_for_devices"<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Even though this issue exist in Grizzly but we observed this issue only after 200+ active instances (200 network ports), but in Havana<o:p></o:p></p>
<p class="MsoNormal">We are having these issues with less than 100 active ports. <o:p>
</o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In Havana, if we reboot the instance it’s not able to get the IP Address even though its network port entry is already exist in<o:p></o:p></p>
<p class="MsoNormal">dnsmasq hosts file. We can't even ping the IP Address now which we were able to ping before the instance reboot. After restarting the<o:p></o:p></p>
<p class="MsoNormal">'neutron-dhcp-agent' service which will restart the 'dnsmasq' and reboot of the instance could get the IP<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This clear shows we have performance regression in neutron/havana compared to quantum/grizzly<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"HP Simplified","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal">FYI, I attached the my notes for one of the instance which could not get IP during first boot with this email<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"HP Simplified","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal">I am happy to share the results of my grizzly tests and logs during recent havana tests<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks & Regards,<o:p></o:p></p>
<p class="MsoNormal">Sreedhar Nathani<o:p></o:p></p>
</div>
</body>
</html>