<div dir="ltr">On Thu, Jul 25, 2013 at 7:30 PM, Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span> wrote:<br>
<div class="gmail_quote"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
You will need to provide more details than "old" vs. "new" OpenStack.<br>
<br>
For example...<br>
<br>
1) What is your network model in the old vs. new<br></blockquote></div><div>Old: nova-network + FlatDHCP <br></div><div>New: Quantum + OpenvSwitch + network-namespace + metadata-ns-proxy<br><br></div><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
2) What version of OpenStack is the old<br></blockquote></div><div>Old: Essex<br></div><div>New: Grizzly <br><br></div><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
3) Is Keystone used in old and new? If so, what drivers are used in Keystone?<br></blockquote></div><div>Yes, both use Keystone with SQL backend<br><br></div><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
4) Do you have errors in any of your log files (usually an indication that something like a timeout or failure on </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
RPC which may affect performance)<br></blockquote></div><div>No, i didn't found it.<br> <br></div><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
5) Are you using nova-conductor in the new?<br></blockquote></div><div>I have nova-conductor process run on Controller node, but seem nova-compute don't use this.<br> <br></div><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
6) What database backend are you using?<br></blockquote></div><div>MySQL.<br> <br></div><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
7) Do a time keystone user-list on both old and new<br></blockquote></div><div># keystone user-list<br></div><div>Old: <br></div><div>Run 1st time:<br>real <span style="color:rgb(255,0,0)"><b>0m2.494s</b></span><br>user 0m0.112s<br>
sys 0m0.040s<br></div><div>2nd time:<br>real <span style="color:rgb(0,0,255)"> <b>0m0.279s</b></span><br>user 0m0.112s<br>sys 0m0.020s<br><br></div><div><br></div><div>New:<br>Run 1st time:<br>real <b><span style="color:rgb(255,0,0)">0m1.687s</span></b><br>
user 0m0.176s<br>sys 0m0.012s<br></div><div>2nd time:<br>real <b><span style="color:rgb(0,0,255)">0m0.213s</span></b><br>user 0m0.160s<br>sys 0m0.040s<br></div><div class="im"><div><br></div><div><br></div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
8) Pastebin your conf files, with passwords removed<br></blockquote></div><div>My config files are quite large.<br></div><div>I can sum up that all services & API are HA-ed using HAproxy, RabbitMQ Cluster, Database replication, v.v...<br>
<br></div>This morning, i just do some tuning memcached for Dashboard service (HA-ed using HAProxy), then my Dashboard become faster significantly. <br></div><div class="gmail_quote"><br></div><div class="gmail_quote"><div>
The questions remain are:<br></div><div>1. Why metadata query so slow? Possible problems? ( "curl <a href="http://169.254.169.254/openstack" target="_blank">http://169.254.169.254/openstack</a>" took 2.5-5 seconds to complete, "ec2metadata" took 14-17 seconds to complete - i tested many times.) <br>
</div><div>2. Why API calls using *-client reduce time from 2nd time on my old Essex deployment, but not on my new Grizzly deployment? Maybe I need some "cache" settings?<br><br></div><div>Thank you very much!<br>
</div><div><div class="h5"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
The more information you give, the better folks can help you.<br>
<br>
Best,<br>
-jay<div><br>
<br>
On 07/25/2013 07:14 AM, Chu Duc Minh wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
Check some more API (I run these command below from Controller node):<br>
# time quantum subnet-list<br>
(...have 4 subnet)<br>
real 0m0.676s<br>
user 0m0.196s<br>
sys 0m0.020s<br>
<br>
# time quantum router-list<br>
(...have 1 router)<br>
real 0m0.496s<br>
user 0m0.164s<br>
sys 0m0.052s<br>
<br>
# time nova list --all_tenants=1<br>
(...have 5 instances)<br>
real 0m1.290s<br>
user 0m0.308s<br>
sys 0m0.040s<br>
<br>
Compare with my old OpenStack deployment on weaker servers, it took 1/3<br>
in times.<br>
<br>
<br>
<br>
On Thu, Jul 25, 2013 at 5:43 PM, Peter Cheung <<a href="mailto:mcheung63@hotmail.com" target="_blank">mcheung63@hotmail.com</a><br></div><div>
<mailto:<a href="mailto:mcheung63@hotmail.com" target="_blank">mcheung63@hotmail.com</a>><u></u>> wrote:<br>
<br>
I am having a problem about calling API speed is up and down,<br>
something need 0.1s, something it needs 3s<br>
<br>
Thanks<br>
from Peter<br>
<br>
<br></div>
------------------------------<u></u>------------------------------<u></u>------------<div><br>
Date: Thu, 25 Jul 2013 17:41:11 +0700<br></div>
From: <a href="mailto:chu.ducminh@gmail.com" target="_blank">chu.ducminh@gmail.com</a> <mailto:<a href="mailto:chu.ducminh@gmail.com" target="_blank">chu.ducminh@gmail.com</a>><br>
To: <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
<mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.<u></u>launchpad.net</a>>;<br>
<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a> <mailto:<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.<u></u>openstack.org</a>><div>
<br>
Subject: [Openstack] Call to API very slow [Grizzly]<br>
<br>
<br>
All operations in my Openstack dashboard very slow (compare to my<br>
old Openstack deployment)<br>
Then i do some check on an instance:<br>
<br>
$ time curl <a href="http://169.254.169.254/openstack" target="_blank">http://169.254.169.254/<u></u>openstack</a><br>
2012-08-10<br>
2013-04-04<br>
latest<br></div>
real 0m*5.605s*<div><br>
user 0m0.004s<br>
sys 0m0.004s<br>
<br>
5 seconds for a simple API query !??<br>
<br>
<br>
in quantum-ns-metadata-proxyxxxx.<u></u>log, i saw:<br></div>
2013-07-25 *17:17:09 * DEBUG<div><br>
[quantum.agent.metadata.<u></u>namespace_proxy] Request: GET /openstack<br>
HTTP/1.0<br>
Accept: */*<br>
Content-Type: text/plain<br>
Host: 169.254.169.254<br>
User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0<br></div>
OpenSSL/1.0.1 zlib/<a href="http://1.2.3.4" target="_blank">1.2.3.4</a> <<a href="http://1.2.3.4" target="_blank">http://1.2.3.4</a>> libidn/1.23 librtmp/2.3<br>
2013-07-25 *17:17:14* DEBUG<div><br>
[quantum.agent.metadata.<u></u>namespace_proxy] {'date': 'Thu, 25 Jul 2013<br>
10:17:14 GMT', 'status': '200', 'content-length': '28',<br>
'content-type': 'text/html; charset=UTF-8', 'content-location':<br>
u'<a href="http://169.254.169.254/openstack" target="_blank">http://169.254.169.254/<u></u>openstack</a>'}<br>
2013-07-25 17:17:14 DEBUG<br>
[quantum.agent.metadata.<u></u>namespace_proxy] 2012-08-10<br>
2013-04-04<br>
latest<br>
<br>
I take a look at metadata-agent.log, and saw almost request/response<br></div>
finished @*17:17:09<br>
*<br>
But the last finished *@**17:17:14<br>
*2013-07-25 *17:17:14* DEBUG [quantum.agent.metadata.agent]<div><br>
{'date': 'Thu, 25 Jul 2013 10:17:14 GMT', 'status': '200',<br>
'content-length': '28', 'content-type': 'text/html; charset=UTF-8',<br>
'content-location': u'<a href="http://172.30.1.14:8775/openstack" target="_blank">http://172.30.1.14:8775/<u></u>openstack</a>'}<br></div>
*<br>
*<div><br>
I enabled slow query log on MySql, but can't find any slow query.<br>
<br>
Do you know possible problems in this situation?<br>
Thank you very much!<br>
<br>
<br>
______________________________<u></u>_________________ Mailing list:<br>
<a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a> Post to :<br></div>
<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a> <mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.<u></u>launchpad.net</a>><div>
<br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a> More help :<br>
<a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/<u></u>ListHelp</a><br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/<u></u>ListHelp</a><br>
<br>
</div></blockquote><div><div>
<br>
<br>
______________________________<u></u>_________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/<u></u>ListHelp</a><br>
</div></div></blockquote></div></div></div><br></div></div>
</div><br></div>