<div dir="ltr"><div>If for whatever reason you don't want to use the memcache token driver, I've seen an order-of-magnitude increase in performance simply from adding an index to the "expires" column in the token table. I opened a blueprint (<a href="https://blueprints.launchpad.net/keystone/+spec/index-token-expiry">https://blueprints.launchpad.net/keystone/+spec/index-token-expiry</a>) to get this in mainline, but until that lands in the version you're using, you may benefit from adding the index manually.<br>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jul 26, 2013 at 12:03 PM, Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On 07/25/2013 11:23 PM, Chu Duc Minh wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
On Thu, Jul 25, 2013 at 7:30 PM, Jay Pipes <<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a><br></div><div class="im">
<mailto:<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>>> wrote:<br>
<br>
You will need to provide more details than "old" vs. "new" OpenStack.<br>
<br>
For example...<br>
<br>
1) What is your network model in the old vs. new<br>
<br>
Old: nova-network + FlatDHCP<br>
New: Quantum + OpenvSwitch + network-namespace + metadata-ns-proxy<br>
</div></blockquote>
<br>
OK. If you do a time ping -c1 <IP> from a VM to the metadata IP, what kind of times do you see in each setup?<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
2) What version of OpenStack is the old<br>
<br>
Old: Essex<br>
New: Grizzly<br>
</blockquote>
<br></div>
OK.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
3) Is Keystone used in old and new? If so, what drivers are used in<br>
Keystone?<br>
<br>
Yes, both use Keystone with SQL backend<br>
</blockquote>
<br></div>
I recommend switching to the memcache Token driver in Keystone.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
4) Do you have errors in any of your log files (usually an<br>
indication that something like a timeout or failure on<br>
<br>
RPC which may affect performance)<br>
<br>
No, i didn't found it.<br>
</blockquote>
<br></div>
OK.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
5) Are you using nova-conductor in the new?<br>
<br>
I have nova-conductor process run on Controller node, but seem<br>
nova-compute don't use this.<br>
</blockquote>
<br></div>
OK, good to know, but unlikely to be the cause of a metadata API query slowdown.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
6) What database backend are you using?<br>
<br>
MySQL.<br>
</blockquote>
<br></div>
OK.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
7) Do a time keystone user-list on both old and new<br>
<br>
# keystone user-list<br>
Old:<br>
Run 1st time:<br></div>
real *0m2.494s*<div class="im"><br>
user 0m0.112s<br>
sys 0m0.040s<br>
2nd time:<br></div>
real *0m0.279s*<div class="im"><br>
user 0m0.112s<br>
sys 0m0.020s<br>
<br>
<br>
New:<br>
Run 1st time:<br></div>
real *0m1.687s*<div class="im"><br>
user 0m0.176s<br>
sys 0m0.012s<br>
2nd time:<br></div>
real *0m0.213s*<br>
user 0m0.160s<br>
sys 0m0.040s<br>
</blockquote>
<br>
Interesting, so Keystone is actually faster in Grizzly than in Essex... or at least it seems to be given the above rudimentary timing.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
8) Pastebin your conf files, with passwords removed<br>
<br>
My config files are quite large.<br>
I can sum up that all services & API are HA-ed using HAproxy, RabbitMQ<br>
Cluster, Database replication, v.v...<br>
</blockquote>
<br></div>
OK.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
This morning, i just do some tuning memcached for Dashboard service<br>
(HA-ed using HAProxy), then my Dashboard become faster significantly.<br>
<br>
The questions remain are:<br>
1. Why metadata query so slow? Possible problems? ( "curl<br>
<a href="http://169.254.169.254/openstack" target="_blank">http://169.254.169.254/<u></u>openstack</a>" took 2.5-5 seconds to complete,<br>
"ec2metadata" took 14-17 seconds to complete - i tested many times.)<br>
</blockquote>
<br></div>
Could be a number of things. I thought Vish had redesigned the metadata API endpoint in Folsom and removed the problematic slow queries that used to be in there. If you check your MySQL slow log (hopefully you have it enabled), look to see if any of the queries in there reference the instance_metadata table.<div class="im">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
2. Why API calls using *-client reduce time from 2nd time on my old<br>
Essex deployment, but not on my new Grizzly deployment? Maybe I need<br>
some "cache" settings?<br>
</blockquote>
<br></div>
Looks to me that both 2nd time calls were reduced...in Essex as well as Grizzly zones.<br>
<br>
Best,<br>
-jay<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Thank you very much!<br>
<br>
<br>
The more information you give, the better folks can help you.<br>
<br>
Best,<br>
-jay<br>
<br>
<br>
On 07/25/2013 07:14 AM, Chu Duc Minh wrote:<br>
<br>
Check some more API (I run these command below from Controller<br>
node):<br>
# time quantum subnet-list<br>
(...have 4 subnet)<br>
real 0m0.676s<br>
user 0m0.196s<br>
sys 0m0.020s<br>
<br>
# time quantum router-list<br>
(...have 1 router)<br>
real 0m0.496s<br>
user 0m0.164s<br>
sys 0m0.052s<br>
<br>
# time nova list --all_tenants=1<br>
(...have 5 instances)<br>
real 0m1.290s<br>
user 0m0.308s<br>
sys 0m0.040s<br>
<br>
Compare with my old OpenStack deployment on weaker servers, it<br>
took 1/3<br>
in times.<br>
<br>
<br>
<br>
On Thu, Jul 25, 2013 at 5:43 PM, Peter Cheung<br>
<<a href="mailto:mcheung63@hotmail.com" target="_blank">mcheung63@hotmail.com</a> <mailto:<a href="mailto:mcheung63@hotmail.com" target="_blank">mcheung63@hotmail.com</a>><br></div></div>
<mailto:<a href="mailto:mcheung63@hotmail.com" target="_blank">mcheung63@hotmail.com</a> <mailto:<a href="mailto:mcheung63@hotmail.com" target="_blank">mcheung63@hotmail.com</a>><u></u>>__><div class="im">
<br>
wrote:<br>
<br>
I am having a problem about calling API speed is up and down,<br>
something need 0.1s, something it needs 3s<br>
<br>
Thanks<br>
from Peter<br>
<br>
<br>
<br></div>
------------------------------<u></u>__----------------------------<u></u>--__------------<div class="im"><br>
<br>
Date: Thu, 25 Jul 2013 17:41:11 +0700<br>
From: <a href="mailto:chu.ducminh@gmail.com" target="_blank">chu.ducminh@gmail.com</a> <mailto:<a href="mailto:chu.ducminh@gmail.com" target="_blank">chu.ducminh@gmail.com</a>><br></div>
<mailto:<a href="mailto:chu.ducminh@gmail.com" target="_blank">chu.ducminh@gmail.com</a> <mailto:<a href="mailto:chu.ducminh@gmail.com" target="_blank">chu.ducminh@gmail.com</a>><u></u>><br>
To: <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
<mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.<u></u>launchpad.net</a>><br>
<mailto:<a href="mailto:openstack@lists." target="_blank">openstack@lists.</a>__<a href="http://launchpad.net" target="_blank">laun<u></u>chpad.net</a><div class="im"><br>
<mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.<u></u>launchpad.net</a>>>;<br>
<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
<mailto:<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.<u></u>openstack.org</a>><br></div>
<mailto:<a href="mailto:openstack@lists." target="_blank">openstack@lists.</a>__<a href="http://openstack.org" target="_blank">open<u></u>stack.org</a><div class="im"><br>
<mailto:<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.<u></u>openstack.org</a>>><br>
<br>
Subject: [Openstack] Call to API very slow [Grizzly]<br>
<br>
<br>
All operations in my Openstack dashboard very slow (compare<br>
to my<br>
old Openstack deployment)<br>
Then i do some check on an instance:<br>
<br></div>
$ time curl <a href="http://169.254.169.254/__openstack" target="_blank">http://169.254.169.254/__<u></u>openstack</a><div class="im"><br>
<<a href="http://169.254.169.254/openstack" target="_blank">http://169.254.169.254/<u></u>openstack</a>><br>
2012-08-10<br>
2013-04-04<br>
latest<br>
real 0m*5.605s*<br>
<br>
user 0m0.004s<br>
sys 0m0.004s<br>
<br>
5 seconds for a simple API query !??<br>
<br>
<br></div>
in quantum-ns-metadata-proxyxxxx.<u></u>__log, i saw:<div class="im"><br>
2013-07-25 *17:17:09 * DEBUG<br>
<br></div>
[quantum.agent.metadata.__<u></u>namespace_proxy] Request: GET<div class="im"><br>
/openstack<br>
HTTP/1.0<br>
Accept: */*<br>
Content-Type: text/plain<br>
Host: 169.254.169.254<br>
User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0<br>
OpenSSL/1.0.1 zlib/<a href="http://1.2.3.4" target="_blank">1.2.3.4</a> <<a href="http://1.2.3.4" target="_blank">http://1.2.3.4</a>><br>
<<a href="http://1.2.3.4" target="_blank">http://1.2.3.4</a>> libidn/1.23 librtmp/2.3<br>
2013-07-25 *17:17:14* DEBUG<br>
<br></div>
[quantum.agent.metadata.__<u></u>namespace_proxy] {'date': 'Thu,<div class="im"><br>
25 Jul 2013<br>
10:17:14 GMT', 'status': '200', 'content-length': '28',<br>
'content-type': 'text/html; charset=UTF-8', 'content-location':<br></div>
u'<a href="http://169.254.169.254/__openstack" target="_blank">http://169.254.169.254/__<u></u>openstack</a><div class="im"><br>
<<a href="http://169.254.169.254/openstack" target="_blank">http://169.254.169.254/<u></u>openstack</a>>'}<br>
2013-07-25 17:17:14 DEBUG<br></div>
[quantum.agent.metadata.__<u></u>namespace_proxy] 2012-08-10<div class="im"><br>
2013-04-04<br>
latest<br>
<br>
I take a look at metadata-agent.log, and saw almost<br>
request/response<br>
finished @*17:17:09<br>
*<br>
But the last finished *@**17:17:14<br>
*2013-07-25 *17:17:14* DEBUG [quantum.agent.metadata.agent]<br>
<br>
{'date': 'Thu, 25 Jul 2013 10:17:14 GMT', 'status': '200',<br>
'content-length': '28', 'content-type': 'text/html;<br>
charset=UTF-8',<br></div>
'content-location': u'<a href="http://172.30.1.14:8775/__openstack" target="_blank">http://172.30.1.14:8775/__<u></u>openstack</a><div class="im"><br>
<<a href="http://172.30.1.14:8775/openstack" target="_blank">http://172.30.1.14:8775/<u></u>openstack</a>>'}<br>
*<br>
*<br>
<br>
I enabled slow query log on MySql, but can't find any slow<br>
query.<br>
<br>
Do you know possible problems in this situation?<br>
Thank you very much!<br>
<br>
<br></div>
______________________________<u></u>___________________ Mailing list:<br>
<a href="https://launchpad.net/~__openstack" target="_blank">https://launchpad.net/~__<u></u>openstack</a><div class="im"><br>
<<a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a>> Post to :<br>
<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
<mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.<u></u>launchpad.net</a>><br></div>
<mailto:<a href="mailto:openstack@lists." target="_blank">openstack@lists.</a>__<a href="http://launchpad.net" target="_blank">laun<u></u>chpad.net</a><br>
<mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.<u></u>launchpad.net</a>>><br>
<br>
Unsubscribe : <a href="https://launchpad.net/~__openstack" target="_blank">https://launchpad.net/~__<u></u>openstack</a><br>
<<a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a>> More help :<br>
<a href="https://help.launchpad.net/__ListHelp" target="_blank">https://help.launchpad.net/__<u></u>ListHelp</a><br>
<<a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/<u></u>ListHelp</a>><br>
<br>
<br>
<br>
<br>
______________________________<u></u>___________________<br>
Mailing list: <a href="https://launchpad.net/~__openstack" target="_blank">https://launchpad.net/~__<u></u>openstack</a><div class="im"><br>
<<a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a>><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
<mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.<u></u>launchpad.net</a>><br></div>
Unsubscribe : <a href="https://launchpad.net/~__openstack" target="_blank">https://launchpad.net/~__<u></u>openstack</a><br>
<<a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a>><br>
More help : <a href="https://help.launchpad.net/__ListHelp" target="_blank">https://help.launchpad.net/__<u></u>ListHelp</a><br>
<<a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/<u></u>ListHelp</a>><br>
<br>
<br>
<br>
______________________________<u></u>___________________<br>
Mailing list: <a href="https://launchpad.net/~__openstack" target="_blank">https://launchpad.net/~__<u></u>openstack</a><div class="im"><br>
<<a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a>><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
<mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.<u></u>launchpad.net</a>><br></div>
Unsubscribe : <a href="https://launchpad.net/~__openstack" target="_blank">https://launchpad.net/~__<u></u>openstack</a><br>
<<a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a>><br>
More help : <a href="https://help.launchpad.net/__ListHelp" target="_blank">https://help.launchpad.net/__<u></u>ListHelp</a><br>
<<a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/<u></u>ListHelp</a>><br>
<br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Mailing list: %(list_owner)s<br>
Post to : %(list_post)s<br>
Unsubscribe : %(list_unsubscribe)s<br>
More help : %(list_help)s<br>
<br>
</blockquote>
<br>
<br>
______________________________<u></u>_________________<br>
Mailing list: %(list_owner)s<br>
Post to : %(list_post)s<br>
Unsubscribe : %(list_unsubscribe)s<br>
More help : %(list_help)s<br>
<br>
</blockquote></div><br></div></div>