<div dir="ltr"><div>This is slower now because of the quantum metadata agent. In Essex when a vm did a request for 169.254.169.254 the request would go straight to the nova-metadata server and nova would determine which metadata to return based on the instances source ip. With quantum you are allowed to have overlapping ip addresses so this method does not work anymore. In order for metadata to work with overlapping ip addresses the quantum metadata agent intercepts the request and inserts a special header so that nova-metadata can determine which instance this request belongs to (since ips can overlap). In order for the quantum metadata agent to determine which header values to insert he needs to query quantum to determine the device_id of the instance that made the request (which is an extra query). My guess is this coupled with proxying the data is is why you are seeing an extra second or so. Is this presenting a problem for you? If you don't want to allow overlapping ips you can disable that in quantum and still use nova-metadata directly.<br>
<br></div>Thanks, <br><br>Aaron<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 25, 2013 at 8:19 PM, Chu Duc Minh <span dir="ltr"><<a href="mailto:chu.ducminh@gmail.com" target="_blank">chu.ducminh@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><div class="gmail_quote"><div class="im">On Thu, Jul 25, 2013 at 7:30 PM, Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">You will need to provide more details than "old" vs. "new" OpenStack.<br>
<br>
For example...<br>
<br>
1) What is your network model in the old vs. new<br></blockquote></div><div>Old: nova-network + FlatDHCP <br></div><div>New: Quantum + OpenvSwitch + network-namespace + metadata-ns-proxy<br><br></div><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
2) What version of OpenStack is the old<br></blockquote></div><div>Old: Essex<br></div><div>New: Grizzly <br><br></div><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
3) Is Keystone used in old and new? If so, what drivers are used in Keystone?<br></blockquote></div><div>Yes, both use Keystone with SQL backend<br><br></div><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
4) Do you have errors in any of your log files (usually an indication that something like a timeout or failure on </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
RPC which may affect performance)<br></blockquote></div><div>No, i didn't found it.<br> <br></div><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
5) Are you using nova-conductor in the new?<br></blockquote></div><div>I have nova-conductor process run on Controller node, but seem nova-compute don't use this.<br> <br></div><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
6) What database backend are you using?<br></blockquote></div><div>MySQL.<br> <br></div><div class="im"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
7) Do a time keystone user-list on both old and new<br></blockquote></div><div># keystone user-list<br></div><div>Old: <br></div><div>Run 1st time:<br>real <span style="color:rgb(255,0,0)"><b>0m2.494s</b></span><br>user 0m0.112s<br>
sys 0m0.040s<br></div><div>2nd time:<br>real <span style="color:rgb(0,0,255)"> <b>0m0.279s</b></span><br>user 0m0.112s<br>sys 0m0.020s<br><br></div><div><br></div><div>New:<br>Run 1st time:<br>real <b><span style="color:rgb(255,0,0)">0m1.687s</span></b><br>
user 0m0.176s<br>sys 0m0.012s<br></div><div>2nd time:<br>real <b><span style="color:rgb(0,0,255)">0m0.213s</span></b><br>user 0m0.160s<br>sys 0m0.040s<br></div><div class="im"><div><br></div><div><br></div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
8) Pastebin your conf files, with passwords removed<br></blockquote></div><div>My config files are quite large.<br></div><div>I can sum up that all services & API are HA-ed using HAproxy, RabbitMQ Cluster, Database replication, v.v...<br>
<br></div>This morning, i just do some tuning memcached for Dashboard service (HA-ed using HAProxy), then my Dashboard become faster significantly. <br></div><div class="gmail_quote"><br></div><div class="gmail_quote"><div>
The questions remain are:<br></div><div>1. Why metadata query so slow? Possible problems? ( "curl <a href="http://169.254.169.254/openstack" target="_blank">http://169.254.169.254/openstack</a>" took 2.5-5 seconds to complete, "ec2metadata" took 14-17 seconds to complete - i tested many times.) <br>
</div><div>2. Why API calls using *-client reduce time from 2nd time on my old Essex deployment, but not on my new Grizzly deployment? Maybe I need some "cache" settings?<br><br></div><div>Thank you very much!<br>
</div><div><div class="h5"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
The more information you give, the better folks can help you.<br>
<br>
Best,<br>
-jay<div><br>
<br>
On 07/25/2013 07:14 AM, Chu Duc Minh wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
Check some more API (I run these command below from Controller node):<br>
# time quantum subnet-list<br>
(...have 4 subnet)<br>
real 0m0.676s<br>
user 0m0.196s<br>
sys 0m0.020s<br>
<br>
# time quantum router-list<br>
(...have 1 router)<br>
real 0m0.496s<br>
user 0m0.164s<br>
sys 0m0.052s<br>
<br>
# time nova list --all_tenants=1<br>
(...have 5 instances)<br>
real 0m1.290s<br>
user 0m0.308s<br>
sys 0m0.040s<br>
<br>
Compare with my old OpenStack deployment on weaker servers, it took 1/3<br>
in times.<br>
<br>
<br>
<br>
On Thu, Jul 25, 2013 at 5:43 PM, Peter Cheung <<a href="mailto:mcheung63@hotmail.com" target="_blank">mcheung63@hotmail.com</a><br></div><div>
<mailto:<a href="mailto:mcheung63@hotmail.com" target="_blank">mcheung63@hotmail.com</a>><u></u>> wrote:<br>
<br>
I am having a problem about calling API speed is up and down,<br>
something need 0.1s, something it needs 3s<br>
<br>
Thanks<br>
from Peter<br>
<br>
<br></div>
------------------------------<u></u>------------------------------<u></u>------------<div><br>
Date: Thu, 25 Jul 2013 17:41:11 +0700<br></div>
From: <a href="mailto:chu.ducminh@gmail.com" target="_blank">chu.ducminh@gmail.com</a> <mailto:<a href="mailto:chu.ducminh@gmail.com" target="_blank">chu.ducminh@gmail.com</a>><br>
To: <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
<mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.<u></u>launchpad.net</a>>;<br>
<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a> <mailto:<a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.<u></u>openstack.org</a>><div>
<br>
Subject: [Openstack] Call to API very slow [Grizzly]<br>
<br>
<br>
All operations in my Openstack dashboard very slow (compare to my<br>
old Openstack deployment)<br>
Then i do some check on an instance:<br>
<br>
$ time curl <a href="http://169.254.169.254/openstack" target="_blank">http://169.254.169.254/<u></u>openstack</a><br>
2012-08-10<br>
2013-04-04<br>
latest<br></div>
real 0m*5.605s*<div><br>
user 0m0.004s<br>
sys 0m0.004s<br>
<br>
5 seconds for a simple API query !??<br>
<br>
<br>
in quantum-ns-metadata-proxyxxxx.<u></u>log, i saw:<br></div>
2013-07-25 *17:17:09 * DEBUG<div><br>
[quantum.agent.metadata.<u></u>namespace_proxy] Request: GET /openstack<br>
HTTP/1.0<br>
Accept: */*<br>
Content-Type: text/plain<br>
Host: 169.254.169.254<br>
User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0<br></div>
OpenSSL/1.0.1 zlib/<a href="http://1.2.3.4" target="_blank">1.2.3.4</a> <<a href="http://1.2.3.4" target="_blank">http://1.2.3.4</a>> libidn/1.23 librtmp/2.3<br>
2013-07-25 *17:17:14* DEBUG<div><br>
[quantum.agent.metadata.<u></u>namespace_proxy] {'date': 'Thu, 25 Jul 2013<br>
10:17:14 GMT', 'status': '200', 'content-length': '28',<br>
'content-type': 'text/html; charset=UTF-8', 'content-location':<br>
u'<a href="http://169.254.169.254/openstack" target="_blank">http://169.254.169.254/<u></u>openstack</a>'}<br>
2013-07-25 17:17:14 DEBUG<br>
[quantum.agent.metadata.<u></u>namespace_proxy] 2012-08-10<br>
2013-04-04<br>
latest<br>
<br>
I take a look at metadata-agent.log, and saw almost request/response<br></div>
finished @*17:17:09<br>
*<br>
But the last finished *@**17:17:14<br>
*2013-07-25 *17:17:14* DEBUG [quantum.agent.metadata.agent]<div><br>
{'date': 'Thu, 25 Jul 2013 10:17:14 GMT', 'status': '200',<br>
'content-length': '28', 'content-type': 'text/html; charset=UTF-8',<br>
'content-location': u'<a href="http://172.30.1.14:8775/openstack" target="_blank">http://172.30.1.14:8775/<u></u>openstack</a>'}<br></div>
*<br>
*<div><br>
I enabled slow query log on MySql, but can't find any slow query.<br>
<br>
Do you know possible problems in this situation?<br>
Thank you very much!<br>
<br>
<br>
______________________________<u></u>_________________ Mailing list:<br>
<a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a> Post to :<br></div>
<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a> <mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.<u></u>launchpad.net</a>><div>
<br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a> More help :<br>
<a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/<u></u>ListHelp</a><br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/<u></u>ListHelp</a><br>
<br>
</div></blockquote><div><div>
<br>
<br>
______________________________<u></u>_________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/<u></u>ListHelp</a><br>
</div></div></blockquote></div></div></div><br></div></div>
<br>_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br></blockquote></div><br></div>