[Openstack] Fwd: Call to API very slow [Grizzly]
Jay Pipes
jaypipes at gmail.com
Fri Jul 26 16:03:16 UTC 2013
On 07/25/2013 11:23 PM, Chu Duc Minh wrote:
> On Thu, Jul 25, 2013 at 7:30 PM, Jay Pipes <jaypipes at gmail.com
> <mailto:jaypipes at gmail.com>> wrote:
>
> You will need to provide more details than "old" vs. "new" OpenStack.
>
> For example...
>
> 1) What is your network model in the old vs. new
>
> Old: nova-network + FlatDHCP
> New: Quantum + OpenvSwitch + network-namespace + metadata-ns-proxy
OK. If you do a time ping -c1 <IP> from a VM to the metadata IP, what
kind of times do you see in each setup?
> 2) What version of OpenStack is the old
>
> Old: Essex
> New: Grizzly
OK.
> 3) Is Keystone used in old and new? If so, what drivers are used in
> Keystone?
>
> Yes, both use Keystone with SQL backend
I recommend switching to the memcache Token driver in Keystone.
> 4) Do you have errors in any of your log files (usually an
> indication that something like a timeout or failure on
>
> RPC which may affect performance)
>
> No, i didn't found it.
OK.
> 5) Are you using nova-conductor in the new?
>
> I have nova-conductor process run on Controller node, but seem
> nova-compute don't use this.
OK, good to know, but unlikely to be the cause of a metadata API query
slowdown.
> 6) What database backend are you using?
>
> MySQL.
OK.
> 7) Do a time keystone user-list on both old and new
>
> # keystone user-list
> Old:
> Run 1st time:
> real *0m2.494s*
> user 0m0.112s
> sys 0m0.040s
> 2nd time:
> real *0m0.279s*
> user 0m0.112s
> sys 0m0.020s
>
>
> New:
> Run 1st time:
> real *0m1.687s*
> user 0m0.176s
> sys 0m0.012s
> 2nd time:
> real *0m0.213s*
> user 0m0.160s
> sys 0m0.040s
Interesting, so Keystone is actually faster in Grizzly than in Essex...
or at least it seems to be given the above rudimentary timing.
> 8) Pastebin your conf files, with passwords removed
>
> My config files are quite large.
> I can sum up that all services & API are HA-ed using HAproxy, RabbitMQ
> Cluster, Database replication, v.v...
OK.
> This morning, i just do some tuning memcached for Dashboard service
> (HA-ed using HAProxy), then my Dashboard become faster significantly.
>
> The questions remain are:
> 1. Why metadata query so slow? Possible problems? ( "curl
> http://169.254.169.254/openstack" took 2.5-5 seconds to complete,
> "ec2metadata" took 14-17 seconds to complete - i tested many times.)
Could be a number of things. I thought Vish had redesigned the metadata
API endpoint in Folsom and removed the problematic slow queries that
used to be in there. If you check your MySQL slow log (hopefully you
have it enabled), look to see if any of the queries in there reference
the instance_metadata table.
> 2. Why API calls using *-client reduce time from 2nd time on my old
> Essex deployment, but not on my new Grizzly deployment? Maybe I need
> some "cache" settings?
Looks to me that both 2nd time calls were reduced...in Essex as well as
Grizzly zones.
Best,
-jay
> Thank you very much!
>
>
> The more information you give, the better folks can help you.
>
> Best,
> -jay
>
>
> On 07/25/2013 07:14 AM, Chu Duc Minh wrote:
>
> Check some more API (I run these command below from Controller
> node):
> # time quantum subnet-list
> (...have 4 subnet)
> real 0m0.676s
> user 0m0.196s
> sys 0m0.020s
>
> # time quantum router-list
> (...have 1 router)
> real 0m0.496s
> user 0m0.164s
> sys 0m0.052s
>
> # time nova list --all_tenants=1
> (...have 5 instances)
> real 0m1.290s
> user 0m0.308s
> sys 0m0.040s
>
> Compare with my old OpenStack deployment on weaker servers, it
> took 1/3
> in times.
>
>
>
> On Thu, Jul 25, 2013 at 5:43 PM, Peter Cheung
> <mcheung63 at hotmail.com <mailto:mcheung63 at hotmail.com>
> <mailto:mcheung63 at hotmail.com <mailto:mcheung63 at hotmail.com>>__>
> wrote:
>
> I am having a problem about calling API speed is up and down,
> something need 0.1s, something it needs 3s
>
> Thanks
> from Peter
>
>
>
> ------------------------------__------------------------------__------------
>
> Date: Thu, 25 Jul 2013 17:41:11 +0700
> From: chu.ducminh at gmail.com <mailto:chu.ducminh at gmail.com>
> <mailto:chu.ducminh at gmail.com <mailto:chu.ducminh at gmail.com>>
> To: openstack at lists.launchpad.net
> <mailto:openstack at lists.launchpad.net>
> <mailto:openstack at lists.__launchpad.net
> <mailto:openstack at lists.launchpad.net>>;
> openstack at lists.openstack.org
> <mailto:openstack at lists.openstack.org>
> <mailto:openstack at lists.__openstack.org
> <mailto:openstack at lists.openstack.org>>
>
> Subject: [Openstack] Call to API very slow [Grizzly]
>
>
> All operations in my Openstack dashboard very slow (compare
> to my
> old Openstack deployment)
> Then i do some check on an instance:
>
> $ time curl http://169.254.169.254/__openstack
> <http://169.254.169.254/openstack>
> 2012-08-10
> 2013-04-04
> latest
> real 0m*5.605s*
>
> user 0m0.004s
> sys 0m0.004s
>
> 5 seconds for a simple API query !??
>
>
> in quantum-ns-metadata-proxyxxxx.__log, i saw:
> 2013-07-25 *17:17:09 * DEBUG
>
> [quantum.agent.metadata.__namespace_proxy] Request: GET
> /openstack
> HTTP/1.0
> Accept: */*
> Content-Type: text/plain
> Host: 169.254.169.254
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0
> OpenSSL/1.0.1 zlib/1.2.3.4 <http://1.2.3.4>
> <http://1.2.3.4> libidn/1.23 librtmp/2.3
> 2013-07-25 *17:17:14* DEBUG
>
> [quantum.agent.metadata.__namespace_proxy] {'date': 'Thu,
> 25 Jul 2013
> 10:17:14 GMT', 'status': '200', 'content-length': '28',
> 'content-type': 'text/html; charset=UTF-8', 'content-location':
> u'http://169.254.169.254/__openstack
> <http://169.254.169.254/openstack>'}
> 2013-07-25 17:17:14 DEBUG
> [quantum.agent.metadata.__namespace_proxy] 2012-08-10
> 2013-04-04
> latest
>
> I take a look at metadata-agent.log, and saw almost
> request/response
> finished @*17:17:09
> *
> But the last finished *@**17:17:14
> *2013-07-25 *17:17:14* DEBUG [quantum.agent.metadata.agent]
>
> {'date': 'Thu, 25 Jul 2013 10:17:14 GMT', 'status': '200',
> 'content-length': '28', 'content-type': 'text/html;
> charset=UTF-8',
> 'content-location': u'http://172.30.1.14:8775/__openstack
> <http://172.30.1.14:8775/openstack>'}
> *
> *
>
> I enabled slow query log on MySql, but can't find any slow
> query.
>
> Do you know possible problems in this situation?
> Thank you very much!
>
>
> _________________________________________________ Mailing list:
> https://launchpad.net/~__openstack
> <https://launchpad.net/~openstack> Post to :
> openstack at lists.launchpad.net
> <mailto:openstack at lists.launchpad.net>
> <mailto:openstack at lists.__launchpad.net
> <mailto:openstack at lists.launchpad.net>>
>
> Unsubscribe : https://launchpad.net/~__openstack
> <https://launchpad.net/~openstack> More help :
> https://help.launchpad.net/__ListHelp
> <https://help.launchpad.net/ListHelp>
>
>
>
>
> _________________________________________________
> Mailing list: https://launchpad.net/~__openstack
> <https://launchpad.net/~openstack>
> Post to : openstack at lists.launchpad.net
> <mailto:openstack at lists.launchpad.net>
> Unsubscribe : https://launchpad.net/~__openstack
> <https://launchpad.net/~openstack>
> More help : https://help.launchpad.net/__ListHelp
> <https://help.launchpad.net/ListHelp>
>
>
>
> _________________________________________________
> Mailing list: https://launchpad.net/~__openstack
> <https://launchpad.net/~openstack>
> Post to : openstack at lists.launchpad.net
> <mailto:openstack at lists.launchpad.net>
> Unsubscribe : https://launchpad.net/~__openstack
> <https://launchpad.net/~openstack>
> More help : https://help.launchpad.net/__ListHelp
> <https://help.launchpad.net/ListHelp>
>
>
>
>
>
> _______________________________________________
> Mailing list: %(list_owner)s
> Post to : %(list_post)s
> Unsubscribe : %(list_unsubscribe)s
> More help : %(list_help)s
>
More information about the Openstack
mailing list