[Openstack] Fwd: Call to API very slow [Grizzly]
Dan Reif
launchpad-mailinglist-20130227 at angelfaq.com
Fri Jul 26 17:40:49 UTC 2013
If for whatever reason you don't want to use the memcache token driver,
I've seen an order-of-magnitude increase in performance simply from adding
an index to the "expires" column in the token table. I opened a blueprint (
https://blueprints.launchpad.net/keystone/+spec/index-token-expiry) to get
this in mainline, but until that lands in the version you're using, you may
benefit from adding the index manually.
On Fri, Jul 26, 2013 at 12:03 PM, Jay Pipes <jaypipes at gmail.com> wrote:
> On 07/25/2013 11:23 PM, Chu Duc Minh wrote:
>
>> On Thu, Jul 25, 2013 at 7:30 PM, Jay Pipes <jaypipes at gmail.com
>> <mailto:jaypipes at gmail.com>> wrote:
>>
>> You will need to provide more details than "old" vs. "new" OpenStack.
>>
>> For example...
>>
>> 1) What is your network model in the old vs. new
>>
>> Old: nova-network + FlatDHCP
>> New: Quantum + OpenvSwitch + network-namespace + metadata-ns-proxy
>>
>
> OK. If you do a time ping -c1 <IP> from a VM to the metadata IP, what kind
> of times do you see in each setup?
>
>
> 2) What version of OpenStack is the old
>>
>> Old: Essex
>> New: Grizzly
>>
>
> OK.
>
>
> 3) Is Keystone used in old and new? If so, what drivers are used in
>> Keystone?
>>
>> Yes, both use Keystone with SQL backend
>>
>
> I recommend switching to the memcache Token driver in Keystone.
>
>
> 4) Do you have errors in any of your log files (usually an
>> indication that something like a timeout or failure on
>>
>> RPC which may affect performance)
>>
>> No, i didn't found it.
>>
>
> OK.
>
>
> 5) Are you using nova-conductor in the new?
>>
>> I have nova-conductor process run on Controller node, but seem
>> nova-compute don't use this.
>>
>
> OK, good to know, but unlikely to be the cause of a metadata API query
> slowdown.
>
>
> 6) What database backend are you using?
>>
>> MySQL.
>>
>
> OK.
>
> 7) Do a time keystone user-list on both old and new
>>
>> # keystone user-list
>> Old:
>> Run 1st time:
>> real *0m2.494s*
>>
>> user 0m0.112s
>> sys 0m0.040s
>> 2nd time:
>> real *0m0.279s*
>>
>> user 0m0.112s
>> sys 0m0.020s
>>
>>
>> New:
>> Run 1st time:
>> real *0m1.687s*
>>
>> user 0m0.176s
>> sys 0m0.012s
>> 2nd time:
>> real *0m0.213s*
>> user 0m0.160s
>> sys 0m0.040s
>>
>
> Interesting, so Keystone is actually faster in Grizzly than in Essex... or
> at least it seems to be given the above rudimentary timing.
>
>
> 8) Pastebin your conf files, with passwords removed
>>
>> My config files are quite large.
>> I can sum up that all services & API are HA-ed using HAproxy, RabbitMQ
>> Cluster, Database replication, v.v...
>>
>
> OK.
>
>
> This morning, i just do some tuning memcached for Dashboard service
>> (HA-ed using HAProxy), then my Dashboard become faster significantly.
>>
>> The questions remain are:
>> 1. Why metadata query so slow? Possible problems? ( "curl
>> http://169.254.169.254/**openstack <http://169.254.169.254/openstack>"
>> took 2.5-5 seconds to complete,
>> "ec2metadata" took 14-17 seconds to complete - i tested many times.)
>>
>
> Could be a number of things. I thought Vish had redesigned the metadata
> API endpoint in Folsom and removed the problematic slow queries that used
> to be in there. If you check your MySQL slow log (hopefully you have it
> enabled), look to see if any of the queries in there reference the
> instance_metadata table.
>
>
> 2. Why API calls using *-client reduce time from 2nd time on my old
>> Essex deployment, but not on my new Grizzly deployment? Maybe I need
>> some "cache" settings?
>>
>
> Looks to me that both 2nd time calls were reduced...in Essex as well as
> Grizzly zones.
>
> Best,
> -jay
>
> Thank you very much!
>>
>>
>> The more information you give, the better folks can help you.
>>
>> Best,
>> -jay
>>
>>
>> On 07/25/2013 07:14 AM, Chu Duc Minh wrote:
>>
>> Check some more API (I run these command below from Controller
>> node):
>> # time quantum subnet-list
>> (...have 4 subnet)
>> real 0m0.676s
>> user 0m0.196s
>> sys 0m0.020s
>>
>> # time quantum router-list
>> (...have 1 router)
>> real 0m0.496s
>> user 0m0.164s
>> sys 0m0.052s
>>
>> # time nova list --all_tenants=1
>> (...have 5 instances)
>> real 0m1.290s
>> user 0m0.308s
>> sys 0m0.040s
>>
>> Compare with my old OpenStack deployment on weaker servers, it
>> took 1/3
>> in times.
>>
>>
>>
>> On Thu, Jul 25, 2013 at 5:43 PM, Peter Cheung
>> <mcheung63 at hotmail.com <mailto:mcheung63 at hotmail.com>
>> <mailto:mcheung63 at hotmail.com <mailto:mcheung63 at hotmail.com>**
>> >__>
>>
>> wrote:
>>
>> I am having a problem about calling API speed is up and down,
>> something need 0.1s, something it needs 3s
>>
>> Thanks
>> from Peter
>>
>>
>>
>> ------------------------------**__----------------------------**
>> --__------------
>>
>>
>> Date: Thu, 25 Jul 2013 17:41:11 +0700
>> From: chu.ducminh at gmail.com <mailto:chu.ducminh at gmail.com>
>> <mailto:chu.ducminh at gmail.com <mailto:chu.ducminh at gmail.com>**>
>> To: openstack at lists.launchpad.net
>> <mailto:openstack at lists.**launchpad.net<openstack at lists.launchpad.net>
>> >
>> <mailto:openstack at lists.__laun**chpad.net<http://launchpad.net>
>>
>> <mailto:openstack at lists.**launchpad.net<openstack at lists.launchpad.net>
>> >>;
>> openstack at lists.openstack.org
>> <mailto:openstack at lists.**openstack.org<openstack at lists.openstack.org>
>> >
>> <mailto:openstack at lists.__open**stack.org <http://openstack.org>
>>
>> <mailto:openstack at lists.**openstack.org<openstack at lists.openstack.org>
>> >>
>>
>> Subject: [Openstack] Call to API very slow [Grizzly]
>>
>>
>> All operations in my Openstack dashboard very slow (compare
>> to my
>> old Openstack deployment)
>> Then i do some check on an instance:
>>
>> $ time curl http://169.254.169.254/__**openstack<http://169.254.169.254/__openstack>
>>
>> <http://169.254.169.254/**openstack<http://169.254.169.254/openstack>
>> >
>> 2012-08-10
>> 2013-04-04
>> latest
>> real 0m*5.605s*
>>
>> user 0m0.004s
>> sys 0m0.004s
>>
>> 5 seconds for a simple API query !??
>>
>>
>> in quantum-ns-metadata-proxyxxxx.**__log, i saw:
>>
>> 2013-07-25 *17:17:09 * DEBUG
>>
>> [quantum.agent.metadata.__**namespace_proxy] Request: GET
>>
>> /openstack
>> HTTP/1.0
>> Accept: */*
>> Content-Type: text/plain
>> Host: 169.254.169.254
>> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0
>> OpenSSL/1.0.1 zlib/1.2.3.4 <http://1.2.3.4>
>> <http://1.2.3.4> libidn/1.23 librtmp/2.3
>> 2013-07-25 *17:17:14* DEBUG
>>
>> [quantum.agent.metadata.__**namespace_proxy] {'date': 'Thu,
>>
>> 25 Jul 2013
>> 10:17:14 GMT', 'status': '200', 'content-length': '28',
>> 'content-type': 'text/html; charset=UTF-8',
>> 'content-location':
>> u'http://169.254.169.254/__**openstack<http://169.254.169.254/__openstack>
>>
>> <http://169.254.169.254/**openstack<http://169.254.169.254/openstack>
>> >'}
>> 2013-07-25 17:17:14 DEBUG
>> [quantum.agent.metadata.__**namespace_proxy] 2012-08-10
>>
>> 2013-04-04
>> latest
>>
>> I take a look at metadata-agent.log, and saw almost
>> request/response
>> finished @*17:17:09
>> *
>> But the last finished *@**17:17:14
>> *2013-07-25 *17:17:14* DEBUG
>> [quantum.agent.metadata.agent]
>>
>> {'date': 'Thu, 25 Jul 2013 10:17:14 GMT', 'status': '200',
>> 'content-length': '28', 'content-type': 'text/html;
>> charset=UTF-8',
>> 'content-location': u'http://172.30.1.14:8775/__**openstack<http://172.30.1.14:8775/__openstack>
>>
>> <http://172.30.1.14:8775/**openstack<http://172.30.1.14:8775/openstack>
>> >'}
>> *
>> *
>>
>> I enabled slow query log on MySql, but can't find any slow
>> query.
>>
>> Do you know possible problems in this situation?
>> Thank you very much!
>>
>>
>> ______________________________**___________________ Mailing
>> list:
>> https://launchpad.net/~__**openstack<https://launchpad.net/~__openstack>
>>
>> <https://launchpad.net/~**openstack<https://launchpad.net/~openstack>>
>> Post to :
>> openstack at lists.launchpad.net
>> <mailto:openstack at lists.**launchpad.net<openstack at lists.launchpad.net>
>> >
>> <mailto:openstack at lists.__laun**chpad.net <http://launchpad.net>
>> <mailto:openstack at lists.**launchpad.net<openstack at lists.launchpad.net>
>> >>
>>
>> Unsubscribe : https://launchpad.net/~__**openstack<https://launchpad.net/~__openstack>
>> <https://launchpad.net/~**openstack<https://launchpad.net/~openstack>>
>> More help :
>> https://help.launchpad.net/__**ListHelp<https://help.launchpad.net/__ListHelp>
>> <https://help.launchpad.net/**ListHelp<https://help.launchpad.net/ListHelp>
>> >
>>
>>
>>
>>
>> ______________________________**___________________
>> Mailing list: https://launchpad.net/~__**openstack<https://launchpad.net/~__openstack>
>>
>> <https://launchpad.net/~**openstack<https://launchpad.net/~openstack>
>> >
>> Post to : openstack at lists.launchpad.net
>> <mailto:openstack at lists.**launchpad.net<openstack at lists.launchpad.net>
>> >
>> Unsubscribe : https://launchpad.net/~__**openstack<https://launchpad.net/~__openstack>
>> <https://launchpad.net/~**openstack<https://launchpad.net/~openstack>
>> >
>> More help : https://help.launchpad.net/__**ListHelp<https://help.launchpad.net/__ListHelp>
>> <https://help.launchpad.net/**ListHelp<https://help.launchpad.net/ListHelp>
>> >
>>
>>
>>
>> ______________________________**___________________
>> Mailing list: https://launchpad.net/~__**openstack<https://launchpad.net/~__openstack>
>>
>> <https://launchpad.net/~**openstack<https://launchpad.net/~openstack>
>> >
>> Post to : openstack at lists.launchpad.net
>> <mailto:openstack at lists.**launchpad.net<openstack at lists.launchpad.net>
>> >
>> Unsubscribe : https://launchpad.net/~__**openstack<https://launchpad.net/~__openstack>
>> <https://launchpad.net/~**openstack<https://launchpad.net/~openstack>
>> >
>> More help : https://help.launchpad.net/__**ListHelp<https://help.launchpad.net/__ListHelp>
>> <https://help.launchpad.net/**ListHelp<https://help.launchpad.net/ListHelp>
>> >
>>
>>
>>
>>
>>
>> ______________________________**_________________
>> Mailing list: %(list_owner)s
>> Post to : %(list_post)s
>> Unsubscribe : %(list_unsubscribe)s
>> More help : %(list_help)s
>>
>>
>
> ______________________________**_________________
> Mailing list: %(list_owner)s
> Post to : %(list_post)s
> Unsubscribe : %(list_unsubscribe)s
> More help : %(list_help)s
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130726/62fb1853/attachment.html>
More information about the Openstack
mailing list