[Openstack] Call to API very slow [Grizzly]
Aaron Rosen
arosen at nicira.com
Fri Jul 26 18:09:44 UTC 2013
This is slower now because of the quantum metadata agent. In Essex when a
vm did a request for 169.254.169.254 the request would go straight to the
nova-metadata server and nova would determine which metadata to return
based on the instances source ip. With quantum you are allowed to have
overlapping ip addresses so this method does not work anymore. In order for
metadata to work with overlapping ip addresses the quantum metadata agent
intercepts the request and inserts a special header so that nova-metadata
can determine which instance this request belongs to (since ips can
overlap). In order for the quantum metadata agent to determine which
header values to insert he needs to query quantum to determine the
device_id of the instance that made the request (which is an extra query).
My guess is this coupled with proxying the data is is why you are seeing an
extra second or so. Is this presenting a problem for you? If you don't
want to allow overlapping ips you can disable that in quantum and still use
nova-metadata directly.
Thanks,
Aaron
On Thu, Jul 25, 2013 at 8:19 PM, Chu Duc Minh <chu.ducminh at gmail.com> wrote:
>
> On Thu, Jul 25, 2013 at 7:30 PM, Jay Pipes <jaypipes at gmail.com> wrote:
>
>> You will need to provide more details than "old" vs. "new" OpenStack.
>>
>> For example...
>>
>> 1) What is your network model in the old vs. new
>>
> Old: nova-network + FlatDHCP
> New: Quantum + OpenvSwitch + network-namespace + metadata-ns-proxy
>
> 2) What version of OpenStack is the old
>>
> Old: Essex
> New: Grizzly
>
> 3) Is Keystone used in old and new? If so, what drivers are used in
>> Keystone?
>>
> Yes, both use Keystone with SQL backend
>
> 4) Do you have errors in any of your log files (usually an indication that
>> something like a timeout or failure on
>
> RPC which may affect performance)
>>
> No, i didn't found it.
>
>
>> 5) Are you using nova-conductor in the new?
>>
> I have nova-conductor process run on Controller node, but seem
> nova-compute don't use this.
>
>
>> 6) What database backend are you using?
>>
> MySQL.
>
>
>> 7) Do a time keystone user-list on both old and new
>>
> # keystone user-list
> Old:
> Run 1st time:
> real *0m2.494s*
> user 0m0.112s
> sys 0m0.040s
> 2nd time:
> real *0m0.279s*
> user 0m0.112s
> sys 0m0.020s
>
>
> New:
> Run 1st time:
> real *0m1.687s*
> user 0m0.176s
> sys 0m0.012s
> 2nd time:
> real *0m0.213s*
> user 0m0.160s
> sys 0m0.040s
>
>
> 8) Pastebin your conf files, with passwords removed
>>
> My config files are quite large.
> I can sum up that all services & API are HA-ed using HAproxy, RabbitMQ
> Cluster, Database replication, v.v...
>
> This morning, i just do some tuning memcached for Dashboard service (HA-ed
> using HAProxy), then my Dashboard become faster significantly.
>
> The questions remain are:
> 1. Why metadata query so slow? Possible problems? ( "curl
> http://169.254.169.254/openstack" took 2.5-5 seconds to complete,
> "ec2metadata" took 14-17 seconds to complete - i tested many times.)
> 2. Why API calls using *-client reduce time from 2nd time on my old Essex
> deployment, but not on my new Grizzly deployment? Maybe I need some "cache"
> settings?
>
> Thank you very much!
>
>
>> The more information you give, the better folks can help you.
>>
>> Best,
>> -jay
>>
>>
>> On 07/25/2013 07:14 AM, Chu Duc Minh wrote:
>>
>>> Check some more API (I run these command below from Controller node):
>>> # time quantum subnet-list
>>> (...have 4 subnet)
>>> real 0m0.676s
>>> user 0m0.196s
>>> sys 0m0.020s
>>>
>>> # time quantum router-list
>>> (...have 1 router)
>>> real 0m0.496s
>>> user 0m0.164s
>>> sys 0m0.052s
>>>
>>> # time nova list --all_tenants=1
>>> (...have 5 instances)
>>> real 0m1.290s
>>> user 0m0.308s
>>> sys 0m0.040s
>>>
>>> Compare with my old OpenStack deployment on weaker servers, it took 1/3
>>> in times.
>>>
>>>
>>>
>>> On Thu, Jul 25, 2013 at 5:43 PM, Peter Cheung <mcheung63 at hotmail.com
>>> <mailto:mcheung63 at hotmail.com>**> wrote:
>>>
>>> I am having a problem about calling API speed is up and down,
>>> something need 0.1s, something it needs 3s
>>>
>>> Thanks
>>> from Peter
>>>
>>>
>>> ------------------------------**------------------------------**
>>> ------------
>>>
>>> Date: Thu, 25 Jul 2013 17:41:11 +0700
>>> From: chu.ducminh at gmail.com <mailto:chu.ducminh at gmail.com>
>>> To: openstack at lists.launchpad.net
>>> <mailto:openstack at lists.**launchpad.net<openstack at lists.launchpad.net>
>>> >;
>>> openstack at lists.openstack.org <mailto:openstack at lists.**
>>> openstack.org <openstack at lists.openstack.org>>
>>>
>>> Subject: [Openstack] Call to API very slow [Grizzly]
>>>
>>>
>>> All operations in my Openstack dashboard very slow (compare to my
>>> old Openstack deployment)
>>> Then i do some check on an instance:
>>>
>>> $ time curl http://169.254.169.254/**openstack<http://169.254.169.254/openstack>
>>> 2012-08-10
>>> 2013-04-04
>>> latest
>>> real 0m*5.605s*
>>>
>>> user 0m0.004s
>>> sys 0m0.004s
>>>
>>> 5 seconds for a simple API query !??
>>>
>>>
>>> in quantum-ns-metadata-proxyxxxx.**log, i saw:
>>> 2013-07-25 *17:17:09 * DEBUG
>>>
>>> [quantum.agent.metadata.**namespace_proxy] Request: GET /openstack
>>> HTTP/1.0
>>> Accept: */*
>>> Content-Type: text/plain
>>> Host: 169.254.169.254
>>> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0
>>> OpenSSL/1.0.1 zlib/1.2.3.4 <http://1.2.3.4> libidn/1.23 librtmp/2.3
>>> 2013-07-25 *17:17:14* DEBUG
>>>
>>> [quantum.agent.metadata.**namespace_proxy] {'date': 'Thu, 25 Jul
>>> 2013
>>> 10:17:14 GMT', 'status': '200', 'content-length': '28',
>>> 'content-type': 'text/html; charset=UTF-8', 'content-location':
>>> u'http://169.254.169.254/**openstack<http://169.254.169.254/openstack>
>>> '}
>>> 2013-07-25 17:17:14 DEBUG
>>> [quantum.agent.metadata.**namespace_proxy] 2012-08-10
>>> 2013-04-04
>>> latest
>>>
>>> I take a look at metadata-agent.log, and saw almost request/response
>>> finished @*17:17:09
>>> *
>>> But the last finished *@**17:17:14
>>> *2013-07-25 *17:17:14* DEBUG [quantum.agent.metadata.agent]
>>>
>>> {'date': 'Thu, 25 Jul 2013 10:17:14 GMT', 'status': '200',
>>> 'content-length': '28', 'content-type': 'text/html; charset=UTF-8',
>>> 'content-location': u'http://172.30.1.14:8775/**openstack<http://172.30.1.14:8775/openstack>
>>> '}
>>> *
>>> *
>>>
>>> I enabled slow query log on MySql, but can't find any slow query.
>>>
>>> Do you know possible problems in this situation?
>>> Thank you very much!
>>>
>>>
>>> ______________________________**_________________ Mailing list:
>>> https://launchpad.net/~**openstack<https://launchpad.net/~openstack>Post to :
>>> openstack at lists.launchpad.net <mailto:openstack at lists.**
>>> launchpad.net <openstack at lists.launchpad.net>>
>>>
>>> Unsubscribe : https://launchpad.net/~**openstack<https://launchpad.net/~openstack>More help :
>>> https://help.launchpad.net/**ListHelp<https://help.launchpad.net/ListHelp>
>>>
>>>
>>>
>>>
>>> ______________________________**_________________
>>> Mailing list: https://launchpad.net/~**openstack<https://launchpad.net/~openstack>
>>> Post to : openstack at lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~**openstack<https://launchpad.net/~openstack>
>>> More help : https://help.launchpad.net/**ListHelp<https://help.launchpad.net/ListHelp>
>>>
>>>
>>
>> ______________________________**_________________
>> Mailing list: https://launchpad.net/~**openstack<https://launchpad.net/~openstack>
>> Post to : openstack at lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~**openstack<https://launchpad.net/~openstack>
>> More help : https://help.launchpad.net/**ListHelp<https://help.launchpad.net/ListHelp>
>>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help : https://help.launchpad.net/ListHelp
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130726/48e020a7/attachment.html>
More information about the Openstack
mailing list