Open Stack

Dear all,
I managed to have some workaround for this.
Well, after I looked into code, I noticed there are two implementations 
which will make significant impact on my system:

        instances = 
self.compute_api.get_all(context,search_opts=search_opts,sort_dir='asc')





         sys_metas = self.compute_api.get_all_system_metadata(
             context, search_filts=[{'key': ['EC2_client_token']},
                                    {'value': [client_token]}])


Since my database has a lot of records, such as over 1M metadata, and 
over 300k instances(deleted + error.., yes, we have a lot of short term 
VMs...  ), so, each search takes ages, after I archived those old 
records and delete them, now, it's better.


Best regards,
Felix Lee ~


On 2014年01月10日 11:21, Felix Lee wrote:
> Dear Openstack experts,
> Recently, I upgraded Openstack from Grizzy to Havana(2013.2-2, from RDO
> repository).
> Everything is running pretty well excepting ec2api.
> I noticed that ec2api will suffer heavy load and respond slowly with
> some particular ec2 requests(RunInstances), for instance:
>
>
> 2014-01-10 09:11:45.072 129745 INFO nova.ec2.wsgi.server
> [req-84d16d16-3808-426b-b7af-3b90a11b83b0
> 0c6e7dba03c24c6a9bce299747499e8a 7052bd6714e7460caeb16242e68124f9]
> 117.103.103.29 "GET
> /services/Cloud?AWSAccessKeyId=<something>&Action=RunInstances&ClientToken=<something>&ImageId=ami-00000001&InstanceInitiatedShutdownBehavior=terminate&InstanceType=m8.pilot&MaxCount=1&MinCount=1&SecurityGroup.2=gridWN&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2014-01-10T09%3A09%3A26Z&UserData=<something>&Version=2014-10-01&Signature=Nxs8cRDvcyHikmyOKwbut0U6DTSfBxZpAGbwLjymQBs%3D
>
> HTTP/1.1" status: 200 len: 1109 time: 138.5970151
>
>
>
> It took over 2 minutes to process this request, but in my another
> co-existing Grizzy instance(the same hardware and system configuration),
> it doesn't have such problem.
>
> 2014-01-08 11:15:15.704 INFO nova.ec2.wsgi.server
> [req-ccac9790-3357-4aa8-84bd-cdaab1aa394e
> ebbd729575cb404081a45c9ada0849b7 8175953c209044358ab5e0ec19d52c37]
> 117.103.103.29 "GET
> /services/Cloud?AWSAccessKeyId=<something>&Action=RunInstances&ClientToken=<something>&ImageId=ami-00000007&InstanceInitiatedShutdownBehavior=terminate&InstanceType=m1.large&MaxCount=1&MinCount=1&SecurityGroup.2=amsWN&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2014-01-08T11%3A15%3A11Z&UserData=<something>%3D%3D&Version=2012-10-01&Signature=uQ%2Bipw9buWTflC83tI76%2B9%2F66Bj8I%2FF8ChUUfEmK2v4%3D
>
> HTTP/1.1" status: 200 len: 931 time: 3.9426181
>
>
>
>
> Besides, the it seems to me that the ec2api will not release memory(or
> it doesn't manage that in good way..) properly while processing such
> requests, so, if it gets many requests, the memory usage will be growing
> pretty fast until all system memory is consumed and goes to swap, then
> entire system will be running like hell until I restart nova-api (I have
> 48GB memory, and it can eat all of them in minutes). So, I am wondering
> what changes are made for ec2api in Havana and is there anyway to
> improve this if that is not a bug?
>
> It would be very appreciated if having any suggestion/reply for this
> problem.
>
> Best regards,
> Felix Lee ~
>
>