Open Stack

Dear Openstack experts,
Recently, I upgraded Openstack from Grizzy to Havana(2013.2-2, from RDO
repository).
Everything is running pretty well excepting ec2api.
I noticed that ec2api will suffer heavy load and respond slowly with 
some particular ec2 requests(RunInstances), for instance:


2014-01-10 09:11:45.072 129745 INFO nova.ec2.wsgi.server
[req-84d16d16-3808-426b-b7af-3b90a11b83b0
0c6e7dba03c24c6a9bce299747499e8a 7052bd6714e7460caeb16242e68124f9]
117.103.103.29 "GET
/services/Cloud?AWSAccessKeyId=<something>&Action=RunInstances&ClientToken=<something>&ImageId=ami-00000001&InstanceInitiatedShutdownBehavior=terminate&InstanceType=m8.pilot&MaxCount=1&MinCount=1&SecurityGroup.2=gridWN&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2014-01-10T09%3A09%3A26Z&UserData=<something>&Version=2014-10-01&Signature=Nxs8cRDvcyHikmyOKwbut0U6DTSfBxZpAGbwLjymQBs%3D 

HTTP/1.1" status: 200 len: 1109 time: 138.5970151



It took over 2 minutes to process this request, but in my another 
co-existing Grizzy instance(the same hardware and system configuration), 
it doesn't have such problem.

2014-01-08 11:15:15.704 INFO nova.ec2.wsgi.server
[req-ccac9790-3357-4aa8-84bd-cdaab1aa394e
ebbd729575cb404081a45c9ada0849b7 8175953c209044358ab5e0ec19d52c37]
117.103.103.29 "GET
/services/Cloud?AWSAccessKeyId=<something>&Action=RunInstances&ClientToken=<something>&ImageId=ami-00000007&InstanceInitiatedShutdownBehavior=terminate&InstanceType=m1.large&MaxCount=1&MinCount=1&SecurityGroup.2=amsWN&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2014-01-08T11%3A15%3A11Z&UserData=<something>%3D%3D&Version=2012-10-01&Signature=uQ%2Bipw9buWTflC83tI76%2B9%2F66Bj8I%2FF8ChUUfEmK2v4%3D 

HTTP/1.1" status: 200 len: 931 time: 3.9426181




Besides, the it seems to me that the ec2api will not release memory(or 
it doesn't manage that in good way..) properly while processing such 
requests, so, if it gets many requests, the memory usage will be growing 
pretty fast until all system memory is consumed and goes to swap, then 
entire system will be running like hell until I restart nova-api (I have 
48GB memory, and it can eat all of them in minutes). So, I am wondering 
what changes are made for ec2api in Havana and is there anyway to 
improve this if that is not a bug?

It would be very appreciated if having any suggestion/reply for this 
problem.

Best regards,
Felix Lee ~


-- 
Felix Lee                               Academia Sinica Grid & Cloud.
Tel: +886-2-27898308
Office: Room P111, Institute of Physics, 128 Academia Road, Section 2,
Nankang, Taipei 115, Taiwan

-- 
Felix Lee                               Academia Sinica Grid & Cloud.
Tel: +886-2-27898308
Office: Room P111, Institute of Physics, 128 Academia Road, Section 2, 
Nankang, Taipei 115, Taiwan