Open Stack

From: Jiang, Yunhong [yunhong.jiang at intel.com]
Sent: Monday, November 26, 2012 4:35 AM
>>
>> Thanks for this Jiang ... I agree, data good. Several questions/observations:
>>
>> 1. I agree that there should be no architectural guarantees for periodic_task.
>> But it should be "within reason". It looks like about a 1 minute periodic task
>
>Thanks for reply.
>I think if "within reason" depends on the implementation of periodic_task, especially currently the periodic_task is >serialized and each periodic_task can sleep to give chance to RPC, which is higher priority.

gotcha.

>> with about a 30 second margin of error. If it was a 5 minute periodic task would
>> we still only see a 30s margin of error? I think a 1 minute periodic task is too
>> high-resolution for this mechanism.
>
>Instead of 30 seconds margin of error, it's two minutes error, see below example, the periodic task start at >07:09:42 and stopped at 07:11:49, so 2m 7s in total.
>
>periodic tasks 2012-11-23 07:09:42.055921
>PPP periodic tasks done 2012-11-23 07:11:49.662566
>
>I'm beginning working on a patch on this, and will submit it after it's done.

Right, but is it still 2 minutes if the periodic task is longer (like 5 min / 10 min vs the 1 minute)

>
>>
>> 2. Which service is this coming from? I'm assuming Compute? Is the service
>> especially busy? Or are this lags just inherent even under low load?
>
>Yes, it's from nova compute. I didn't do anything special in the service, except boot 20 instances. Considering it's >a 32-core system, that number is not big IMHO.

The hypervisor could still be I/O bound by disk or network (pulling down the images for example). 

>
>I did quick check for the delay, seems 2 periodic task contributed most. One is update_available resources and >another is report_driver_status and I'm still investigating the root cause..
>

Yeah, it would be nice to understand the root cause here. 

>>
>> 3. Are you dipping down into user-space (on the instances?) Could this be
>> causing a explosion of green threads that is adding to the latency? My personal
>
>No, all instances are idle.

Right, but what does your periodic task actually do? github link perhaps?

>
>--jyh


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev