Open Stack

> 
> Thanks for this Jiang ... I agree, data good. Several questions/observations:
> 
> 1. I agree that there should be no architectural guarantees for periodic_task.
> But it should be "within reason". It looks like about a 1 minute periodic task

Thanks for reply.
I think if "within reason" depends on the implementation of periodic_task, especially currently the periodic_task is serialized and each periodic_task can sleep to give chance to RPC, which is higher priority.

> with about a 30 second margin of error. If it was a 5 minute periodic task would
> we still only see a 30s margin of error? I think a 1 minute periodic task is too
> high-resolution for this mechanism.

Instead of 30 seconds margin of error, it's two minutes error, see below example, the periodic task start at 07:09:42 and stopped at 07:11:49, so 2m 7s in total.

periodic tasks 2012-11-23 07:09:42.055921 
PPP periodic tasks done 2012-11-23 07:11:49.662566

I'm beginning working on a patch on this, and will submit it after it's done.

> 
> 2. Which service is this coming from? I'm assuming Compute? Is the service
> especially busy? Or are this lags just inherent even under low load?

Yes, it's from nova compute. I didn't do anything special in the service, except boot 20 instances. Considering it's a 32-core system, that number is not big IMHO.

I did quick check for the delay, seems 2 periodic task contributed most. One is update_available resources and another is report_driver_status and I'm still investigating the root cause..

> 
> 3. Are you dipping down into user-space (on the instances?) Could this be
> causing a explosion of green threads that is adding to the latency? My personal

No, all instances are idle.

--jyh