Open Stack

> Regardless, could you provide some info on the services that are suffering so
> badly that eventlet lags the periodic_task for so long? Sounds like a nova bug
> we should fix? Actually, if we're monitoring green thread counts via the periodic
> task we should be able to warn upstream monitoring before it becomes an
> issue.


Data tells everything.

I did some quick experiment on my SandyBridge EP system. The period_all.txt gives the time it cost to finish one periodic_task. Nova_list is the output of my noav list. 

I paste the length of periodic tasks below, it takes more than 2 minutes to finish a round!

Of course, you can call it as "nova bug", and we can tune the periodic tasks to shorten the time, but the key thing is, "there is no architecture guarantee of the timeliness of the periodic task" in the eventlet and greenthread, and we need make sure periodic task will not block the RPC call, will not block status report ...... 


--jyh

periodic tasks 2012-11-23 06:53:12.209065 
PPP periodic tasks done 2012-11-23 06:53:35.225337
periodic tasks 2012-11-23 06:54:35.226013 
PPP periodic tasks done 2012-11-23 06:55:37.826033
periodic tasks 2012-11-23 06:56:37.826802 
PPP periodic tasks done 2012-11-23 06:57:40.110641
periodic tasks 2012-11-23 06:58:40.113476 
PPP periodic tasks done 2012-11-23 06:59:23.248379
periodic tasks 2012-11-23 07:00:23.248486 
PPP periodic tasks done 2012-11-23 07:01:21.104802
periodic tasks 2012-11-23 07:02:21.105146 
PPP periodic tasks done 2012-11-23 07:03:31.180916
periodic tasks 2012-11-23 07:04:31.181067 
PPP periodic tasks done 2012-11-23 07:05:57.834720
periodic tasks 2012-11-23 07:06:57.834820 
PPP periodic tasks done 2012-11-23 07:08:42.046682
periodic tasks 2012-11-23 07:09:42.055921 
PPP periodic tasks done 2012-11-23 07:11:49.662566
periodic tasks 2012-11-23 07:12:49.667298 
PPP periodic tasks done 2012-11-23 07:14:42.141007
periodic tasks 2012-11-23 07:15:42.141126 
PPP periodic tasks done 2012-11-23 07:17:54.201270
periodic tasks 2012-11-23 07:18:54.201386 
PPP periodic tasks done 2012-11-23 07:20:54.931626
periodic tasks 2012-11-23 07:21:54.937052


--jyh

> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: period_all.txt
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121123/c09d361b/attachment.txt>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nova_list
Type: application/octet-stream
Size: 1925 bytes
Desc: nova_list
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121123/c09d361b/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sampling.diff
Type: application/octet-stream
Size: 2552 bytes
Desc: sampling.diff
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121123/c09d361b/attachment-0001.obj>