Open Stack

> Date: Wed, 10 Jun 2015 21:33:44 +1200
> From: robertc at robertcollins.net
> To: openstack-dev at lists.openstack.org
> Subject: Re: [openstack-dev] [all][python3] use of six.iteritems()
> 
> On 10 June 2015 at 17:22, gordon chung <gord at live.ca> wrote:
> > maybe the suggestion should be "don't blindly apply six.iteritems or items" rather than don't apply iteritems at all. admittedly, it's a massive eyesore, but it's a very real use case that some projects deal with large data results and to enforce the latter policy can have negative effects[1].  one "million item dictionary" might be negligible but in a multi-user, multi-* environment that can have a significant impact on the amount memory required to store everything.
> 
> > [1] disclaimer: i have no real world results but i assume memory management was the reason for the switch in logic from py2 to py3
> 
> I wouldn't make that assumption.
> 
> And no, memory isn't an issue. If you have a million item dict,
> ignoring the internal overheads, the dict needs 1 million object
> pointers. The size of a list with those pointers in it is 1M (pointer
> size in bytes). E.g. 4M or 8M. Nothing to worry about given the
> footprint of such a program :)
iiuc, items() (in py2) will create a copy of  the dictionary in memory to be processed. this is useful for cases such as concurrency where you want to ensure consistency but doing a quick test i noticed a massive spike in memory usage between items() and iteritems.
'for i in dict(enumerate(range(1000000))).items(): pass' consumes significantly more memory than 'for i in dict(enumerate(range(1000000))).iteritems(): pass'. on my system, the difference in memory consumption was double when using items() vs iteritems() and the cpu util was significantly more as well... let me know if there's anything that stands out as inaccurate.
unless there's something wrong with my ignorant testing above, i think it's something projects should consider when mass applying any iteritems/items patch.
cheers,gord
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150610/275fb9d0/attachment.html>