Open Stack

On 06/09/2015 08:15 PM, Robert Collins wrote:
> I'm very glad folk are working on Python3 ports.
> 
> I'd like to call attention to one little wart in that process: I get
> the feeling that folk are applying a massive regex to find things like
> d.iteritems() and convert that to six.iteritems(d).
> 
> I'd very much prefer that such a regex approach move things to
> d.items(), which is much easier to read.
> 
> Here's why. Firstly, very very very few of our dict iterations are
> going to be performance sensitive in the way that iteritems() matters.
> Secondly, no really - unless you're doing HUGE dicts, it doesn't
> matter. Thirdly. Really, it doesn't.
> 
> At 1 million items the overhead is 54ms[1]. If we're doing inner loops
> on million item dictionaries anywhere in OpenStack today, we have a
> problem. We might want to in e.g. the scheduler... if it held
> in-memory state on a million hypervisors at once, because I don't
> really to to imagine it pulling a million rows from a DB on every
> action. But then, we'd be looking at a whole 54ms. I think we could
> survive, if we did that (which we don't).
> 
> So - please, no six.iteritems().
> 
> Thanks,
> Rob
> 
> 
> [1]
> python2.7 -m timeit -s 'd=dict(enumerate(range(1000000)))' 'for i in
> d.items(): pass'
> 10 loops, best of 3: 76.6 msec per loop
> python2.7 -m timeit -s 'd=dict(enumerate(range(1000000)))' 'for i in
> d.iteritems(): pass'
> 100 loops, best of 3: 22.6 msec per loop
> python3.4 -m timeit -s 'd=dict(enumerate(range(1000000)))' 'for i in
> d.items(): pass'
> 10 loops, best of 3: 18.9 msec per loop
> pypy2.3 -m timeit -s 'd=dict(enumerate(range(1000000)))' 'for i in
> d.items(): pass'
> 10 loops, best of 3: 65.8 msec per loop
> # and out of interest, assuming that that hadn't triggered the JIT....
> but it had.
>  pypy -m timeit -n 1000 -s 'd=dict(enumerate(range(1000000)))' 'for i
> in d.items(): pass'
> 1000 loops, best of 3: 64.3 msec per loop
> 

That's awesome, because those six.iteritems loops make me want to throw
up a little. Very happy to have our code just use items instead.

	-Sean

-- 
Sean Dague
http://dague.net