Open Stack

On Wed, Oct 31, 2012 at 8:17 AM, Sean Dague <sdague at linux.vnet.ibm.com>wrote:

> On 10/30/2012 07:07 PM, Doug Hellmann wrote:
> <snip>
>
> >     While I get that in production, the class loading will disappear in
>
>>     the noise, that huge hit on the unit tests seems really problematic
>>     from a developer point of view. And a big problem from a CI
>>     perspective. Anyone have ideas on why this is such a huge hit, or
>>     thoughts on how to mitigate it?
>>
>>
>> I haven't profiled it, but my guess is that the extra time is from
>> scanning the entrypoints registry. That would, as you say, hardly matter
>> when it is only done once on startup. Perhaps we need to cache driver
>> loaded from the config outside of the manager (?) instance where it is
>> used to avoid having to rescan the registry each time the tests set up
>> the manager.
>>
>
> Given that in our use case we already know the one class we are loading,
> is there any reason all the entrypoints are being scanned anyway? That just
> seems wasteful. And it also seems like it only gets worse as the entrypoint
> list grows.
>

The code for the entry points are not all being loaded. The *registry* of
entry points is being scanned. To perform the scan, pkg_resources looks at
the entry point definitions in all packages installed on the system, since
part of the point of the registry is your code doesn't have to live inside
any particular python package.


> As the stevedore author, could you add fast path loading into the library,
> and fail back to the full scan if something goes wrong on that?


I'm not sure what you mean by "fast path loading." The name of an entry
point does not include enough information, by itself, to find the class;
the registry has to be scanned.

Typically way to cache that is to hold on to the extension manager
instance, since it already knows about all of the plugins. I'm not sure if
that works for the driver manager, if it is trying to load different
drivers, since it really only caches the one it is configured to load.

Doug


>
>
>         -Sean
>
> --
> Sean Dague
> IBM Linux Technology Center
> email: sdague at linux.vnet.ibm.com
> alt-email: sldague at us.ibm.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121031/cc847615/attachment-0001.html>