Open Stack

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 14/08/14 18:33, Ben Nemec wrote:
> On 08/14/2014 08:37 AM, Ihar Hrachyshka wrote:
>> Hi all,
> 
>> some plugins depend on modules that are not mentioned in 
>> requirements.txt. Among them, Cisco Nexus (ncclient), Brocade 
>> (ncclient), Embrane (heleosapi)... Some other plugins put their 
>> dependencies in requirements.txt though (like Arista depending on
>>  jsonrpclib).
> 
>> There are pros and cons in both cases. The obvious issue with not
>>  putting those requirements in the file is that packagers are
>> left uninformed about those implicit requirements existing,
>> meaning plugins are shipped to users with broken dependencies. It
>> also means we ship code that depends on unknown modules grabbed
>> from random places in the internet instead of relying on what's 
>> available on pypi, which is a bit scary.
> 
>> With my packager hat on, I would like to suggest to make those 
>> dependencies explicit by filling in requirements.txt. This will 
>> make packaging a bit easier. Of course, runtime dependencies
>> being set correctly do not mean plugins are working and tested,
>> but at least we give them chance to be tested and used.
> 
>> But, maybe there are valid concerns against doing so. In that
>> case, I would be glad to know how packagers are expected to track
>> those implicit dependencies.
> 
>> I would like to ask community to decide what's the right way to 
>> handle those cases.
> 
> So I raised a similar issue about six months ago and completely
> failed to follow up on the direction everyone seemed to be onboard
> with: 
> http://lists.openstack.org/pipermail/openstack-dev/2014-February/026976.html
>
>  I did add support to pbr for using nested requirements files, and
> I had posted a PoC for oslo.messaging to allow requirements files
> for different backends, but some of our CI jobs don't know how to
> handle that and I never got around to addressing the limitation.
> 
> - From the packaging perspective, I think you could do a
> requirements file that basically rolls up requirements.d/*.txt
> minus test.txt and get all the runtime dependencies that the
> project knows about, assuming we finished the implementation for
> this and started using it in projects.
> 
> I don't really anticipate having time to pursue this in the near 
> future, so if you wanted to pick up the ball and run with it that 
> would be great! :-)

Thanks for the back reference!

Though I think it's overkill in Neutron case. There, we are not
interested in Qpid vs. Rabbitmq issue since we depend on
oslo.messaging that is to solve this specific dependency hell.

The case of Neutron plugins depending on specific code that is not
mentioned in requirements.txt is also a bit different. It's not about
alternative implementations of the same library for users to choose
from (as it's in case of qpid vs. rabbitmq). Instead, it's plugins
here that are really optional, but their dependencies are still strict
(=no alternative dependency implementations).

So what we need to make optional here are plugins. I've heard that
Kyle Mestery was going to propose splitting Neutron codebase in parts,
with core plugins sitting in the tree, and vendor specific stuff
maintained in separate repos. That would effectively make plugins
optional.

Till that time, I would make dependencies straight and explicit,
putting them in global requirements list. If distributions are
interested in not installing optional code for all plugins, they would
just separate core from plugins and attach those dependencies to
plugin-specific packages (as they already do btw, at least in Red Hat
world).

As for pip and devstack installations (which are not meant to be the
proper way to deploy openstack anyway), I don't see much value in
separating requirements. It's ok to have optional libraries installed
in your development environment, if only to be able to write unit
tests without mocking out the whole underlying library and praying
that your assumptions about its API are correct.

/Ihar
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)

iQEcBAEBCgAGBQJT8frcAAoJEC5aWaUY1u57m9oH/0w/RCdJcfZOAwVtGWgd2pH/
BkMpEChlQl40Eb7Vz/9UnLA1jdnnaWmhu9CPEFYs8mYQy/ZMMfB1Ww0e1wl5w0VD
pnbepRyYhmmyupZpsG8ywx5lOdSHNWJzUSEsJpK/ZHszgMiZUB/l++mMr+YfjyYK
h4SzFpoiz7Dnr3qYfowJQuoeH/yFK3Qd03WAybm39waxj+CX/0DXFVS7xoFFokGa
sQWPrMf+xhH2iOVgpgZl2YMzVVDl/bu2NWFkyDepXbTX/gWYI67rx4SbidwWlRYG
ifRQbrruV34n6wdBZ8C5s1WonpOpqX6Unay6qxPsn1qmgPKP4M2SckJoZqbvKVk=
=A0Gf
-----END PGP SIGNATURE-----