Open Stack

This thread was unfortunately hidden under a project specific tag (I
have thus stripped all the tags).

The crux of the argument here is the following:

Is a stackforge project project able to propose additions to
global-requirements.txt that aren't used by any projects in OpenStack.

I believe the answer is firmly *no*.

global-requirements.txt provides a way for us to have a single point of
vetting for requirements for OpenStack. It lets us assess licensing,
maturity, current state of packaging, python3 support, all in one place.
And it lets us enforce that integration of OpenStack projects all run
under a well understood set of requirements.

The requirements sync that happens after requirements land is basically
just a nicety for getting openstack projects to the tested state by
eventual consistency.

If a stackforge project wants to be limited by global-requirements,
that's cool. We have a mechanism for that. However, they are accepting
that they will be limited by it. That means they live with how the
OpenStack project establishes that list. It specifically means they
*don't* get to propose any new requirements.

Basically in this case Solum wants to have it's cake and eat it to. Both
be enforced on requirements, and not be enforced. Or some 3rd thing that
means the same as that.

The near term fix is to remove solum from projects.txt.

On 06/26/2014 02:00 AM, Adrian Otto wrote:
> Ok,
> 
> I submitted and abandoned a couple of reviews[1][2] for a solution aimed
> to meet my goals without adding a new per-project requirements file. The
> flaw with this approach is that pip may install other requirements when
> installing the one(s) loaded from the fallback mirror, and those may
> conflict with the ones loaded from the primary mirror.
> 
> After discussing this further in #openstack-infra this evening, we
> should give serious consideration to adding python-mistralclient to
> global requirements. I have posted a review[3] for that to get input
> from the requirements review team.
> 
> Thanks,
> 
> Adrian
> 
> [1] https://review.openstack.org/102716
> [2] https://review.openstack.org/102719
> [3] https://review.openstack.org/102738
> <https://review.openstack.org/1027387>
> 
> On Jun 25, 2014, at 9:51 PM, Matthew Oliver <matt at oliver.net.au
> <mailto:matt at oliver.net.au>> wrote:
> 
>>
>> On Jun 26, 2014 12:12 PM, "Angus Salkeld" <angus.salkeld at rackspace.com
>> <mailto:angus.salkeld at rackspace.com>> wrote:
>> >
> On 25/06/14 15:13, Clark Boylan wrote:
>> On Tue, Jun 24, 2014 at 9:54 PM, Adrian Otto
>>> <adrian.otto at rackspace.com <mailto:adrian.otto at rackspace.com>> wrote:
>>> Hello,
>>>
>>> Solum has run into a constraint with the current scheme for
>>> requirements management within the OpenStack CI system. We have a
>>> proposal for dealing with this constraint that involves making a
>>> contribution to openstack-infra. This message explains the constraint,
>>> and our proposal for addressing it.
>>>
>>> == Background ==
>>>
>>> OpenStack uses a list of global requirements in the requirements
>>> repo[1], and each project has it’s own requirements.txt and
>>> test-requirements.txt files. The requirements are satisfied by gate
>>> jobs using pip configured to use the pypi.openstack.org
>>> <http://pypi.openstack.org/> mirror, which is periodically updated
>>> with new content from pypi.python.org <http://pypi.python.org/>. One
>>> motivation for doing this is that pypi.python.org
>>> <http://pypi.python.org/> may not be as fast or as reliable as a local
>>> mirror. The gate/check jobs for the projects use the OpenStack
>>> internal pypi mirror to ensure stability.
>>>
>>> The OpenStack CI system will sync up the requirements across all
>>> the official projects and will create reviews in the participating
>>> projects for any mis-matches. Solum is one of these projects, and
>>> enjoys this feature.
>>>
>>> Another motivation is so that users of OpenStack will have one
>>> single set of python package requirements/dependencies to install and
>>> run the individual OpenStack components.
>>>
>>> == Problem ==
>>>
>>> Stackforge projects listed in openstack/requirements/projects.txt
>>> that decide to depend on each other (for example, Solum wanting to
>>> list mistralclient as a requirement) are unable to, because they are
>>> not yet integrated, and are not listed in
>>> openstack/requirements/global-requirements.txt yet. This means that in
>>> order to depend on each other, a project must withdraw from
>>> projects.txt and begin using pip with pypi.poython.org
>>> <http://pypi.poython.org/> to satisfy all of their requirements.I
>>> strongly dislike this option.
>>>
>>> Mistral is still evolving rapidly, and we don’t think it makes
>>> sense for them to pursue integration wight now. The upstream
>>> distributions who include packages to support OpenStack will also
>>> prefer not to deal with a requirement that will be cutting a new
>>> version every week or two in order to satisfy evolving needs as Solum
>>> and other consumers of Mistral help refine how it works.
>>>
>>> == Proposal ==
>>>
>>> We want the best of both worlds. We want the freedom to innovate
>>> and use new software for a limited selection of stackforge projects,
>>> and still use the OpenStack pypi server to satisfy my regular
>>> requirements. We want the speed and reliability of using our local
>>> mirror, and users of Solum to use a matching set of requirements for
>>> all the things that we use, and integrated projects use. We want to
>>> continue getting the reviews that bring us up to date with new
>>> requirements versions.
>>>
>>> We propose that we submit an enhancement to the gate/check job
>>> setup that will:
>>>
>>> 1) Begin (as it does today) by satisfying global-requirements.txt
>>> and my local project’s requirements.txt and test-requirements.txt
>>> using the local OpenStack pypi mirror.
>>> 2) After all requirements are satisfied, check the name of my
>>> project. If it begins with ‘stackforge/‘ then look for a
>>> stackforge-requirements.txt file. If one exists, reconfigure pip to
>>> switch to use pypi.python.org <http://pypi.python.org/>, and satisfy
>>> the requirements listed in the file. We will list mistralclient there,
>>> and get the latest tagged/released version of that.
>>>
>> I am reasonably sure that if you remove yourself from the
>> openstack/requirements project list this is basically how it will
>> work. Pip is configured to use the OpenStack mirror and fall back on
>> pypi.python.org <http://pypi.python.org/> for packages not
>>> available on the OpenStack mirror
>> [2]. So I don't think there is any work to do here with additional
>> requirements files. It should just work. Adding a new requirements
>> file will just make things more confusing for packagers and consumers
>> of your software.
> 
> Adrian I know this is not the optimal solution, but I think this is
> the most pragmatic solution (esp. given we need to progress and not
>>> be held
> up by this), most stackforge projects are in the same boat as us.
> As far as pypi breakages (most are easily fixable by restricting the
> package versions if we get an issue with a new release
> of *random-api-breaking-package*).
> 
>>>
>>> I've looked through the infra choose mirror code, and Clark is
>>> correct. If the project isn't in the projects.txt file they will only
>>> access to pypi.openstack.org <http://pypi.openstack.org/> however if
>>> removed then it will first check pypi.openstack.org
>>> <http://pypi.openstack.org/> and then fall back to to pypi.python.org
>>> <http://pypi.python.org/>. I think the only real solution is what
>>> Angus mentioned, remove yourself from projects.txt at least until all
>>> your dependencies can be provided by pypi.openstack.org
>>> <http://pypi.openstack.org/> or another solution is put into place. In
>>> the mean time you can at least progress and continue development.
>>>
>>> If you code requires a direct dependency (rather then an optional
>>> dependency) of some non integrated project, then your stuck until they
>>> are.
>>>
>>>
> 
>>>
>>> == Call To Action ==
>>>
>>> What do you think of this approach to satisfy a balance of
>>> interests? Everything remains the same for OpenStack projects, and
>>> Stackforge projects get a new feature that allows them to require
>>> software that has not yet been integrated. Are there even better
>>> options that we should consider?
>>>
>>> Thanks,
>>>
>>> Adrian Otto
>>>
>>>
>>> References:
>>> [1] https://review.openstack.org/openstack/requirements
> 
>> For what it is worth the Infra team has also been looking at
>> potentially using something like bandersnatch to mirror all of pypi
>> which is now a possibility because OpenStack doesn't depend on
>> packages that are hosted external to pypi. We would then do
>> requirements enforcement via checks rather than explicit use of a
>> restricted mirror. There are some things to sort out like platform
>> dependent wheels (I am not sure that any OpenStack project directly
>> consumes these but I have found them to be quite handy) and the
>> potential need for more enforcement to keep this working, but I think
>> this is a possibility.
> 
> This would be neat.
> 
> -Angus
> 
> 
>> Clark
> 
>> [2]
>>> https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/slave_scripts/select-mirror.sh#n54
> 
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>>> <mailto:OpenStack-dev at lists.openstack.org>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
>> >
>> > _______________________________________________
>> > OpenStack-dev mailing list
>> > OpenStack-dev at lists.openstack.org
>> <mailto:OpenStack-dev at lists.openstack.org>
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> <mailto:OpenStack-dev at lists.openstack.org>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-- 
Sean Dague
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140707/6b6e3c06/attachment.pgp>