Open Stack

It may not have been clear from the below email, but clarkb clarifies on
https://bugs.launchpad.net/openstack-ci/+bug/1294381 that the infra team is
no longer maintaining pypi-mirror

This has been a very useful tool for tripleo. It's much simpler for new
developers to set up and use than a full bandersnatch mirror (and requires
less disk space), and it can create a local cache of wheels which saves
build time.

But it's now unsupported.

To me it seems like we have two options:

A) Deprecate usage of pypi-mirror; update docs to instruct new devs in
setting up a local bandersnatch mirror instead
or
B) Take on care-and-feeding of the tool.
or, I guess,
C) Continue to recommend people use an unsupported unmaintained known-buggy
tool (it works reasonably well for us today, but it's going to work less
and less well as time goes by)

Are there other options I haven't thought of?

Do you have thoughts on which option is preferred?


---------- Forwarded message ----------
From: Clark Boylan <clark.boylan at gmail.com>
Date: Tue, Jul 8, 2014 at 8:50 AM
Subject: Re: [openstack-dev] Policy around Requirements Adds (was: New
class of requirements for Stackforge projects)
To: "OpenStack Development Mailing List (not for usage questions)" <
openstack-dev at lists.openstack.org>


On Mon, Jul 7, 2014 at 3:45 PM, Joe Gordon <joe.gordon0 at gmail.com> wrote:
>
> On Jul 7, 2014 4:48 PM, "Sean Dague" <sean at dague.net> wrote:
>>
>> This thread was unfortunately hidden under a project specific tag (I
>> have thus stripped all the tags).
>>
>> The crux of the argument here is the following:
>>
>> Is a stackforge project project able to propose additions to
>> global-requirements.txt that aren't used by any projects in OpenStack.
>>
>> I believe the answer is firmly *no*.
>
> ++
>
>>
>> global-requirements.txt provides a way for us to have a single point of
>> vetting for requirements for OpenStack. It lets us assess licensing,
>> maturity, current state of packaging, python3 support, all in one place.
>> And it lets us enforce that integration of OpenStack projects all run
>> under a well understood set of requirements.
>>
>> The requirements sync that happens after requirements land is basically
>> just a nicety for getting openstack projects to the tested state by
>> eventual consistency.
>>
>> If a stackforge project wants to be limited by global-requirements,
>> that's cool. We have a mechanism for that. However, they are accepting
>> that they will be limited by it. That means they live with how the
>> OpenStack project establishes that list. It specifically means they
>> *don't* get to propose any new requirements.
>>
>> Basically in this case Solum wants to have it's cake and eat it to. Both
>> be enforced on requirements, and not be enforced. Or some 3rd thing that
>> means the same as that.
>>
>> The near term fix is to remove solum from projects.txt.
>
> The email included below mentions an additional motivation for using
> global-requirements is to avoid using pypi.python.org and instead use
> pypi.openstack.org for speed and reliability. Perhaps there is a way we
can
> support use case for stackforge projects not in projects.txt? I thought I
> saw something the other day about adding a full pypi mirror to OpenStack
> infra.
>
This is done. All tests are now run against a bandersnatch built full
mirror of pypi. Enforcement of the global requirements is performed
via the enforcement jobs.
>>
>> On 06/26/2014 02:00 AM, Adrian Otto wrote:
>> > Ok,
>> >
>> > I submitted and abandoned a couple of reviews[1][2] for a solution
aimed
>> > to meet my goals without adding a new per-project requirements file.
The
>> > flaw with this approach is that pip may install other requirements when
>> > installing the one(s) loaded from the fallback mirror, and those may
>> > conflict with the ones loaded from the primary mirror.
>> >
>> > After discussing this further in #openstack-infra this evening, we
>> > should give serious consideration to adding python-mistralclient to
>> > global requirements. I have posted a review[3] for that to get input
>> > from the requirements review team.
>> >
>> > Thanks,
>> >
>> > Adrian
>> >
>> > [1] https://review.openstack.org/102716
>> > [2] https://review.openstack.org/102719
>> > [3] https://review.openstack.org/102738
>> > <https://review.openstack.org/1027387>
>> >
>> > On Jun 25, 2014, at 9:51 PM, Matthew Oliver <matt at oliver.net.au
>> > <mailto:matt at oliver.net.au>> wrote:
>> >
>> >>
>> >> On Jun 26, 2014 12:12 PM, "Angus Salkeld" <angus.salkeld at rackspace.com
>> >> <mailto:angus.salkeld at rackspace.com>> wrote:
>> >> >
>> > On 25/06/14 15:13, Clark Boylan wrote:
>> >> On Tue, Jun 24, 2014 at 9:54 PM, Adrian Otto
>> >>> <adrian.otto at rackspace.com <mailto:adrian.otto at rackspace.com>> wrote:
>> >>> Hello,
>> >>>
>> >>> Solum has run into a constraint with the current scheme for
>> >>> requirements management within the OpenStack CI system. We have a
>> >>> proposal for dealing with this constraint that involves making a
>> >>> contribution to openstack-infra. This message explains the
constraint,
>> >>> and our proposal for addressing it.
>> >>>
>> >>> == Background ==
>> >>>
>> >>> OpenStack uses a list of global requirements in the requirements
>> >>> repo[1], and each project has it’s own requirements.txt and
>> >>> test-requirements.txt files. The requirements are satisfied by gate
>> >>> jobs using pip configured to use the pypi.openstack.org
>> >>> <http://pypi.openstack.org/> mirror, which is periodically updated
>> >>> with new content from pypi.python.org <http://pypi.python.org/>. One
>> >>> motivation for doing this is that pypi.python.org
>> >>> <http://pypi.python.org/> may not be as fast or as reliable as a
local
>> >>> mirror. The gate/check jobs for the projects use the OpenStack
>> >>> internal pypi mirror to ensure stability.
>> >>>
>> >>> The OpenStack CI system will sync up the requirements across all
>> >>> the official projects and will create reviews in the participating
>> >>> projects for any mis-matches. Solum is one of these projects, and
>> >>> enjoys this feature.
>> >>>
>> >>> Another motivation is so that users of OpenStack will have one
>> >>> single set of python package requirements/dependencies to install and
>> >>> run the individual OpenStack components.
>> >>>
>> >>> == Problem ==
>> >>>
>> >>> Stackforge projects listed in openstack/requirements/projects.txt
>> >>> that decide to depend on each other (for example, Solum wanting to
>> >>> list mistralclient as a requirement) are unable to, because they are
>> >>> not yet integrated, and are not listed in
>> >>> openstack/requirements/global-requirements.txt yet. This means that
in
>> >>> order to depend on each other, a project must withdraw from
>> >>> projects.txt and begin using pip with pypi.poython.org
>> >>> <http://pypi.poython.org/> to satisfy all of their requirements.I
>> >>> strongly dislike this option.
>> >>>
>> >>> Mistral is still evolving rapidly, and we don’t think it makes
>> >>> sense for them to pursue integration wight now. The upstream
>> >>> distributions who include packages to support OpenStack will also
>> >>> prefer not to deal with a requirement that will be cutting a new
>> >>> version every week or two in order to satisfy evolving needs as Solum
>> >>> and other consumers of Mistral help refine how it works.
>> >>>
>> >>> == Proposal ==
>> >>>
>> >>> We want the best of both worlds. We want the freedom to innovate
>> >>> and use new software for a limited selection of stackforge projects,
>> >>> and still use the OpenStack pypi server to satisfy my regular
>> >>> requirements. We want the speed and reliability of using our local
>> >>> mirror, and users of Solum to use a matching set of requirements for
>> >>> all the things that we use, and integrated projects use. We want to
>> >>> continue getting the reviews that bring us up to date with new
>> >>> requirements versions.
>> >>>
>> >>> We propose that we submit an enhancement to the gate/check job
>> >>> setup that will:
>> >>>
>> >>> 1) Begin (as it does today) by satisfying global-requirements.txt
>> >>> and my local project’s requirements.txt and test-requirements.txt
>> >>> using the local OpenStack pypi mirror.
>> >>> 2) After all requirements are satisfied, check the name of my
>> >>> project. If it begins with ‘stackforge/‘ then look for a
>> >>> stackforge-requirements.txt file. If one exists, reconfigure pip to
>> >>> switch to use pypi.python.org <http://pypi.python.org/>, and satisfy
>> >>> the requirements listed in the file. We will list mistralclient
there,
>> >>> and get the latest tagged/released version of that.
>> >>>
>> >> I am reasonably sure that if you remove yourself from the
>> >> openstack/requirements project list this is basically how it will
>> >> work. Pip is configured to use the OpenStack mirror and fall back on
>> >> pypi.python.org <http://pypi.python.org/> for packages not
>> >>> available on the OpenStack mirror
>> >> [2]. So I don't think there is any work to do here with additional
>> >> requirements files. It should just work. Adding a new requirements
>> >> file will just make things more confusing for packagers and consumers
>> >> of your software.
>> >
>> > Adrian I know this is not the optimal solution, but I think this is
>> > the most pragmatic solution (esp. given we need to progress and not
>> >>> be held
>> > up by this), most stackforge projects are in the same boat as us.
>> > As far as pypi breakages (most are easily fixable by restricting the
>> > package versions if we get an issue with a new release
>> > of *random-api-breaking-package*).
>> >
>> >>>
>> >>> I've looked through the infra choose mirror code, and Clark is
>> >>> correct. If the project isn't in the projects.txt file they will only
>> >>> access to pypi.openstack.org <http://pypi.openstack.org/> however if
>> >>> removed then it will first check pypi.openstack.org
>> >>> <http://pypi.openstack.org/> and then fall back to to pypi.python.org
>> >>> <http://pypi.python.org/>. I think the only real solution is what
>> >>> Angus mentioned, remove yourself from projects.txt at least until all
>> >>> your dependencies can be provided by pypi.openstack.org
>> >>> <http://pypi.openstack.org/> or another solution is put into place.
In
>> >>> the mean time you can at least progress and continue development.
>> >>>
>> >>> If you code requires a direct dependency (rather then an optional
>> >>> dependency) of some non integrated project, then your stuck until
they
>> >>> are.
>> >>>
>> >>>
>> >
>> >>>
>> >>> == Call To Action ==
>> >>>
>> >>> What do you think of this approach to satisfy a balance of
>> >>> interests? Everything remains the same for OpenStack projects, and
>> >>> Stackforge projects get a new feature that allows them to require
>> >>> software that has not yet been integrated. Are there even better
>> >>> options that we should consider?
>> >>>
>> >>> Thanks,
>> >>>
>> >>> Adrian Otto
>> >>>
>> >>>
>> >>> References:
>> >>> [1] https://review.openstack.org/openstack/requirements
>> >
>> >> For what it is worth the Infra team has also been looking at
>> >> potentially using something like bandersnatch to mirror all of pypi
>> >> which is now a possibility because OpenStack doesn't depend on
>> >> packages that are hosted external to pypi. We would then do
>> >> requirements enforcement via checks rather than explicit use of a
>> >> restricted mirror. There are some things to sort out like platform
>> >> dependent wheels (I am not sure that any OpenStack project directly
>> >> consumes these but I have found them to be quite handy) and the
>> >> potential need for more enforcement to keep this working, but I think
>> >> this is a possibility.
>> >
>> > This would be neat.
>> >
>> > -Angus
>> >
>> >
>> >> Clark
>> >
>> >> [2]
>> >>>
>> >>>
https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/slave_scripts/select-mirror.sh#n54
>> >
>> >> _______________________________________________
>> >> OpenStack-dev mailing list
>> >> OpenStack-dev at lists.openstack.org
>> >>> <mailto:OpenStack-dev at lists.openstack.org>
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> >
>> >> >
>> >> > _______________________________________________
>> >> > OpenStack-dev mailing list
>> >> > OpenStack-dev at lists.openstack.org
>> >> <mailto:OpenStack-dev at lists.openstack.org>
>> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/
>> >>
>> >> _______________________________________________
>> >> OpenStack-dev mailing list
>> >> OpenStack-dev at lists.openstack.org
>> >> <mailto:OpenStack-dev at lists.openstack.org>
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> >
>> >
>> > _______________________________________________
>> > OpenStack-dev mailing list
>> > OpenStack-dev at lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> --
>> Sean Dague
>> http://dague.net
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140709/92d8ac1d/attachment.html>