Open Stack

On Tue, Jul 8, 2014 at 8:54 PM, James Polley <jp at jamezpolley.com> wrote:

> It may not have been clear from the below email, but clarkb clarifies on
> https://bugs.launchpad.net/openstack-ci/+bug/1294381 that the infra team
> is no longer maintaining pypi-mirror
>
> This has been a very useful tool for tripleo. It's much simpler for new
> developers to set up and use than a full bandersnatch mirror (and requires
> less disk space), and it can create a local cache of wheels which saves
> build time.
>
> But it's now unsupported.
>
> To me it seems like we have two options:
>
> A) Deprecate usage of pypi-mirror; update docs to instruct new devs in
> setting up a local bandersnatch mirror instead
> or
> B) Take on care-and-feeding of the tool.
> or, I guess,
> C) Continue to recommend people use an unsupported unmaintained
> known-buggy tool (it works reasonably well for us today, but it's going to
> work less and less well as time goes by)
>
> Are there other options I haven't thought of?
>

I don't know if this fits your requirements but I use
http://doc.devpi.net/latest/quickstart-pypimirror.html for my development
needs.


>
> Do you have thoughts on which option is preferred?
>
>
> ---------- Forwarded message ----------
> From: Clark Boylan <clark.boylan at gmail.com>
> Date: Tue, Jul 8, 2014 at 8:50 AM
> Subject: Re: [openstack-dev] Policy around Requirements Adds (was: New
> class of requirements for Stackforge projects)
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev at lists.openstack.org>
>
>
> On Mon, Jul 7, 2014 at 3:45 PM, Joe Gordon <joe.gordon0 at gmail.com> wrote:
> >
> > On Jul 7, 2014 4:48 PM, "Sean Dague" <sean at dague.net> wrote:
> >>
> >> This thread was unfortunately hidden under a project specific tag (I
> >> have thus stripped all the tags).
> >>
> >> The crux of the argument here is the following:
> >>
> >> Is a stackforge project project able to propose additions to
> >> global-requirements.txt that aren't used by any projects in OpenStack.
> >>
> >> I believe the answer is firmly *no*.
> >
> > ++
> >
> >>
> >> global-requirements.txt provides a way for us to have a single point of
> >> vetting for requirements for OpenStack. It lets us assess licensing,
> >> maturity, current state of packaging, python3 support, all in one place.
> >> And it lets us enforce that integration of OpenStack projects all run
> >> under a well understood set of requirements.
> >>
> >> The requirements sync that happens after requirements land is basically
> >> just a nicety for getting openstack projects to the tested state by
> >> eventual consistency.
> >>
> >> If a stackforge project wants to be limited by global-requirements,
> >> that's cool. We have a mechanism for that. However, they are accepting
> >> that they will be limited by it. That means they live with how the
> >> OpenStack project establishes that list. It specifically means they
> >> *don't* get to propose any new requirements.
> >>
> >> Basically in this case Solum wants to have it's cake and eat it to. Both
> >> be enforced on requirements, and not be enforced. Or some 3rd thing that
> >> means the same as that.
> >>
> >> The near term fix is to remove solum from projects.txt.
> >
> > The email included below mentions an additional motivation for using
> > global-requirements is to avoid using pypi.python.org and instead use
> > pypi.openstack.org for speed and reliability. Perhaps there is a way we
> can
> > support use case for stackforge projects not in projects.txt? I thought I
> > saw something the other day about adding a full pypi mirror to OpenStack
> > infra.
> >
> This is done. All tests are now run against a bandersnatch built full
> mirror of pypi. Enforcement of the global requirements is performed
> via the enforcement jobs.
> >>
> >> On 06/26/2014 02:00 AM, Adrian Otto wrote:
> >> > Ok,
> >> >
> >> > I submitted and abandoned a couple of reviews[1][2] for a solution
> aimed
> >> > to meet my goals without adding a new per-project requirements file.
> The
> >> > flaw with this approach is that pip may install other requirements
> when
> >> > installing the one(s) loaded from the fallback mirror, and those may
> >> > conflict with the ones loaded from the primary mirror.
> >> >
> >> > After discussing this further in #openstack-infra this evening, we
> >> > should give serious consideration to adding python-mistralclient to
> >> > global requirements. I have posted a review[3] for that to get input
> >> > from the requirements review team.
> >> >
> >> > Thanks,
> >> >
> >> > Adrian
> >> >
> >> > [1] https://review.openstack.org/102716
> >> > [2] https://review.openstack.org/102719
> >> > [3] https://review.openstack.org/102738
> >> > <https://review.openstack.org/1027387>
> >> >
> >> > On Jun 25, 2014, at 9:51 PM, Matthew Oliver <matt at oliver.net.au
> >> > <mailto:matt at oliver.net.au>> wrote:
> >> >
> >> >>
> >> >> On Jun 26, 2014 12:12 PM, "Angus Salkeld" <
> angus.salkeld at rackspace.com
> >> >> <mailto:angus.salkeld at rackspace.com>> wrote:
> >> >> >
> >> > On 25/06/14 15:13, Clark Boylan wrote:
> >> >> On Tue, Jun 24, 2014 at 9:54 PM, Adrian Otto
> >> >>> <adrian.otto at rackspace.com <mailto:adrian.otto at rackspace.com>>
> wrote:
> >> >>> Hello,
> >> >>>
> >> >>> Solum has run into a constraint with the current scheme for
> >> >>> requirements management within the OpenStack CI system. We have a
> >> >>> proposal for dealing with this constraint that involves making a
> >> >>> contribution to openstack-infra. This message explains the
> constraint,
> >> >>> and our proposal for addressing it.
> >> >>>
> >> >>> == Background ==
> >> >>>
> >> >>> OpenStack uses a list of global requirements in the requirements
> >> >>> repo[1], and each project has it’s own requirements.txt and
> >> >>> test-requirements.txt files. The requirements are satisfied by gate
> >> >>> jobs using pip configured to use the pypi.openstack.org
> >> >>> <http://pypi.openstack.org/> mirror, which is periodically updated
> >> >>> with new content from pypi.python.org <http://pypi.python.org/>.
> One
> >> >>> motivation for doing this is that pypi.python.org
> >> >>> <http://pypi.python.org/> may not be as fast or as reliable as a
> local
> >> >>> mirror. The gate/check jobs for the projects use the OpenStack
> >> >>> internal pypi mirror to ensure stability.
> >> >>>
> >> >>> The OpenStack CI system will sync up the requirements across all
> >> >>> the official projects and will create reviews in the participating
> >> >>> projects for any mis-matches. Solum is one of these projects, and
> >> >>> enjoys this feature.
> >> >>>
> >> >>> Another motivation is so that users of OpenStack will have one
> >> >>> single set of python package requirements/dependencies to install
> and
> >> >>> run the individual OpenStack components.
> >> >>>
> >> >>> == Problem ==
> >> >>>
> >> >>> Stackforge projects listed in openstack/requirements/projects.txt
> >> >>> that decide to depend on each other (for example, Solum wanting to
> >> >>> list mistralclient as a requirement) are unable to, because they are
> >> >>> not yet integrated, and are not listed in
> >> >>> openstack/requirements/global-requirements.txt yet. This means that
> in
> >> >>> order to depend on each other, a project must withdraw from
> >> >>> projects.txt and begin using pip with pypi.poython.org
> >> >>> <http://pypi.poython.org/> to satisfy all of their requirements.I
> >> >>> strongly dislike this option.
> >> >>>
> >> >>> Mistral is still evolving rapidly, and we don’t think it makes
> >> >>> sense for them to pursue integration wight now. The upstream
> >> >>> distributions who include packages to support OpenStack will also
> >> >>> prefer not to deal with a requirement that will be cutting a new
> >> >>> version every week or two in order to satisfy evolving needs as
> Solum
> >> >>> and other consumers of Mistral help refine how it works.
> >> >>>
> >> >>> == Proposal ==
> >> >>>
> >> >>> We want the best of both worlds. We want the freedom to innovate
> >> >>> and use new software for a limited selection of stackforge projects,
> >> >>> and still use the OpenStack pypi server to satisfy my regular
> >> >>> requirements. We want the speed and reliability of using our local
> >> >>> mirror, and users of Solum to use a matching set of requirements for
> >> >>> all the things that we use, and integrated projects use. We want to
> >> >>> continue getting the reviews that bring us up to date with new
> >> >>> requirements versions.
> >> >>>
> >> >>> We propose that we submit an enhancement to the gate/check job
> >> >>> setup that will:
> >> >>>
> >> >>> 1) Begin (as it does today) by satisfying global-requirements.txt
> >> >>> and my local project’s requirements.txt and test-requirements.txt
> >> >>> using the local OpenStack pypi mirror.
> >> >>> 2) After all requirements are satisfied, check the name of my
> >> >>> project. If it begins with ‘stackforge/‘ then look for a
> >> >>> stackforge-requirements.txt file. If one exists, reconfigure pip to
> >> >>> switch to use pypi.python.org <http://pypi.python.org/>, and
> satisfy
> >> >>> the requirements listed in the file. We will list mistralclient
> there,
> >> >>> and get the latest tagged/released version of that.
> >> >>>
> >> >> I am reasonably sure that if you remove yourself from the
> >> >> openstack/requirements project list this is basically how it will
> >> >> work. Pip is configured to use the OpenStack mirror and fall back on
> >> >> pypi.python.org <http://pypi.python.org/> for packages not
> >> >>> available on the OpenStack mirror
> >> >> [2]. So I don't think there is any work to do here with additional
> >> >> requirements files. It should just work. Adding a new requirements
> >> >> file will just make things more confusing for packagers and consumers
> >> >> of your software.
> >> >
> >> > Adrian I know this is not the optimal solution, but I think this is
> >> > the most pragmatic solution (esp. given we need to progress and not
> >> >>> be held
> >> > up by this), most stackforge projects are in the same boat as us.
> >> > As far as pypi breakages (most are easily fixable by restricting the
> >> > package versions if we get an issue with a new release
> >> > of *random-api-breaking-package*).
> >> >
> >> >>>
> >> >>> I've looked through the infra choose mirror code, and Clark is
> >> >>> correct. If the project isn't in the projects.txt file they will
> only
> >> >>> access to pypi.openstack.org <http://pypi.openstack.org/> however
> if
> >> >>> removed then it will first check pypi.openstack.org
> >> >>> <http://pypi.openstack.org/> and then fall back to to
> pypi.python.org
> >> >>> <http://pypi.python.org/>. I think the only real solution is what
> >> >>> Angus mentioned, remove yourself from projects.txt at least until
> all
> >> >>> your dependencies can be provided by pypi.openstack.org
> >> >>> <http://pypi.openstack.org/> or another solution is put into
> place. In
> >> >>> the mean time you can at least progress and continue development.
> >> >>>
> >> >>> If you code requires a direct dependency (rather then an optional
> >> >>> dependency) of some non integrated project, then your stuck until
> they
> >> >>> are.
> >> >>>
> >> >>>
> >> >
> >> >>>
> >> >>> == Call To Action ==
> >> >>>
> >> >>> What do you think of this approach to satisfy a balance of
> >> >>> interests? Everything remains the same for OpenStack projects, and
> >> >>> Stackforge projects get a new feature that allows them to require
> >> >>> software that has not yet been integrated. Are there even better
> >> >>> options that we should consider?
> >> >>>
> >> >>> Thanks,
> >> >>>
> >> >>> Adrian Otto
> >> >>>
> >> >>>
> >> >>> References:
> >> >>> [1] https://review.openstack.org/openstack/requirements
> >> >
> >> >> For what it is worth the Infra team has also been looking at
> >> >> potentially using something like bandersnatch to mirror all of pypi
> >> >> which is now a possibility because OpenStack doesn't depend on
> >> >> packages that are hosted external to pypi. We would then do
> >> >> requirements enforcement via checks rather than explicit use of a
> >> >> restricted mirror. There are some things to sort out like platform
> >> >> dependent wheels (I am not sure that any OpenStack project directly
> >> >> consumes these but I have found them to be quite handy) and the
> >> >> potential need for more enforcement to keep this working, but I think
> >> >> this is a possibility.
> >> >
> >> > This would be neat.
> >> >
> >> > -Angus
> >> >
> >> >
> >> >> Clark
> >> >
> >> >> [2]
> >> >>>
> >> >>>
> https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/slave_scripts/select-mirror.sh#n54
> >> >
> >> >> _______________________________________________
> >> >> OpenStack-dev mailing list
> >> >> OpenStack-dev at lists.openstack.org
> >> >>> <mailto:OpenStack-dev at lists.openstack.org>
> >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >
> >> >
> >> >> >
> >> >> > _______________________________________________
> >> >> > OpenStack-dev mailing list
> >> >> > OpenStack-dev at lists.openstack.org
> >> >> <mailto:OpenStack-dev at lists.openstack.org>
> >> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/
> >> >>
> >> >> _______________________________________________
> >> >> OpenStack-dev mailing list
> >> >> OpenStack-dev at lists.openstack.org
> >> >> <mailto:OpenStack-dev at lists.openstack.org>
> >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > OpenStack-dev mailing list
> >> > OpenStack-dev at lists.openstack.org
> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >
> >>
> >> --
> >> Sean Dague
> >> http://dague.net
> >>
> >>
> >> _______________________________________________
> >> OpenStack-dev mailing list
> >> OpenStack-dev at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140708/86cb29dd/attachment-0001.html>