<div dir="ltr">It may not have been clear from the below email, but clarkb clarifies on <a href="https://bugs.launchpad.net/openstack-ci/+bug/1294381">https://bugs.launchpad.net/openstack-ci/+bug/1294381</a> that the infra team is no longer maintaining pypi-mirror<div>
<br></div><div>This has been a very useful tool for tripleo. It's much simpler for new developers to set up and use than a full bandersnatch mirror (and requires less disk space), and it can create a local cache of wheels which saves build time.<br>
<br>But it's now unsupported.<br><br>To me it seems like we have two options:</div><div><br></div><div>A) Deprecate usage of pypi-mirror; update docs to instruct new devs in setting up a local bandersnatch mirror instead</div>
<div>or</div><div>B) Take on care-and-feeding of the tool.</div><div>or, I guess,<br>C) Continue to recommend people use an unsupported unmaintained known-buggy tool (it works reasonably well for us today, but it's going to work less and less well as time goes by)</div>
<div><br>Are there other options I haven't thought of?</div><div><br></div><div>Do you have thoughts on which option is preferred?</div><div><br><br><div class="gmail_quote">---------- Forwarded message ----------<br>
From: <b class="gmail_sendername">Clark Boylan</b> <span dir="ltr"><<a href="mailto:clark.boylan@gmail.com">clark.boylan@gmail.com</a>></span><br>Date: Tue, Jul 8, 2014 at 8:50 AM<br>Subject: Re: [openstack-dev] Policy around Requirements Adds (was: New class of requirements for Stackforge projects)<br>
To: "OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br><br><br><div class=""><div class="h5">On Mon, Jul 7, 2014 at 3:45 PM, Joe Gordon <<a href="mailto:joe.gordon0@gmail.com">joe.gordon0@gmail.com</a>> wrote:<br>
><br>
> On Jul 7, 2014 4:48 PM, "Sean Dague" <<a href="mailto:sean@dague.net">sean@dague.net</a>> wrote:<br>
>><br>
>> This thread was unfortunately hidden under a project specific tag (I<br>
>> have thus stripped all the tags).<br>
>><br>
>> The crux of the argument here is the following:<br>
>><br>
>> Is a stackforge project project able to propose additions to<br>
>> global-requirements.txt that aren't used by any projects in OpenStack.<br>
>><br>
>> I believe the answer is firmly *no*.<br>
><br>
> ++<br>
><br>
>><br>
>> global-requirements.txt provides a way for us to have a single point of<br>
>> vetting for requirements for OpenStack. It lets us assess licensing,<br>
>> maturity, current state of packaging, python3 support, all in one place.<br>
>> And it lets us enforce that integration of OpenStack projects all run<br>
>> under a well understood set of requirements.<br>
>><br>
>> The requirements sync that happens after requirements land is basically<br>
>> just a nicety for getting openstack projects to the tested state by<br>
>> eventual consistency.<br>
>><br>
>> If a stackforge project wants to be limited by global-requirements,<br>
>> that's cool. We have a mechanism for that. However, they are accepting<br>
>> that they will be limited by it. That means they live with how the<br>
>> OpenStack project establishes that list. It specifically means they<br>
>> *don't* get to propose any new requirements.<br>
>><br>
>> Basically in this case Solum wants to have it's cake and eat it to. Both<br>
>> be enforced on requirements, and not be enforced. Or some 3rd thing that<br>
>> means the same as that.<br>
>><br>
>> The near term fix is to remove solum from projects.txt.<br>
><br>
> The email included below mentions an additional motivation for using<br>
> global-requirements is to avoid using <a href="http://pypi.python.org" target="_blank">pypi.python.org</a> and instead use<br>
> <a href="http://pypi.openstack.org" target="_blank">pypi.openstack.org</a> for speed and reliability. Perhaps there is a way we can<br>
> support use case for stackforge projects not in projects.txt? I thought I<br>
> saw something the other day about adding a full pypi mirror to OpenStack<br>
> infra.<br>
><br>
</div></div>This is done. All tests are now run against a bandersnatch built full<br>
mirror of pypi. Enforcement of the global requirements is performed<br>
via the enforcement jobs.<br>
<div class=""><div class="h5">>><br>
>> On 06/26/2014 02:00 AM, Adrian Otto wrote:<br>
>> > Ok,<br>
>> ><br>
>> > I submitted and abandoned a couple of reviews[1][2] for a solution aimed<br>
>> > to meet my goals without adding a new per-project requirements file. The<br>
>> > flaw with this approach is that pip may install other requirements when<br>
>> > installing the one(s) loaded from the fallback mirror, and those may<br>
>> > conflict with the ones loaded from the primary mirror.<br>
>> ><br>
>> > After discussing this further in #openstack-infra this evening, we<br>
>> > should give serious consideration to adding python-mistralclient to<br>
>> > global requirements. I have posted a review[3] for that to get input<br>
>> > from the requirements review team.<br>
>> ><br>
>> > Thanks,<br>
>> ><br>
>> > Adrian<br>
>> ><br>
>> > [1] <a href="https://review.openstack.org/102716" target="_blank">https://review.openstack.org/102716</a><br>
>> > [2] <a href="https://review.openstack.org/102719" target="_blank">https://review.openstack.org/102719</a><br>
>> > [3] <a href="https://review.openstack.org/102738" target="_blank">https://review.openstack.org/102738</a><br>
>> > <<a href="https://review.openstack.org/1027387" target="_blank">https://review.openstack.org/1027387</a>><br>
>> ><br>
>> > On Jun 25, 2014, at 9:51 PM, Matthew Oliver <<a href="mailto:matt@oliver.net.au">matt@oliver.net.au</a><br>
>> > <mailto:<a href="mailto:matt@oliver.net.au">matt@oliver.net.au</a>>> wrote:<br>
>> ><br>
>> >><br>
>> >> On Jun 26, 2014 12:12 PM, "Angus Salkeld" <<a href="mailto:angus.salkeld@rackspace.com">angus.salkeld@rackspace.com</a><br>
>> >> <mailto:<a href="mailto:angus.salkeld@rackspace.com">angus.salkeld@rackspace.com</a>>> wrote:<br>
>> >> ><br>
>> > On 25/06/14 15:13, Clark Boylan wrote:<br>
>> >> On Tue, Jun 24, 2014 at 9:54 PM, Adrian Otto<br>
>> >>> <<a href="mailto:adrian.otto@rackspace.com">adrian.otto@rackspace.com</a> <mailto:<a href="mailto:adrian.otto@rackspace.com">adrian.otto@rackspace.com</a>>> wrote:<br>
>> >>> Hello,<br>
>> >>><br>
>> >>> Solum has run into a constraint with the current scheme for<br>
>> >>> requirements management within the OpenStack CI system. We have a<br>
>> >>> proposal for dealing with this constraint that involves making a<br>
>> >>> contribution to openstack-infra. This message explains the constraint,<br>
>> >>> and our proposal for addressing it.<br>
>> >>><br>
>> >>> == Background ==<br>
>> >>><br>
>> >>> OpenStack uses a list of global requirements in the requirements<br>
>> >>> repo[1], and each project has it’s own requirements.txt and<br>
>> >>> test-requirements.txt files. The requirements are satisfied by gate<br>
>> >>> jobs using pip configured to use the <a href="http://pypi.openstack.org" target="_blank">pypi.openstack.org</a><br>
>> >>> <<a href="http://pypi.openstack.org/" target="_blank">http://pypi.openstack.org/</a>> mirror, which is periodically updated<br>
>> >>> with new content from <a href="http://pypi.python.org" target="_blank">pypi.python.org</a> <<a href="http://pypi.python.org/" target="_blank">http://pypi.python.org/</a>>. One<br>
>> >>> motivation for doing this is that <a href="http://pypi.python.org" target="_blank">pypi.python.org</a><br>
>> >>> <<a href="http://pypi.python.org/" target="_blank">http://pypi.python.org/</a>> may not be as fast or as reliable as a local<br>
>> >>> mirror. The gate/check jobs for the projects use the OpenStack<br>
>> >>> internal pypi mirror to ensure stability.<br>
>> >>><br>
>> >>> The OpenStack CI system will sync up the requirements across all<br>
>> >>> the official projects and will create reviews in the participating<br>
>> >>> projects for any mis-matches. Solum is one of these projects, and<br>
>> >>> enjoys this feature.<br>
>> >>><br>
>> >>> Another motivation is so that users of OpenStack will have one<br>
>> >>> single set of python package requirements/dependencies to install and<br>
>> >>> run the individual OpenStack components.<br>
>> >>><br>
>> >>> == Problem ==<br>
>> >>><br>
>> >>> Stackforge projects listed in openstack/requirements/projects.txt<br>
>> >>> that decide to depend on each other (for example, Solum wanting to<br>
>> >>> list mistralclient as a requirement) are unable to, because they are<br>
>> >>> not yet integrated, and are not listed in<br>
>> >>> openstack/requirements/global-requirements.txt yet. This means that in<br>
>> >>> order to depend on each other, a project must withdraw from<br>
>> >>> projects.txt and begin using pip with <a href="http://pypi.poython.org" target="_blank">pypi.poython.org</a><br>
>> >>> <<a href="http://pypi.poython.org/" target="_blank">http://pypi.poython.org/</a>> to satisfy all of their requirements.I<br>
>> >>> strongly dislike this option.<br>
>> >>><br>
>> >>> Mistral is still evolving rapidly, and we don’t think it makes<br>
>> >>> sense for them to pursue integration wight now. The upstream<br>
>> >>> distributions who include packages to support OpenStack will also<br>
>> >>> prefer not to deal with a requirement that will be cutting a new<br>
>> >>> version every week or two in order to satisfy evolving needs as Solum<br>
>> >>> and other consumers of Mistral help refine how it works.<br>
>> >>><br>
>> >>> == Proposal ==<br>
>> >>><br>
>> >>> We want the best of both worlds. We want the freedom to innovate<br>
>> >>> and use new software for a limited selection of stackforge projects,<br>
>> >>> and still use the OpenStack pypi server to satisfy my regular<br>
>> >>> requirements. We want the speed and reliability of using our local<br>
>> >>> mirror, and users of Solum to use a matching set of requirements for<br>
>> >>> all the things that we use, and integrated projects use. We want to<br>
>> >>> continue getting the reviews that bring us up to date with new<br>
>> >>> requirements versions.<br>
>> >>><br>
>> >>> We propose that we submit an enhancement to the gate/check job<br>
>> >>> setup that will:<br>
>> >>><br>
>> >>> 1) Begin (as it does today) by satisfying global-requirements.txt<br>
>> >>> and my local project’s requirements.txt and test-requirements.txt<br>
>> >>> using the local OpenStack pypi mirror.<br>
>> >>> 2) After all requirements are satisfied, check the name of my<br>
>> >>> project. If it begins with ‘stackforge/‘ then look for a<br>
>> >>> stackforge-requirements.txt file. If one exists, reconfigure pip to<br>
>> >>> switch to use <a href="http://pypi.python.org" target="_blank">pypi.python.org</a> <<a href="http://pypi.python.org/" target="_blank">http://pypi.python.org/</a>>, and satisfy<br>
>> >>> the requirements listed in the file. We will list mistralclient there,<br>
>> >>> and get the latest tagged/released version of that.<br>
>> >>><br>
>> >> I am reasonably sure that if you remove yourself from the<br>
>> >> openstack/requirements project list this is basically how it will<br>
>> >> work. Pip is configured to use the OpenStack mirror and fall back on<br>
>> >> <a href="http://pypi.python.org" target="_blank">pypi.python.org</a> <<a href="http://pypi.python.org/" target="_blank">http://pypi.python.org/</a>> for packages not<br>
>> >>> available on the OpenStack mirror<br>
>> >> [2]. So I don't think there is any work to do here with additional<br>
>> >> requirements files. It should just work. Adding a new requirements<br>
>> >> file will just make things more confusing for packagers and consumers<br>
>> >> of your software.<br>
>> ><br>
>> > Adrian I know this is not the optimal solution, but I think this is<br>
>> > the most pragmatic solution (esp. given we need to progress and not<br>
>> >>> be held<br>
>> > up by this), most stackforge projects are in the same boat as us.<br>
>> > As far as pypi breakages (most are easily fixable by restricting the<br>
>> > package versions if we get an issue with a new release<br>
>> > of *random-api-breaking-package*).<br>
>> ><br>
>> >>><br>
>> >>> I've looked through the infra choose mirror code, and Clark is<br>
>> >>> correct. If the project isn't in the projects.txt file they will only<br>
>> >>> access to <a href="http://pypi.openstack.org" target="_blank">pypi.openstack.org</a> <<a href="http://pypi.openstack.org/" target="_blank">http://pypi.openstack.org/</a>> however if<br>
>> >>> removed then it will first check <a href="http://pypi.openstack.org" target="_blank">pypi.openstack.org</a><br>
>> >>> <<a href="http://pypi.openstack.org/" target="_blank">http://pypi.openstack.org/</a>> and then fall back to to <a href="http://pypi.python.org" target="_blank">pypi.python.org</a><br>
>> >>> <<a href="http://pypi.python.org/" target="_blank">http://pypi.python.org/</a>>. I think the only real solution is what<br>
>> >>> Angus mentioned, remove yourself from projects.txt at least until all<br>
>> >>> your dependencies can be provided by <a href="http://pypi.openstack.org" target="_blank">pypi.openstack.org</a><br>
>> >>> <<a href="http://pypi.openstack.org/" target="_blank">http://pypi.openstack.org/</a>> or another solution is put into place. In<br>
>> >>> the mean time you can at least progress and continue development.<br>
>> >>><br>
>> >>> If you code requires a direct dependency (rather then an optional<br>
>> >>> dependency) of some non integrated project, then your stuck until they<br>
>> >>> are.<br>
>> >>><br>
>> >>><br>
>> ><br>
>> >>><br>
>> >>> == Call To Action ==<br>
>> >>><br>
>> >>> What do you think of this approach to satisfy a balance of<br>
>> >>> interests? Everything remains the same for OpenStack projects, and<br>
>> >>> Stackforge projects get a new feature that allows them to require<br>
>> >>> software that has not yet been integrated. Are there even better<br>
>> >>> options that we should consider?<br>
>> >>><br>
>> >>> Thanks,<br>
>> >>><br>
>> >>> Adrian Otto<br>
>> >>><br>
>> >>><br>
>> >>> References:<br>
>> >>> [1] <a href="https://review.openstack.org/openstack/requirements" target="_blank">https://review.openstack.org/openstack/requirements</a><br>
>> ><br>
>> >> For what it is worth the Infra team has also been looking at<br>
>> >> potentially using something like bandersnatch to mirror all of pypi<br>
>> >> which is now a possibility because OpenStack doesn't depend on<br>
>> >> packages that are hosted external to pypi. We would then do<br>
>> >> requirements enforcement via checks rather than explicit use of a<br>
>> >> restricted mirror. There are some things to sort out like platform<br>
>> >> dependent wheels (I am not sure that any OpenStack project directly<br>
>> >> consumes these but I have found them to be quite handy) and the<br>
>> >> potential need for more enforcement to keep this working, but I think<br>
>> >> this is a possibility.<br>
>> ><br>
>> > This would be neat.<br>
>> ><br>
>> > -Angus<br>
>> ><br>
>> ><br>
>> >> Clark<br>
>> ><br>
>> >> [2]<br>
>> >>><br>
>> >>> <a href="https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/slave_scripts/select-mirror.sh#n54" target="_blank">https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/slave_scripts/select-mirror.sh#n54</a><br>
>> ><br>
>> >> _______________________________________________<br>
>> >> OpenStack-dev mailing list<br>
>> >> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> >>> <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>><br>
>> >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>> ><br>
>> ><br>
>> >> ><br>
>> >> > _______________________________________________<br>
>> >> > OpenStack-dev mailing list<br>
>> >> > <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> >> <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>><br>
>> >> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/</a><br>
>> >><br>
>> >> _______________________________________________<br>
>> >> OpenStack-dev mailing list<br>
>> >> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> >> <mailto:<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>><br>
>> >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>> ><br>
>> ><br>
>> ><br>
>> > _______________________________________________<br>
>> > OpenStack-dev mailing list<br>
>> > <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>> ><br>
>><br>
>> --<br>
>> Sean Dague<br>
>> <a href="http://dague.net" target="_blank">http://dague.net</a><br>
>><br>
>><br>
>> _______________________________________________<br>
>> OpenStack-dev mailing list<br>
>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>><br>
><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></div><br></div></div>