[openstack-dev] [TripleO] pypi-mirror is now unsupported - what do we do now?

Donald Stufft donald at stufft.io
Wed Jul 9 16:24:25 UTC 2014 

You’re aware pip has a built in command to create a local cache of wheels yea?

Not sure what your requirements are or what you’re using it for but if you detail
your requirements I can tell you how well pip can handle the use case out of the box.

On Jul 9, 2014, at 12:19 PM, Ben Nemec <openstack at nemebean.com> wrote:

> On 07/08/2014 11:05 PM, Joe Gordon wrote:
>> On Tue, Jul 8, 2014 at 8:54 PM, James Polley <jp at jamezpolley.com> wrote:
>> 
>>> It may not have been clear from the below email, but clarkb clarifies on
>>> https://bugs.launchpad.net/openstack-ci/+bug/1294381 that the infra team
>>> is no longer maintaining pypi-mirror
>>> 
>>> This has been a very useful tool for tripleo. It's much simpler for new
>>> developers to set up and use than a full bandersnatch mirror (and requires
>>> less disk space), and it can create a local cache of wheels which saves
>>> build time.
>>> 
>>> But it's now unsupported.
>>> 
>>> To me it seems like we have two options:
>>> 
>>> A) Deprecate usage of pypi-mirror; update docs to instruct new devs in
>>> setting up a local bandersnatch mirror instead
>>> or
>>> B) Take on care-and-feeding of the tool.
>>> or, I guess,
>>> C) Continue to recommend people use an unsupported unmaintained
>>> known-buggy tool (it works reasonably well for us today, but it's going to
>>> work less and less well as time goes by)
>>> 
>>> Are there other options I haven't thought of?
>>> 
>> 
>> I don't know if this fits your requirements but I use
>> http://doc.devpi.net/latest/quickstart-pypimirror.html for my development
>> needs.
> 
> Will that also cache wheels?  In my experience, wheels are one of the
> big time savers in tripleo so I would consider it an important feature
> to maintain, however we decide to proceed.
> 
>> 
>> 
>>> 
>>> Do you have thoughts on which option is preferred?
>>> 
>>> 
>>> ---------- Forwarded message ----------
>>> From: Clark Boylan <clark.boylan at gmail.com>
>>> Date: Tue, Jul 8, 2014 at 8:50 AM
>>> Subject: Re: [openstack-dev] Policy around Requirements Adds (was: New
>>> class of requirements for Stackforge projects)
>>> To: "OpenStack Development Mailing List (not for usage questions)" <
>>> openstack-dev at lists.openstack.org>
>>> 
>>> 
>>> On Mon, Jul 7, 2014 at 3:45 PM, Joe Gordon <joe.gordon0 at gmail.com> wrote:
>>>> 
>>>> On Jul 7, 2014 4:48 PM, "Sean Dague" <sean at dague.net> wrote:
>>>>> 
>>>>> This thread was unfortunately hidden under a project specific tag (I
>>>>> have thus stripped all the tags).
>>>>> 
>>>>> The crux of the argument here is the following:
>>>>> 
>>>>> Is a stackforge project project able to propose additions to
>>>>> global-requirements.txt that aren't used by any projects in OpenStack.
>>>>> 
>>>>> I believe the answer is firmly *no*.
>>>> 
>>>> ++
>>>> 
>>>>> 
>>>>> global-requirements.txt provides a way for us to have a single point of
>>>>> vetting for requirements for OpenStack. It lets us assess licensing,
>>>>> maturity, current state of packaging, python3 support, all in one place.
>>>>> And it lets us enforce that integration of OpenStack projects all run
>>>>> under a well understood set of requirements.
>>>>> 
>>>>> The requirements sync that happens after requirements land is basically
>>>>> just a nicety for getting openstack projects to the tested state by
>>>>> eventual consistency.
>>>>> 
>>>>> If a stackforge project wants to be limited by global-requirements,
>>>>> that's cool. We have a mechanism for that. However, they are accepting
>>>>> that they will be limited by it. That means they live with how the
>>>>> OpenStack project establishes that list. It specifically means they
>>>>> *don't* get to propose any new requirements.
>>>>> 
>>>>> Basically in this case Solum wants to have it's cake and eat it to. Both
>>>>> be enforced on requirements, and not be enforced. Or some 3rd thing that
>>>>> means the same as that.
>>>>> 
>>>>> The near term fix is to remove solum from projects.txt.
>>>> 
>>>> The email included below mentions an additional motivation for using
>>>> global-requirements is to avoid using pypi.python.org and instead use
>>>> pypi.openstack.org for speed and reliability. Perhaps there is a way we
>>> can
>>>> support use case for stackforge projects not in projects.txt? I thought I
>>>> saw something the other day about adding a full pypi mirror to OpenStack
>>>> infra.
>>>> 
>>> This is done. All tests are now run against a bandersnatch built full
>>> mirror of pypi. Enforcement of the global requirements is performed
>>> via the enforcement jobs.
>>>>> 
>>>>> On 06/26/2014 02:00 AM, Adrian Otto wrote:
>>>>>> Ok,
>>>>>> 
>>>>>> I submitted and abandoned a couple of reviews[1][2] for a solution
>>> aimed
>>>>>> to meet my goals without adding a new per-project requirements file.
>>> The
>>>>>> flaw with this approach is that pip may install other requirements
>>> when
>>>>>> installing the one(s) loaded from the fallback mirror, and those may
>>>>>> conflict with the ones loaded from the primary mirror.
>>>>>> 
>>>>>> After discussing this further in #openstack-infra this evening, we
>>>>>> should give serious consideration to adding python-mistralclient to
>>>>>> global requirements. I have posted a review[3] for that to get input
>>>>>> from the requirements review team.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Adrian
>>>>>> 
>>>>>> [1] https://review.openstack.org/102716
>>>>>> [2] https://review.openstack.org/102719
>>>>>> [3] https://review.openstack.org/102738
>>>>>> <https://review.openstack.org/1027387>
>>>>>> 
>>>>>> On Jun 25, 2014, at 9:51 PM, Matthew Oliver <matt at oliver.net.au
>>>>>> <mailto:matt at oliver.net.au>> wrote:
>>>>>> 
>>>>>>> 
>>>>>>> On Jun 26, 2014 12:12 PM, "Angus Salkeld" <
>>> angus.salkeld at rackspace.com
>>>>>>> <mailto:angus.salkeld at rackspace.com>> wrote:
>>>>>>>> 
>>>>>> On 25/06/14 15:13, Clark Boylan wrote:
>>>>>>> On Tue, Jun 24, 2014 at 9:54 PM, Adrian Otto
>>>>>>>> <adrian.otto at rackspace.com <mailto:adrian.otto at rackspace.com>>
>>> wrote:
>>>>>>>> Hello,
>>>>>>>> 
>>>>>>>> Solum has run into a constraint with the current scheme for
>>>>>>>> requirements management within the OpenStack CI system. We have a
>>>>>>>> proposal for dealing with this constraint that involves making a
>>>>>>>> contribution to openstack-infra. This message explains the
>>> constraint,
>>>>>>>> and our proposal for addressing it.
>>>>>>>> 
>>>>>>>> == Background ==
>>>>>>>> 
>>>>>>>> OpenStack uses a list of global requirements in the requirements
>>>>>>>> repo[1], and each project has it’s own requirements.txt and
>>>>>>>> test-requirements.txt files. The requirements are satisfied by gate
>>>>>>>> jobs using pip configured to use the pypi.openstack.org
>>>>>>>> <http://pypi.openstack.org/> mirror, which is periodically updated
>>>>>>>> with new content from pypi.python.org <http://pypi.python.org/>.
>>> One
>>>>>>>> motivation for doing this is that pypi.python.org
>>>>>>>> <http://pypi.python.org/> may not be as fast or as reliable as a
>>> local
>>>>>>>> mirror. The gate/check jobs for the projects use the OpenStack
>>>>>>>> internal pypi mirror to ensure stability.
>>>>>>>> 
>>>>>>>> The OpenStack CI system will sync up the requirements across all
>>>>>>>> the official projects and will create reviews in the participating
>>>>>>>> projects for any mis-matches. Solum is one of these projects, and
>>>>>>>> enjoys this feature.
>>>>>>>> 
>>>>>>>> Another motivation is so that users of OpenStack will have one
>>>>>>>> single set of python package requirements/dependencies to install
>>> and
>>>>>>>> run the individual OpenStack components.
>>>>>>>> 
>>>>>>>> == Problem ==
>>>>>>>> 
>>>>>>>> Stackforge projects listed in openstack/requirements/projects.txt
>>>>>>>> that decide to depend on each other (for example, Solum wanting to
>>>>>>>> list mistralclient as a requirement) are unable to, because they are
>>>>>>>> not yet integrated, and are not listed in
>>>>>>>> openstack/requirements/global-requirements.txt yet. This means that
>>> in
>>>>>>>> order to depend on each other, a project must withdraw from
>>>>>>>> projects.txt and begin using pip with pypi.poython.org
>>>>>>>> <http://pypi.poython.org/> to satisfy all of their requirements.I
>>>>>>>> strongly dislike this option.
>>>>>>>> 
>>>>>>>> Mistral is still evolving rapidly, and we don’t think it makes
>>>>>>>> sense for them to pursue integration wight now. The upstream
>>>>>>>> distributions who include packages to support OpenStack will also
>>>>>>>> prefer not to deal with a requirement that will be cutting a new
>>>>>>>> version every week or two in order to satisfy evolving needs as
>>> Solum
>>>>>>>> and other consumers of Mistral help refine how it works.
>>>>>>>> 
>>>>>>>> == Proposal ==
>>>>>>>> 
>>>>>>>> We want the best of both worlds. We want the freedom to innovate
>>>>>>>> and use new software for a limited selection of stackforge projects,
>>>>>>>> and still use the OpenStack pypi server to satisfy my regular
>>>>>>>> requirements. We want the speed and reliability of using our local
>>>>>>>> mirror, and users of Solum to use a matching set of requirements for
>>>>>>>> all the things that we use, and integrated projects use. We want to
>>>>>>>> continue getting the reviews that bring us up to date with new
>>>>>>>> requirements versions.
>>>>>>>> 
>>>>>>>> We propose that we submit an enhancement to the gate/check job
>>>>>>>> setup that will:
>>>>>>>> 
>>>>>>>> 1) Begin (as it does today) by satisfying global-requirements.txt
>>>>>>>> and my local project’s requirements.txt and test-requirements.txt
>>>>>>>> using the local OpenStack pypi mirror.
>>>>>>>> 2) After all requirements are satisfied, check the name of my
>>>>>>>> project. If it begins with ‘stackforge/‘ then look for a
>>>>>>>> stackforge-requirements.txt file. If one exists, reconfigure pip to
>>>>>>>> switch to use pypi.python.org <http://pypi.python.org/>, and
>>> satisfy
>>>>>>>> the requirements listed in the file. We will list mistralclient
>>> there,
>>>>>>>> and get the latest tagged/released version of that.
>>>>>>>> 
>>>>>>> I am reasonably sure that if you remove yourself from the
>>>>>>> openstack/requirements project list this is basically how it will
>>>>>>> work. Pip is configured to use the OpenStack mirror and fall back on
>>>>>>> pypi.python.org <http://pypi.python.org/> for packages not
>>>>>>>> available on the OpenStack mirror
>>>>>>> [2]. So I don't think there is any work to do here with additional
>>>>>>> requirements files. It should just work. Adding a new requirements
>>>>>>> file will just make things more confusing for packagers and consumers
>>>>>>> of your software.
>>>>>> 
>>>>>> Adrian I know this is not the optimal solution, but I think this is
>>>>>> the most pragmatic solution (esp. given we need to progress and not
>>>>>>>> be held
>>>>>> up by this), most stackforge projects are in the same boat as us.
>>>>>> As far as pypi breakages (most are easily fixable by restricting the
>>>>>> package versions if we get an issue with a new release
>>>>>> of *random-api-breaking-package*).
>>>>>> 
>>>>>>>> 
>>>>>>>> I've looked through the infra choose mirror code, and Clark is
>>>>>>>> correct. If the project isn't in the projects.txt file they will
>>> only
>>>>>>>> access to pypi.openstack.org <http://pypi.openstack.org/> however
>>> if
>>>>>>>> removed then it will first check pypi.openstack.org
>>>>>>>> <http://pypi.openstack.org/> and then fall back to to
>>> pypi.python.org
>>>>>>>> <http://pypi.python.org/>. I think the only real solution is what
>>>>>>>> Angus mentioned, remove yourself from projects.txt at least until
>>> all
>>>>>>>> your dependencies can be provided by pypi.openstack.org
>>>>>>>> <http://pypi.openstack.org/> or another solution is put into
>>> place. In
>>>>>>>> the mean time you can at least progress and continue development.
>>>>>>>> 
>>>>>>>> If you code requires a direct dependency (rather then an optional
>>>>>>>> dependency) of some non integrated project, then your stuck until
>>> they
>>>>>>>> are.
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>>>> 
>>>>>>>> == Call To Action ==
>>>>>>>> 
>>>>>>>> What do you think of this approach to satisfy a balance of
>>>>>>>> interests? Everything remains the same for OpenStack projects, and
>>>>>>>> Stackforge projects get a new feature that allows them to require
>>>>>>>> software that has not yet been integrated. Are there even better
>>>>>>>> options that we should consider?
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Adrian Otto
>>>>>>>> 
>>>>>>>> 
>>>>>>>> References:
>>>>>>>> [1] https://review.openstack.org/openstack/requirements
>>>>>> 
>>>>>>> For what it is worth the Infra team has also been looking at
>>>>>>> potentially using something like bandersnatch to mirror all of pypi
>>>>>>> which is now a possibility because OpenStack doesn't depend on
>>>>>>> packages that are hosted external to pypi. We would then do
>>>>>>> requirements enforcement via checks rather than explicit use of a
>>>>>>> restricted mirror. There are some things to sort out like platform
>>>>>>> dependent wheels (I am not sure that any OpenStack project directly
>>>>>>> consumes these but I have found them to be quite handy) and the
>>>>>>> potential need for more enforcement to keep this working, but I think
>>>>>>> this is a possibility.
>>>>>> 
>>>>>> This would be neat.
>>>>>> 
>>>>>> -Angus
>>>>>> 
>>>>>> 
>>>>>>> Clark
>>>>>> 
>>>>>>> [2]
>>>>>>>> 
>>>>>>>> 
>>> https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/slave_scripts/select-mirror.sh#n54
>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> OpenStack-dev mailing list
>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>> <mailto:OpenStack-dev at lists.openstack.org>
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>> 
>>>>>> 
>>>>>>>> 
>>>>>>>> _______________________________________________
>>>>>>>> OpenStack-dev mailing list
>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>> <mailto:OpenStack-dev at lists.openstack.org>
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> OpenStack-dev mailing list
>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>> <mailto:OpenStack-dev at lists.openstack.org>
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>> 
>>>>> 
>>>>> --
>>>>> Sean Dague
>>>>> http://dague.net
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> 
>>> 
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> 
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> 
>> 
>> 
>> 
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140709/8a187290/attachment-0001.pgp>

More information about the OpenStack-dev mailing list