Open Stack

On 07/08/2014 11:05 PM, Joe Gordon wrote:
> On Tue, Jul 8, 2014 at 8:54 PM, James Polley <jp at jamezpolley.com> wrote:
> 
>> It may not have been clear from the below email, but clarkb clarifies on
>> https://bugs.launchpad.net/openstack-ci/+bug/1294381 that the infra team
>> is no longer maintaining pypi-mirror
>>
>> This has been a very useful tool for tripleo. It's much simpler for new
>> developers to set up and use than a full bandersnatch mirror (and requires
>> less disk space), and it can create a local cache of wheels which saves
>> build time.
>>
>> But it's now unsupported.
>>
>> To me it seems like we have two options:
>>
>> A) Deprecate usage of pypi-mirror; update docs to instruct new devs in
>> setting up a local bandersnatch mirror instead
>> or
>> B) Take on care-and-feeding of the tool.
>> or, I guess,
>> C) Continue to recommend people use an unsupported unmaintained
>> known-buggy tool (it works reasonably well for us today, but it's going to
>> work less and less well as time goes by)
>>
>> Are there other options I haven't thought of?
>>
> 
> I don't know if this fits your requirements but I use
> http://doc.devpi.net/latest/quickstart-pypimirror.html for my development
> needs.

Will that also cache wheels?  In my experience, wheels are one of the
big time savers in tripleo so I would consider it an important feature
to maintain, however we decide to proceed.

> 
> 
>>
>> Do you have thoughts on which option is preferred?
>>
>>
>> ---------- Forwarded message ----------
>> From: Clark Boylan <clark.boylan at gmail.com>
>> Date: Tue, Jul 8, 2014 at 8:50 AM
>> Subject: Re: [openstack-dev] Policy around Requirements Adds (was: New
>> class of requirements for Stackforge projects)
>> To: "OpenStack Development Mailing List (not for usage questions)" <
>> openstack-dev at lists.openstack.org>
>>
>>
>> On Mon, Jul 7, 2014 at 3:45 PM, Joe Gordon <joe.gordon0 at gmail.com> wrote:
>>>
>>> On Jul 7, 2014 4:48 PM, "Sean Dague" <sean at dague.net> wrote:
>>>>
>>>> This thread was unfortunately hidden under a project specific tag (I
>>>> have thus stripped all the tags).
>>>>
>>>> The crux of the argument here is the following:
>>>>
>>>> Is a stackforge project project able to propose additions to
>>>> global-requirements.txt that aren't used by any projects in OpenStack.
>>>>
>>>> I believe the answer is firmly *no*.
>>>
>>> ++
>>>
>>>>
>>>> global-requirements.txt provides a way for us to have a single point of
>>>> vetting for requirements for OpenStack. It lets us assess licensing,
>>>> maturity, current state of packaging, python3 support, all in one place.
>>>> And it lets us enforce that integration of OpenStack projects all run
>>>> under a well understood set of requirements.
>>>>
>>>> The requirements sync that happens after requirements land is basically
>>>> just a nicety for getting openstack projects to the tested state by
>>>> eventual consistency.
>>>>
>>>> If a stackforge project wants to be limited by global-requirements,
>>>> that's cool. We have a mechanism for that. However, they are accepting
>>>> that they will be limited by it. That means they live with how the
>>>> OpenStack project establishes that list. It specifically means they
>>>> *don't* get to propose any new requirements.
>>>>
>>>> Basically in this case Solum wants to have it's cake and eat it to. Both
>>>> be enforced on requirements, and not be enforced. Or some 3rd thing that
>>>> means the same as that.
>>>>
>>>> The near term fix is to remove solum from projects.txt.
>>>
>>> The email included below mentions an additional motivation for using
>>> global-requirements is to avoid using pypi.python.org and instead use
>>> pypi.openstack.org for speed and reliability. Perhaps there is a way we
>> can
>>> support use case for stackforge projects not in projects.txt? I thought I
>>> saw something the other day about adding a full pypi mirror to OpenStack
>>> infra.
>>>
>> This is done. All tests are now run against a bandersnatch built full
>> mirror of pypi. Enforcement of the global requirements is performed
>> via the enforcement jobs.
>>>>
>>>> On 06/26/2014 02:00 AM, Adrian Otto wrote:
>>>>> Ok,
>>>>>
>>>>> I submitted and abandoned a couple of reviews[1][2] for a solution
>> aimed
>>>>> to meet my goals without adding a new per-project requirements file.
>> The
>>>>> flaw with this approach is that pip may install other requirements
>> when
>>>>> installing the one(s) loaded from the fallback mirror, and those may
>>>>> conflict with the ones loaded from the primary mirror.
>>>>>
>>>>> After discussing this further in #openstack-infra this evening, we
>>>>> should give serious consideration to adding python-mistralclient to
>>>>> global requirements. I have posted a review[3] for that to get input
>>>>> from the requirements review team.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Adrian
>>>>>
>>>>> [1] https://review.openstack.org/102716
>>>>> [2] https://review.openstack.org/102719
>>>>> [3] https://review.openstack.org/102738
>>>>> <https://review.openstack.org/1027387>
>>>>>
>>>>> On Jun 25, 2014, at 9:51 PM, Matthew Oliver <matt at oliver.net.au
>>>>> <mailto:matt at oliver.net.au>> wrote:
>>>>>
>>>>>>
>>>>>> On Jun 26, 2014 12:12 PM, "Angus Salkeld" <
>> angus.salkeld at rackspace.com
>>>>>> <mailto:angus.salkeld at rackspace.com>> wrote:
>>>>>>>
>>>>> On 25/06/14 15:13, Clark Boylan wrote:
>>>>>> On Tue, Jun 24, 2014 at 9:54 PM, Adrian Otto
>>>>>>> <adrian.otto at rackspace.com <mailto:adrian.otto at rackspace.com>>
>> wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> Solum has run into a constraint with the current scheme for
>>>>>>> requirements management within the OpenStack CI system. We have a
>>>>>>> proposal for dealing with this constraint that involves making a
>>>>>>> contribution to openstack-infra. This message explains the
>> constraint,
>>>>>>> and our proposal for addressing it.
>>>>>>>
>>>>>>> == Background ==
>>>>>>>
>>>>>>> OpenStack uses a list of global requirements in the requirements
>>>>>>> repo[1], and each project has it’s own requirements.txt and
>>>>>>> test-requirements.txt files. The requirements are satisfied by gate
>>>>>>> jobs using pip configured to use the pypi.openstack.org
>>>>>>> <http://pypi.openstack.org/> mirror, which is periodically updated
>>>>>>> with new content from pypi.python.org <http://pypi.python.org/>.
>> One
>>>>>>> motivation for doing this is that pypi.python.org
>>>>>>> <http://pypi.python.org/> may not be as fast or as reliable as a
>> local
>>>>>>> mirror. The gate/check jobs for the projects use the OpenStack
>>>>>>> internal pypi mirror to ensure stability.
>>>>>>>
>>>>>>> The OpenStack CI system will sync up the requirements across all
>>>>>>> the official projects and will create reviews in the participating
>>>>>>> projects for any mis-matches. Solum is one of these projects, and
>>>>>>> enjoys this feature.
>>>>>>>
>>>>>>> Another motivation is so that users of OpenStack will have one
>>>>>>> single set of python package requirements/dependencies to install
>> and
>>>>>>> run the individual OpenStack components.
>>>>>>>
>>>>>>> == Problem ==
>>>>>>>
>>>>>>> Stackforge projects listed in openstack/requirements/projects.txt
>>>>>>> that decide to depend on each other (for example, Solum wanting to
>>>>>>> list mistralclient as a requirement) are unable to, because they are
>>>>>>> not yet integrated, and are not listed in
>>>>>>> openstack/requirements/global-requirements.txt yet. This means that
>> in
>>>>>>> order to depend on each other, a project must withdraw from
>>>>>>> projects.txt and begin using pip with pypi.poython.org
>>>>>>> <http://pypi.poython.org/> to satisfy all of their requirements.I
>>>>>>> strongly dislike this option.
>>>>>>>
>>>>>>> Mistral is still evolving rapidly, and we don’t think it makes
>>>>>>> sense for them to pursue integration wight now. The upstream
>>>>>>> distributions who include packages to support OpenStack will also
>>>>>>> prefer not to deal with a requirement that will be cutting a new
>>>>>>> version every week or two in order to satisfy evolving needs as
>> Solum
>>>>>>> and other consumers of Mistral help refine how it works.
>>>>>>>
>>>>>>> == Proposal ==
>>>>>>>
>>>>>>> We want the best of both worlds. We want the freedom to innovate
>>>>>>> and use new software for a limited selection of stackforge projects,
>>>>>>> and still use the OpenStack pypi server to satisfy my regular
>>>>>>> requirements. We want the speed and reliability of using our local
>>>>>>> mirror, and users of Solum to use a matching set of requirements for
>>>>>>> all the things that we use, and integrated projects use. We want to
>>>>>>> continue getting the reviews that bring us up to date with new
>>>>>>> requirements versions.
>>>>>>>
>>>>>>> We propose that we submit an enhancement to the gate/check job
>>>>>>> setup that will:
>>>>>>>
>>>>>>> 1) Begin (as it does today) by satisfying global-requirements.txt
>>>>>>> and my local project’s requirements.txt and test-requirements.txt
>>>>>>> using the local OpenStack pypi mirror.
>>>>>>> 2) After all requirements are satisfied, check the name of my
>>>>>>> project. If it begins with ‘stackforge/‘ then look for a
>>>>>>> stackforge-requirements.txt file. If one exists, reconfigure pip to
>>>>>>> switch to use pypi.python.org <http://pypi.python.org/>, and
>> satisfy
>>>>>>> the requirements listed in the file. We will list mistralclient
>> there,
>>>>>>> and get the latest tagged/released version of that.
>>>>>>>
>>>>>> I am reasonably sure that if you remove yourself from the
>>>>>> openstack/requirements project list this is basically how it will
>>>>>> work. Pip is configured to use the OpenStack mirror and fall back on
>>>>>> pypi.python.org <http://pypi.python.org/> for packages not
>>>>>>> available on the OpenStack mirror
>>>>>> [2]. So I don't think there is any work to do here with additional
>>>>>> requirements files. It should just work. Adding a new requirements
>>>>>> file will just make things more confusing for packagers and consumers
>>>>>> of your software.
>>>>>
>>>>> Adrian I know this is not the optimal solution, but I think this is
>>>>> the most pragmatic solution (esp. given we need to progress and not
>>>>>>> be held
>>>>> up by this), most stackforge projects are in the same boat as us.
>>>>> As far as pypi breakages (most are easily fixable by restricting the
>>>>> package versions if we get an issue with a new release
>>>>> of *random-api-breaking-package*).
>>>>>
>>>>>>>
>>>>>>> I've looked through the infra choose mirror code, and Clark is
>>>>>>> correct. If the project isn't in the projects.txt file they will
>> only
>>>>>>> access to pypi.openstack.org <http://pypi.openstack.org/> however
>> if
>>>>>>> removed then it will first check pypi.openstack.org
>>>>>>> <http://pypi.openstack.org/> and then fall back to to
>> pypi.python.org
>>>>>>> <http://pypi.python.org/>. I think the only real solution is what
>>>>>>> Angus mentioned, remove yourself from projects.txt at least until
>> all
>>>>>>> your dependencies can be provided by pypi.openstack.org
>>>>>>> <http://pypi.openstack.org/> or another solution is put into
>> place. In
>>>>>>> the mean time you can at least progress and continue development.
>>>>>>>
>>>>>>> If you code requires a direct dependency (rather then an optional
>>>>>>> dependency) of some non integrated project, then your stuck until
>> they
>>>>>>> are.
>>>>>>>
>>>>>>>
>>>>>
>>>>>>>
>>>>>>> == Call To Action ==
>>>>>>>
>>>>>>> What do you think of this approach to satisfy a balance of
>>>>>>> interests? Everything remains the same for OpenStack projects, and
>>>>>>> Stackforge projects get a new feature that allows them to require
>>>>>>> software that has not yet been integrated. Are there even better
>>>>>>> options that we should consider?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Adrian Otto
>>>>>>>
>>>>>>>
>>>>>>> References:
>>>>>>> [1] https://review.openstack.org/openstack/requirements
>>>>>
>>>>>> For what it is worth the Infra team has also been looking at
>>>>>> potentially using something like bandersnatch to mirror all of pypi
>>>>>> which is now a possibility because OpenStack doesn't depend on
>>>>>> packages that are hosted external to pypi. We would then do
>>>>>> requirements enforcement via checks rather than explicit use of a
>>>>>> restricted mirror. There are some things to sort out like platform
>>>>>> dependent wheels (I am not sure that any OpenStack project directly
>>>>>> consumes these but I have found them to be quite handy) and the
>>>>>> potential need for more enforcement to keep this working, but I think
>>>>>> this is a possibility.
>>>>>
>>>>> This would be neat.
>>>>>
>>>>> -Angus
>>>>>
>>>>>
>>>>>> Clark
>>>>>
>>>>>> [2]
>>>>>>>
>>>>>>>
>> https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/slave_scripts/select-mirror.sh#n54
>>>>>
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>> <mailto:OpenStack-dev at lists.openstack.org>
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> OpenStack-dev mailing list
>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> <mailto:OpenStack-dev at lists.openstack.org>
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/
>>>>>>
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> <mailto:OpenStack-dev at lists.openstack.org>
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>> --
>>>> Sean Dague
>>>> http://dague.net
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>