Open Stack

I'll follow-up on the spec, but one thing Donald has been pointing out
for a while is that we don't use requirements.txt the way that pip
anticipates: the expected use is that a specific install (e.g. the
gate) will have a very specific list of requirements, caps etc, but
that the install_requires will be as minimal as possible to ensure the
project builds and self-tests ok.

I see the issues here as being related to that.

-Rob

On 26 February 2015 at 10:04, Joe Gordon <joe.gordon0 at gmail.com> wrote:
>
>
> On Tue, Feb 24, 2015 at 7:00 AM, Doug Hellmann <doug at doughellmann.com>
> wrote:
>>
>>
>>
>> On Mon, Feb 23, 2015, at 06:31 PM, Joe Gordon wrote:
>> > On Mon, Feb 23, 2015 at 11:04 AM, Doug Hellmann <doug at doughellmann.com>
>> > wrote:
>> >
>> > >
>> > >
>> > > On Mon, Feb 23, 2015, at 12:26 PM, Joe Gordon wrote:
>> > > > On Mon, Feb 23, 2015 at 8:49 AM, Ihar Hrachyshka
>> > > > <ihrachys at redhat.com>
>> > > > wrote:
>> > > >
>> > > > > -----BEGIN PGP SIGNED MESSAGE-----
>> > > > > Hash: SHA1
>> > > > >
>> > > > > On 02/20/2015 07:16 PM, Joshua Harlow wrote:
>> > > > > > Sean Dague wrote:
>> > > > > >> On 02/20/2015 12:26 AM, Adam Gandelman wrote:
>> > > > > >>> Its more than just the naming.  In the original proposal,
>> > > > > >>> requirements.txt is the compiled list of all pinned deps
>> > > > > >>> (direct and transitive), while
>> > > > > >>> requirements.in<http://requirements.in>  reflects what people
>> > > > > >>> will actually use.  Whatever is in requirements.txt affects
>> > > > > >>> the
>> > > > > >>> egg's requires.txt. Instead, we can keep requirements.txt
>> > > > > >>> unchanged and have it still be the canonical list of
>> > > > > >>> dependencies, while
>> > > > > >>> reqiurements.out/requirements.gate/requirements.whatever is an
>> > > > > >>> upstream utility we produce and use to keep things sane on our
>> > > > > >>> slaves.
>> > > > > >>>
>> > > > > >>> Maybe all we need is:
>> > > > > >>>
>> > > > > >>> * update the existing post-merge job on the requirements repo
>> > > > > >>> to produce a requirements.txt (as it does now) as well the
>> > > > > >>> compiled version.
>> > > > > >>>
>> > > > > >>> * modify devstack in some way with a toggle to have it process
>> > > > > >>> dependencies from the compiled version when necessary
>> > > > > >>>
>> > > > > >>> I'm not sure how the second bit jives with the existing
>> > > > > >>> devstack installation code, specifically with the libraries
>> > > > > >>> from git-or-master but we can probably add something to warm
>> > > > > >>> the system with dependencies from the compiled version prior
>> > > > > >>> to
>> > > > > >>> calling pip/setup.py/etc <http://setup.py/etc>
>> > > > > >>
>> > > > > >> It sounds like you are suggesting we take the tool we use to
>> > > > > >> ensure that all of OpenStack is installable together in a
>> > > > > >> unified
>> > > > > >> way, and change it's installation so that it doesn't do that
>> > > > > >> any
>> > > > > >> more.
>> > > > > >>
>> > > > > >> Which I'm fine with.
>> > > > > >>
>> > > > > >> But if we are doing that we should just whole hog give up on
>> > > > > >> the
>> > > > > >> idea that OpenStack can be run all together in a single
>> > > > > >> environment, and just double down on the devstack venv work
>> > > > > >> instead.
>> > > > > >
>> > > > > > It'd be interesting to see what a distribution (canonical,
>> > > > > > redhat...) would think about this movement. I know yahoo! has
>> > > > > > been
>> > > > > > looking into it for similar reasons (but we are more flexibly
>> > > > > > then
>> > > > > > I think a packager such as canonical/redhat/debian/...
>> > > > > > would/culd
>> > > > > > be). With a move to venv's that seems like it would just offload
>> > > > > > the work to find the set of dependencies that work together (in
>> > > > > > a
>> > > > > > single-install) to packagers instead.
>> > > > > >
>> > > > > > Is that ok/desired at this point?
>> > > > > >
>> > > > >
>> > > > > Honestly, I failed to track all the different proposals. Just
>> > > > > saying
>> > > > > from packager perspective: we absolutely rely on requirements.txt
>> > > > > not
>> > > > > being a list of hardcoded values from pip freeze, but present us a
>> > > > > reasonable freedom in choosing versions we want to run in packaged
>> > > > > products.
>> > > > >
>> > > > >
>> > > > in short the current proposal for stable branches is:
>> > > >
>> > > > keep requirements.txt as is, except maybe put some upper bounds on
>> > > > the
>> > > > requirements.
>> > > >
>> > > > Add requirements.gate to specify the *exact* versions we are gating
>> > > > against
>> > > > (this would be a full list including all transitive dependencies).
>> > >
>> > > The gate syncs requirements into projects before installing them.
>> > > Would
>> > > we change the sync script for the gate to work from the
>> > > requirements.gate file, or keep it pulling from requirements.txt?
>> > >
>> >
>> > We would only add requirements.gate for stable branches (because we
>> > don't
>> > want to cap/pin  things on master). So I think the answer is sync script
>> > should work for both.  I am not sure on the exact mechanics of how this
>> > would work. Whoever ends up driving this bit of work (I think Adam G),
>> > will
>> > have to sort out the details.
>>
>> OK. I think it's probably worth a spec, then, so we can think it through
>> before starting work. Maybe in the cross-project specs repo, to avoid
>> having to create one just for requirements? Or we could modify the
>> README or something, but the specs repo seems more visible.
>
>
> Start of the cross project spec https://review.openstack.org/159249
>
>>
>>
>> Doug
>>
>> >
>> >
>> > > Doug
>> > >
>> > > >
>> > > >
>> > > > > That's why I asked before we should have caps and not pins.
>> > > > >
>> > > > > /Ihar
>> > > > > -----BEGIN PGP SIGNATURE-----
>> > > > > Version: GnuPG v1
>> > > > >
>> > > > > iQEcBAEBAgAGBQJU61oJAAoJEC5aWaUY1u57T7cIALySnlpLV0tjrsTH2gZxskH+
>> > > > > zY+L6E/DukFNZsWxB2XSaOuVdVaP3Oj4eYCZ2iL8OoxLrBotiOYyRFH29f9vjNSX
>> > > > > h++dErBr0SwIeUtcnEjbk9re6fNP6y5Hqhk1Ac+NSxwL75KlS3bgKnJAhLA08MVB
>> > > > > 5xkGRR7xl2cuYf9eylPlQaAy9rXPCyyRdxZs6mNjZ2vlY6hZx/w/Y7V28R/V4gO4
>> > > > > qsvMg6Kv+3urDTRuJdEsV6HbN/cXr2+o543Unzq7gcPpDYXRFTLkpCRV2k8mnmA1
>> > > > > pO9W10F1FCQZiBnLk0c6OypFz9rQmKxpwlNUN5MTMF15Et6DOxGBxMcfr7TpRaQ=
>> > > > > =WHOH
>> > > > > -----END PGP SIGNATURE-----
>> > > > >
>> > > > >
>> > >
>> > > __________________________________________________________________________
>> > > > > OpenStack Development Mailing List (not for usage questions)
>> > > > > Unsubscribe:
>> > > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > > > >
>> > > >
>> > >
>> > > __________________________________________________________________________
>> > > > OpenStack Development Mailing List (not for usage questions)
>> > > > Unsubscribe:
>> > > > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >
>> > >
>> > > __________________________________________________________________________
>> > > OpenStack Development Mailing List (not for usage questions)
>> > > Unsubscribe:
>> > > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >
>> >
>> > __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud