Open Stack

On Wed, Jan 14, 2015, at 04:50 AM, Sean Dague wrote:
> On 01/14/2015 03:55 AM, Flavio Percoco wrote:
> > On 13/01/15 14:31 -0500, Sean Dague wrote:
> >> On 01/13/2015 02:10 PM, Jay Pipes wrote:
> >>> On 01/13/2015 12:39 PM, Zane Bitter wrote:
> >>>> On 13/01/15 10:01, Jeremy Stanley wrote:
> >>>>> On 2015-01-13 07:46:38 -0500 (-0500), Sean Dague wrote:
> >>>>>> Why doesn't rally just remove itself from projects.txt, then there
> >>>>>> would be no restrictions on what it adds.
> >>>>>
> >>>>> I second this recommendation. If Rally wants to depend on things
> >>>>> which are not part of OpenStack and not required to run or test
> >>>>> OpenStack in an official capacity (and since it's not an official
> >>>>> OpenStack project it's entirely within its rights to want to do
> >>>>> that), then forcing it to comply with the list of requirements for
> >>>>> official OpenStack projects is an unnecessarily restrictive choice.
> >>>>
> >>>> FWIW I don't really agree with this advice. The purpose of
> >>>> openstack/requirements is to ensure that it's possible to create a
> >>>> distribution of OpenStack without conflicting version requirements, not
> >>>> to force every project to use every dependency listed. As such, some
> >>>> co-ordination around client libraries for related projects seems
> >>>> like it
> >>>> ought to be an uncontroversial addition. (Obviously it's easy to
> >>>> imagine
> >>>> potential additions that would legitimately be highly controversial,
> >>>> but
> >>>> IMHO this isn't one of them.) Saying that we require people to use our
> >>>> tools to get into the club but that our tools are not going to
> >>>> accommodate them in any way until they are in sounds a bit too close to
> >>>> "Go away" to my ears.
> >>>
> >>> +1
> >>
> >> I think as we grow global-requirements probably needs to be broken up
> >> into functional lists. What's allowed in base-iass, what's allowed in
> >> application services, what's allowed in docs, etc, etc.
> >>
> >> The complexity of keeping the g-r list actually working goes up sort of
> >> n^2 as the size of it, because pip doesn't have a real solver. This is a
> >> barely functional set of plate spinning so "just add everything" misses
> >> the point that the breaks get more and more difficult to unwind.
> >>
> >> If someone is up for stepping up and contributing to breaking up into
> >> "domains", please do. But I think while this remains a global list we
> >> really do need to be a bit careful here.
> > 
> > Do we really need a per-domain g-r?
> > 
> > With the upcoming changes in the openstack governance model, I believe
> > this won't scale even if we break it into per-domain basis. Some
> > questions that need answeres are:
> > 
> > - Who's going to review the per domain g-r ?
> > 
> > If it's a centralized team, then it won't definitely scale. If it's
> > the domain team then, what's the real point of having these
> > requirements?
> > 
> > - How are we going to support the creation and maintnance of these g-r
> >  in the gate? Are they actually worth it?
> > 
> > 
> > While I understand why we have g-r, I really don't think it'll scale
> > for a broader community as we're envisioning it. With that in mind,
> > I'd probably recommend to have 1 requirements group for the base-caas
> > integrated gate and let other projects and groups have their own
> > requirements. That is, non base-caas projects should get the versions
> > of their common requirements from g-r so that they won't conflict if
> > they're installed alongside with any of the base-caas projects. But
> > other than those base requirements, I'd let non base-caas projects
> > have what they need (which is pretty much what heat's team has done
> > with the contrib packages, AFAICT).
> > 
> > I know this opens the doors for version conflicts between the
> > requirements of non base-caas projects but I think this is a more
> > welcoming (and non-blocking) way to do things until we've a better
> > one.
> > 
> > Hope the above makes some sense.... I blame the lack of coffee if it
> > doesn't,
> > Flavio
> 
> We have the base infrastructure of that in devstack today with a SOFT
> requirements update -
> https://github.com/openstack-dev/devstack/blob/master/functions-common#L1696-L1704
> 
> If you are not in projects.txt, and you set your REQUIREMENTS_MODE=soft
> in your devstack run, we will leave extra dependencies untouched.
> Probably a few more bits required to make it really easy to expose, but
> that direction is also feasable.
> 
> The reason I brought up domains though is that some groups really want
> the facility to have common requirements lists across a domain. Like the
> OpenStack Docs team. They've currently inserted a ton of stuff in g-r
> that really shouldn't be there because they have a lot of git trees, and
> the synchronization is a nice feature.
> 
> However, if there were domain specific lists, that would be fine. A
> collection of projects that want to coordinate could all agree on a
> domain specific set of lists, and off we go. Maybe that list wouldn't
> even need to be contained in the main repo, it could be in a sub repo so
> have different approvers.
> 
This ml thread came up in the TC meeting today and I am responding here
to catch the thread up with the meeting. The soft update option is the
suggested fix for non openstack projects that want to have most of their
requirements managed by global requirements.

For the project structure reform opening things up we should consider
loosening the criteria to get on the list and make it primarily based on
technical criteria such as py3k support, license compatibility, upstream
support/activity, and so on (basically the current criteria with less of
a focus on where the project comes from if it is otherwise healthy).
Then individual projects would choose the subset they need to depend on.
This model should be viable with different domains as well if we go that
route.

The following is not from the TC meeting but addressing other portions
of this conversation:

At least one concern with this option is that as the number of total
requirements goes up is the difficulty in debugging installation
conflicts becomes more difficult too. I have suggested that we could
write tools to help with this. Install bisection based on pip logs for
example, but these tools are still theoretical so I may be
overestimating their usefulness.

To address the community scaling aspect I think you push a lot of work
back on deployers/users if we don't curate requirements for anything
that ends up tagged as "production ready" (or whatever the equivalent
tag becomes). Essentially we are saying "this doesn't scale for us so
now you deal with the fallout. Have fun", which isn't very friendly to
people consuming the software. We already have an absurd number of
requirements and management of them has appeared to scale. I don't
foresee my workload going up if we open up the list as suggested.

Thanks,
Clark