[openstack-dev] [tripleo][releases] Remove diskimage-builder from releases
greg at greghaynes.net
Tue Apr 19 20:11:49 UTC 2016
On Tue, Apr 19, 2016, at 10:25 AM, Doug Hellmann wrote:
> Excerpts from Jeremy Stanley's message of 2016-04-19 15:41:26 +0000:
> > On 2016-04-19 09:22:57 -0400 (-0400), Doug Hellmann wrote:
> > > Excerpts from Ian Wienand's message of 2016-04-19 12:11:35 +1000:
> > [...]
> > > > I don't expect the stable release team to be involved with all this;
> > > > but if we miss windows then we're left either going to efforts getting
> > > > one of a handful of people with permissions to do manual rebuilds or
> > > > waiting yet another day to get something fixed. Add some timezones
> > > > into this, and simple fixes are taking many days to get into builds.
> > > > Thus adding points where we can extend this by another 24 hours
> > > > really, well, sucks.
> > >
> > > How often does that situation actually come up?
> > Semi-often. The project is officially under TripleO but it's sort of
> > a shared jurisdiction between some TripleO and Infra contributors. I
> > think the release team for diskimage-builder used to shoot for
> > tagging weekly (sans emergencies), though that's slacked off a bit
> > and is more like every 2 weeks lately.
> That's about the same as or less often than we tag Oslo libraries.
> > DIB is an unfortunate combination of a mostly stable framework and a
> > large pre-written set of scripts and declarative data which is
> > constantly evolving for widespread use outside the OpenStack
> > ecosystem (so most of the change volume is to the latter). As Ian
> > points out, the Infra team has already been tempted to stop relying
> > on DIB releases at all (or worse, maintain a fork) to reduce overall
> > latency for getting emergency fixes reflected in our worker images.
> Sure, that's a compelling argument. I'm not opposed to making it easier
> for timely releases, just trying to understand the pressure.
> > I suspect that most of the concern over using OpenStack release
> > process for DIB (and similarly Infra projects) is that the added
> > complexities introduce delays, especially if there's not a release
> > team member available to do on-the-spot approvals on weekends and
> > such. I don't know whether extending that to add tagging ACLs for
> > the library-release group would help? That would bring the total up
> > to 6 people, two more of whom are in non-American timezones, so
> > might be worth a try.
> > It's also worth keeping in mind that we've sort of already
> > identified two classes of official OpenStack projects. One is
> > "OpenStack the Product" only able to be distributed under the Apache
> > license and its contributors bound by a contributor license
> > agreement. The other is the output of a loose collective of groups
> > writing ancillary tooling consumed by the OpenStack community and
> > also often used for a lot of other things completely unrelated to
> > OpenStack. I can see where strict coordinated release process and
> > consistency for the former makes sense, but a lot of projects in the
> > latter category likely see it as unnecessary overkill for their
> > releases.
> It's not just about control, it's also about communication. One of
> the most frequent refrains we hear is "what is OpenStack", and one
> way we're trying to answer that is to publicize all of the things
> we release through releases.openstack.org. Centralizing tagging
> also helps us ensure consistent versioning rules, good timing, good
> release announcements, etc.
> Since dib is part of tripleo, and at least 2 other projects depend
> on it directly (sahara-image-elements and manila-image-elements),
> I would expect the tripleo team to want it included on the site,
> to publish release announcements, etc.
> On the other hand, dib is using the release:independent model, which
> indicates that the team in fact doesn't think it should be considered
> part of the "product." Maybe we can use that as our flag for which
> projects should really be managed by the release team and which
> should not, but we don't want projects that want to be part of official
> releases to use that model.
> With what I know today, I can't tell which side of the line dib is
> really on. Maybe someone can clarify?
There is a bit more nuance to getting releases out for downstream
consumers than just getting DIB fixes out quickly. Often there is a
situation where a downstream needs a fix/feature soonish but not
critically - maybe DIB is creating images for infra where networking is
not working due to a dib bug. Infra can delete the last round of images
and still function but its worthwhile to get things fixed soon. In that
case we don't want to just rush a release out the door (there's often
other things which have been merged and carry some risk), and we (or at
least I) like to wait to cut a DIB release until a morning, preferably
when a DIB core is around to help debug / verify the fix and any
fallout. I am not sure there is an easy fix where we can keep this model
without being release:independent - otherwise we would be coordinating
between 3 groups to get a fix out at an ideal time rather than 2, and
the only way I could see this working is for us to give up on releases
being made at a convenient time for DIB cores.
The reason for this is partially what fungi mentioned - DIB has a fair
amount of integration code with the things it installs. Some of this
could be removed out of the in-tree elements which would help, but at
the end of the day DIB will always be responsible for a fair amount of
distro-specific logic to make a workable image. As a result, there will
always be a need for some integration between DIB and downstreams who
gate using it (as with any other tool to build images / packages / etc).
In an ideal world, we would solve this with tighter integration testing
between DIB and its downstreams, but this isn't possible right now due
to the fact that there isn't a way for us to build DIB images in the
gate without hitting a ton of upsteam resources (such as package
mirrors), and as a result the DIB tests are inherently unstable. There
are new package mirrors being worked on, and this will solve some of the
issues, but I think we are a *long* way from DIB tests being stable
enough to be OK to run in a gate for other projects. The best fix we
have been able to come up with for now is to make sure DIB tests arent
integrated in the gate with others, and try and be vocal about potential
breaks with our downstreams.
I hope this clarifies a bit (although maybe it just made things more
confusing given all that text ;)).
More information about the OpenStack-dev