Open Stack

On 27/05/15 09:14, Thomas Goirand wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Hi all,
>
> tl;dr:
> - - We'd like to push distribution packaging of OpenStack on upstream
> gerrit with reviews.
> - - The intention is to better share the workload, and improve the overall
> QA for packaging *and* upstream.
> - - The goal is *not* to publish packages upstream
> - - There's an ongoing discussion about using stackforge or openstack.
> This isn't, IMO, that important, what's important is to get started.
> - - There's an ongoing discussion about using a distribution specific
> namespace, my own opinion here is that using /openstack-pkg-{deb,rpm} or
> /stackforge-pkg-{deb,rpm} would be the most convenient because of a
> number of technical reasons like the amount of Git repository.
> - - Finally, let's not discuss for too long and let's do it!!! :)
>
> Longer version:
>
> Before I start: some stuff below is just my own opinion, others are just
> given facts. I'm sure the reader is smart enough to guess which is what,
> and we welcome anyone involved in the project to voice an opinion if
> he/she differs.
>
> During the Vancouver summit, operation, Canonical, Fedora and Debian
> people gathered and collectively expressed the will to maintain
> packaging artifacts within upstream OpenStack Gerrit infrastructure. We
> haven't decided all the details of the implementation, but spent the
> Friday morning together with members of the infra team (hi Paul!) trying
> to figure out what and how.
>
> A number of topics have been raised, which needs to be shared.
>
> First, we've been told that such a topic deserved a message to the dev
> list, in order to let groups who were not present at the summit. Yes,
> there was a consensus among distributions that this should happen, but
> still, it's always nice to let everyone know.
>
> So here it is. Suse people (and other distributions), you're welcome to
> join the effort.
>
> - - Why doing this
> ================
> It's been clear to both Canonical/Ubuntu teams, and Debian (ie: myself)
> that we'd be a way more effective if we worked better together, on a
> collaborative fashion using a review process like on upstream Gerrit.
> But also, we'd like to welcome anyone, and especially the operation
> folks, to contribute and give feedback. Using Gerrit is the obvious way
> to give everyone a say on what we're implementing.
>
> As OpenStack is welcoming every day more and more projects, it's making
> even more sense to spread the workload.
>
> This is becoming easier for Ubuntu guys as Launchpad now understand not
> only BZR, but also Git.
>
> We'd start by merging all of our packages that aren't core packages
> (like all the non-OpenStack maintained dependencies, then the Oslo libs,
> then the clients). Then we'll see how we can try merging core packages.
>
> Another reason is that we believe working with the infra of OpenStack
> upstream will improve the overall quality of the packages. We want to be
> able to run a set of tests at build time, which we already do on each
> distribution, but now we want this on every proposed patch. Later on,
> when we have everything implemented and working, we may explore doing a
> package based CI on every upstream patch (though, we're far from doing
> this, so let's not discuss this right now please, this is a very long
> term goal only, and we will have a huge improvement already *before*
> this is implemented).
>
> - - What it will *not* be
> =======================
> We do not have the intention (yet?) to publish the resulting packages
> built on upstream infra. Yes, we will share the same Git repositories,
> and yes, the infra will need to keep a copy of all builds (for example,
> because core packages will need oslo.db to build and run unit tests).
> But we will still upload on each distributions on separate repositories.
> So published packages by the infra isn't currently discussed. We could
> get to this topic once everything is implemented, which may be nice
> (because we'd have packages following trunk), though please, refrain to
> engage in this topic right now: having the implementation done is more
> important for the moment. Let's try to stay on tracks and be constructive.
>
> - - Let's keep efficiency in mind
> ===============================
> Over the last few years, I've been able to maintain all of OpenStack in
> Debian with little to no external contribution. Let's hope that the
> Gerrit workflow will not slow down too much the packaging work, even if
> there's an unavoidable overhead. Hopefully, we can implement some
> liberal ACL policies for the core reviewers so that the Gerrit workflow
> don't slow down anyone too much. For example we may be able to create
> new repositories very fast, and it may be possible to self-approve some
> of the most trivial patches (for things like typo in a package
> description, adding new debconf translations, and such obvious fixes, we
> shouldn't waste our time).
>
> There's a middle ground between the current system (ie: only write
> access ACLs for git.debian.org with no other check what so ever) and a
> too restrictive fully protected gerrit workflow that may slow down
> everyone too much. Currently, there's a small amount of people involved
> in the packaging. While we can expect a lot of operators will be
> interested to work on core packages like Nova, Neutron and so on, I am
> at the same time expecting that nobody will care about a given indirect
> python module dependency (and I may still continue to be the only one
> working on these...). We really don't want to be stuck in situations
> with nobody reviewing these.
>
> - - /stackforge or /openstack namespace
> =====================================
> There's been this question floating around. Should we try joining
> directly as part of OpenStack, as many suggested, or should we go
> through a first round (during one dev cycle, until Liberty is released?)
> through stackforge.
>
> I have to admit that I'm not strongly opinionated about this myself.
> Sure, being an official OpenStack big-tent project would be nice, and it
> would feel worm, but also there's the issue that we don't have any
> commits within the project yet, and therefore finding who's core and
> who's the PTL could be an issue if we want to follow the official
> guidelines.
>
> However, without even discussing the topic between us, it's obvious that
> people like James Page (plus others from his team) and myself would be
> core approvers for the Debian packaging. Fedora guys can decide for
> themselves (I must write names I have in mind for the RPM based-OS side
> of things: M. Runge, Haikel Gemmar and Alan Pevec, for example. Please
> add the missing ones here...).
>
> Another thing is that going for stackforge now means that one day, we'll
> have to suffer from a migration to the openstack namespace, which may be
> 1/ a lot of work and 2/ disruptive. Avoiding such a migration would be good.
>
> All together, the packaging team will happily accept whatever is decided
> by the TC. What counts for us is that a decision is taken quickly, so
> that we can start implementing it and share our repositories. So, dear
> TC, please let us know ASAP after your next meeting.
>
> Anyhow, please feel free to discuss the issue within the governance
> patch which I started:
> https://review.openstack.org/#/c/185187/
>
> - - Specific packaging namespace
> ==============================
> Since distribution packaging through Git implies running a lot of git
> repository (one per package), it's been discussed that we may want to
> use a specific namespace for distributions. For example,
> /stackforge-packaging or /openstack-packaging.
>
> Let's keep in mind that, for OpenStack packaging by the distributions,
> we're talking about more than 200 packages. 235 so far on the OpenStack
> packaging group in Debian [1], and it's increasing all the time...
>
> Personally, I believe it'd be nice to have a specifc namespace for
> distributions, for multiple reasons. The first one is that we already
> have a lot of projects inside /stackforge or /openstack, and that
> searching through them isn't easy. Now, consider that we'll have
> duplicates for each and every project. Also, we don't have a complete
> match between OpenStack package names. For example, in Debian/Ubuntu, we
> have "python-" as prefix for each and every python libraries (including
> for example all oslo packages, but not only). Plus we'd be forced to use
> a prefix, for example "pkg-nova", to avoid collision. However, it'd be
> nice to keep the same repository names as per the name of the source
> package in the distributions, otherwise we'd have to fix our tooling
> which obviously will become slightly more complex.
>
> - - Distribution specific git repositories or specific branches
> =============================================================
> There's 2 issues if we keep the same repository names for multiple
> distribution.
>
> First, we'd like to have a different set of core reviewers for a given
> distribution family. This implies that, if we use the same repository
> for RPM and DEB, then we'd have to set ACLs on per-branch basis, which
> may be harder to maintain (compared to giving access to the full set of
> Git repositories for a given distro family). Yes, we could give access
> to both for core reviewers, and trust them to behave. This trust model
> has been proved as successful within the Debian community, but still,
> adding and removing ACLs for new core reviewers would be harder to maintain.
>
> Then, as written above, it is desirable to have a matching pair between
> distribution source package names and the name of the underlying git
> repository (so that for building X, you git clone X, not prefix-X).
> However, distributions don't have the same names for packages. For
> example, RPM supports upper case, while dpkg systems don't. Also, Red
> Hat uses a prefix "openstack-" for all core projects (like
> "openstack-nova"), while Debian based distribution don't, and so on...
>
> - - Finally
> =========
> Finally, well, please don't discuss the color of the bike-shed for too
> long. We don't really care the color as long as we have a shed to share
> the hosting of our packages Git.
>
> Yes, I'm open to discussing all of the issues above, but if and only if
> it doesn't become a blocker for implementing packaging on upstream
> Gerrit. For the /openstack namespace, can we agree to set a deadline in
> 6 days, before the next TC meeting? We really need to get started soon,
> at the beginning of the liberty cycle, so that we can achieve important
> tasks like merging our source packages between Debian and Ubuntu (for
> example) and still be on-time for the release of Liberty. The task will
> be non-trivial, if we want to accommodate all parties, and retain the
> features our users had enjoyed over the past few years.
>
> Last word: thanks to everyone on the infra team (and others) which were
> supportive of the project. All of this is very exciting, and it's really
> great that this is finally happening.

This all sounds great thanks for kicking it off

Ihar gave some good points of how we are doing most of this in RDO, I'll 
summarize this a bit here and add a proposal of how I think we can make 
some more progress towards your goal

As part of the RDO project we have been doing pretty much most of what 
you described for the last 6 months, because of the number of git 
repositories involved we decided to trial it on github[1] using 
gerrithub[2] for peer review. Effectively we now have a yum repository 
representing the collection of openstack projects per upstream commit 
for both Fedora and Centos[3][4]

After some initial teething problems I think we have settled on a 
process that appears to be working.

The tool which we are using to tie the flow together is called 
delorean[5], essentially it monitors git repositories and triggers a 
build when a commit occurs on either the upstream project or the 
packaging itself, it shouldn't be too hard to expand it to build deb's 
in what ever way is most appropriate. The main thing I think that is 
important here is that we all agree on a common flow. We will also work 
on adding support so it can build packages based on ZUUL REFs so we can 
support CI triggered by upstream gerrit.


I propose we do the following,

1. rename all of the packaging repositories in[1] to prefix them with 
something like rdorpm-

2. Approach infra to explore the possibility of having a new namespace 
for this, I think the number of repositories would justify it (depending 
on what dependencies are needed it could be well over 100 per distro), 
if they agree then we could give infra control of the 
"openstack-packages" github organization, or alternatively create a new one.

3. for rpm trunk builds we can point our tools at the new location and 
continue as we have been. This would have to be changed in rdoinfo[6] 
and is currently RDO specific so so we would need some work here if 
adding support to delorean for other distros.

4. For deb packages you can create new repositories along side the 
rdorpm-* repositories

 From a packaging point of view I think the collaboration may stop there 
but from the point of view of flow around the packaging we can (and 
should) continue to collaborate e.g.

   5. Add deb support to delorean, I know of at least one person who has 
already explored this (steve cc'd), if delorean is too far off the path 
of what we want to achieve and there is a better tool then I'm open to 
change.

   6. Explore what ci jobs could be run against the packaging

The trunk repositories have proven very valuable to us over the last few 
months, I hope we can make this work in a way that will be valuable to a 
wider audience. How would this sound to you?

thanks,
Derek.

[1] - https://github.com/openstack-packages
[2] - https://review.gerrithub.io/#/q/project:^openstack-packages/.*,n,z
[3] - http://trunk.rdoproject.org/f21/report.html
[4] - http://trunk.rdoproject.org/centos/report.html
[5] - https://github.com/openstack-packages/delorean
[6] - https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml



>
> Cheers,
>
> Thomas Goirand (zigo)
>
> [1]
> https://qa.debian.org/developer.php?login=openstack-devel@lists.alioth.debian.org
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
>
> iQIcBAEBCAAGBQJVZXzYAAoJENQWrRWsa0P+M8gP/21TcEfBs/PTxUFRDMSfP97a
> JCVj5JIy6GCFjDC+8mbX9WGWFEbU0hRn2CG/dRJNbcrkjeQJaDosSd3BMmEAsrQO
> ONyu0Qbm2YgclZ9OyUaewE8NPSH8cGRjxS+IdOm5kzhNqX538ohsxD2XNMknx5rU
> cyq3TTDKWL3ot8+SVdUqfvzmZZdESF+8lucneqOiRk9wU/UVt6fdsovuv3QSqM2h
> k7etO4Oev46nQFCVvSmofv2mfGTWn5iWDbcCj7qf/blI5uiL4GkTkzFbkr/+f2De
> 7C4p9rO8WsEaK+Ptg2hayiOm36i6JPnvO6YSOjm6yTeY4LQ3UkULN2/THvftSkYX
> I+oICj7XbJMhaUDbg9mfJhVZ+O5L0v9YHV2v12NIWMbDGb6EzSYsLqmvANwYN5wM
> Y8zihyETZFYsJVmPdqHN6ZPGeuKAroFo0F5rldMJ/++J5exApf6DJyd+JrE7vmyV
> F3sksS+BhmPgmEBbSvcA0ZP8gqEfPWMK5zT1rWBakKy1OHtDaLwnzsCAkaS6kXUJ
> G45Sr965H6pkO10EMpIbjFeCcVIP7UpmHDxq1PQbhUx4aogjVGA1mIWwuD0Cscaj
> hD18bJyZxe0t0MzhkijxWnQ6holJ1C36SvSD/jha6oZ7k9FHIAwo1MklTJcCEo7J
> PBnrpJTi9IkhDmfp21/s
> =5T0x
> -----END PGP SIGNATURE-----
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>