Open Stack

On 05/27/2015 05:26 PM, Derek Higgins wrote:
> On 27/05/15 09:14, Thomas Goirand wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> Hi all,
>>
>> tl;dr:
>> - - We'd like to push distribution packaging of OpenStack on upstream
>> gerrit with reviews.
>> - - The intention is to better share the workload, and improve the
>> overall
>> QA for packaging *and* upstream.
>> - - The goal is *not* to publish packages upstream
>> - - There's an ongoing discussion about using stackforge or openstack.
>> This isn't, IMO, that important, what's important is to get started.
>> - - There's an ongoing discussion about using a distribution specific
>> namespace, my own opinion here is that using /openstack-pkg-{deb,rpm} or
>> /stackforge-pkg-{deb,rpm} would be the most convenient because of a
>> number of technical reasons like the amount of Git repository.
>> - - Finally, let's not discuss for too long and let's do it!!! :)
>>
>> Longer version:
>>
>> Before I start: some stuff below is just my own opinion, others are just
>> given facts. I'm sure the reader is smart enough to guess which is what,
>> and we welcome anyone involved in the project to voice an opinion if
>> he/she differs.
>>
>> During the Vancouver summit, operation, Canonical, Fedora and Debian
>> people gathered and collectively expressed the will to maintain
>> packaging artifacts within upstream OpenStack Gerrit infrastructure. We
>> haven't decided all the details of the implementation, but spent the
>> Friday morning together with members of the infra team (hi Paul!) trying
>> to figure out what and how.
>>
>> A number of topics have been raised, which needs to be shared.
>>
>> First, we've been told that such a topic deserved a message to the dev
>> list, in order to let groups who were not present at the summit. Yes,
>> there was a consensus among distributions that this should happen, but
>> still, it's always nice to let everyone know.
>>
>> So here it is. Suse people (and other distributions), you're welcome to
>> join the effort.
>>
>> - - Why doing this
>> ================
>> It's been clear to both Canonical/Ubuntu teams, and Debian (ie: myself)
>> that we'd be a way more effective if we worked better together, on a
>> collaborative fashion using a review process like on upstream Gerrit.
>> But also, we'd like to welcome anyone, and especially the operation
>> folks, to contribute and give feedback. Using Gerrit is the obvious way
>> to give everyone a say on what we're implementing.
>>
>> As OpenStack is welcoming every day more and more projects, it's making
>> even more sense to spread the workload.
>>
>> This is becoming easier for Ubuntu guys as Launchpad now understand not
>> only BZR, but also Git.
>>
>> We'd start by merging all of our packages that aren't core packages
>> (like all the non-OpenStack maintained dependencies, then the Oslo libs,
>> then the clients). Then we'll see how we can try merging core packages.
>>
>> Another reason is that we believe working with the infra of OpenStack
>> upstream will improve the overall quality of the packages. We want to be
>> able to run a set of tests at build time, which we already do on each
>> distribution, but now we want this on every proposed patch. Later on,
>> when we have everything implemented and working, we may explore doing a
>> package based CI on every upstream patch (though, we're far from doing
>> this, so let's not discuss this right now please, this is a very long
>> term goal only, and we will have a huge improvement already *before*
>> this is implemented).
>>
>> - - What it will *not* be
>> =======================
>> We do not have the intention (yet?) to publish the resulting packages
>> built on upstream infra. Yes, we will share the same Git repositories,
>> and yes, the infra will need to keep a copy of all builds (for example,
>> because core packages will need oslo.db to build and run unit tests).
>> But we will still upload on each distributions on separate repositories.
>> So published packages by the infra isn't currently discussed. We could
>> get to this topic once everything is implemented, which may be nice
>> (because we'd have packages following trunk), though please, refrain to
>> engage in this topic right now: having the implementation done is more
>> important for the moment. Let's try to stay on tracks and be
>> constructive.
>>
>> - - Let's keep efficiency in mind
>> ===============================
>> Over the last few years, I've been able to maintain all of OpenStack in
>> Debian with little to no external contribution. Let's hope that the
>> Gerrit workflow will not slow down too much the packaging work, even if
>> there's an unavoidable overhead. Hopefully, we can implement some
>> liberal ACL policies for the core reviewers so that the Gerrit workflow
>> don't slow down anyone too much. For example we may be able to create
>> new repositories very fast, and it may be possible to self-approve some
>> of the most trivial patches (for things like typo in a package
>> description, adding new debconf translations, and such obvious fixes, we
>> shouldn't waste our time).
>>
>> There's a middle ground between the current system (ie: only write
>> access ACLs for git.debian.org with no other check what so ever) and a
>> too restrictive fully protected gerrit workflow that may slow down
>> everyone too much. Currently, there's a small amount of people involved
>> in the packaging. While we can expect a lot of operators will be
>> interested to work on core packages like Nova, Neutron and so on, I am
>> at the same time expecting that nobody will care about a given indirect
>> python module dependency (and I may still continue to be the only one
>> working on these...). We really don't want to be stuck in situations
>> with nobody reviewing these.
>>
>> - - /stackforge or /openstack namespace
>> =====================================
>> There's been this question floating around. Should we try joining
>> directly as part of OpenStack, as many suggested, or should we go
>> through a first round (during one dev cycle, until Liberty is released?)
>> through stackforge.
>>
>> I have to admit that I'm not strongly opinionated about this myself.
>> Sure, being an official OpenStack big-tent project would be nice, and it
>> would feel worm, but also there's the issue that we don't have any
>> commits within the project yet, and therefore finding who's core and
>> who's the PTL could be an issue if we want to follow the official
>> guidelines.
>>
>> However, without even discussing the topic between us, it's obvious that
>> people like James Page (plus others from his team) and myself would be
>> core approvers for the Debian packaging. Fedora guys can decide for
>> themselves (I must write names I have in mind for the RPM based-OS side
>> of things: M. Runge, Haikel Gemmar and Alan Pevec, for example. Please
>> add the missing ones here...).
>>
>> Another thing is that going for stackforge now means that one day, we'll
>> have to suffer from a migration to the openstack namespace, which may be
>> 1/ a lot of work and 2/ disruptive. Avoiding such a migration would be
>> good.
>>
>> All together, the packaging team will happily accept whatever is decided
>> by the TC. What counts for us is that a decision is taken quickly, so
>> that we can start implementing it and share our repositories. So, dear
>> TC, please let us know ASAP after your next meeting.
>>
>> Anyhow, please feel free to discuss the issue within the governance
>> patch which I started:
>> https://review.openstack.org/#/c/185187/
>>
>> - - Specific packaging namespace
>> ==============================
>> Since distribution packaging through Git implies running a lot of git
>> repository (one per package), it's been discussed that we may want to
>> use a specific namespace for distributions. For example,
>> /stackforge-packaging or /openstack-packaging.
>>
>> Let's keep in mind that, for OpenStack packaging by the distributions,
>> we're talking about more than 200 packages. 235 so far on the OpenStack
>> packaging group in Debian [1], and it's increasing all the time...
>>
>> Personally, I believe it'd be nice to have a specifc namespace for
>> distributions, for multiple reasons. The first one is that we already
>> have a lot of projects inside /stackforge or /openstack, and that
>> searching through them isn't easy. Now, consider that we'll have
>> duplicates for each and every project. Also, we don't have a complete
>> match between OpenStack package names. For example, in Debian/Ubuntu, we
>> have "python-" as prefix for each and every python libraries (including
>> for example all oslo packages, but not only). Plus we'd be forced to use
>> a prefix, for example "pkg-nova", to avoid collision. However, it'd be
>> nice to keep the same repository names as per the name of the source
>> package in the distributions, otherwise we'd have to fix our tooling
>> which obviously will become slightly more complex.
>>
>> - - Distribution specific git repositories or specific branches
>> =============================================================
>> There's 2 issues if we keep the same repository names for multiple
>> distribution.
>>
>> First, we'd like to have a different set of core reviewers for a given
>> distribution family. This implies that, if we use the same repository
>> for RPM and DEB, then we'd have to set ACLs on per-branch basis, which
>> may be harder to maintain (compared to giving access to the full set of
>> Git repositories for a given distro family). Yes, we could give access
>> to both for core reviewers, and trust them to behave. This trust model
>> has been proved as successful within the Debian community, but still,
>> adding and removing ACLs for new core reviewers would be harder to
>> maintain.
>>
>> Then, as written above, it is desirable to have a matching pair between
>> distribution source package names and the name of the underlying git
>> repository (so that for building X, you git clone X, not prefix-X).
>> However, distributions don't have the same names for packages. For
>> example, RPM supports upper case, while dpkg systems don't. Also, Red
>> Hat uses a prefix "openstack-" for all core projects (like
>> "openstack-nova"), while Debian based distribution don't, and so on...
>>
>> - - Finally
>> =========
>> Finally, well, please don't discuss the color of the bike-shed for too
>> long. We don't really care the color as long as we have a shed to share
>> the hosting of our packages Git.
>>
>> Yes, I'm open to discussing all of the issues above, but if and only if
>> it doesn't become a blocker for implementing packaging on upstream
>> Gerrit. For the /openstack namespace, can we agree to set a deadline in
>> 6 days, before the next TC meeting? We really need to get started soon,
>> at the beginning of the liberty cycle, so that we can achieve important
>> tasks like merging our source packages between Debian and Ubuntu (for
>> example) and still be on-time for the release of Liberty. The task will
>> be non-trivial, if we want to accommodate all parties, and retain the
>> features our users had enjoyed over the past few years.
>>
>> Last word: thanks to everyone on the infra team (and others) which were
>> supportive of the project. All of this is very exciting, and it's really
>> great that this is finally happening.
>
> This all sounds great thanks for kicking it off
>
> Ihar gave some good points of how we are doing most of this in RDO, I'll
> summarize this a bit here and add a proposal of how I think we can make
> some more progress towards your goal
>
> As part of the RDO project we have been doing pretty much most of what
> you described for the last 6 months, because of the number of git
> repositories involved we decided to trial it on github[1] using
> gerrithub[2] for peer review. Effectively we now have a yum repository
> representing the collection of openstack projects per upstream commit
> for both Fedora and Centos[3][4]
>
> After some initial teething problems I think we have settled on a
> process that appears to be working.
>
> The tool which we are using to tie the flow together is called
> delorean[5], essentially it monitors git repositories and triggers a
> build when a commit occurs on either the upstream project or the
> packaging itself, it shouldn't be too hard to expand it to build deb's
> in what ever way is most appropriate. The main thing I think that is
> important here is that we all agree on a common flow. We will also work
> on adding support so it can build packages based on ZUUL REFs so we can
> support CI triggered by upstream gerrit.
>
>
> I propose we do the following,
>
> 1. rename all of the packaging repositories in[1] to prefix them with
> something like rdorpm-
>
Not sure I'm a fan of rdorpm, seems too specific to RDO and would not 
foster other people using the git repo for packaging. Personally, I 
simple say rpm- prefix, allowing for branches to be used for distro 
specific changes.

> 2. Approach infra to explore the possibility of having a new namespace
> for this, I think the number of repositories would justify it (depending
> on what dependencies are needed it could be well over 100 per distro),
> if they agree then we could give infra control of the
> "openstack-packages" github organization, or alternatively create a new
> one.
>
> 3. for rpm trunk builds we can point our tools at the new location and
> continue as we have been. This would have to be changed in rdoinfo[6]
> and is currently RDO specific so so we would need some work here if
> adding support to delorean for other distros.
>
+1

> 4. For deb packages you can create new repositories along side the
> rdorpm-* repositories
>
>  From a packaging point of view I think the collaboration may stop there
> but from the point of view of flow around the packaging we can (and
> should) continue to collaborate e.g.
>
>    5. Add deb support to delorean, I know of at least one person who has
> already explored this (steve cc'd), if delorean is too far off the path
> of what we want to achieve and there is a better tool then I'm open to
> change.
>
Personally, I'm a fan of mock. Is there plan to add support for it? 
Also, currently containers are not being used in -infra.  Not saying it 
is a show stopper, but could see some initial planning that is required 
for it.

>    6. Explore what ci jobs could be run against the packaging
>
> The trunk repositories have proven very valuable to us over the last few
> months, I hope we can make this work in a way that will be valuable to a
> wider audience. How would this sound to you?
>
> thanks,
> Derek.
>
> [1] - https://github.com/openstack-packages
> [2] - https://review.gerrithub.io/#/q/project:^openstack-packages/.*,n,z
> [3] - http://trunk.rdoproject.org/f21/report.html
> [4] - http://trunk.rdoproject.org/centos/report.html
> [5] - https://github.com/openstack-packages/delorean
> [6] - https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml
>
>
>
>>
>> Cheers,
>>
>> Thomas Goirand (zigo)
>>
>> [1]
>> https://qa.debian.org/developer.php?login=openstack-devel@lists.alioth.debian.org
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1
>>
>> iQIcBAEBCAAGBQJVZXzYAAoJENQWrRWsa0P+M8gP/21TcEfBs/PTxUFRDMSfP97a
>> JCVj5JIy6GCFjDC+8mbX9WGWFEbU0hRn2CG/dRJNbcrkjeQJaDosSd3BMmEAsrQO
>> ONyu0Qbm2YgclZ9OyUaewE8NPSH8cGRjxS+IdOm5kzhNqX538ohsxD2XNMknx5rU
>> cyq3TTDKWL3ot8+SVdUqfvzmZZdESF+8lucneqOiRk9wU/UVt6fdsovuv3QSqM2h
>> k7etO4Oev46nQFCVvSmofv2mfGTWn5iWDbcCj7qf/blI5uiL4GkTkzFbkr/+f2De
>> 7C4p9rO8WsEaK+Ptg2hayiOm36i6JPnvO6YSOjm6yTeY4LQ3UkULN2/THvftSkYX
>> I+oICj7XbJMhaUDbg9mfJhVZ+O5L0v9YHV2v12NIWMbDGb6EzSYsLqmvANwYN5wM
>> Y8zihyETZFYsJVmPdqHN6ZPGeuKAroFo0F5rldMJ/++J5exApf6DJyd+JrE7vmyV
>> F3sksS+BhmPgmEBbSvcA0ZP8gqEfPWMK5zT1rWBakKy1OHtDaLwnzsCAkaS6kXUJ
>> G45Sr965H6pkO10EMpIbjFeCcVIP7UpmHDxq1PQbhUx4aogjVGA1mIWwuD0Cscaj
>> hD18bJyZxe0t0MzhkijxWnQ6holJ1C36SvSD/jha6oZ7k9FHIAwo1MklTJcCEo7J
>> PBnrpJTi9IkhDmfp21/s
>> =5T0x
>> -----END PGP SIGNATURE-----
>>
>> __________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev