Open Stack

On Fri, Jan 29, 2016 at 2:12 AM, Bogdan Dobrelya <bdobrelia at mirantis.com> wrote:
> This is a continuation of the forked discussion [0].
>
> The idea is to relax Fuel-library downstream policy and implement a
> "lazy downstreaming", which is to not create a downstream fork of a
> puppet module referenced in the librarian Puppetfile unless we have to
> do so.
>

So what exactly would be the change to the policy? Just allowing for
different source git repo?

Currently, we have basically been enforcing the following:
1) requiring a fuel-infra mirror
2) requiring a tag

See https://wiki.openstack.org/wiki/Fuel/Library_and_Upstream_Modules

This is being validated by the verify-fuel-library-puppetfile fuel-ci
job.  See https://github.com/openstack/fuel-library/blob/master/utils/jenkins/fuel_validate_puppetfile.rb

If the change you're proposing is to just relax the first requirement,
I could agree with that.  If it's to allow for the relaxing of both,
then no I don't agree. When we first switched to using using a
Puppetfile, we had discussions specifically around the tag requirement
and that is there to ensure that when we build we are always pulling
in the same expected version for every build.  While we could support
a hash instead of a tag, I would prefer that we not do that.

> The relaxed policy would unblock an upstream switching of the rabbitmq
> [1] and corosync [2] modules as well.
>
> Pros:
> - Reduced costs of maintaining downstream forks because of the laziness
> of the forking process. This much better distributes load to Fuel
> engineering and HW resources in time. Even though related tasks may be
> one-time, HW resources and network bandwidth shall not be wasted for
> keeping and cloning unnecessary forks (unless we have no choice but to
> fork things)

We aren't maintaining forks for every module, we are using normal git
sync processes to pull in the tags from the upstream modules as well.
This is completely hands off so we're not managing tags for most
upstream modules, and these days I think we're only creating tags and
managing some minor forks only the openstack modules.  And this is
primarily because the upstream openstack modules do not get new tags
the previous stable versions.  I think we only have maybe 4 commits
that are being maintained across 2 or 3 of the modules? The puppet
guys have a list.  Doing a quick look, it appears we're managing 18
custom tags (13 of which are openstack modules) and relying on 17
upstream tags.

> -  A module might remain as direct upstream reference in the Puppetfile
> for quite a while. This as well shows an amount of the hidden technical
> debt in much more clean representation - downstream vs upstream refs in
> the Puppetfile.

I'm not sure how this is addressed by using the upstream source repo
and a tagged version. This is the same problem with any versions. The
only way this gets fixed is to switch to branches across the board and
deal with the changes that pop up or have a periodic job that tests
this. But this completely violates what we agreed to when we switched
to librarian in the first place.

> - Some generic, non OpenStack puppet modules will barely require
> backporting of separate patches by older tags as they work just
> straightforward and the latest version works for every supported Fuel
> release as well. Those would save resources because of the laziness of
> the relaxed process will never require to create downstream forks for
> such modules!

What resources are we saving?  We don't create forks unless we need
to. We're using tags, it's just that the git repo is not the original
source because of the policy we put in place when we switched to
librarian.  And like I said, I'm a little more flexible on that part.

>
> Cons:
> - I see none. For "unlucky" modules, the *same* amount of work shall be
> done anyway, although with the lazy process in place it will be
> distributed in time much better.
>

A large con is that we're moving the work of creating a mirror (not
necessarily a fork as you call it), from the front of the dev cycle
when we have more time to the end where we're in a pinch because we've
run across something that evidently needs to be fixed asap.  Getting a
mirror created is currently just a ticket to fuel-infra...

As I've stated multiple times now, I tend to agree with the premise of
this proposal but it needs to be thought out and well defined as to
what we are expecting and what should be enforced. Today we have a
process that is 1) documented, 2) tested in CI, and 3) has workflows
and infrastructure to support it. I'd like to see more concrete
details on what changes you're proposing and make sure we properly
understand the impact from these changes as part of the development
process.

The reason why in the previous thread I mentioned the
upstream/downstream items needs to be solidified before this policy
change is that I believe the current policy should turn into the
downstream policy and the upstream fuel policy should be changed to
require the upstream git repo and allow for the use of a branch.  This
would allow the downstream version to be much more stable and we would
be able to cherrypick/fix issues as necessary while allowing the
upstream to consume more current modules for development purposes.

Thanks,
-Alex

> [0]
> http://lists.openstack.org/pipermail/openstack-dev/2016-January/085249.html
> [1] https://review.openstack.org/#/c/271217
> [2] https://review.openstack.org/#/c/273036/
>
> --
> Best regards,
> Bogdan Dobrelya,
> Irc #bogdando
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev