Open Stack

On Mon, Feb 29, 2016 at 01:19:29PM +0300, Vladimir Kuklin wrote:
> Thanks for bringing this up. Frankly, I think that we hurried a little bit
> by making our integration with upstream puppet manifests too continuous. I
> would suppose that we used a little bit different technique.

I don't think "hurried" is applicable here: the only way to become more
ready to track upstream than we already are is to *start* tracking
upstream. Postponing that leaves us in a Catch-22 situation where we
can't stay in sync with upstream because we're not continuously catching
up with upstream.

> First of all, we need to have a set of stable Fuel commits against which
> the changes to upstream manifests should be tested. Could you please
> elaborate on whether we are doing this already?

We are using the same known-good Fuel ISO for puppet-openstack and
fuel-library, so in that sense, yes, we are doing this already.

We use the HEAD of master branch of fuel-library in these tests, because
using the fuel-library commit from the known-good ISO would mean having
to update that ISO every time we merge a commit that brings fuel-library
up to date with recent changes in puppet-openstack.

> Secondly, we need to have FUEL CI working with a set of stable commits of
> puppet openstack manifests which passed those upstream tests as we should
> not have too much moving parts or we will get into situation similar to
> requirements.txt updates when we have pypi updated with new library, e.g.
> oslo-serialization.

That would lock us into that Catch-22 situation where we can't allow 
Fuel CI to vote on puppet-openstack commits because fuel-library is
always too far behind puppet-openstack for its votes to mean anything
useful.

We have to approach this from the opposite direction: make Fuel CI
stable and meaningful enough so that, 9 times out of 10, Fuel CI failure
indicates a real problem with the code, and the remaining cases can be
quickly unblocked by pushing a catch-up commit to fuel-library (ideally
with a Depends-On tag).

It is a matter of trust between projects: do we trust Puppet OpenStack
project to take Fuel's problems seriously and to avoid breaking our CI
more often than necessary? Can Puppet OpenStack project trust us with
the same? So far, our collaboration track record has been pretty good
bordering on examplary, and I think we should build on that and move 
forward instead of clinging to the old ways.

> In this case, we will be able to do proper testing against frozen code-base
> for each piece thus avoiding such issues while retaining fair amount of
> integration with upstream puppet manifests for OpenStack.

The problem with moving only one piece at a time is that you end up so
far behind that you're slowing everyone down. BKL and GIL are not the
only way to deal with concurrency, we can do better than that.

-- 
Dmitry Borodaenko


> On Fri, Feb 26, 2016 at 1:28 PM, Ivan Berezovskiy <iberezovskiy at mirantis.com
> > wrote:
> 
> > Hello, Fuelers!
> >
> > Yesterday we've faced an issue which came from puppet-neutron
> > module: LP #1549934 <https://bugs.launchpad.net/fuel/+bug/1549934>. Fix
> > was prepared very fast:
> > https://review.openstack.org/#/c/284882/ (thanks Sergey for this).
> > So, If CI is red on your patch please re-base it on top of master.
> >
> > Anyway, this issue affected a lot of patches and blocked some developers,
> > because BVT and neutron_smoke tests was also broken. We need to find
> > a way how to minimize risks and affection of such changes on fuel-library.
> > We have jobs which monitors upstream patches:
> > https://ci.fuel-infra.org/view/puppet-openstack/
> > Let's start to monitor those jobs on daily basis. We should have at least 1
> > (ideally 2 or more) engineers which are responsible for analysis of those
> > CI failures. If patch to puppet module is incorrect - we should review it
> > with explanation what is actually wrong. If patch is correct, but breaks
> > current Fuel CI, it means that problem is in our side and we should prepare
> > fuel-library adapt patch to fix the issue. Ideally, we should have an
> > ability
> > to test this fuel-library patch together with upstream one e.g. using
> > 'Depends on'
> > in commit message.
> >
> > Thoughts?
> >
> > --
> > Thanks, Ivan Berezovskiy
> > MOS Puppet Team Lead
> > at Mirantis <https://www.mirantis.com/>
> >
> > slack: iberezovskiy
> > skype: bouhforever
> > phone: + 7-960-343-42-46
> >
> >
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> 
> 
> -- 
> Yours Faithfully,
> Vladimir Kuklin,
> Fuel Library Tech Lead,
> Mirantis, Inc.
> +7 (495) 640-49-04
> +7 (926) 702-39-68
> Skype kuklinvv
> 35bk3, Vorontsovskaya Str.
> Moscow, Russia,
> www.mirantis.com <http://www.mirantis.ru/>
> www.mirantis.ru
> vkuklin at mirantis.com

> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev