Open Stack

I think we need to address two separate concerns here:

1) smooth integration of puppet-openstack with Fuel master;
2)  tests for changes in fuel-library.

For 1) we need to build and test Fuel against master of
puppet-openstack and treat any integration issues as critical/blocker.
And this is what regular ISO builds and BVT tests are for.

But for 2) we need to test new commit to fuel-library against fixed
baseline - which means stable everything including Ubuntu upstream
mirror, QA framework, fuel code, mos packages or puppet-openstack
modules.

So while I think we should build ISO from current latest HEAD, debug
BVT failures and address them ASAP, we need also to pin puppet modules
used for fuel-library tests to provide targeted feedback on
fuel-library reviews.

To do so we can rely on the same process as we use now for fixing
ubuntu upstream repo:

We have a jenkins job which defines environment for deployment tests:

  https://ci.fuel-infra.org/view/devops/job/devops.master.env/

And additionally to fuel-qa commit, ubuntu mirror id and iso magnet
link we can provide there also the versions list or the entire tarball
of puppet modules which were tested in the latest bvt.

Then we will update the environment as a whole via the same process as
we have now for ISO images.
(Actually I hope it will become a much better process soon as we plan
to speed up and fully automate environment update in the nearest
future).

On Tue, Mar 1, 2016 at 11:36 PM, Sergey Kolekonov
<skolekonov at mirantis.com> wrote:
> I think we should also look at the root cause of these CI failures.
> They are connected with difference in packages and not with manifests or
> deployment process.
> So another possible solution is to stay as close as possible to the package
> sources used by OpenStack Puppet modules CI.
> For example, we have a BP [0] that adds an ability to deploy UCA packages
> with Fuel.
> Current package sources used by openstack-modules CI can be found here [1]
>
> Just my 2c.
> Thanks.
>
> [0] https://blueprints.launchpad.net/fuel/+spec/deploy-with-uca-packages
> [1]
> https://github.com/openstack/puppet-openstack-integration/blob/master/manifests/repos.pp#L4
>
> On Tue, Mar 1, 2016 at 2:21 PM, Vladimir Kuklin <vkuklin at mirantis.com>
> wrote:
>>
>> Dmitry
>>
>> >I don't think "hurried" is applicable here: the only way to become more
>> >ready to track upstream than we already are is to *start* tracking
>> >upstream. Postponing that leaves us in a Catch-22 situation where we
>> >can't stay in sync with upstream because we're not continuously catching
>> >up with upstream.
>>
>> First of all, if you read my email again, you will see that I propose a
>> way of tracking upstream in less continuous mode with nightly testing and
>> switching to it based on automated integration testing which will leave us 0
>> opportunity to face the aforementioned issues.
>>
>> >That would lock us into that Catch-22 situation where we can't allow
>> >Fuel CI to vote on puppet-openstack commits because fuel-library is
>> >always too far behind puppet-openstack for its votes to mean anything
>> >useful.
>>
>> This is not true. We can run FUEL CI against any set of commits.
>>
>> > We have to approach this from the opposite direction: make Fuel CI
>> > stable and meaningful enough so that, 9 times out of 10, Fuel CI failure
>> > indicates a real problem with the code, and the remaining cases can be
>> > quickly unblocked by pushing a catch-up commit to fuel-library (ideally
>> > with a Depends-On tag).
>>
>> Dmitry, could you please point me at the person who will be strictly
>> responsible for creating this 'ketchup' commit? Do you know that this may
>> take up the whole day (couple of hours to do RCA, couple of hours on writing
>> and debugging and couple of hours for FUEL CI tests run) and block the
>> entire Fuel project from having ANY code merged? Taking into consideration
>> that openstack infra is currently under really high load it may take even
>> several days for the fix to land into master. How do you expect us to have
>> any feature merged prior to FF?
>>
>> > It is a matter of trust between projects: do we trust Puppet OpenStack
>> > project to take Fuel's problems seriously and to avoid breaking our CI
>> > more often than necessary? Can Puppet OpenStack project trust us with
>> > the same? So far, our collaboration track record has been pretty good
>> > bordering on examplary, and I think we should build on that and move
>> > forward instead of clinging to the old ways.
>> > The problem with moving only one piece at a time is that you end up so
>> > far behind that you're slowing everyone down. BKL and GIL are not the
>> > only way to deal with concurrency, we can do better than that.
>>
>> I have always thought that buliding software is about verification being
>> more important than 'trust'. There should not be any humanitarian stuff
>> invloved - we are not in a relationship with Puppet-OpenStack folks,
>> although I really admire their work very much. We should not follow sliding
>> git references without being 100% sure that we have mutual gating of the
>> code.
>>
>> Moreover, having such git ref as a source in our Puppetfile will lead to
>> the situation when we have UNREPRODUCIBLE build of Fuel project. Each build
>> may and will result in different code and you will not be able to tell which
>> one without actually looking into the build logs. I think, that this is
>> unacceptable as it may lead to impossibilty of debugging and troubleshooting
>> of anything because it will be impossible to reproduce things.
>>
>> Additionally, we do not have:
>>
>> 1) depends-on support
>> 2) any process instantiated for monitoring of the Puppet-Openstack FUEL CI
>> job
>> 3) any person responsible for monitoring of that job
>>
>> Finally, we have another blocker issue [0] which essentially killed March
>> 1st in EU timezone from code merge point of view as our master is blocked
>> right now by non-boostrapping master node.
>>
>> Having that said I propose that we:
>>
>> 1) immediately pick a set of stable commits to puppet-openstck
>> 2) immediately update puppetfile  with this set
>> 3) unblock other fuel developers
>> 4) fix the aforementioned issues
>> 5) turn tracking of upstream commits on when we have all the pieces set up
>> and when we are sure that we will not ever break master with this change.
>>
>>
>> [0] https://bugs.launchpad.net/fuel/+bug/1551584
>>
>> On Tue, Mar 1, 2016 at 5:10 AM, Dmitry Borodaenko
>> <dborodaenko at mirantis.com> wrote:
>>>
>>> On Mon, Feb 29, 2016 at 01:19:29PM +0300, Vladimir Kuklin wrote:
>>> > Thanks for bringing this up. Frankly, I think that we hurried a little
>>> > bit
>>> > by making our integration with upstream puppet manifests too
>>> > continuous. I
>>> > would suppose that we used a little bit different technique.
>>>
>>> I don't think "hurried" is applicable here: the only way to become more
>>> ready to track upstream than we already are is to *start* tracking
>>> upstream. Postponing that leaves us in a Catch-22 situation where we
>>> can't stay in sync with upstream because we're not continuously catching
>>> up with upstream.
>>>
>>> > First of all, we need to have a set of stable Fuel commits against
>>> > which
>>> > the changes to upstream manifests should be tested. Could you please
>>> > elaborate on whether we are doing this already?
>>>
>>> We are using the same known-good Fuel ISO for puppet-openstack and
>>> fuel-library, so in that sense, yes, we are doing this already.
>>>
>>> We use the HEAD of master branch of fuel-library in these tests, because
>>> using the fuel-library commit from the known-good ISO would mean having
>>> to update that ISO every time we merge a commit that brings fuel-library
>>> up to date with recent changes in puppet-openstack.
>>>
>>> > Secondly, we need to have FUEL CI working with a set of stable commits
>>> > of
>>> > puppet openstack manifests which passed those upstream tests as we
>>> > should
>>> > not have too much moving parts or we will get into situation similar to
>>> > requirements.txt updates when we have pypi updated with new library,
>>> > e.g.
>>> > oslo-serialization.
>>>
>>> That would lock us into that Catch-22 situation where we can't allow
>>> Fuel CI to vote on puppet-openstack commits because fuel-library is
>>> always too far behind puppet-openstack for its votes to mean anything
>>> useful.
>>>
>>> We have to approach this from the opposite direction: make Fuel CI
>>> stable and meaningful enough so that, 9 times out of 10, Fuel CI failure
>>> indicates a real problem with the code, and the remaining cases can be
>>> quickly unblocked by pushing a catch-up commit to fuel-library (ideally
>>> with a Depends-On tag).
>>>
>>> It is a matter of trust between projects: do we trust Puppet OpenStack
>>> project to take Fuel's problems seriously and to avoid breaking our CI
>>> more often than necessary? Can Puppet OpenStack project trust us with
>>> the same? So far, our collaboration track record has been pretty good
>>> bordering on examplary, and I think we should build on that and move
>>> forward instead of clinging to the old ways.
>>>
>>> > In this case, we will be able to do proper testing against frozen
>>> > code-base
>>> > for each piece thus avoiding such issues while retaining fair amount of
>>> > integration with upstream puppet manifests for OpenStack.
>>>
>>> The problem with moving only one piece at a time is that you end up so
>>> far behind that you're slowing everyone down. BKL and GIL are not the
>>> only way to deal with concurrency, we can do better than that.
>>>
>>> --
>>> Dmitry Borodaenko
>>>
>>>
>>> > On Fri, Feb 26, 2016 at 1:28 PM, Ivan Berezovskiy
>>> > <iberezovskiy at mirantis.com
>>> > > wrote:
>>> >
>>> > > Hello, Fuelers!
>>> > >
>>> > > Yesterday we've faced an issue which came from puppet-neutron
>>> > > module: LP #1549934 <https://bugs.launchpad.net/fuel/+bug/1549934>.
>>> > > Fix
>>> > > was prepared very fast:
>>> > > https://review.openstack.org/#/c/284882/ (thanks Sergey for this).
>>> > > So, If CI is red on your patch please re-base it on top of master.
>>> > >
>>> > > Anyway, this issue affected a lot of patches and blocked some
>>> > > developers,
>>> > > because BVT and neutron_smoke tests was also broken. We need to find
>>> > > a way how to minimize risks and affection of such changes on
>>> > > fuel-library.
>>> > > We have jobs which monitors upstream patches:
>>> > > https://ci.fuel-infra.org/view/puppet-openstack/
>>> > > Let's start to monitor those jobs on daily basis. We should have at
>>> > > least 1
>>> > > (ideally 2 or more) engineers which are responsible for analysis of
>>> > > those
>>> > > CI failures. If patch to puppet module is incorrect - we should
>>> > > review it
>>> > > with explanation what is actually wrong. If patch is correct, but
>>> > > breaks
>>> > > current Fuel CI, it means that problem is in our side and we should
>>> > > prepare
>>> > > fuel-library adapt patch to fix the issue. Ideally, we should have an
>>> > > ability
>>> > > to test this fuel-library patch together with upstream one e.g. using
>>> > > 'Depends on'
>>> > > in commit message.
>>> > >
>>> > > Thoughts?
>>> > >
>>> > > --
>>> > > Thanks, Ivan Berezovskiy
>>> > > MOS Puppet Team Lead
>>> > > at Mirantis <https://www.mirantis.com/>
>>> > >
>>> > > slack: iberezovskiy
>>> > > skype: bouhforever
>>> > > phone: + 7-960-343-42-46
>>> > >
>>> > >
>>> > >
>>> > > __________________________________________________________________________
>>> > > OpenStack Development Mailing List (not for usage questions)
>>> > > Unsubscribe:
>>> > > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> > >
>>> > >
>>> >
>>> >
>>> > --
>>> > Yours Faithfully,
>>> > Vladimir Kuklin,
>>> > Fuel Library Tech Lead,
>>> > Mirantis, Inc.
>>> > +7 (495) 640-49-04
>>> > +7 (926) 702-39-68
>>> > Skype kuklinvv
>>> > 35bk3, Vorontsovskaya Str.
>>> > Moscow, Russia,
>>> > www.mirantis.com <http://www.mirantis.ru/>
>>> > www.mirantis.ru
>>> > vkuklin at mirantis.com
>>>
>>> >
>>> > __________________________________________________________________________
>>> > OpenStack Development Mailing List (not for usage questions)
>>> > Unsubscribe:
>>> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>
>> --
>> Yours Faithfully,
>> Vladimir Kuklin,
>> Fuel Library Tech Lead,
>> Mirantis, Inc.
>> +7 (495) 640-49-04
>> +7 (926) 702-39-68
>> Skype kuklinvv
>> 35bk3, Vorontsovskaya Str.
>> Moscow, Russia,
>> www.mirantis.com
>> www.mirantis.ru
>> vkuklin at mirantis.com
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
> Regards,
> Sergey Kolekonov
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Aleksandra Fedorova
Fuel CI Engineer
bookwar