Open Stack

It is time for update!
The previous idea with the committed state and automatic cross-repo
merge hooks in zuul seems too complex to implement. So, the "CI gate for
blah blah" magically becomes now a manual helper tool for
reviewers/developers, see the docs update [0], [1].

You may start using it right now, as described in the docs. Hopefully,
it will help to visualize data changes for complex patches better.

[0] https://review.openstack.org/#/c/299912/
[1] http://goo.gl/Pj3lNf

> On 01.12.2015 11:28, Aleksandr Didenko wrote:
>> Hi,
>> 
>>> pregenerated catalogs for the Noop tests to become the very first
>>> committed state in the data regression process has to be put in the
>>> *separate repo*
>> 
>> +1 to that, we can put this new repo into .fixtures.yml
>> 
>>> note, we could as well move the tests/noop/astute.yaml/ there
>> 
>> +1 here too, astute.yaml files are basically configuration fixtures, we
>> can put them into .fixtures.yml as well
> 
> I found the better -and easier for patch authors- way to use the data
> regression checks. Originally suggested workflow was:
> 
> 1.
> "The check should be done for every modular component (aka deployment
> task). Data generated in the noop catalog run for all classes and
> defines of a given deployment task should be verified against its
> "acknowledged" (committed) state."
> 
> This part remains the same with the only comment that the astute.yaml
> fixtures of deployment cases should be fetched from the
> fuel-noop-fixtures repo. And the committed state for generated catalogs
> should be
> stored there as well.
> 
> 2.
> "And fail the test gate, if changes has been found, like new parameter
> with a defined value, removed a parameter, changed a parameter's value."
> 
> This should be changed as following:
> - the data checks gate should be just a non voting helper for reviewers
> and patch authors. The only its task would be to show inducted data
> changes in a pretty and fast view to help accept/update/reject a patch
> on review.
> - the data checks gate job should fetch the committed data state from
> the fuel-noop-fixtures repo and run regressions check with the patch
> under review checked out on fuel-library repo.
> - the Noop tests gate should be changed to fetch the astute.yaml
> fixtures from the fuel-noop-fixtures repo in order to run noop tests as
> usual.
> 
> 3.
> "In order to remove a regression, a patch author will have to add (and
> reviewers should acknowledge) detected changes in the committed state of
> the deployment data. This may be done manually, with a tool like [3] or
> by a pre-commit hook, or even at the CI side!"
> 
> Instead, the patch authors should do nothing additionally. Once accepted
> with wf+1, the patch on reivew should be merged with a pre-commit zuul
> hook (is it possible?). The hook should just regenerate catalogs with
> the changes introduced by the patch and update the committed state of
> data in the fuel-noop-fixtures repo. After that, the patch may be safely
> merged to the fuel-library and everything will be up to date with the
> committed data state.
> 
> 4.
> "The regression check should show the diff between committed state and a
> new state proposed in a patch. Changed state should be *reviewed* and
> accepted with a patch, to became a committed one. So the deployment data
> will evolve with *only* approved changes. And those changes would be
> very easy to be discovered for each patch under review process!"
> 
> So this part would work even better now, with no additional actions
> required from the review process sides.
> 
>> 
>> Regards,
>> Alex
>> 
>> 
>> On Mon, Nov 30, 2015 at 1:03 PM, Bogdan Dobrelya <bdobrelia at mirantis.com
>> <mailto:bdobrelia at mirantis.com>> wrote:
>> 
>>     On 20.11.2015 17:41, Bogdan Dobrelya wrote:
>>     >> Hi,
>>     >>
>>     >> let me try to rephrase this a bit and Bogdan will correct me if
>>     I'm wrong
>>     >> or missing something.
>>     >>
>>     >> We have a set of top-scope manifests (called Fuel puppet tasks)
>>     that we use
>>     >> for OpenStack deployment. We execute those tasks with "puppet
>>     apply". Each
>>     >> task supposed to bring target system into some desired state, so
>>     puppet
>>     >> compiles a catalog and applies it. So basically, puppet catalog =
>>     desired
>>     >> system state.
>>     >>
>>     >> So we can compile* catalogs for all top-scope manifests in master
>>     branch
>>     >> and store those compiled* catalogs in fuel-library repo. Then for
>>     each
>>     >> proposed patch CI will compare new catalogs with stored ones and
>>     print out
>>     >> the difference if any. This will pretty much show what is going to be
>>     >> changed in system configuration by proposed patch.
>>     >>
>>     >> We were discussing such checks before several times, iirc, but we
>>     did not
>>     >> have right tools to implement such thing before. Well, now we do
>>     :) I think
>>     >> it could be quite useful even in non-voting mode.
>>     >>
>>     >> * By saying compiled catalogs I don't mean actual/real puppet
>>     catalogs, I
>>     >> mean sorted lists of all classes/resources with all parameters
>>     that we find
>>     >> during puppet-rspec tests in our noop test framework, something like
>>     >> standard puppet-rspec coverage. See example [0] for networks.pp
>>     task [1].
>>     >>
>>     >> Regards,
>>     >> Alex
>>     >>
>>     >> [0] http://paste.openstack.org/show/477839/
>>     >> [1]
>>     >>
>>     https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular/openstack-network/networks.pp
>>     >
>>     > Thank you, Alex.
>>     > Yes, the composition layer is a top-scope manifests, known as a Fuel
>>     > library modular tasks [0].
>>     >
>>     > The "deployment data checks", is nothing more than comparing the
>>     > committed vs changed states of fixtures [1] of puppet catalogs for
>>     known
>>     > deployment paths under test with rspecs written for each modular
>>     task [2].
>>     >
>>     > And the *current status* is:
>>     > - the script for data layer checks now implemented [3]
>>     > - how-to is being documented here [4]
>>     > - a fix to make catalogs compilation idempotent submitted [5]
>> 
>>     The status update:
>>     - the issue [0] is the data regression checks blocker and is only the
>>     Noop tests specific. It has been reworked to not use custom facts [1].
>>     New uuid will be still generated each time in the catalog, but the
>>     augeas ensures it will be processed in idempotent way. Let's make this
>>     change [2] to the upstream puppet-nova as well please.
>> 
>>     [0] https://bugs.launchpad.net/fuel/+bug/1517915
>>     [1] https://review.openstack.org/251314
>>     [2] https://review.openstack.org/131710
>> 
>>     - pregenerated catalogs for the Noop tests to become the very first
>>     committed state in the data regression process has to be put in the
>>     *separate repo*. Otherwise, the stackalytics would go mad as that would
>>     be a 600k-liner patch to an OpenStack project, which is the Fuel-library
>>     now :)
>> 
>>     So, I'm planning to use the separate repo for the templates. Note, we
>>     could as well move the tests/noop/astute.yaml/ there. Thoughts?
>> 
>>     > - and there is my WIP branch [6] with the initial committed state of
>>     > deploy data pre-generated. So, you can checkout, make any test changes
>>     > to manifests and run the data check (see the README [4]). It works for
>>     > me, there is no issues with idempotent re-checks of a clean committed
>>     > state or tests failing when unexpected.
>>     >
>>     > So the plan is to implement this noop tests extention as a
>>     non-voting CI
>>     > gate after I make an example workflow update for developers to the
>>     > Fuel wiki. Thoughts?
>>     >
>>     > [0]
>>     >
>>     https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular
>>     > [1]
>>     >
>>     https://github.com/openstack/fuel-library/tree/master/tests/noop/astute.yaml
>>     > [2]
>>     https://github.com/openstack/fuel-library/tree/master/tests/noop/spec
>>     > [3] https://review.openstack.org/240015
>>     > [4]
>>     >
>>     https://github.com/openstack/fuel-library/blob/master/tests/noop/README.rst
>>     > [5] https://review.openstack.org/247989
>>     > [6] https://github.com/bogdando/fuel-library-1/commits/data_checks
>>     >
>>     >
>> 
>> 
>>     --
>>     Best regards,
>>     Bogdan Dobrelya,
>>     Irc #bogdando
>> 
>>     __________________________________________________________________________
>>     OpenStack Development Mailing List (not for usage questions)
>>     Unsubscribe:
>>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> 
>> 
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
> 
> 
> -- 
> Best regards,
> Bogdan Dobrelya,
> Irc #bogdando

-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando