Open Stack

On 01.12.2015 11:28, Aleksandr Didenko wrote:
> Hi,
> 
>> pregenerated catalogs for the Noop tests to become the very first
>> committed state in the data regression process has to be put in the
>> *separate repo*
> 
> +1 to that, we can put this new repo into .fixtures.yml
> 
>> note, we could as well move the tests/noop/astute.yaml/ there
> 
> +1 here too, astute.yaml files are basically configuration fixtures, we
> can put them into .fixtures.yml as well

I found the better -and easier for patch authors- way to use the data
regression checks. Originally suggested workflow was:

1.
"The check should be done for every modular component (aka deployment
task). Data generated in the noop catalog run for all classes and
defines of a given deployment task should be verified against its
"acknowledged" (committed) state."

This part remains the same with the only comment that the astute.yaml
fixtures of deployment cases should be fetched from the
fuel-noop-fixtures repo. And the committed state for generated catalogs
should be
stored there as well.

2.
"And fail the test gate, if changes has been found, like new parameter
with a defined value, removed a parameter, changed a parameter's value."

This should be changed as following:
- the data checks gate should be just a non voting helper for reviewers
and patch authors. The only its task would be to show inducted data
changes in a pretty and fast view to help accept/update/reject a patch
on review.
- the data checks gate job should fetch the committed data state from
the fuel-noop-fixtures repo and run regressions check with the patch
under review checked out on fuel-library repo.
- the Noop tests gate should be changed to fetch the astute.yaml
fixtures from the fuel-noop-fixtures repo in order to run noop tests as
usual.

3.
"In order to remove a regression, a patch author will have to add (and
reviewers should acknowledge) detected changes in the committed state of
the deployment data. This may be done manually, with a tool like [3] or
by a pre-commit hook, or even at the CI side!"

Instead, the patch authors should do nothing additionally. Once accepted
with wf+1, the patch on reivew should be merged with a pre-commit zuul
hook (is it possible?). The hook should just regenerate catalogs with
the changes introduced by the patch and update the committed state of
data in the fuel-noop-fixtures repo. After that, the patch may be safely
merged to the fuel-library and everything will be up to date with the
committed data state.

4.
"The regression check should show the diff between committed state and a
new state proposed in a patch. Changed state should be *reviewed* and
accepted with a patch, to became a committed one. So the deployment data
will evolve with *only* approved changes. And those changes would be
very easy to be discovered for each patch under review process!"

So this part would work even better now, with no additional actions
required from the review process sides.

> 
> Regards,
> Alex
> 
> 
> On Mon, Nov 30, 2015 at 1:03 PM, Bogdan Dobrelya <bdobrelia at mirantis.com
> <mailto:bdobrelia at mirantis.com>> wrote:
> 
>     On 20.11.2015 17:41, Bogdan Dobrelya wrote:
>     >> Hi,
>     >>
>     >> let me try to rephrase this a bit and Bogdan will correct me if
>     I'm wrong
>     >> or missing something.
>     >>
>     >> We have a set of top-scope manifests (called Fuel puppet tasks)
>     that we use
>     >> for OpenStack deployment. We execute those tasks with "puppet
>     apply". Each
>     >> task supposed to bring target system into some desired state, so
>     puppet
>     >> compiles a catalog and applies it. So basically, puppet catalog =
>     desired
>     >> system state.
>     >>
>     >> So we can compile* catalogs for all top-scope manifests in master
>     branch
>     >> and store those compiled* catalogs in fuel-library repo. Then for
>     each
>     >> proposed patch CI will compare new catalogs with stored ones and
>     print out
>     >> the difference if any. This will pretty much show what is going to be
>     >> changed in system configuration by proposed patch.
>     >>
>     >> We were discussing such checks before several times, iirc, but we
>     did not
>     >> have right tools to implement such thing before. Well, now we do
>     :) I think
>     >> it could be quite useful even in non-voting mode.
>     >>
>     >> * By saying compiled catalogs I don't mean actual/real puppet
>     catalogs, I
>     >> mean sorted lists of all classes/resources with all parameters
>     that we find
>     >> during puppet-rspec tests in our noop test framework, something like
>     >> standard puppet-rspec coverage. See example [0] for networks.pp
>     task [1].
>     >>
>     >> Regards,
>     >> Alex
>     >>
>     >> [0] http://paste.openstack.org/show/477839/
>     >> [1]
>     >>
>     https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular/openstack-network/networks.pp
>     >
>     > Thank you, Alex.
>     > Yes, the composition layer is a top-scope manifests, known as a Fuel
>     > library modular tasks [0].
>     >
>     > The "deployment data checks", is nothing more than comparing the
>     > committed vs changed states of fixtures [1] of puppet catalogs for
>     known
>     > deployment paths under test with rspecs written for each modular
>     task [2].
>     >
>     > And the *current status* is:
>     > - the script for data layer checks now implemented [3]
>     > - how-to is being documented here [4]
>     > - a fix to make catalogs compilation idempotent submitted [5]
> 
>     The status update:
>     - the issue [0] is the data regression checks blocker and is only the
>     Noop tests specific. It has been reworked to not use custom facts [1].
>     New uuid will be still generated each time in the catalog, but the
>     augeas ensures it will be processed in idempotent way. Let's make this
>     change [2] to the upstream puppet-nova as well please.
> 
>     [0] https://bugs.launchpad.net/fuel/+bug/1517915
>     [1] https://review.openstack.org/251314
>     [2] https://review.openstack.org/131710
> 
>     - pregenerated catalogs for the Noop tests to become the very first
>     committed state in the data regression process has to be put in the
>     *separate repo*. Otherwise, the stackalytics would go mad as that would
>     be a 600k-liner patch to an OpenStack project, which is the Fuel-library
>     now :)
> 
>     So, I'm planning to use the separate repo for the templates. Note, we
>     could as well move the tests/noop/astute.yaml/ there. Thoughts?
> 
>     > - and there is my WIP branch [6] with the initial committed state of
>     > deploy data pre-generated. So, you can checkout, make any test changes
>     > to manifests and run the data check (see the README [4]). It works for
>     > me, there is no issues with idempotent re-checks of a clean committed
>     > state or tests failing when unexpected.
>     >
>     > So the plan is to implement this noop tests extention as a
>     non-voting CI
>     > gate after I make an example workflow update for developers to the
>     > Fuel wiki. Thoughts?
>     >
>     > [0]
>     >
>     https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular
>     > [1]
>     >
>     https://github.com/openstack/fuel-library/tree/master/tests/noop/astute.yaml
>     > [2]
>     https://github.com/openstack/fuel-library/tree/master/tests/noop/spec
>     > [3] https://review.openstack.org/240015
>     > [4]
>     >
>     https://github.com/openstack/fuel-library/blob/master/tests/noop/README.rst
>     > [5] https://review.openstack.org/247989
>     > [6] https://github.com/bogdando/fuel-library-1/commits/data_checks
>     >
>     >
> 
> 
>     --
>     Best regards,
>     Bogdan Dobrelya,
>     Irc #bogdando
> 
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando