<div dir="ltr"><div><div><div>Hi,<br><br></div>btw, right now we have 33 outdated astute.yaml fixtures for noop rspec tests [0] - and this number is based on a single parameter of network_metadata['vips'] hash, actual amount of outdated fixtures could be bigger. So I've registered a new BP [1] to create a script that will generate up-to-date fixtures on demand and/or periodically.<br><br></div>Regards,<br></div>Alex<br><div><div><br>[0] <a href="http://paste.openstack.org/show/482508/" target="_blank">http://paste.openstack.org/show/482508/</a><br>[1] <a href="https://blueprints.launchpad.net/fuel/+spec/deployment-dryrun-fixtures-generator">https://blueprints.launchpad.net/fuel/+spec/deployment-dryrun-fixtures-generator</a><br><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 7, 2015 at 11:51 AM, Bogdan Dobrelya <span dir="ltr"><<a href="mailto:bdobrelia@mirantis.com" target="_blank">bdobrelia@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 02.12.2015 17:03, Bogdan Dobrelya wrote:<br>
> On 01.12.2015 11:28, Aleksandr Didenko wrote:<br>
>> Hi,<br>
>><br>
>>> pregenerated catalogs for the Noop tests to become the very first<br>
>>> committed state in the data regression process has to be put in the<br>
>>> *separate repo*<br>
>><br>
>> +1 to that, we can put this new repo into .fixtures.yml<br>
>><br>
>>> note, we could as well move the tests/noop/astute.yaml/ there<br>
>><br>
>> +1 here too, astute.yaml files are basically configuration fixtures, we<br>
>> can put them into .fixtures.yml as well<br>
<br>
</span>Folks, the patch to create the fuel-noop-fixtures [0] is in trouble.<br>
I'm not sure I've answered Andreas's questions correct:<br>
<br>
- Would it be OK to keep Noop tests fixtures for fuel-library as a<br>
separate Fuel-related repo but *not* as a part of the Fuel project?<br>
<br>
- Should we require the contribution license agreement for fixtures<br>
which are only be used by tests?<br>
<br>
[0] <a href="https://review.openstack.org/252992" rel="noreferrer" target="_blank">https://review.openstack.org/252992</a><br>
<div class="HOEnZb"><div class="h5"><br>
><br>
> I found the better -and easier for patch authors- way to use the data<br>
> regression checks. Originally suggested workflow was:<br>
><br>
> 1.<br>
> "The check should be done for every modular component (aka deployment<br>
> task). Data generated in the noop catalog run for all classes and<br>
> defines of a given deployment task should be verified against its<br>
> "acknowledged" (committed) state."<br>
><br>
> This part remains the same with the only comment that the astute.yaml<br>
> fixtures of deployment cases should be fetched from the<br>
> fuel-noop-fixtures repo. And the committed state for generated catalogs<br>
> should be<br>
> stored there as well.<br>
><br>
> 2.<br>
> "And fail the test gate, if changes has been found, like new parameter<br>
> with a defined value, removed a parameter, changed a parameter's value."<br>
><br>
> This should be changed as following:<br>
> - the data checks gate should be just a non voting helper for reviewers<br>
> and patch authors. The only its task would be to show inducted data<br>
> changes in a pretty and fast view to help accept/update/reject a patch<br>
> on review.<br>
> - the data checks gate job should fetch the committed data state from<br>
> the fuel-noop-fixtures repo and run regressions check with the patch<br>
> under review checked out on fuel-library repo.<br>
> - the Noop tests gate should be changed to fetch the astute.yaml<br>
> fixtures from the fuel-noop-fixtures repo in order to run noop tests as<br>
> usual.<br>
><br>
> 3.<br>
> "In order to remove a regression, a patch author will have to add (and<br>
> reviewers should acknowledge) detected changes in the committed state of<br>
> the deployment data. This may be done manually, with a tool like [3] or<br>
> by a pre-commit hook, or even at the CI side!"<br>
><br>
> Instead, the patch authors should do nothing additionally. Once accepted<br>
> with wf+1, the patch on reivew should be merged with a pre-commit zuul<br>
> hook (is it possible?). The hook should just regenerate catalogs with<br>
> the changes introduced by the patch and update the committed state of<br>
> data in the fuel-noop-fixtures repo. After that, the patch may be safely<br>
> merged to the fuel-library and everything will be up to date with the<br>
> committed data state.<br>
><br>
> 4.<br>
> "The regression check should show the diff between committed state and a<br>
> new state proposed in a patch. Changed state should be *reviewed* and<br>
> accepted with a patch, to became a committed one. So the deployment data<br>
> will evolve with *only* approved changes. And those changes would be<br>
> very easy to be discovered for each patch under review process!"<br>
><br>
> So this part would work even better now, with no additional actions<br>
> required from the review process sides.<br>
><br>
>><br>
>> Regards,<br>
>> Alex<br>
>><br>
>><br>
>> On Mon, Nov 30, 2015 at 1:03 PM, Bogdan Dobrelya <<a href="mailto:bdobrelia@mirantis.com">bdobrelia@mirantis.com</a><br>
>> <mailto:<a href="mailto:bdobrelia@mirantis.com">bdobrelia@mirantis.com</a>>> wrote:<br>
>><br>
>> On 20.11.2015 17:41, Bogdan Dobrelya wrote:<br>
>> >> Hi,<br>
>> >><br>
>> >> let me try to rephrase this a bit and Bogdan will correct me if<br>
>> I'm wrong<br>
>> >> or missing something.<br>
>> >><br>
>> >> We have a set of top-scope manifests (called Fuel puppet tasks)<br>
>> that we use<br>
>> >> for OpenStack deployment. We execute those tasks with "puppet<br>
>> apply". Each<br>
>> >> task supposed to bring target system into some desired state, so<br>
>> puppet<br>
>> >> compiles a catalog and applies it. So basically, puppet catalog =<br>
>> desired<br>
>> >> system state.<br>
>> >><br>
>> >> So we can compile* catalogs for all top-scope manifests in master<br>
>> branch<br>
>> >> and store those compiled* catalogs in fuel-library repo. Then for<br>
>> each<br>
>> >> proposed patch CI will compare new catalogs with stored ones and<br>
>> print out<br>
>> >> the difference if any. This will pretty much show what is going to be<br>
>> >> changed in system configuration by proposed patch.<br>
>> >><br>
>> >> We were discussing such checks before several times, iirc, but we<br>
>> did not<br>
>> >> have right tools to implement such thing before. Well, now we do<br>
>> :) I think<br>
>> >> it could be quite useful even in non-voting mode.<br>
>> >><br>
>> >> * By saying compiled catalogs I don't mean actual/real puppet<br>
>> catalogs, I<br>
>> >> mean sorted lists of all classes/resources with all parameters<br>
>> that we find<br>
>> >> during puppet-rspec tests in our noop test framework, something like<br>
>> >> standard puppet-rspec coverage. See example [0] for networks.pp<br>
>> task [1].<br>
>> >><br>
>> >> Regards,<br>
>> >> Alex<br>
>> >><br>
>> >> [0] <a href="http://paste.openstack.org/show/477839/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/477839/</a><br>
>> >> [1]<br>
>> >><br>
>> <a href="https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular/openstack-network/networks.pp" rel="noreferrer" target="_blank">https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular/openstack-network/networks.pp</a><br>
>> ><br>
>> > Thank you, Alex.<br>
>> > Yes, the composition layer is a top-scope manifests, known as a Fuel<br>
>> > library modular tasks [0].<br>
>> ><br>
>> > The "deployment data checks", is nothing more than comparing the<br>
>> > committed vs changed states of fixtures [1] of puppet catalogs for<br>
>> known<br>
>> > deployment paths under test with rspecs written for each modular<br>
>> task [2].<br>
>> ><br>
>> > And the *current status* is:<br>
>> > - the script for data layer checks now implemented [3]<br>
>> > - how-to is being documented here [4]<br>
>> > - a fix to make catalogs compilation idempotent submitted [5]<br>
>><br>
>> The status update:<br>
>> - the issue [0] is the data regression checks blocker and is only the<br>
>> Noop tests specific. It has been reworked to not use custom facts [1].<br>
>> New uuid will be still generated each time in the catalog, but the<br>
>> augeas ensures it will be processed in idempotent way. Let's make this<br>
>> change [2] to the upstream puppet-nova as well please.<br>
>><br>
>> [0] <a href="https://bugs.launchpad.net/fuel/+bug/1517915" rel="noreferrer" target="_blank">https://bugs.launchpad.net/fuel/+bug/1517915</a><br>
>> [1] <a href="https://review.openstack.org/251314" rel="noreferrer" target="_blank">https://review.openstack.org/251314</a><br>
>> [2] <a href="https://review.openstack.org/131710" rel="noreferrer" target="_blank">https://review.openstack.org/131710</a><br>
>><br>
>> - pregenerated catalogs for the Noop tests to become the very first<br>
>> committed state in the data regression process has to be put in the<br>
>> *separate repo*. Otherwise, the stackalytics would go mad as that would<br>
>> be a 600k-liner patch to an OpenStack project, which is the Fuel-library<br>
>> now :)<br>
>><br>
>> So, I'm planning to use the separate repo for the templates. Note, we<br>
>> could as well move the tests/noop/astute.yaml/ there. Thoughts?<br>
>><br>
>> > - and there is my WIP branch [6] with the initial committed state of<br>
>> > deploy data pre-generated. So, you can checkout, make any test changes<br>
>> > to manifests and run the data check (see the README [4]). It works for<br>
>> > me, there is no issues with idempotent re-checks of a clean committed<br>
>> > state or tests failing when unexpected.<br>
>> ><br>
>> > So the plan is to implement this noop tests extention as a<br>
>> non-voting CI<br>
>> > gate after I make an example workflow update for developers to the<br>
>> > Fuel wiki. Thoughts?<br>
>> ><br>
>> > [0]<br>
>> ><br>
>> <a href="https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular" rel="noreferrer" target="_blank">https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular</a><br>
>> > [1]<br>
>> ><br>
>> <a href="https://github.com/openstack/fuel-library/tree/master/tests/noop/astute.yaml" rel="noreferrer" target="_blank">https://github.com/openstack/fuel-library/tree/master/tests/noop/astute.yaml</a><br>
>> > [2]<br>
>> <a href="https://github.com/openstack/fuel-library/tree/master/tests/noop/spec" rel="noreferrer" target="_blank">https://github.com/openstack/fuel-library/tree/master/tests/noop/spec</a><br>
>> > [3] <a href="https://review.openstack.org/240015" rel="noreferrer" target="_blank">https://review.openstack.org/240015</a><br>
>> > [4]<br>
>> ><br>
>> <a href="https://github.com/openstack/fuel-library/blob/master/tests/noop/README.rst" rel="noreferrer" target="_blank">https://github.com/openstack/fuel-library/blob/master/tests/noop/README.rst</a><br>
>> > [5] <a href="https://review.openstack.org/247989" rel="noreferrer" target="_blank">https://review.openstack.org/247989</a><br>
>> > [6] <a href="https://github.com/bogdando/fuel-library-1/commits/data_checks" rel="noreferrer" target="_blank">https://github.com/bogdando/fuel-library-1/commits/data_checks</a><br>
>> ><br>
>> ><br>
>><br>
>><br>
>> --<br>
>> Best regards,<br>
>> Bogdan Dobrelya,<br>
>> Irc #bogdando<br>
>><br>
>> __________________________________________________________________________<br>
>> OpenStack Development Mailing List (not for usage questions)<br>
>> Unsubscribe:<br>
>> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>> <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>><br>
>><br>
>><br>
>><br>
>> __________________________________________________________________________<br>
>> OpenStack Development Mailing List (not for usage questions)<br>
>> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>><br>
><br>
><br>
<br>
<br>
--<br>
Best regards,<br>
Bogdan Dobrelya,<br>
Irc #bogdando<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>