Open Stack

Hi,

following our docs/workflow plus writing rspec tests for every new option
we add/modify in our manifests could help with regressions. For example:

   - we add new keystone config option in openstack::keystone class -
   keystone_config {'cache/backend': value => 'keystone.cache.memcache_pool';}
   - we create new test for openstack::keystone class, something like this:
      - should
      contain_keystone_config("cache/backend").with_value('keystone.cache.memcache_pool')

So with such test, if for some reason we lose
keystone_config("cache/backend") option, 'rake spec' would alert us about
it right away and we'll get "-1" from CI. Of course we should also
implement 'rake spec' CI gate for this.

But from the other hand, if someone changes option in manifests and updates
rspec tests accordingly, then such commit will pass 'rake spec' test and we
can still lose some specific option.

> We should speed up development of some modular testing framework that
will check that corresponding change affects only particular pieces.

Such test would not catch this particular regressions with "keystone_config
{'cache/backend': value => 'keystone.cache.memcache_pool';}", because even
with regression (i.e. dogpile backend) keystone was working OK. It has
passed several BVTs and custom system tests, because 'dogpile' cache
backend was working just fine while all memcached servers are up and
running. So it looks like we need some kind of tests that will ensure that
particular config options (or particular puppet resources) have some
particular values (like "backend = keystone.cache.memcache_pool" in [cache]
block of keystone.conf).

So I would go with rspec testing for specific resources but I would write
them in 'openstack' module. Those tests should check that needed
(nova/cinder/keystone/glance)_config resources have needed values in the
puppet catalog. Since we're not going to sync 'openstack' module with the
upstream, such tests will remain intact until we change them, and they
won't be affected by other modules sync/merge (keystone, cinder, nova, etc).

--
Regards,
Aleksandr Didenko


On Wed, Nov 19, 2014 at 5:45 PM, Vladimir Kuklin <vkuklin at mirantis.com>
wrote:

> Fuelers
>
> I am writing that we had a really sad incident - we noticed that after we
> merged upstream keystone module we lost modifications (Change-Id:
> Idfe4b54caa0d96a93e93bfff12d8b6216f83e2f1
> <https://review.openstack.org/#/q/Idfe4b54caa0d96a93e93bfff12d8b6216f83e2f1,n,z>)
> for memcached dogpile driver which are crucial for us. And here I can see 2
> problems:
>
> 1) how can we ensure that we did not lose anything else?
> 2) how can we ensure that this will never happen again?
>
> Sadly, it seems that the first question implies that we recheck all the
> upstream merge/adaptation commits by hand and check that we did not lose
> anything.
>
> Regarding question number 2 we do already have established process for
> upstream code merge:
> http://docs.mirantis.com/fuel-dev/develop/module_structure.html#contributing-to-existing-fuel-library-modules.
> It seems that this process had  not been established when keystone code was
> reviewed. I see two ways here:
>
> 1) We should enforce code review workflow and specifically say that
> upstream merges can be accepted only after we have 2 '+2s' from core
> reviewers after they recheck that corresponding change does not introduce
> any regressions.
> 2) We should speed up development of some modular testing framework that
> will check that corresponding change affects only particular pieces. It
> seems much easier if we split deployment into stages (oh my, I am again
> talking about granular deployment feature) and each particular commit
> affects only one of the stages, so that we can see the difference and catch
> regressions eariler.
>
>
>
>
>
> --
> Yours Faithfully,
> Vladimir Kuklin,
> Fuel Library Tech Lead,
> Mirantis, Inc.
> +7 (495) 640-49-04
> +7 (926) 702-39-68
> Skype kuklinvv
> 45bk3, Vorontsovskaya Str.
> Moscow, Russia,
> www.mirantis.com <http://www.mirantis.ru/>
> www.mirantis.ru
> vkuklin at mirantis.com
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20141121/628fb362/attachment.html>