Open Stack

Vladimir,

You have identified the root cause of the problem:

> The key issue here is that our CI tests are using some code not from
packages of particular releases, but from other sources, e.g. pypi for
python.

Using frozen pypi, rubygems, nodejs and other language specific package
mirrors is a workaround that doesn't address the root cause. Making that
workaround even more complex by implementing a system to manage, freeze,
and unfreeze these mirrors is a waste of effort since it still does nothing
to address the root cause.

Addressing the root cause means modifying the tests to use the same package
repositories that are used in production, and for which we already have all
the necessary tools.




On Fri, Jul 10, 2015 at 9:44 AM Vladimir Kuklin <vkuklin at mirantis.com>
wrote:

> Folks
>
> I am writing this email to discuss very important topic which appeared to
> be rather hot with several bugs associated with it.
>
> For example these ones:
> https://bugs.launchpad.net/fuel/+bug/1472018
> https://bugs.launchpad.net/fuel/+bug/1468053
>
> There were several fixes applied. One of them was merged (unfortunately)
> and one was stopped by me and other reviewers of stable branches
> (fortunately),
>
> I want to  describe why these bugs happened, why such fixes should not be
> applied and which solution I envision the best for such cases.
>
> The key issue here is that our CI tests are using some code not from
> packages of particular releases, but from other sources, e.g. pypi for
> python.
> Libraries being installed are determined by using particular
> requirements.txt/test-requirements.txt files. These files usually do not
> contain upper bound for libraries versions, thus the newest versions are
> getting installed. This leads to the issues which are especially no good
> for already released code:
>
> 1) if there is a new conflicting version, you need to set this
> upper-bound, thus you need to modify bits which get released
> 2) you are actually testing your code by linking it with libraries which
> are different from those that users are really using when running your code
> 3) even if you specify an upper bound (or even fix the version) for this
> particular library, you may still fetch its newer dependency implicitly (by
> traversing indirect dependencies) with which you will be linking your
> libraries and which will actually be different from the code that you (and
> your users) use
> 4) you may also break production installation if you fix some library
> version as it may not exist in the code bundle which gets delivered to your
> users as a set of packages
>
> All this means that you will get false-negative (you are lucky in this
> case) -1's from CI or false-positive (oops, it can ruin the production
> installation) +1's from CI.
>
> So far I suggest that we do the following (which, strictly speaking, seems
> to be the only reasonable one): in order to do good testing of components,
> we need to do complete freeze of all other libraries and components code we
> are linking/running against. Especially, this should happen for already
> released code which gets into state of being maintained after that.
>
> This means that we need to not only freeze our repositories, but use a
> frozen copy of rubygems/pypi/nodejs repositories for all types of tests we
> run. This will require certain amount of effort from our Infra teams to
> build up a mechanism of freezing/unfreezing the mirrors, determining how we
> should add new pieces of code in such mirrors, may be, add some additional
> code-review policies, create real mirroring scripts and so on.
>
> But I think we should start doing this now in order to finish it at least
> before 8.0 timeframe.
>
> As a temporary solution, I suggest that we apply a series of changes into
> these requirements.txt files before running actual tests instead of
> modifying product code.
>
> Please provide your comments and thoughts.
>
>
> --
> Yours Faithfully,
> Vladimir Kuklin,
> Fuel Library Tech Lead,
> Mirantis, Inc.
> +7 (495) 640-49-04
> +7 (926) 702-39-68
> Skype kuklinvv
> 35bk3, Vorontsovskaya Str.
> Moscow, Russia,
> www.mirantis.com <http://www.mirantis.ru/>
> www.mirantis.ru
> vkuklin at mirantis.com
>  __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150710/26c463bd/attachment.html>