Open Stack

When we (infra) changed the unit test jobs to not set up databases by
default, we created special python-db and tox-db jobs that set up both
MySQL and PostgreSQL databases. And that complicated the setup of those
projects and lead to problems like setting projects up via bindep for
both databases even if one was used.

We had last week an IRC discussion [1] and came up with the following
approach:

  Projects can use a tools/test-setup.sh script that is called from
  our unit test (tox, python27, python34, python35) targets. The
  script is executed as root and should set up the needed databases -
  or whatever is needed. The script needs to reside in the repository -
  and thus might need to get backported to older branches.

  This setup should be used for any kind of repo specific unit test
  setup.

  Projects are suggested to add to their developer documents, e.g. the
  README or CONTRIBUTING or TESTING file, the usage of
  tools/testsetup.sh. Developers should be able to use the script to
  set up prerequisites for unit tests locally.

  Long term goal is for projects to not use the -db jobs anymore, new
  changes for them should not be accepted.

This is implemented in project-config [2], an example usage in
nodepool [3,4], which leads to a cleanup [5].

Further investigation shows that the special searchlight setup can be
solved with the same approach (searchlight change [6], project-config
[7]). Here it's interesting to note that moving the setup in the
repository, found a problem: The repo needs elasticsearch 1 for
liberty and 2 for newer branches, this can now be done inside the
repository.

The xfs TMPDIR setup of swift [2] could been done in general this way as
well but that change needs to set TMPDIR for the unittests, passing
information from the set up builder to the tox builder. This is
currently not possible using only the proposed solution, and so would
still require a custom tox job. Alternative, this could be changed with
some other way of passing the value of TMPDIR between these different
invocations.

Today, a change was proposed [8,9] that would setup docker for kolla
and kolla-ansible. I suggest to not merge it and instead use the same
approach here.

Credits for the proposal go to Jeremy - and this got triggered by
comments by Jim. Thanks!

Andreas

[1]
http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-11-17.log.html#t2016-11-17T15:07:38

[2] https://review.openstack.org/399105
[3] https://review.openstack.org/399079
[4] https://review.openstack.org/399177
[5] https://review.openstack.org/399180
[6] https://review.openstack.org/399159
[7] https://review.openstack.org/399169
[8] https://review.openstack.org/400128
[9] https://review.openstack.org/400474
[10] https://review.openstack.org/394600
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
  SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Felix Imendörffer, Jane Smithard, Graham Norton,
       HRB 21284 (AG Nürnberg)
    GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126