---- On Fri, 22 Feb 2019 02:55:35 +0900 Clark Boylan cboylan@sapwetik.org wrote ----

On Thu, Feb 21, 2019, at 9:26 AM, Boden Russell wrote:

...

Question: What's the proper way to install "siblings" [1] in devstack based zuul v3 jobs for projects that also require the siblings via requirements.txt?

Background: Following the zuul v3 migration guide for "sibling requirements" [1] works fine for non-devstack based jobs. However, jobs that use devstack must take other measures to install those siblings in their playbooks.

Based on what I see projects like oslo.messaging doing for cross testing, they are using the PROJECTS env var to specify the siblings in their playbook (example [2]). This approach may work if those siblings are not in requirements.txt, but for projects that also require the siblings at runtime (in requirements.txt) it appears the version from the requirements.txt is used rather than the sibling's source.

By default devstack installs "libraries" (mostly things listed in requirements files) from pypi to ensure that our software works with released libraries. However, it is often important to also test that the next version of our own libraries will work with existing software. For this devstack has the LIBS_FROM_GIT [5] variable which overrides the install via pypi behavior.

Note that I believe you must handle this flag in your devstack plugins.

In addition to what Clark mentioned, all repo defined in "required-projects" variable in zuul v3 job gets appended to devstack's LIBS_FROM_GIT variable by default.

-gmann

...

For example the changes in [3][4].

Thanks

[1] https://docs.openstack.org/infra/manual/zuulv3.html#installation-of-sibling-... [2] https://github.com/openstack/oslo.messaging/blob/master/playbooks/oslo.messa... [3] https://review.openstack.org/#/c/638099 [4] http://logs.openstack.org/99/638099/6/check/tricircle-functional/0b34687/log...

[5] https://docs.openstack.org/devstack/latest/development.html#testing-changes-...

---- On Fri, 22 Feb 2019 22:11:33 +0900 Boden Russell bodenvmw@gmail.com wrote ----

On 2/21/19 4:54 PM, Ghanshyam Mann wrote:

...

In addition to what Clark mentioned, all repo defined in "required-projects"

variable in zuul v3 job gets appended to devstack's LIBS_FROM_GIT variable by default.

Thanks for the info.

However, based on trial and error, using LIBS_FROM_GIT only works if those projects are not in requirements.txt. If the projects used in LIBS_FROM_GIT are also in requirements.txt; the versions from requirements.txt are used; not the source from git.

For example the tricircle-functional job passes when neutron and networking-sfc are removed from requirements.txt [1], but fails if they are in requirements.txt [2]. I've also tried moving those required projects into their own requirements file [3], but that does not work either.

That said; the only solution I see at the moment is to remove those required projects from requirements.txt until we are ready to release the given project and then specify the versions for these source projects.

Am I missing something here; it seems there must be a better solution?

I do not think LIBS_FROM_GIT and requirement.txt are two conflict entity. repo mentioned in LIBS_FROM_GIT will be checked against the requirement.txt version and they should satisfy with the latest master version of repo mentioned in LIBS_FROM_GIT.

For example, in your case neutron is mentioned in LIBS_FROM_GIT so devstack will pickup the neutron master verison which should be compatible with the requirement.txt (>=neutron-released-version)

I saw in your patch (taking the example of PS7), neutron is installed form master[4] and it did satisfy the requirement.txt version [5]. So the final installed version of neutron is 14.0.0.0b2.dev243 which is the latest master.

similar case with required_project which end up appending in LIBS_FROM_GIT by devstack so all repo mentioned in required_projects are installed from source until installing project has explicitly constrained them by upper_constarinted etc.

The problem I see in your patch is networking-sfc latest version is not picked up even that is installed from source. i found networking-sfc-7.0.0 has neutron.db.api imported which has been changed to neutron_lib.db.api in networking-sfc-8.0.0 In the failure, networking-sfc-7.0.0 is being picked up[6] instead of networking-sfc-8.0.0 which fail with the latest neutron 14.0.0.0b2.dev243.

All other PS8-11 [7], I cannot find the networking-sfc installed in that job and so does no error. I am not sure how that is passing without networking-sfc.

But in term of installation, devstack pickup the source version and then apply constraint according to what installation repo has.

[1] https://review.openstack.org/#/c/638099/8 [2] https://review.openstack.org/#/c/638099/7 [3] https://review.openstack.org/#/c/638099/9

[4] http://logs.openstack.org/99/638099/7/check/tricircle-functional/5b20269/log... [5] http://logs.openstack.org/99/638099/7/check/tricircle-functional/5b20269/log... [6] http://logs.openstack.org/99/638099/7/check/tricircle-functional/5b20269/log... [7] http://logs.openstack.org/99/638099/8/check/tricircle-functional/92841bb/log...

-gmann