[neutron][devstack] Using git source for networking devstack based jobs
If your networking project uses required projects (aka "Siblings") [1], please read on. It's recently come to our attention [2][3][4] that some required networking projects are not being pulled from git source in devstack based jobs. As a result, a number of networking gate jobs were impacted [3]. While we have a short term fix in place, it doesn't solve the issue longer term. Today it's still not clear why some dependencies are coming from source and others aren't [2]. However, before digging deeper, we should probably step back and ensure we are heading down the right path. The question is; do we want these devstack jobs using their dependencies from git source, or should they be using tagged branches/releases of them instead?? Using git source seems to increase development velocity; not having to wait/coordinate on a tagged dependency, but at the same time it's also a moving target. My personal vote is to use git source; it will help maintain the velocity of neutron-lib and other cross-project efforts. It would be helpful to get some feedback from the other networking teams on what approach they think would work best. Thanks [1] https://docs.openstack.org/infra/manual/zuulv3.html#installation-of-sibling-... [2] https://bugs.launchpad.net/devstack/+bug/1817555 [3] https://bugs.launchpad.net/tricircle/+bug/1816644 [4] http://lists.openstack.org/pipermail/openstack-discuss/2019-February/003010....
On 2019-03-05 11:31:16 -0700 (-0700), Boden Russell wrote: [...]
The question is; do we want these devstack jobs using their dependencies from git source, or should they be using tagged branches/releases of them instead?? [...]
The primary risk with developing only against unreleased source of your dependencies, when your users expect to deploy your releases with released dependencies, is that you can end up accidentally releasing something which only works with unreleased and potentially unsupported states of its dependencies. You do still need some means of confirming your releases can work when used strictly with released versions of its dependencies, so as to prevent this. Testing against released dependencies instead side-steps this problem, but at the loss of easy testing against upcoming dependency versions. -- Jeremy Stanley
On Tue, 2019-03-05 at 18:52 +0000, Jeremy Stanley wrote:
On 2019-03-05 11:31:16 -0700 (-0700), Boden Russell wrote: [...]
The question is; do we want these devstack jobs using their dependencies from git source, or should they be using tagged branches/releases of them instead??
[...]
The primary risk with developing only against unreleased source of your dependencies, when your users expect to deploy your releases with released dependencies, is that you can end up accidentally releasing something which only works with unreleased and potentially unsupported states of its dependencies. You do still need some means of confirming your releases can work when used strictly with released versions of its dependencies, so as to prevent this. Testing against released dependencies instead side-steps this problem, but at the loss of easy testing against upcoming dependency versions. so personally i would expect the majority of neutrons gate jobs to used the released versions but we proablly coudl have a neutron-next jobs that was bassically the full tempst job that specifically used master
that is basically what we do with os-vif. nova uses os-vif form pip we have an experimental job we can run to use os-vif form souce on any nova chagne and obviosly os-vif itself uses os-vif from gerrit + nova master we dont use os-vif master in the default nova jobs but we could if it was needed. sofar that has not been an issue since we have the experimental job we can alway do a spot check.
On 3/5/19 12:15 PM, Sean Mooney wrote:> so personally i would expect the majority of neutrons gate jobs to used the released versions
but we proablly coudl have a neutron-next jobs that was bassically the full tempst job that specifically used master
I wasn't talking about neutron's jobs, but rather other networking projects that use neutron + other networking projects... But I suspect the same applies to them as well.
On Tue, 2019-03-05 at 18:52 +0000, Jeremy Stanley wrote:
The primary risk with developing only against unreleased source of your dependencies, when your users expect to deploy your releases with released dependencies, is that you can end up accidentally releasing something which only works with unreleased and potentially unsupported states of its dependencies.
If I understand correctly, that risk only applies to consumers of "beta releases" during the dev cycle (other openstack projects for testing I'd guess). What gets packaged/installed in terms of a release deliverable would still be tested with dependencies pinned via stable branch contents, not them from git source. Is this correct? I think we may have a special case here in neutron land as part of our neutron-lib effort [1][2]. We are aggressively rehoming [2] common code from neutron into neutron-lib and then consuming it in neutron and other networking projects (currently 20 projects in total). Due to the volume of code changes involved and the potential for cross-project dependencies, testing only with tagged releases implies an extra step in this process that could hinder velocity. Am I wrong to think that maybe this special case could warrant testing from git master source rather than tagged releases during the dev cycle? Or perhaps we just start doing a lot more releases of these networking projects during the dev cycle? [1] https://blueprints.launchpad.net/neutron/+spec/neutron-lib-decouple-db [2] https://docs.openstack.org/neutron-lib/latest/contributor/contributing.html
On 2019-03-05 14:23:50 -0700 (-0700), Boden Russell wrote: [...]
On Tue, 2019-03-05 at 18:52 +0000, Jeremy Stanley wrote:
The primary risk with developing only against unreleased source of your dependencies, when your users expect to deploy your releases with released dependencies, is that you can end up accidentally releasing something which only works with unreleased and potentially unsupported states of its dependencies.
If I understand correctly, that risk only applies to consumers of "beta releases" during the dev cycle (other openstack projects for testing I'd guess). What gets packaged/installed in terms of a release deliverable would still be tested with dependencies pinned via stable branch contents, not them from git source.
Is this correct?
If (arbitrary example) you only ever test neutron against the master branch state of neutron-lib and not against the released state of neutron-lib, then when it comes time to tag neutron for a new release there's a chance you've grown to expect some unreleased behavior change in neutron-lib which was introduced since its most recent release. Then when users attempt to deploy and run neutron it breaks because their installation of the most recent version of neutron-lib is still too old to support their installation of the most recent version of neutron.
I think we may have a special case here in neutron land as part of our neutron-lib effort [1][2]. We are aggressively rehoming [2] common code from neutron into neutron-lib and then consuming it in neutron and other networking projects (currently 20 projects in total). Due to the volume of code changes involved and the potential for cross-project dependencies, testing only with tagged releases implies an extra step in this process that could hinder velocity.
Am I wrong to think that maybe this special case could warrant testing from git master source rather than tagged releases during the dev cycle?
Or perhaps we just start doing a lot more releases of these networking projects during the dev cycle?
[1] https://blueprints.launchpad.net/neutron/+spec/neutron-lib-decouple-db [2] https://docs.openstack.org/neutron-lib/latest/contributor/contributing.html
If you freeze neutron-lib after its last release of the cycle and don't accept any new changes to it until after everything which tests against its branch tip also releases, then you might be protected from this class of problem. -- Jeremy Stanley
On 2019-03-05 14:23:50 -0700 (-0700), Boden Russell wrote: [...]
On Tue, 2019-03-05 at 18:52 +0000, Jeremy Stanley wrote:
The primary risk with developing only against unreleased source of your dependencies, when your users expect to deploy your releases with released dependencies, is that you can end up accidentally releasing something which only works with unreleased and potentially unsupported states of its dependencies.
If I understand correctly, that risk only applies to consumers of "beta releases" during the dev cycle (other openstack projects for testing I'd guess). What gets packaged/installed in terms of a release deliverable would still be tested with dependencies pinned via stable branch contents, not them from git source.
Is this correct?
If (arbitrary example) you only ever test neutron against the master branch state of neutron-lib and not against the released state of neutron-lib, then when it comes time to tag neutron for a new release there's a chance you've grown to expect some unreleased behavior change in neutron-lib which was introduced since its most recent release. Then when users attempt to deploy and run neutron it breaks because their installation of the most recent version of neutron-lib is still too old to support their installation of the most recent version of neutron. well you could get around that by having the lower-constaints job use
On Tue, 2019-03-05 at 22:21 +0000, Jeremy Stanley wrote: the pypi releases. by convention we run the full unittest suite in lower constrains so it might not be as much coverage as you might like but it should catch alot of incompatibilities. i personly woudl still want at least on voting tempest job that used the latest package and perferally the oldest package however we don't have the latter today.
I think we may have a special case here in neutron land as part of our neutron-lib effort [1][2]. We are aggressively rehoming [2] common code from neutron into neutron-lib and then consuming it in neutron and other networking projects (currently 20 projects in total). Due to the volume of code changes involved and the potential for cross-project dependencies, testing only with tagged releases implies an extra step in this process that could hinder velocity.
Am I wrong to think that maybe this special case could warrant testing from git master source rather than tagged releases during the dev cycle?
Or perhaps we just start doing a lot more releases of these networking projects during the dev cycle?
[1] https://blueprints.launchpad.net/neutron/+spec/neutron-lib-decouple-db [2] https://docs.openstack.org/neutron-lib/latest/contributor/contributing.html
If you freeze neutron-lib after its last release of the cycle and don't accept any new changes to it until after everything which tests against its branch tip also releases, then you might be protected from this class of problem.
On 2019-03-06 13:08:49 +0000 (+0000), Sean Mooney wrote:
On Tue, 2019-03-05 at 22:21 +0000, Jeremy Stanley wrote: [...]
If (arbitrary example) you only ever test neutron against the master branch state of neutron-lib and not against the released state of neutron-lib, then when it comes time to tag neutron for a new release there's a chance you've grown to expect some unreleased behavior change in neutron-lib which was introduced since its most recent release. Then when users attempt to deploy and run neutron it breaks because their installation of the most recent version of neutron-lib is still too old to support their installation of the most recent version of neutron.
well you could get around that by having the lower-constaints job use the pypi releases. by convention we run the full unittest suite in lower constrains so it might not be as much coverage as you might like but it should catch alot of incompatibilities. i personly woudl still want at least on voting tempest job that used the latest package and perferally the oldest package however we don't have the latter today. [...]
I guess my point was that if you're developing two interdependent projects in parallel which won't release at the same time, you need to be mindful that work to implement support for a new feature of one in the other includes a graceful fallback behavior so that deployments with already-released dependencies lacking that feature will still be usable (and resist the temptation to instead simply drop support for versions in which it does not appear). Yes, your lower-constraints unit testing helps ensure this is the case with imported libraries like neutron-lib (as long as you have sufficient coverage over calls you make into it, of course). -- Jeremy Stanley
participants (3)
-
Boden Russell
-
Jeremy Stanley
-
Sean Mooney