Hi Jeremy,
I agree, that is the point I was trying to make... or moreso that the "snapshot in time" is the purpose upper-constraints.txt was intended to serve for stable branches so we can keep them... stable.
I would replace "stable" with "working" in that sentence (e.g. existing test coverage still passes). I do agree with that in general.
On the other hand "user expects it to be updated asap on security vulnerabilities" sounds like a misconception we need to better document isn't the reason we have that mechanism.
I am not sure this was implied. The current suggested mode was to clear out known security vulnerabilities, which is a different angle than guaranteeing that everything is immediately secure. Fixing known issues still means there might be unfixed issues left. For me the point of that exercise is not to ensure that pure pip/upper-constraints built environments are *secure*, but that they still *work*. I would not want to extrapolate that just because we're fixing known vulnerable versions in our dependency chain by fixed versions that we're magically secure.
contemporary branches. Many (I expect most?) of our external Python dependencies do not follow a similar pattern
For many projects with security problems there is actually a sane backporting strategy implemented. As an example there is django, where we opt for the LTS version and that one is very cautiously maintained with only security and critical bugfixes (data-loss/data corruption type). I would rather up the constraints on stable branches to verify that our test coverage is still passing with that change than to rely on distributors magically doing better backports of the security fixes than upstream does and then not shipping that to their customers without finding that the security patch actually broke some useful functionality of the project. django might be a good example and a bad example here, because I actually know that horizon *ignores* upper-constraints so the django version we're having in upper-constraints is not the one that is actually being used in the testing, which is another reason why I would like to make it actually match* the actually used version because thats the one the code is most likely working with and it then serves at least as a documentation type thing.
Yes, and those vendors (at least for the versions of their distros we claim to test against) generally maintain a snapshot-in-time fork of those packages and selectively backport fixes to them, which is *why* we can depend on them being a generally stable test bed for us.
I do happen to work for one of those vendors, and yes, upper-constraints is an important consideration for the choice of versions that are being shipped. I happen to come accross a situation every once in a while where a security fix the upstream project wasn't such a super great thing and caused regressions in some other project somewhere that was accidentally or intentionally using some of the previously vulnerable functionality. OpenStack can be such a case. Thats another reason why I would like to ensure that the "stable version with the vulnerable behavior fixed" version is actually used in testing so that we know it doesn't break functionality and that also OpenStack does not introduce changes that rely on behavior that has been patched out by vulnerability fixes in the downstream vendor distributions. A security fix could intentionally change the observed behavior of the dependency because that is the essential part of the security fix. One example of that could be that it rejects a certain invalid input that it sloppily ignored beforehand. By running our testing against the version that still allows the invalid input we're not finding the issue. Distribution vendors that are careful and provide security fixes to their user/customer base will do the security backport and release that as an update, either with some testing or with the implicit assumption that necessary cross-checking has already been performed. If it does cause a problem, its is going to be a security fix that explodes right into the face of the user of OpenStack in this case. Eventually, every once in a while there is a production outage due to that, and customers are generally not happy about that. the naive reaction of the user is either "Ok, this vendor is not able to patch things properly" or "OpenStack is broken by the security fix". Both are uncomfortable situations to be in. I don't ask you to walk in the shoes of the distribution vendor that goes through that customer conversation, but I would like to ask you to consider the case that OpenStack needs to *deal* with behavior changes in its dependencies that are introduced as an essential part or the side effects of security fixes being done there. I can give some theoretical examples. There were cases where dependencies ran code that left files with insecure permissions. Quite an obvious problem, and quite easy to fix. Suddenly files have secure permissions and openstack breaks. There were cases where a url was sanitized and slightly malformatted urls were rejected, and openstack generated those and fell into the trap. Yes, all of that can be fixed by patches, but there is no testing of those backports because theyre being executed against the *still vulnerable* version.
Right, again we seem to agree on the risk, just not the source of the problem. I continue to argue that the underlying issue is the choice to reuse the existing upper-constraints.txt (which was invented for a different, conflicting purpose) rather than creating a solid solution to their problem.
Perhaps the projects that currently use upper constraints don't care about a secure virtualenv/container build, and thats fine. It still does have a point to test against the versions end users will most likely have, and they most likely have security fixed versions (because they're good users and run against a stable security maintained enterprise operating system). We'd be doing ourselves a favor by testing a situation that is coming close to the end user situation in our CI. Yes, we can debate that upping "requests" to a 3 version newer version is stretching the limits. I think thats a very useful conversation to have, and it doesn't take me a lot to feel convinced that its going outside stable policy. It isn't the only case however.
OpenStack chose to use pip to *test* its Python dependency chain so that we can evaluate newer versions of those dependencies than the distros in question are carrying.
So that would not change then at all. with by-demand including version updates of our dependencies we're still going to do exactly that.
The stable branches are meant as a place for distro package maintainers to collaborate on selective backports of patches for their packaged versions of our software, and suddenly starting to have them depend on newer versions of external dependencies which don't follow the same branching model and cadence creates new challenges for verifying that critical and security fixes for *our* software continues to remain compatible with deps contemporary to when we initially created those branches.
I'm not saying we should bump all of our 500+ dependencies regularly on stable branches. the number of packages the pypi tool "safety" considers vulnerable is a very small quantity of dependencies that are in that bucket, I would say less than 1%.Lets not make the problem bigger than it is.
And I think we're better off picking a different solution for coordinating security updates to external dependencies of our stable branches, rather than trying to turn upper-constraints.txt into that while destabilizing its intended use.
The intended use was to keep stable branches "stable" (or in my interpretation: "working"). distribution vendors don't have much choice. they do have to patch security vulnerabilities. It is their core responsibility. It makes their life easier by having upstream testing and known working against those patched versions (which can not 100% be reflected in the current pip installed fashion, but we can approximate that once we agree on the policy that would describe that OpenStack CI is testing what users are likely going to hit). Greetings, Dirk