On 2019-05-07 22:50:21 +0200 (+0200), Dirk Müller wrote:

Am Di., 7. Mai 2019 um 22:30 Uhr schrieb Matthew Thode mthode@mthode.org:

...

Pike - 2.18.2 -> 2.20.1 - https://review.opendev.org/640727 Queens - 2.18.4 -> 2.20.1 - https://review.opendev.org/640710

Specifically it looks like we're already at the next issue, as tracked here:

https://github.com/kennethreitz/requests/issues/5065

Any concerns from anyone on these newer urllib3 updates? I guess we'll do them a bit later though.

It's still unclear to me why we're doing this at all. Our stable constraints lists are supposed to be a snapshot in time from when we released, modulo stable point release updates of the libraries we're maintaining. Agreeing to bump random dependencies on stable branches because of security vulnerabilities in them is a slippery slope toward our users expecting the project to be on top of vulnerability announcements for every one of the ~600 packages in our constraints list. Deployment projects already should not depend on our requirements team tracking security vulnerabilities, so need to have a mechanism to override constraints entries anyway if they're making such guarantees to their users (and I would also caution against doing that too).

Distributions are far better equipped than our project to handle such tracking, as they generally get advance notice of vulnerabilities and selectively backport fixes for them. Trying to accomplish the same with a mix of old and new dependency versions in our increasingly aging stable and extended maintenance branches seems like a disaster waiting to happen.</soapbox>

On 9 May 2019, at 13:10, Thierry Carrez thierry@openstack.org wrote:

I agree it is a bit of a slippery slope... We historically did not do that (stable branches are a convenience, not a product), because it is a lot of work to track and test vulnerable dependencies across multiple stable branches in a comprehensive manner.

Why update requests 2.18.4 for CVE-2018-18074, and not Jinja2 2.10.0 for CVE-2019-8341 ?

I'm not sure doing it on a case-by-case basis is a good idea either, as it might set unreasonable expectations.

A lot of operators make use of u-c for source-based builds to ensure consistency in the builds and to ensure that they’re using the same packages as those which were tested upstream. It makes sense to collaborate on something this important as far upstream as possible.

If we think of this as a community effort similar to the extended maintenance policy - the development community doesn’t *have* to implement the infrastructure to actively monitor for the vulnerabilities and respond to them. It can be maintained on a best effort basis by those interested in doing so.

To limit the effort involved we could agree to limit the scope to only allow changes to the current ‘maintained’ releases. For all other branches we can encourage an upgrade to a ‘maintained’ release by adding a release note.

To manage the 'unreasonable expectations’, we should document a policy to this effect.

On 19-05-09 13:48:09, Jeremy Stanley wrote:

On 2019-05-09 12:38:29 +0000 (+0000), Jesse Pretorius wrote: [...]

...

A lot of operators make use of u-c for source-based builds to ensure consistency in the builds and to ensure that they’re using the same packages as those which were tested upstream. It makes sense to collaborate on something this important as far upstream as possible.

[...]

See, this is what frightens me. We should *strongly* discourage them from doing this, period. If your deployment relies on distribution packages of dependencies then your distro's package maintainers have almost certainly received advance notice of many of these vulnerabilities and have fixes ready for you to download the moment they're made public. They're in most cases selectively backporting the fixes to the versions they carry so as to make them otherwise backward compatible and avoid knock-on effects involving a need to upgrade other transitive dependencies which are not involved in the vulnerability.

To extend on this, I thought that OSA had the ability to override certian constraints (meaning they could run the check and maintain the overrides on their end).

...

If we think of this as a community effort similar to the extended maintenance policy - the development community doesn’t *have* to implement the infrastructure to actively monitor for the vulnerabilities and respond to them. It can be maintained on a best effort basis by those interested in doing so.

By the time we find out and work through the transitive dependency bumps implied by this sort of change (because many of these ~600 dependencies of ours don't backport fixes or maintain multiple stable series of their own and so our only option is to upgrade to the latest version, and this brings with it removal of old features or reliance on newer versions of other transitive dependencies), we're long past public disclosure and the vulnerability has likely been getting exploited in the wild for some time. If a deployer/operator can't rely on our constraints list for having a timely and complete picture of a secure dependency tree then they already need local workarounds which are probably superior regardless. There are also plenty of non-Python dependencies for our software which can have vulnerabilities of their own, and those aren't reflected at all in our constraints lists. How are said users updating those?

There's also the problem for knock on dependencies. Update foo, which pulls in a new version of bar as required. Either of which can break the world (and on down the dep tree)

On 2019-05-13 09:42:30 -0400 (-0400), Jean-Philippe Evrard wrote: [...]

I agree with Jesse, we should do as much upstream as we can, so that the whole community benefits from it. If things are updated on a best effort basis in u-c, more than a single project benefits from this. If things are not updated on a best effort basis, then source based deployment projects should discuss together on making this a reality. In all cases, this deserves documentation if it's not documented already (I totally missed that part of the documentation myself).

I don't see anything wrong with a best-effort attempt by folks who build or rely on source-based deployments from stable branches, my primary concerns remain:

1. This goal is tangential to (and even conflicting with) the purpose of the requirements repository's upper-constraints.txt file so should probably be managed independently of that.

2. As a project we should be clear that this is a not-at-all-timely post-hoc attempt at reflecting somewhat secure deployment sets and can't guarantee we will always be able to find a solution for (or perhaps even notice) many future vulnerabilities in the transitive dependency tree where stable branches of our software are concerned.

Dirk Müller dirk@dmllr.de writes:

Jeremy wrote:

...

It's still unclear to me why we're doing this at all. Our stable constraints lists are supposed to be a snapshot in time from when we released, modulo stable point release updates of the libraries we're maintaining. Agreeing to bump random dependencies on stable branches because of security vulnerabilities in them is a slippery slope toward our users expecting the project to be on top of vulnerability announcements for every one of the ~600 packages in our constraints list.

I think this is combining two different viewpoints in one: "snapshot in time" and "user expects it to be updated asap on security vulnerabilities". We are already updating upper-constraints on bugfixes for projects that openstack maintains, and we do so similarly for security fixes for packages that are part of openstack. Also, distribution vendor provided (non-pip installed bindeps) are maintained and updated by security fixes.

But our motivation for updating the list when *we* release a package is that we want to test that package with the rest of our code. That's consistent with the original purpose of the list, which was to control which things we run in CI so we can have more control over when releases "break" us, and it isn't related to the reason for the releases.

On 2019-05-13 23:31:41 +0200 (+0200), Dirk Müller wrote: [...]

I think this is combining two different viewpoints in one: "snapshot in time" and "user expects it to be updated asap on security vulnerabilities".

I agree, that is the point I was trying to make... or moreso that the "snapshot in time" is the purpose upper-constraints.txt was intended to serve for stable branches so we can keep them... stable. On the other hand "user expects it to be updated asap on security vulnerabilities" sounds like a misconception we need to better document isn't the reason we have that mechanism.

We are already updating upper-constraints on bugfixes for projects that openstack maintains, and we do so similarly for security fixes for packages that are part of openstack.

Yes, in those cases we have selectively backported those fixes by themselves into the affected projects so as to minimize disruption to any other projects depending on them, and we update the constraints list so that they are tested with other projects' contemporary branches. Many (I expect most?) of our external Python dependencies do not follow a similar pattern, and those which do may not have the same opinions as to what constitutes a backward-incompatible change or may maintain different lifetimes for their various stable backport branches.

Also, distribution vendor provided (non-pip installed bindeps) are maintained and updated by security fixes.

Yes, and those vendors (at least for the versions of their distros we claim to test against) generally maintain a snapshot-in-time fork of those packages and selectively backport fixes to them, which is *why* we can depend on them being a generally stable test bed for us.

...

Deployment projects already should not depend on our requirements team tracking security vulnerabilities, so need to have a mechanism to override constraints entries anyway if they're making such guarantees to their users (and I would also caution against doing that too).

for traditional openstack deployment projects that might be well the case. However several virtualenv / container / dockerbuild based projects are using upperconstraints to generate a stable, coherent and tested container. Without adjustments those will not be including security fixes though.

Right, again we seem to agree on the risk, just not the source of the problem. I continue to argue that the underlying issue is the choice to reuse the existing upper-constraints.txt (which was invented for a different, conflicting purpose) rather than creating a solid solution to their problem. It's not a good idea to make the current constraints list less effective at solving its intended problem just so that it can be used to solve an unrelated one, regardless of how important solving that other problem might be.

...

Distributions are far better equipped than our project to handle such tracking, as they generally get advance notice of vulnerabilities and selectively backport fixes for them.

Agreed, still OpenStack chose to use pip for managing its dependencies, so I think it is preferable to find a solution within that ecosystem. Not every dependency with security issues is so offensive as "requests" is which does not maintain any stable branch but asks you to update to the latest version instead. Most others do have backports available on micro versions as pip installable project.

OpenStack chose to use pip to *test* its Python dependency chain so that we can evaluate newer versions of those dependencies than the distros in question are carrying. It sticks with the same model on stable branches to avoid having to maintain two fundamentally different mechanisms for installing Python dependencies for tests. This doesn't mean we necessarily think its a good model for production deployments of stable branches of our software (for exactly the security-related reasons being discussed in this thread). The stable branches are meant as a place for distro package maintainers to collaborate on selective backports of patches for their packaged versions of our software, and suddenly starting to have them depend on newer versions of external dependencies which don't follow the same branching model and cadence creates new challenges for verifying that critical and security fixes for *our* software continues to remain compatible with deps contemporary to when we initially created those branches.

...

accomplish the same with a mix of old and new dependency versions in our increasingly aging stable and extended maintenance branches seems like a disaster waiting to happen.

I agree it is a difficult exercise and we need to document a clear policy. For me documenting that it upper-constraints is maintained with security fix version updates as a best effort/risk bases is good enough, fwiw.

And I think we're better off picking a different solution for coordinating security updates to external dependencies of our stable branches, rather than trying to turn upper-constraints.txt into that while destabilizing its intended use.

Matthew Thode wrote:

[...] I don't like the idea of conflating the stability promise of upper-constraints.txt with the not quite fully tested-ness of adding security updates after the fact (while we do some cross testing, we do not and should not have 100% coverage, boiling the ocean).

Me neither.

The only way I can see this working is to have a separate file for security updates.

The idea I had (and don't like too much) is to do the following.

1. Keep upper-constraints.txt as is a. rename to tox-constraints possibly
2. add a new file, let's call it 'security-updates.txt' a. in this file goes security updates and all the knock on updates that it causes (foo pulls in a new bersion of bar and baz). b. the file needs to maintain co-installability of openstack. It is laid over the upper-constraints file and tested the same way upper-constraints is. This testing is NOT perfect. The generated file could be called something like 'somewhat-tested-secureconstraints.txt'
3. global-requirements.txt remains the same (minimum not updated for security issues)

This would increase test sprawl quite a bit (tests need to be run on any constraints change on this larger set). This also sets up incrased work and scope for the requirements team. Perhaps this could be a sub team type of item or something?

I'm a bit worried that a security-updates.txt would promise more than we can deliver. While we may be able to track vulnerabilities in our direct dependencies, we can't rely on *them* to properly avoid depending on vulnerable second-level dependencies (and so on). And this solution does not cover non-Python dependencies. So saying "use this to be secure" is just misleading our users.

Nothing short of full distribution security work can actually deliver on a "use this to be secure" promise. And that is definitely not the kind of effort we should tackle as a community imho.

I understand the need to signal critical vulnerabilities in our dependencies to our users and distributions. Historically we have used the OSSN (OpenStack Security Notices) to draw attention to select, key vulnerabilities in our dependency chain:

OSSN-0082 - Heap and Stack based buffer overflows in dnsmasq<2.78 OSSN-0044 - Older versions of noVNC allow session theft OSSN-0043 - glibc 'Ghost' vulnerability can allow remote code execution ...

I would rather continue to use that mechanism to communicate about critical vulnerabilities in all our dependencies than implement a complex and costly process to only cover /some/ of our Python dependencies.

Hi,

dependencies, we can't rely on *them* to properly avoid depending on vulnerable second-level dependencies (and so on). And this solution does not cover non-Python dependencies. So saying "use this to be secure" is just misleading our users.

We typically don't use pip constraints for non-python dependencies, so that is an orthogonal problem. Yes, it doesn't tell you that you need to install kernel updates to be secure but since we're not documenting the kernel version that OpenStack requires you to be having installed to begin with its not really part of the problem scope..

Nothing short of full distribution security work can actually deliver on a "use this to be secure" promise. And that is definitely not the kind of effort we should tackle as a community imho.

See my previous reply to Jeremy. I'm not asking for the community to suddenly replace the paid work of distribution vendors. What I was aiming at is that we could align our testing coverage with the situation that end users might likely end up, which is something along the lines of "the upper-constraints version of the dependency + distro specific bugfixes + customer/user requested bugfixes + security backports". Hopefully the vendor follows good practices and upstreams those changes, which means a stable release of the version that OpenStack depends on would pretty closely match end user situation, which is good because we want to test that functionality in our CI.

I would rather continue to use that mechanism to communicate about critical vulnerabilities in all our dependencies than implement a complex and costly process to only cover /some/ of our Python dependencies.

I think announcing such changes in our upper-constraints via OSSN is an excellent idea, totally on board with that.

Greetings, Dirk

On 19-05-22 23:52:47, Dirk Müller wrote:

Hi Matthew,

...

2. add a new file, let's call it 'security-updates.txt'

maybe better call it updates-for-known-insecure-versions.txt ;-)

...

b. the file needs to maintain co-installability of openstack. It is laid over the upper-constraints file and tested the same way upper-constraints is. This testing is NOT perfect. The generated file could be called something like 'somewhat-tested-secureconstraints.txt'

coinstallability is a problem, but I think its not the main one. But I agree we can try that.

...

This also sets up incrased work and scope for the requirements team. Perhaps this could be a sub team type of item or something?

Allowing for additions there doesn't immediately increase work. unless there is somebody actually proposing a change to review, that is. It doesn"t make the team magically fulfill the promise - the policy change would allow the review team to accept such a review as it is within policy.

These are all true, but even before changing anything we'd still have to document the policy. Perhaps that's the next step. Do you mind generating a policy change and proposing it (to this thread) for review?

Hi Jeremy,

I agree, that is the point I was trying to make... or moreso that the "snapshot in time" is the purpose upper-constraints.txt was intended to serve for stable branches so we can keep them... stable.

I would replace "stable" with "working" in that sentence (e.g. existing test coverage still passes). I do agree with that in general.

On the other hand "user expects it to be updated asap on security vulnerabilities" sounds like a misconception we need to better document isn't the reason we have that mechanism.

I am not sure this was implied. The current suggested mode was to clear out known security vulnerabilities, which is a different angle than guaranteeing that everything is immediately secure. Fixing known issues still means there might be unfixed issues left.

For me the point of that exercise is not to ensure that pure pip/upper-constraints built environments are *secure*, but that they still *work*. I would not want to extrapolate that just because we're fixing known vulnerable versions in our dependency chain by fixed versions that we're magically secure.

contemporary branches. Many (I expect most?) of our external Python dependencies do not follow a similar pattern

For many projects with security problems there is actually a sane backporting strategy implemented. As an example there is django, where we opt for the LTS version and that one is very cautiously maintained with only security and critical bugfixes (data-loss/data corruption type). I would rather up the constraints on stable branches to verify that our test coverage is still passing with that change than to rely on distributors magically doing better backports of the security fixes than upstream does and then not shipping that to their customers without finding that the security patch actually broke some useful functionality of the project. django might be a good example and a bad example here, because I actually know that horizon *ignores* upper-constraints so the django version we're having in upper-constraints is not the one that is actually being used in the testing, which is another reason why I would like to make it actually match* the actually used version because thats the one the code is most likely working with and it then serves at least as a documentation type thing.

Yes, and those vendors (at least for the versions of their distros we claim to test against) generally maintain a snapshot-in-time fork of those packages and selectively backport fixes to them, which is *why* we can depend on them being a generally stable test bed for us.

I do happen to work for one of those vendors, and yes, upper-constraints is an important consideration for the choice of versions that are being shipped. I happen to come accross a situation every once in a while where a security fix the upstream project wasn't such a super great thing and caused regressions in some other project somewhere that was accidentally or intentionally using some of the previously vulnerable functionality. OpenStack can be such a case. Thats another reason why I would like to ensure that the "stable version with the vulnerable behavior fixed" version is actually used in testing so that we know it doesn't break functionality and that also OpenStack does not introduce changes that rely on behavior that has been patched out by vulnerability fixes in the downstream vendor distributions.

A security fix could intentionally change the observed behavior of the dependency because that is the essential part of the security fix. One example of that could be that it rejects a certain invalid input that it sloppily ignored beforehand. By running our testing against the version that still allows the invalid input we're not finding the issue. Distribution vendors that are careful and provide security fixes to their user/customer base will do the security backport and release that as an update, either with some testing or with the implicit assumption that necessary cross-checking has already been performed.

If it does cause a problem, its is going to be a security fix that explodes right into the face of the user of OpenStack in this case. Eventually, every once in a while there is a production outage due to that, and customers are generally not happy about that.

the naive reaction of the user is either "Ok, this vendor is not able to patch things properly" or "OpenStack is broken by the security fix". Both are uncomfortable situations to be in. I don't ask you to walk in the shoes of the distribution vendor that goes through that customer conversation, but I would like to ask you to consider the case that OpenStack needs to *deal* with behavior changes in its dependencies that are introduced as an essential part or the side effects of security fixes being done there.

I can give some theoretical examples. There were cases where dependencies ran code that left files with insecure permissions. Quite an obvious problem, and quite easy to fix. Suddenly files have secure permissions and openstack breaks. There were cases where a url was sanitized and slightly malformatted urls were rejected, and openstack generated those and fell into the trap. Yes, all of that can be fixed by patches, but there is no testing of those backports because theyre being executed against the *still vulnerable* version.

Right, again we seem to agree on the risk, just not the source of the problem. I continue to argue that the underlying issue is the choice to reuse the existing upper-constraints.txt (which was invented for a different, conflicting purpose) rather than creating a solid solution to their problem.

Perhaps the projects that currently use upper constraints don't care about a secure virtualenv/container build, and thats fine. It still does have a point to test against the versions end users will most likely have, and they most likely have security fixed versions (because they're good users and run against a stable security maintained enterprise operating system). We'd be doing ourselves a favor by testing a situation that is coming close to the end user situation in our CI.

Yes, we can debate that upping "requests" to a 3 version newer version is stretching the limits. I think thats a very useful conversation to have, and it doesn't take me a lot to feel convinced that its going outside stable policy. It isn't the only case however.

OpenStack chose to use pip to *test* its Python dependency chain so that we can evaluate newer versions of those dependencies than the distros in question are carrying.

So that would not change then at all. with by-demand including version updates of our dependencies we're still going to do exactly that.

The stable branches are meant as a place for distro package maintainers to collaborate on selective backports of patches for their packaged versions of our software, and suddenly starting to have them depend on newer versions of external dependencies which don't follow the same branching model and cadence creates new challenges for verifying that critical and security fixes for *our* software continues to remain compatible with deps contemporary to when we initially created those branches.

I'm not saying we should bump all of our 500+ dependencies regularly on stable branches. the number of packages the pypi tool "safety" considers vulnerable is a very small quantity of dependencies that are in that bucket, I would say less than 1%.Lets not make the problem bigger than it is.

And I think we're better off picking a different solution for coordinating security updates to external dependencies of our stable branches, rather than trying to turn upper-constraints.txt into that while destabilizing its intended use.

The intended use was to keep stable branches "stable" (or in my interpretation: "working"). distribution vendors don't have much choice. they do have to patch security vulnerabilities. It is their core responsibility. It makes their life easier by having upstream testing and known working against those patched versions (which can not 100% be reflected in the current pip installed fashion, but we can approximate that once we agree on the policy that would describe that OpenStack CI is testing what users are likely going to hit).

Greetings, Dirk

Hi Jeremy,

Doing conformance testing on those distros with their packaged versions of our external dependencies would much more closely approximate what I think you want

I think that would also work. Would the community be interested in solving conformance incompatibilities when purely vendored versions are used? I somehow have doubts. Would we track the vendored version/releases in a constraints file to ensure gating issues are not creeping in?

All the existing tooling is around tracking lower and upper constraints as defined by pip and our opendev defined wheel mirrors.

Unless we have a tool that translate pip install commands into the respective distribution equivalent, such a vendored-test also adds significant drag for projects : maintaining two different ways to install things and for X number of vendors to cross-check and help debug solve integration issues. Plus the amount of extra CI load this might cause. Not a fun task.

Considering that I would prefer to volunteer maintaining a pypi/pip wheel fork of the ~5 dependencies with security vulnerabilities that we care about and pull those in instead of exposing the full scope of X vendors downstream specific patching issues to us as a community.

Greetings, Dirk

Hi Jeremy,

DevStack used to support this, but it does indeed seem to have been refactored out some time ago. Reintroducing that, or something like it, could be an alternative solution though.

Most of interesting test coverage is in the project's functional test jobs as well, so "just" devstack alone isn't enough, all the projects need to support this variation as well.

Do we really only care about 5 out of our many hundreds of external Python dependencies?

So running safety against our stable branch spills out this list of packages:

ansible-runner cryptography django numpy msgpack pyOpenSSL urllib3 requests

plus transitive (docker, zhmcclient, kubernetes). numpy is hopefully to ignore in our use case, and django is used unconstrained anyway in the gate, so I would imho remove it from this list. msgpack is an API change, so unsuitable as well. transitive changes are not needed if we have backports in the gate. so this totals in ~5 packages to maintain. I am not looking at branches that are under extended maintenance fwiw.

dependency sets (so that our stable branches don't inadvertently develop a requirement for a new feature in newer versions of these dependencies)

lower-constraints.txt jobs try to ensure that this won't happen, assuming that we somehow bump the version numbers to X.X.X.post1, so thats not an additional thing to worry about.

*and* as long as we make it clear to users that this is not a substitute for running on security-supported distro packages (where someone more accountable and read-in than the OpenStack project is backporting patches for vulnerabilities to forks of those dependencies).

I'm not aware of any statement anywhere that we'd be security maintaining a distro? Where would we state that? in the project's documentation/README?

Greetings, Dirk

On 19-05-22 23:53:50, Jeremy Stanley wrote:

On 2019-05-23 01:09:58 +0200 (+0200), Dirk Müller wrote:

...

Hi Jeremy,

...

Doing conformance testing on those distros with their packaged versions of our external dependencies would much more closely approximate what I think you want

I think that would also work. Would the community be interested in solving conformance incompatibilities when purely vendored versions are used? I somehow have doubts. Would we track the vendored version/releases in a constraints file to ensure gating issues are not creeping in?

I don't know that we need to if the goal is to let us know (e.g. with a periodic job) that a distro we care about has upgraded a dependency in a way that our stable branch targeting that distro version no longer works with.

...

All the existing tooling is around tracking lower and upper constraints as defined by pip and our opendev defined wheel mirrors.

Unless we have a tool that translate pip install commands into the respective distribution equivalent, such a vendored-test also adds significant drag for projects : maintaining two different ways to install things and for X number of vendors to cross-check and help debug solve integration issues. Plus the amount of extra CI load this might cause. Not a fun task.

DevStack used to support this, but it does indeed seem to have been refactored out some time ago. Reintroducing that, or something like it, could be an alternative solution though.

...

Considering that I would prefer to volunteer maintaining a pypi/pip wheel fork of the ~5 dependencies with security vulnerabilities that we care about and pull those in instead of exposing the full scope of X vendors downstream specific patching issues to us as a community.

Do we really only care about 5 out of our many hundreds of external Python dependencies? Or is it that we should assume over years of maintenance, fewer than one percent of them will discover vulnerabilities? At any rate, I'm not opposed to the experiment as long as we can still also run jobs for our original frozen dependency sets (so that our stable branches don't inadvertently develop a requirement for a new feature in newer versions of these dependencies) *and* as long as we make it clear to users that this is not a substitute for running on security-supported distro packages (where someone more accountable and read-in than the OpenStack project is backporting patches for vulnerabilities to forks of those dependencies).

I don't know if we only care about certain things. But if we go forward with this accepting changes that pass tests and go into another constraints file (not upper-constraints) and not actively submitting them ourselves. More opportunistic then active on our part.

The other constraints file should have a LARGE warning saying that this is not a substitute for actual security backports as it is not complete (is opportunistic).