Open Stack

Hi Jeremy,

> I agree, that is the point I was trying to make... or moreso that
> the "snapshot in time" is the purpose upper-constraints.txt was
> intended to serve for stable branches so we can keep them... stable.

I would replace "stable" with "working" in that sentence (e.g.
existing test coverage
still passes). I do agree with that in general.

> On the other hand "user expects it to be updated asap on security
> vulnerabilities" sounds like a misconception we need to better
> document isn't the reason we have that mechanism.

I am not sure this was implied. The current suggested mode was to
clear out known security vulnerabilities, which is a different angle than
guaranteeing that everything is immediately secure. Fixing known issues
still means there might be unfixed issues left.

For me the point of that exercise is not to ensure that pure
pip/upper-constraints built environments are *secure*,
but that they still *work*. I would not want to extrapolate that just
because we're fixing
known vulnerable versions in our dependency chain by fixed versions that
we're magically secure.

> contemporary branches. Many (I expect most?) of our external Python
> dependencies do not follow a similar pattern

For many projects with security problems there is actually a sane backporting
strategy implemented. As an example there is django, where we opt for
the LTS version and that one is very cautiously maintained with only security
and critical bugfixes (data-loss/data corruption type). I would rather
up the constraints on stable branches to verify that our test coverage is still
passing with that change than to rely on distributors magically doing better
backports of the security fixes than upstream does and then not shipping
that to their customers without finding that the security patch actually broke
some useful functionality of the project. django might be a good example and
a bad example here, because I actually know that horizon *ignores*
upper-constraints
so the django version we're having in upper-constraints is not the one that
is actually being used in the testing, which is another reason why I would
like to make it actually match* the actually used version because thats
the one the code is most likely working with and it then serves at least
as a documentation type thing.

> Yes, and those vendors (at least for the versions of their distros
> we claim to test against) generally maintain a snapshot-in-time fork
> of those packages and selectively backport fixes to them, which is
> *why* we can depend on them being a generally stable test bed for
> us.

I do happen to work for one of those vendors, and yes, upper-constraints
is an important consideration for the choice of versions that are being
shipped. I happen to come accross a situation every once in a while where
a security fix the upstream project wasn't such a super great thing
and caused regressions in some other project somewhere that was
accidentally or intentionally using some of the previously vulnerable
functionality. OpenStack can be such a case. Thats another reason
why I would like to ensure that the "stable version with the vulnerable
behavior fixed" version is actually used in testing so that we know
it doesn't break functionality and that also OpenStack does not introduce
changes that rely on behavior that has been patched out by vulnerability
fixes in the downstream vendor distributions.

A security fix could intentionally change the observed behavior of the
dependency because that is the essential part of the security fix. One
example of that could be that it rejects a certain invalid input that it
sloppily ignored beforehand. By running our testing against the version that
still allows the invalid input we're not finding the issue. Distribution
vendors that are careful and provide security fixes to their user/customer
base will do the security backport and release that as an update,
either with some testing or with the implicit assumption that necessary
cross-checking has already been performed.

If it does cause a problem, its is going to be a security fix
that explodes right into the face of the user of OpenStack in this case.
Eventually, every once in a while there is a production outage due to that,
and customers are generally not happy about that.

the naive reaction of the user is either "Ok, this vendor is not able to
patch things properly" or "OpenStack is broken by the security fix". Both
are uncomfortable situations to be in. I don't ask you to walk in the
shoes of the distribution vendor that goes through that customer conversation,
but I would like to ask you to consider the case that OpenStack needs
to *deal* with behavior changes in its dependencies that are introduced
as an essential part or the side effects of security fixes being done there.

I can give some theoretical examples. There were cases where
dependencies ran code that left files with insecure permissions. Quite
an obvious problem, and quite easy to fix. Suddenly files have secure
permissions and openstack breaks. There were cases where a url
was sanitized and slightly malformatted urls were rejected, and openstack
generated those and fell into the trap. Yes, all of that can be fixed by
patches, but there is no testing of those backports because theyre
being executed against the *still vulnerable* version.

> Right, again we seem to agree on the risk, just not the source of
> the problem. I continue to argue that the underlying issue is the
> choice to reuse the existing upper-constraints.txt (which was
> invented for a different, conflicting purpose) rather than creating
> a solid solution to their problem.

Perhaps the projects that currently use upper constraints don't care
about a secure virtualenv/container build, and thats fine. It
still does have a point to test against the versions end users
will most likely have, and they most likely have security
fixed versions (because they're good users and run
against a stable security maintained enterprise operating
system). We'd be doing ourselves a favor by testing
a situation that is coming close to the end user situation in
our CI.

Yes, we can debate that upping "requests" to a 3 version
newer version is stretching the limits. I think thats a very
useful conversation to have, and it doesn't take me a
lot to feel convinced that its going outside stable policy.
It isn't the only case however.

> OpenStack chose to use pip to *test* its Python dependency chain so
> that we can evaluate newer versions of those dependencies than the
> distros in question are carrying.

So that would not change then at all. with by-demand including
version updates of our dependencies we're still going to do exactly
that.

> The stable branches are meant as a place for distro package
> maintainers to collaborate on selective backports of patches for
> their packaged versions of our software, and suddenly starting to
> have them depend on newer versions of external dependencies which
> don't follow the same branching model and cadence creates new
> challenges for verifying that critical and security fixes for *our*
> software continues to remain compatible with deps contemporary to
> when we initially created those branches.

I'm not saying we should bump all of our 500+ dependencies regularly
on stable branches. the number of packages the pypi tool "safety"
considers vulnerable is a very small quantity of
dependencies that are in that bucket, I would say less than 1%.Lets
not make the problem bigger than it is.

> And I think we're better off picking a different solution for
> coordinating security updates to external dependencies of our stable
> branches, rather than trying to turn upper-constraints.txt into that
> while destabilizing its intended use.

The intended use was to keep stable branches "stable" (or in
my interpretation: "working"). distribution vendors don't have
much choice. they do have to patch security vulnerabilities.
It is their core responsibility. It makes their life easier by
having upstream testing and known working against those
patched versions (which can not 100% be reflected
in the current pip installed fashion, but we can approximate
that once we agree on the policy that would describe
that OpenStack CI is testing what users are likely going to
hit).

Greetings,
Dirk