Open Stack

 Hi, I didn't really want to get into this, but we (VF squad) have recently
encountered an issue that might provide some context to this discussion.

Due to omission on my part, we have been building pdf documentation for
validations-common without querying openstack/requirements
upper-constrainst.
This meant that we were always pulling the newest version of packages, and
as the job was triggered only rarely, we didn't see the problem until
promotes began to fail.
A new version of one of our dependencies (I believe it was sphinx)
introduced, by proxy, dependency on the "tgtermes.sty" style file.

Lack of upper constraints made the issue harder to diagnose, as the version
of packages to installed changed, depending on their relative dependencies
and continuous updates.
With several dozen packages, there is considerable chance of one getting a
version bump every week or so.

Which brings me to my point. The openstack/requirements does provide one
rather essential service for us. In the form of upper-constraints for our
pip builds.
While we are mostly installing software through rpm, many CI jobs use pip
in some fashion. Without upper constraints, pip pulls aggressively the
newest version available and compatible with other packages.
Which causes several issues, noted by even pip people.

There is also a question of security. There is a possibility of a bad actor
introducing a package with an extremely high version number.
Such a package would get precedence over the legitimate releases. In fact,
just this sort of attack was spotted in the wild.[1]

Now, nothing is preventing us from using upper requirements, without being
in the openstack/requirements projects.
On the other hand, if we remove ourselves from the covenant, nothing is
stopping the openstack/requirements people from changing versions of the
accepted packages
without considering the impact it could have on our projects.

I believe this is something we need to consider.
[1] (https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610)

On Mon, May 17, 2021 at 8:55 AM Rabi Mishra <ramishra at redhat.com> wrote:

>
>
> On Fri, May 14, 2021 at 9:07 PM Javier Pena <jpena at redhat.com> wrote:
>
>>
>> On Fri, May 14, 2021 at 6:41 AM Marios Andreou <marios at redhat.com> wrote:
>>
>>> On Thu, May 13, 2021 at 9:41 PM James Slagle <james.slagle at gmail.com>
>>> wrote:
>>> >
>>> > I'd like to propose that TripleO opt out of dependency management by
>>> removing tripleo-common from global-requirements.txt.  I do not feel that
>>> the global dependency management brings any advantages or anything needed
>>> for TripleO. I can't think of any reason to enforce the ability to be
>>> globally pip installable with the rest of OpenStack.
>>>
>>> To add a bit more context as to how this discussion came about, we
>>> tried to remove tripleo-common from global-requirements and the
>>> check-requirements jobs in tripleoclient caused [1] as a result.
>>>
>>> So we need to decide whether to continue to be part of that
>>> requirements contract [2] or if we will do what is proposed here and
>>> remove ourselves altogether.
>>> If we decide _not_ to implement this proposal then we will also have
>>> to add the requirements-check jobs in tripleo-ansible [3] and
>>> tripleo-validations [4] as they are currently missing.
>>>
>>> >
>>> > Two of our most critical projects, tripleoclient and tripleo-common do
>>> not even put many of their data files in the right place where our code
>>> expects them when they are pip installed. So, I feel fairly confident that
>>> no one is pip installing TripleO and relying on global requirements
>>> enforcement.
>>>
>>> I don't think this is _just_ about pip installs. It is generally about
>>> the contents of each project requirements.txt. As part of the
>>> requirements contract, it means that those repos with which we are
>>> participating (the ones in projects.txt [5]) are protected against
>>> other projects making any breaking changes in _their_
>>> requirements.txt. Don't the contents of requirements.txt also end up
>>> in the .spec file from which we are building rpm e.g. [6] for tht? In
>>> which case if we remove this and just stop catching any breaking
>>> changes in the check/gate check-requirements jobs, I suspect we will
>>> just move the problem to the rpm build and it will fail there.
>>>
>>
>> I don't see that in the spec file. Unless there is some other automation
>> somewhere that regenerates all of the BuildRequires/Requires and modifies
>> them to match requirements.txt/test-requirements.txt?
>>
>> We run periodic syncs once every cycle, and try our best to make spec
>> requirements match requirements.txt/test-requirements.txt for the project.
>> See
>> https://review.rdoproject.org/r/c/openstack/tripleoclient-distgit/+/33367
>> for a recent example on tripleoclient.
>>
>> I don't have a special opinion on keeping tripleo-common inside
>> global-requirements.txt or not. However, all TripleO projects still need to
>> be co-installable with other OpenStack projects, otherwise we will not be
>> able to build packages for them due to all the dependency issues that could
>> arise.
>>
>
> I think this is probably less of an issue with containerized services(?).
> At present,  there are only two service containers that install
> tripleo-common (mistral,  nova-scheduler). With mistral deprecated and nova
> removed from the undercloud (tripleo-common has a tripleo specific
> scheduler filter used by undercloud nova), we probably won't have many
> issues.
>
> However, as we sync requirements from project requirements to package
> specs regularly, there could be issues with broken  requirements.
>
>
>
>
>> I'm not sure if that was implied in the original post.
>>
>> Regards,
>> Javier
>>
>>
>>
>> >
>>> > One potential advantage of not being in global-requirements.txt is
>>> that our unit tests and functional tests could actually test the same code.
>>> As things stand today, our unit tests in projects that depend on
>>> tripleo-common are pinned to the version in global-requirements.txt, while
>>> our functional tests currently run with tripleo-common from master (or
>>> included depends-on).
>>>
>>> I don't think one in particular is a very valid point though - as
>>> things currently stand in global-requirements we aren't 'pinning' all
>>> we have there is "tripleo-common!=11.3.0  # Apache-2.0" [7] to avoid
>>> (I assume) some bad release we made.
>>>
>>
>> tripleo-common is pinned to the latest release when it's pip installed in
>> the venv, instead of using latest git (and including depends-on). You're
>> right that it's probably what we want to keep doing, and this is probably
>> not related to opting out of g-r. Especially since we don't want to require
>> latest git of a dependency when running unit tests locally. However it is
>> worth noting that our unit tests (tox) and functional tests (tripleo-ci)
>> use different code for the dependencies. That was not obvious to me and
>> others on the surface. Perhaps we could add additional tox jobs that do
>> require the latest tripleo-common from git to also cover that scenario.
>>
>> Here's an example:
>> https://review.opendev.org/c/openstack/python-tripleoclient/+/787907
>> https://review.opendev.org/c/openstack/tripleo-common/+/787906
>>
>> The tripleo-common patch fails unit tests as expected, the tripleoclient
>> which depends-on the tripleo-common patch passes unit tests, but fails
>> functional. I'd rather see that failure caught by a unit test as well.
>>
>>
>> --
>> -- James Slagle
>> --
>>
>>
>>
>
> --
> Regards,
> Rabi Mishra
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210519/2af01280/attachment-0001.html>