On 2021-12-09 16:57:27 +0100 (+0100), Jean-Philippe Evrard wrote: [...]
So the question is: Is OpenStack ready to move forward and adapt to new kinds of packaging/software? If yes, let's modify the PTIs (or python PTI) to cover the case for skyline, and allow that team to move forward. It's not like it's doing something very different than some new python projects.
Yes, OpenStack is mature... I (simply) hope it's also a place for casual contributions, and where developers can have fun with tech. And yes, it won't please everyone from day one (especially distros), but I hope we can work this out together, rather than discard the project because it doesn't fit the mold.
[...]
The reason I raised those points on the review was in order to highlight inconsistencies with OpenStack as a whole. If OpenStack is defined by anything, it's consistent development workflows and automation, which allow small groups of people to take advantage of economies of scale in order to minimize maintenance over time across the whole of the set of projects which comprise it. Inconsistency has an inherent cost which the entire community bears, from tracking integration failures in development to keeping concise recipes for packaging in distributions to operators being able to configure all services the same way and expect the same logging formats.
Most of the benefits a project gets from being "part of OpenStack" are derived directly from the established standards and norms which enable OpenStack to be treated as a cohesive whole rather than a mere "bucket of parts" with tooling choices made independently by their respective development teams. Deviation from these expectations should be deliberate and come with a clear rationale which benefits OpenStack as a whole rather than one project alone.
Let's take the current example point by point:
* Skyline implements its own routines for things like configuration and logging rather than using Oslo libraries. Should all OpenStack projects "move forward" to this model, choose their own configuration and logging formats, implement their own solutions for documenting these, and so on? Does inconsistency here benefit operators in some way?
* Skyline defines the interdependencies between its components with a mix of "monorepo" (multiple packages from a single repository) and Git submodule references (pinning specific commit ids as virtual subtrees of another repository), rather than establishing a loose coupling with explicit version numbers and package relationships. This choice prevents them from taking advantage of features in Zuul like cross-project change dependencies. Is this a pattern the other OpenStack projects should follow? Does it benefit distribution package maintainers to need to track fundamentally different repository layouts and build processes for each of them?
* Skyline performs its release versioning by committing the version strings to files in its repositories rather than assigning them at build time from Git tag metadata. This leads to ambiguity as to what exact state of the repository represents that version, as well as opening up the possibility of forgetting to merge the version update before tagging (a very common problem which drove OpenStack to rely on its current model). Wholly different tools are also needed in order to track versions for these projects as a result. Should the rest of OpenStack's projects follow suit there? Has something changed to make the idea of using Git to signal releases a bad one?
* Skyline uses makefiles to script its build and test workflows rather than tox, adding a tox wrapper around the make targets in order to nominally comply with the OpenStack PTI. As a result, it's unable to rely on most of tox's features for environment isolation, package management, path overrides, et cetera. OpenStack projects have solved a lot of cross-project development workflow challenges through applying consistent changes across the tox.ini files of repositories, should those solutions be abandoned and recreated for make instead?
* Skyline does not put its external Python package dependencies in central lists nor use tools which would allow it to rely on OpenStack's global constraints implementation, making it very hard for them to guarantee that their software can run with the same versions of shared dependencies as the rest of OpenStack. Is this unnecessary? Have we given up on the idea that two pieces of OpenStack software can be coinstalled into one environment, or that Linux distributions won't be stuck having to package many different versions of the same software because OpenStack projects don't agree on which versions work for them any longer?
To be perfectly clear, I don't oppose accepting Skyline as a project in OpenStack, even in its current state, but I do feel like it's not going to end up released as part of OpenStack unless it can address the challenges I outlined above. Most of these are fairly superficial changes to implement, however, and can be done by simply copying the solutions already used in other OpenStack projects or retroactively applying our cookie-cutters. "Moving OpenStack forward" to be more like Skyline, on the other hand, is bound to be far more work than the Skyline team can handle on its own and take far longer, though I'm not opposed to the idea of discussing which of these differences may make sense for OpenStack to adopt in the future.
My deeper concern is that these differences are a clear indication that the software was written outside the OpenStack community, with no original consideration for consistency with OpenStack itself, and so there are probably more significant architectural modifications needed (beyond integrating oslo.config and oslo.log) which I'm not well-placed to spot. Those can addressed in time too, and while Skyline is an official team within OpenStack, but they will still need to be taken care of at some point early in the life of the project either by becoming like the rest of OpenStack or helping the rest of OpenStack become more like Skyline.