[all][tc] What's happening in Technical Committee: summary 15th Oct, 21: Reading: 5 min
Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * TC this week IRC meeting held on Oct 14th Thursday. * Most of the meeting discussions are summarized below (Completed or in-progress activities section). Meeting full logs are available @ - https://meetings.opendev.org/meetings/tc/2021/tc.2021-10-14-15.00.log.html * Next week's meeting is cancelled as we are meeting in PTG. We will have the next IRC meeting on Oct 28th, Thursday 15:00 UTC, feel free the topic on agenda[1] by Oct 27th. 2. What we completed this week: ========================= * None in this week. 3. Activities In progress: ================== TC Tracker for Xena cycle ------------------------------ * TC is using the etherpad[2] for Xena cycle working item. We will be checking it in PTG * Current status is: 9 completed, 3 to-be-discussed in PTG, 1 in-progress Open Reviews ----------------- * Six open reviews for ongoing activities[3]. New project 'Skyline' proposal ------------------------------------ * You might be aware of this new dashboard proposal in the previous month's discussion. * A new project 'Skyline: an OpenStack dashboard optimized by UI and UE' is now proposed in governance to be an official OpenStack project[4]. * Skyline team is planning to meet in PTG on Tue, Wed and Thu at 5UTC, please ask your queries or have feedback/discussion with the team next week. Place to maintain the external hosted ELK, E-R, O-H services ------------------------------------------------------------------------- * We had a final discussion or I will say just a status update on this which was mentioned in last week's email summary[5]. * Now onwards, discussion and migration work will be done in TACT SIG (#openstack-infra IRC channel). Add project health check tool ----------------------------------- * No updates on this, we will continue discussing it in PTG for the next steps on this and what to do with TC liaison things. * Meanwhile, we are reviewing Rico proposal on collecting stats tools [6]. Stable Core team process change --------------------------------------- * Current proposal is under review[7]. Feel free to provide early feedback if you have any. Call for 'Technical Writing' SIG Chair/Maintainers ---------------------------------------------------------- * As agreed in last week's TC meeting, we will be moving this SIG work towards TC. * TC members have been added to the core members list in the SIG repos. * We will be discussing where to move the training repos/work in PTG. TC tags analysis ------------------- * Operator feedback is asked on open infra newsletter too, and we will continue the discussion in PTG and will take the final decision based on feedback we receive, if any[9]. Complete the policy pop up team ---------------------------------------- * Policy pop team has served its purpose and we have new RBAC as one of the the community-wide goal for the Yoga cycle. * We are marking this popup team as completed[10]. Project updates ------------------- * Retiring js-openstack-lib [11] Yoga release community-wide goal ----------------------------------------- * Please add the possible candidates in this etherpad [12]. * Current status: "Secure RBAC" is selected for Yoga cycle[13]. PTG planning ---------------- * We will be meeting in PTG next week, please check the details in this etherpad [14] * Do not forget to join the TC+community leaders sessions on Monday, Oct 18 15 UTC - 17 UTC. Test support for TLS default: ---------------------------------- * Rico has started a separate email thread over testing with tls-proxy enabled[15], we encourage projects to participate in that testing and help to enable the tls-proxy in gate testing. 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[16]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [17] 3. Office hours: The Technical Committee offers a weekly office hour every Tuesday at 0100 UTC [18] 4. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://etherpad.opendev.org/p/tc-xena-tracke [3] https://review.opendev.org/q/projects:openstack/governance+status:open [4] https://review.opendev.org/c/openstack/governance/+/814037 [5]http://lists.openstack.org/pipermail/openstack-discuss/2021-October/025251.h... [6] https://review.opendev.org/c/openstack/governance/+/810037 [7] https://review.opendev.org/c/openstack/governance/+/810721 [9] https://governance.openstack.org/tc/reference/tags/index.html [10] https://review.opendev.org/c/openstack/governance/+/814186 [11] https://review.opendev.org/c/openstack/governance/+/798540 [12] https://review.opendev.org/c/openstack/governance/+/807163 [13] https://etherpad.opendev.org/p/y-series-goals [14] https://etherpad.opendev.org/p/tc-yoga-ptg [15] http://lists.openstack.org/pipermail/openstack-discuss/2021-June/023000.html [16] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [17] http://eavesdrop.openstack.org/#Technical_Committee_Meeting [18] http://eavesdrop.openstack.org/#Technical_Committee_Office_hours -gmann
On 10/15/21 6:07 PM, Ghanshyam Mann wrote:
New project 'Skyline' proposal ------------------------------------ * You might be aware of this new dashboard proposal in the previous month's discussion. * A new project 'Skyline: an OpenStack dashboard optimized by UI and UE' is now proposed in governance to be an official OpenStack project[4]. * Skyline team is planning to meet in PTG on Tue, Wed and Thu at 5UTC, please ask your queries or have feedback/discussion with the team next week.
Skyline looks nice. However, looking nice isn't enough. Before it becomes an OpenStack official component, maybe it should first try to reach our standard. I'm namely thinking about having a proper setuptool integration (using PBR?) for example, and starting tagging releases. I'm very much interested in packaging this for Debian/Ubuntu, if it's not a JS dependency hell. Though the current Makefile thingy doesn't look appealing. I've seen the console has at least 40 JS direct dependency. How many indirect dependency is this? Has anyone looked into it? Is the team ready to help making it package-able in a distro policy compliant way? Your thoughts? Cheers, Thomas Goirand (zigo)
Skyline-apiserver is a pure Python code project, following the Python wheel packaging standard, using pip for installation, and the dependency management of the project using poetry[1] Skyline-console uses npm for dependency management, development and testing. During the packaging and distribution process, webpack will be used to process the source code and dependent library code first, and output the packaged static resource files. These static resource files will be stored in an empty Python module[2]. The file directory is for example: - skyline_console - __init__.py - __main__.py - static - index.html - some_a.css - some_b.js ... Pack this empty module in Python wheel, and additionally include these static resources as "data_files"[3][4][5], so that it can be distributed like a normal Python package without having to deal with JS dependencies. When deploying with Nginx, when you need to fill in the static resource path, use "python -m skyline_console" to find it. There is a packed skyline packag[6] on "tarballs.opendev.org" for you to preview. [1] https://python-poetry.org/ <https://python-poetry.org/> [2] https://opendev.org/skyline/skyline-console/src/branch/master/Makefile#L73-L... <https://opendev.org/skyline/skyline-console/src/branch/master/Makefile#L73-L77> [3] https://packaging.python.org/guides/distributing-packages-using-setuptools/#... <https://packaging.python.org/guides/distributing-packages-using-setuptools/#data-files> [4] https://setuptools.pypa.io/en/latest/deprecated/distutils/setupscript.html#d... <https://setuptools.pypa.io/en/latest/deprecated/distutils/setupscript.html#distutils-additional-files> [5] https://opendev.org/skyline/skyline-console/src/branch/master/pyproject.toml... <https://opendev.org/skyline/skyline-console/src/branch/master/pyproject.toml#L6> [6] https://tarballs.opendev.org/skyline/skyline-apiserver/ <https://tarballs.opendev.org/skyline/skyline-apiserver/>
2021年10月16日 01:23，Thomas Goirand <zigo@debian.org> 写道：
On 10/15/21 6:07 PM, Ghanshyam Mann wrote:
New project 'Skyline' proposal ------------------------------------ * You might be aware of this new dashboard proposal in the previous month's discussion. * A new project 'Skyline: an OpenStack dashboard optimized by UI and UE' is now proposed in governance to be an official OpenStack project[4]. * Skyline team is planning to meet in PTG on Tue, Wed and Thu at 5UTC, please ask your queries or have feedback/discussion with the team next week.
Skyline looks nice. However, looking nice isn't enough. Before it becomes an OpenStack official component, maybe it should first try to reach our standard. I'm namely thinking about having a proper setuptool integration (using PBR?) for example, and starting tagging releases.
I'm very much interested in packaging this for Debian/Ubuntu, if it's not a JS dependency hell. Though the current Makefile thingy doesn't look appealing.
I've seen the console has at least 40 JS direct dependency. How many indirect dependency is this? Has anyone looked into it?
Is the team ready to help making it package-able in a distro policy compliant way?
Your thoughts?
Cheers,
Thomas Goirand (zigo)
On 2021-10-18 11:22:11 +0800 (+0800), 高瀚翔 wrote:
Skyline-apiserver is a pure Python code project, following the Python wheel packaging standard, using pip for installation, and the dependency management of the project using poetry[1]
Skyline-console uses npm for dependency management, development and testing. During the packaging and distribution process, webpack will be used to process the source code and dependent library code first, and output the packaged static resource files. These static resource files will be stored in an empty Python module[2]. [...]
GNU/Linux distributions like Debian are going to want to separately package the original source code for all of these Web components and their dependencies, and recreate them at the time the distro's binary packages are built. I believe the concerns are making it easy for them to find the source for all of it, and to attempt to use dependencies which these distributions already package in order to reduce their workload. Further, it helps to make sure the software is capable of using multiple versions of its dependencies when possible, because it's going to be installed into shared environments with other software which may have some of the same dependencies, so may need to be able to agree on common versions they all support. -- Jeremy Stanley
On 10/18/21 2:18 PM, Jeremy Stanley wrote:
On 2021-10-18 11:22:11 +0800 (+0800), 高瀚翔 wrote:
Skyline-apiserver is a pure Python code project, following the Python wheel packaging standard, using pip for installation, and the dependency management of the project using poetry[1]
Skyline-console uses npm for dependency management, development and testing. During the packaging and distribution process, webpack will be used to process the source code and dependent library code first, and output the packaged static resource files. These static resource files will be stored in an empty Python module[2]. [...]
GNU/Linux distributions like Debian are going to want to separately package the original source code for all of these Web components and their dependencies, and recreate them at the time the distro's binary packages are built. I believe the concerns are making it easy for them to find the source for all of it, and to attempt to use dependencies which these distributions already package in order to reduce their workload. Further, it helps to make sure the software is capable of using multiple versions of its dependencies when possible, because it's going to be installed into shared environments with other software which may have some of the same dependencies, so may need to be able to agree on common versions they all support.
Hi, Thanks Jeremy for summing-up things in a better way that I ever would. Also, using pip is *not* an option for distros. I'm not sure what you mean by "following the Python wheel packaging standard", but to me, we're not there yet. I'd like to have a normal setup.py and setup.cfg file in each Python module so it's easy to call "python3 setup.py install --root $(pwd)/debian/skyline-apiserver --install-layout=deb". I would also expect to see a "normal" requirements.txt and test-requirements.txt like in every other OpenStack project, the use of stestr to run unit tests, and so on. Right now, when looking at the skyline-apiserver as the Debian OpenStack package maintainer, I'd need a lot of manual work to use pyproject.toml instead of my standard tooling. As I understand, dependencies are expressed in the pyproject.toml, but then how do I get the Python code installed under debian/skyline-apiserver? BTW, what made you choose something completely different than the rest of the OpenStack project? Cheers, Thomas Goirand (zigo)
On Wed, Oct 20, 2021 at 5:22 PM Thomas Goirand <zigo@debian.org> wrote:
On 10/18/21 2:18 PM, Jeremy Stanley wrote:
On 2021-10-18 11:22:11 +0800 (+0800), 高瀚翔 wrote:
Skyline-apiserver is a pure Python code project, following the Python wheel packaging standard, using pip for installation, and the dependency management of the project using poetry[1]
Skyline-console uses npm for dependency management, development and testing. During the packaging and distribution process, webpack will be used to process the source code and dependent library code first, and output the packaged static resource files. These static resource files will be stored in an empty Python module[2]. [...]
GNU/Linux distributions like Debian are going to want to separately package the original source code for all of these Web components and their dependencies, and recreate them at the time the distro's binary packages are built. I believe the concerns are making it easy for them to find the source for all of it, and to attempt to use dependencies which these distributions already package in order to reduce their workload. Further, it helps to make sure the software is capable of using multiple versions of its dependencies when possible, because it's going to be installed into shared environments with other software which may have some of the same dependencies, so may need to be able to agree on common versions they all support.
Hi,
Thanks Jeremy for summing-up things in a better way that I ever would.
Also, using pip is *not* an option for distros. I'm not sure what you mean by "following the Python wheel packaging standard", but to me, we're not there yet. I'd like to have a normal setup.py and setup.cfg file in each Python module so it's easy to call "python3 setup.py install --root $(pwd)/debian/skyline-apiserver --install-layout=deb".
Side note: calling setup.py is essentially deprecated: https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html
I would also expect to see a "normal" requirements.txt and test-requirements.txt like in every other OpenStack project, the use of stestr to run unit tests, and so on.
Right now, when looking at the skyline-apiserver as the Debian OpenStack package maintainer, I'd need a lot of manual work to use pyproject.toml instead of my standard tooling.
PyProject is the universal way forward, you'll (and we'll) need to adopt sooner or later. Dmitry
As I understand, dependencies are expressed in the pyproject.toml, but then how do I get the Python code installed under debian/skyline-apiserver?
BTW, what made you choose something completely different than the rest of the OpenStack project?
Cheers,
Thomas Goirand (zigo)
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
On 10/21/21 1:18 PM, Dmitry Tantsur wrote:
Side note: calling setup.py is essentially deprecated: https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html <https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html>
This is essentially bullshit: the replacement method proposed in this page is supposed to be "pip install", which does dependency resolving and download from the internet, which is not useful (and even forbidden at package build time) for downstream distributions. If using pip is the only thing that upstream Python people are proposing to distributions, without any distro-specific options (like the --install-layout=deb option of setuptools, for example), then something really wrong is going on. Unless I missed something and upstream Python likes us from now on?!?
Right now, when looking at the skyline-apiserver as the Debian OpenStack package maintainer, I'd need a lot of manual work to use pyproject.toml instead of my standard tooling.
PyProject is the universal way forward, you'll (and we'll) need to adopt sooner or later.
There's currently an effort going on to get poetry packaged in Debian. However, we're not there yet, but I have hope for it. Anyway, this is orthogonal to what I wrote: IMO skyline needs to conform with everything that's made elsewhere in OpenStack, and adopt the same standards. If OpenStack is ever to move to Poetry (which means: all projects), then maybe I will revise this thinking. Cheers, Thomas Goirand (zigo)
On Fri, Oct 22, 2021, at 3:06 PM, Thomas Goirand wrote:
On 10/21/21 1:18 PM, Dmitry Tantsur wrote:
Side note: calling setup.py is essentially deprecated: https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html <https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html>
This is essentially bullshit: the replacement method proposed in this page is supposed to be "pip install", which does dependency resolving and download from the internet, which is not useful (and even forbidden at package build time) for downstream distributions.
If using pip is the only thing that upstream Python people are proposing to distributions, without any distro-specific options (like the --install-layout=deb option of setuptools, for example), then something really wrong is going on.
Unless I missed something and upstream Python likes us from now on?!?
I think https://pypa-build.readthedocs.io/en/latest/ may be the new tool that is a bit more usable for distros. But I've not used it at all so couldn't comment on its applicability.
On 2021-10-23 00:06:02 +0200 (+0200), Thomas Goirand wrote: [...]
This is essentially bullshit: the replacement method proposed in this page is supposed to be "pip install", which does dependency resolving and download from the internet, which is not useful (and even forbidden at package build time) for downstream distributions.
If using pip is the only thing that upstream Python people are proposing to distributions, without any distro-specific options (like the --install-layout=deb option of setuptools, for example), then something really wrong is going on.
Unless I missed something and upstream Python likes us from now on?!? [...]
Their new answer to "how should we be building binary packages from Python source distributions" is this (already packaged in Debian): https://packages.debian.org/python3-build The idea is that a source distribution may have a variety of different possible build backends per PEP 517, but as long as you run `python3 -m build` (and the relevant build dependencies are made available somehow) then it shouldn't matter. Calling directly into setup.py is only going to work for projects which use Setuptools as their build backend and provide a setup.py, of course, probably not for any other build backend.
IMO skyline needs to conform with everything that's made elsewhere in OpenStack, and adopt the same standards. [...]
I agree, and I attempted to leave some detailed notes on https://review.opendev.org/814037 highlighting the differences I spotted, which I think they're going to want to address for consistency with the rest of OpenStack. -- Jeremy Stanley
---- On Fri, 22 Oct 2021 17:40:29 -0500 Jeremy Stanley <fungi@yuggoth.org> wrote ----
On 2021-10-23 00:06:02 +0200 (+0200), Thomas Goirand wrote: [...]
This is essentially bullshit: the replacement method proposed in this page is supposed to be "pip install", which does dependency resolving and download from the internet, which is not useful (and even forbidden at package build time) for downstream distributions.
If using pip is the only thing that upstream Python people are proposing to distributions, without any distro-specific options (like the --install-layout=deb option of setuptools, for example), then something really wrong is going on.
Unless I missed something and upstream Python likes us from now on?!? [...]
Their new answer to "how should we be building binary packages from Python source distributions" is this (already packaged in Debian):
https://packages.debian.org/python3-build
The idea is that a source distribution may have a variety of different possible build backends per PEP 517, but as long as you run `python3 -m build` (and the relevant build dependencies are made available somehow) then it shouldn't matter. Calling directly into setup.py is only going to work for projects which use Setuptools as their build backend and provide a setup.py, of course, probably not for any other build backend.
IMO skyline needs to conform with everything that's made elsewhere in OpenStack, and adopt the same standards. [...]
I agree, and I attempted to leave some detailed notes on https://review.opendev.org/814037 highlighting the differences I spotted, which I think they're going to want to address for consistency with the rest of OpenStack.
Hello Skyline team, We have not heard from you on the open questions (mentioned in https://review.opendev.org/c/openstack/governance/+/814037 too). It's been ~ 21 days TC has voted -1 in Gerrit with required things to fix/or tell plan so that TC can merge the new project application. Please respond on the Gerrit to proceed next. -gmann
-- Jeremy Stanley
Hello folks, The current governance requires the projects to follow certain guidelines [1], some of which are already covered by skyline. One of such guidelines is the PTI [2], which is detailed in more details for python in [3]. I am interested by the case of Skyline. Skyline is a truely new project for OpenStack, bringing value to the ecosystem. It is forward looking. Yes, it is different than existing projects. Yes, it doesn't pass the pti for python [3]. If you check the PTI for golang [4], many of those commands could be easily wired on Skyline project using a Makefile (of course it wouldn't pass the dependency management part of that document, for which we require glide [5]... unmaintained since 2019, ahem... [6]). So the question is: Is OpenStack ready to move forward and adapt to new kinds of packaging/software? If yes, let's modify the PTIs (or python PTI) to cover the case for skyline, and allow that team to move forward. It's not like it's doing something very different than some new python projects. Yes, OpenStack is mature... I (simply) hope it's also a place for casual contributions, and where developers can have fun with tech. And yes, it won't please everyone from day one (especially distros), but I hope we can work this out together, rather than discard the project because it doesn't fit the mold. Regards, Jean-Philippe Evrard (evrardjp) [1]: https://governance.openstack.org/tc/reference/new-projects-requirements.html [2]: https://governance.openstack.org/tc/reference/project-testing-interface.html [3]: https://governance.openstack.org/tc/reference/pti/python.html [4]: https://governance.openstack.org/tc/reference/pti/golang.html [5]: https://github.com/Masterminds/glide [6]: https://travis-ci.org/github/Masterminds/glide
On Thu, 2021-12-09 at 16:57 +0100, Jean-Philippe Evrard wrote:
Hello folks,
The current governance requires the projects to follow certain guidelines [1], some of which are already covered by skyline. One of such guidelines is the PTI [2], which is detailed in more details for python in [3].
I am interested by the case of Skyline. Skyline is a truely new project for OpenStack, bringing value to the ecosystem. It is forward looking. Yes, it is different than existing projects. Yes, it doesn't pass the pti for python [3].
If you check the PTI for golang [4], many of those commands could be easily wired on Skyline project using a Makefile (of course it wouldn't pass the dependency management part of that document, for which we require glide [5]... unmaintained since 2019, ahem... [6]).
So the question is: Is OpenStack ready to move forward and adapt to new kinds of packaging/software? If yes, let's modify the PTIs (or python PTI) to cover the case for skyline, and allow that team to move forward. It's not like it's doing something very different than some new python projects. using a make file to me woudl be alarge regression in packaging and it does not align with the upstream recomendation for package form the python packaging athority. i support using projects.toml and the new standardised build system and i woudl be open to changeing the pti to allow that but i dont think tha tallow make files would be a good thing. the use of poetry is also an interesting choice that has some advandtges but it will break much of our tooling with regards to upper-constraints.txt.
i woudl personally find it rather suprising as a downstream packageer or end user to consume skyline as currently wtieen as either a standar python proejct or an openstack one if i had to use make to build or install it. i am happy to see that tox is used but since its callign dirctly into make i really dont think it being used the way it is intended to be.
Yes, OpenStack is mature... I (simply) hope it's also a place for casual contributions, and where developers can have fun with tech. And yes, it won't please everyone from day one (especially distros), but I hope we can work this out together, rather than discard the project because it doesn't fit the mold.
is there a reaosbon bveyond its fun to do it this way. to me this add barries to contibutors using the proejct and to packagees consuming it.
Regards, Jean-Philippe Evrard (evrardjp)
[1]: https://governance.openstack.org/tc/reference/new-projects-requirements.html [2]: https://governance.openstack.org/tc/reference/project-testing-interface.html [3]: https://governance.openstack.org/tc/reference/pti/python.html [4]: https://governance.openstack.org/tc/reference/pti/golang.html [5]: https://github.com/Masterminds/glide [6]: https://travis-ci.org/github/Masterminds/glide
On 2021-12-09 16:57:27 +0100 (+0100), Jean-Philippe Evrard wrote: [...]
So the question is: Is OpenStack ready to move forward and adapt to new kinds of packaging/software? If yes, let's modify the PTIs (or python PTI) to cover the case for skyline, and allow that team to move forward. It's not like it's doing something very different than some new python projects.
Yes, OpenStack is mature... I (simply) hope it's also a place for casual contributions, and where developers can have fun with tech. And yes, it won't please everyone from day one (especially distros), but I hope we can work this out together, rather than discard the project because it doesn't fit the mold. [...]
The reason I raised those points on the review was in order to highlight inconsistencies with OpenStack as a whole. If OpenStack is defined by anything, it's consistent development workflows and automation, which allow small groups of people to take advantage of economies of scale in order to minimize maintenance over time across the whole of the set of projects which comprise it. Inconsistency has an inherent cost which the entire community bears, from tracking integration failures in development to keeping concise recipes for packaging in distributions to operators being able to configure all services the same way and expect the same logging formats. Most of the benefits a project gets from being "part of OpenStack" are derived directly from the established standards and norms which enable OpenStack to be treated as a cohesive whole rather than a mere "bucket of parts" with tooling choices made independently by their respective development teams. Deviation from these expectations should be deliberate and come with a clear rationale which benefits OpenStack as a whole rather than one project alone. Let's take the current example point by point: * Skyline implements its own routines for things like configuration and logging rather than using Oslo libraries. Should all OpenStack projects "move forward" to this model, choose their own configuration and logging formats, implement their own solutions for documenting these, and so on? Does inconsistency here benefit operators in some way? * Skyline defines the interdependencies between its components with a mix of "monorepo" (multiple packages from a single repository) and Git submodule references (pinning specific commit ids as virtual subtrees of another repository), rather than establishing a loose coupling with explicit version numbers and package relationships. This choice prevents them from taking advantage of features in Zuul like cross-project change dependencies. Is this a pattern the other OpenStack projects should follow? Does it benefit distribution package maintainers to need to track fundamentally different repository layouts and build processes for each of them? * Skyline performs its release versioning by committing the version strings to files in its repositories rather than assigning them at build time from Git tag metadata. This leads to ambiguity as to what exact state of the repository represents that version, as well as opening up the possibility of forgetting to merge the version update before tagging (a very common problem which drove OpenStack to rely on its current model). Wholly different tools are also needed in order to track versions for these projects as a result. Should the rest of OpenStack's projects follow suit there? Has something changed to make the idea of using Git to signal releases a bad one? * Skyline uses makefiles to script its build and test workflows rather than tox, adding a tox wrapper around the make targets in order to nominally comply with the OpenStack PTI. As a result, it's unable to rely on most of tox's features for environment isolation, package management, path overrides, et cetera. OpenStack projects have solved a lot of cross-project development workflow challenges through applying consistent changes across the tox.ini files of repositories, should those solutions be abandoned and recreated for make instead? * Skyline does not put its external Python package dependencies in central lists nor use tools which would allow it to rely on OpenStack's global constraints implementation, making it very hard for them to guarantee that their software can run with the same versions of shared dependencies as the rest of OpenStack. Is this unnecessary? Have we given up on the idea that two pieces of OpenStack software can be coinstalled into one environment, or that Linux distributions won't be stuck having to package many different versions of the same software because OpenStack projects don't agree on which versions work for them any longer? To be perfectly clear, I don't oppose accepting Skyline as a project in OpenStack, even in its current state, but I do feel like it's not going to end up released as part of OpenStack unless it can address the challenges I outlined above. Most of these are fairly superficial changes to implement, however, and can be done by simply copying the solutions already used in other OpenStack projects or retroactively applying our cookie-cutters. "Moving OpenStack forward" to be more like Skyline, on the other hand, is bound to be far more work than the Skyline team can handle on its own and take far longer, though I'm not opposed to the idea of discussing which of these differences may make sense for OpenStack to adopt in the future. My deeper concern is that these differences are a clear indication that the software was written outside the OpenStack community, with no original consideration for consistency with OpenStack itself, and so there are probably more significant architectural modifications needed (beyond integrating oslo.config and oslo.log) which I'm not well-placed to spot. Those can addressed in time too, and while Skyline is an official team within OpenStack, but they will still need to be taken care of at some point early in the life of the project either by becoming like the rest of OpenStack or helping the rest of OpenStack become more like Skyline. -- Jeremy Stanley
Helllo, On Thu, Dec 9, 2021, at 19:03, Jeremy Stanley wrote:
The reason I raised those points on the review was in order to highlight inconsistencies with OpenStack as a whole. If OpenStack is defined by anything, it's consistent development workflows and automation, which allow small groups of people to take advantage of economies of scale in order to minimize maintenance over time across the whole of the set of projects which comprise it. Inconsistency has an inherent cost which the entire community bears, from tracking integration failures in development to keeping concise recipes for packaging in distributions to operators being able to configure all services the same way and expect the same logging formats.
I agree that consistency is important, and helps us in many ways. I don't think it's the "defining" factor of OpenStack, however. Inconsistency in openstack is in many places, from the users' to the developer's perspective. It doesn't mean I am _looking for_ inconsistency. I am merely embracing the differences (and I am eager to remove them, if we can).
Most of the benefits a project gets from being "part of OpenStack" are derived directly from the established standards and norms which enable OpenStack to be treated as a cohesive whole rather than a mere "bucket of parts" with tooling choices made independently by their respective development teams. Deviation from these expectations should be deliberate and come with a clear rationale which benefits OpenStack as a whole rather than one project alone.
Correct again! :) For the case of skyline, one could made a case to start building packages with a new way, using poetry, rather than setuptools. It could mean we need to ask the skyline team to help on improving the infrastructure and testing. Instead of arriving to a standard testing, we embrace the difference and provide a possible new way of working. We could have two kinds of python projects. Instead of accepting that difference when a single project asking for it, maybe we can just tally who wanted this, and the TC decides at the end "when" it is worth doing. My whole point is that we need to document better and agree which parts benefits OpenStack as a whole. We should also have indicators for checking how the "ecosystem" is behaving. If we are considering the amount of new teams in OpenStack, it's not very encouraging. (It might not be a metric, I don't know, I rather leave Thierry judge there). In other words: Is it really better to change the skyline team to adapt our mold, or should we have multiple molds? We agreed to go the way that skyline should adapt. I am simply raising the fact that some other people/projects won't do the effort to adapt to existing mold, because they won't necessarily see the value. (I won't prevent my teams to work on non-openstack projects ;))
Let's take the current example point by point:
* Skyline implements its own routines for things like configuration and logging rather than using Oslo libraries. Should all OpenStack projects "move forward" to this model, choose their own configuration and logging formats, implement their own solutions for documenting these, and so on? Does inconsistency here benefit operators in some way?
Is Oslo a requirement to be an OpenStack project now? AFAIK it wasn't (see swift). It doesn't mean you can't be consistent for logs/configuration from the user perspective by not using oslo. Side note: Bringing oslo projects for configuration and logging isn't enough to bring consistency from the user perspective.
* Skyline defines the interdependencies between its components with a mix of "monorepo" (multiple packages from a single repository) and Git submodule references (pinning specific commit ids as virtual subtrees of another repository), rather than establishing a loose coupling with explicit version numbers and package relationships. This choice prevents them from taking advantage of features in Zuul like cross-project change dependencies. Is this a pattern the other OpenStack projects should follow?
Which is why I am highlighting that _as of today_, the governance requires certain expectations, which needs clarifications (or exceptions) if we want to integrate Skyline. I know I am a bit pedantic in the following question: Is (nowadays) the expectation for projects to be testable by Zuul (+release-able by zuul, ... list to be defined) or is the expectation for projects to be following the PTI guidelines (tested by zuul)? In other words: Do we need to enforce the tooling or the results (a frequently released and well tested software)? I would like the TC to think about this.
Does it benefit distribution package maintainers to need to track fundamentally different repository layouts and build processes for each of them?
Isn't that exactly what the distros are doing right now with the packages they are maintaining? Not every software is like openstack. Yes, almost all openstack projects are similar to each other, so they can be packaged the same way. Yet, some official openstack projects aren't, as of today. So, some distros decide to not package them? Is that a problem for us? I didn't think so. Projects/Teams need to understand that if they don't follow "x", they will not have "y". Which sounds fair to me. Keep in mind that we are _already doing exceptions_. Not all projects are similar to nova, glance, neutron, keystone, cinder. (Sorry for the many I missed).
* Skyline performs its release versioning by committing the version strings to files in its repositories rather than assigning them at build time from Git tag metadata. This leads to ambiguity as to what exact state of the repository represents that version, as well as opening up the possibility of forgetting to merge the version update before tagging (a very common problem which drove OpenStack to rely on its current model). Wholly different tools are also needed in order to track versions for these projects as a result. Should the rest of OpenStack's projects follow suit there? Has something changed to make the idea of using Git to signal releases a bad one?
You're raising again a valid point, for which the skyline team will have to find a solution. For me, it boils down to defining that projects should follow a "project release interface", which explains the expected behaviour in regards of releasing for projects. We don't have such. I am not claiming we should, I am merely asking us to think about what we are doing. Projects could indeed move to use pbr/setuptools. However, I am not sure many developers understand why. Should we simply explain what is expected in terms of building the version from git? People might even come with interesting solutions. Promoting creativity will create tech debt, but might also reduce it over time. What if we don't need pbr anymore? Will we even notice?
* Skyline uses makefiles to script its build and test workflows rather than tox, adding a tox wrapper around the make targets in order to nominally comply with the OpenStack PTI. As a result, it's unable to rely on most of tox's features for environment isolation, package management, path overrides, et cetera. OpenStack projects have solved a lot of cross-project development workflow challenges through applying consistent changes across the tox.ini files of repositories, should those solutions be abandoned and recreated for make instead?
I didn't say so. I am only partially agreeing with you there, for once :) For me, tox is a test runner that provides many nice features thanks to its code (and integration with setuptools) I would prefer if python projects using tox for testing interface indeed. Should that be the standard? I don't know. Does it make sense to rely on tox features for isolations/package management/other? I don't know. You might be aware that it's possible to integrate tox with poetry. Does it provide all we _need_? I don't know: We aren't explicit in the requirements of what we need, we just talk about tools. If I were to rephrase: Is it easier to fit the existing mold or clarify what we need, why, and how we can deal with the tech debt? Probably the former. Should we do it? I don't know. The TC should however evaluate this seriously.
* Skyline does not put its external Python package dependencies in central lists nor use tools which would allow it to rely on OpenStack's global constraints implementation, making it very hard for them to guarantee that their software can run with the same versions of shared dependencies as the rest of OpenStack. Is this unnecessary? Have we given up on the idea that two pieces of OpenStack software can be coinstalled into one environment, or that Linux distributions won't be stuck having to package many different versions of the same software because OpenStack projects don't agree on which versions work for them any longer?
Now, we are hitting the nail, IMO :) This is a bit of the issue's core, isn't it? I think that's what we need to document. While I (personally) don't believe it's important nowadays (cf. containers), I am sure distros will disagree with me. It's important to know who we are pleasing, as we can't please everyone ;) Let's be clear about the audience. If we intend to please the distros, did we ask about RedHat/Canonical committments on skyline? Are they even remotely interested? If they want to use it, wouldn't it make sense they do the work to make sure skyline is packageable for them, by improving upstream? If they don't, why do we care?
To be perfectly clear, I don't oppose accepting Skyline as a project in OpenStack, even in its current state, but I do feel like it's not going to end up released as part of OpenStack unless it can address the challenges I outlined above. Most of these are fairly superficial changes to implement, however, and can be done by simply copying the solutions already used in other OpenStack projects or retroactively applying our cookie-cutters.
I agree with you. I am merely asking for clarity. However, I want to think as a whole.
"Moving OpenStack forward" to be more like Skyline, on the other hand, is bound to be far more work than the Skyline team can handle on its own and take far longer, though I'm not opposed to the idea of discussing which of these differences may make sense for OpenStack to adopt in the future.
Depends on the definition of "Moving OpenStack forward". Do you mean all modifying all projects to be like skyline + integrate the testing? That was never my point. Do you mean "ensure a single project can be behaving like a majority of other big openstack projects are"? That's indeed more work for a team, compared to if they had their work done outside OpenStack governance. Should we use this energy for thinking(/doing something) about OpenStack future? IMO, absolutely. And for those arriving here: Thanks for reading my rant until the end ;) Jean-Philippe Evrard (evrardjp)
Hi Jean-Philippe. Please read what I'm writing keeping in mind that I'm *very* interested in Skyline, and would love to have it fully packaged in Debian. I probably will do that when I consider the project ready for it, which unfortunately, in the current case, isn't the case (ie: it would need too much effort from my side for it). So I'm writing in the hope to get things improved... :) On 12/10/21 10:43 AM, Jean-Philippe Evrard wrote: > For the case of skyline, one could made a case to start building packages with a new way, using poetry, rather than setuptools. It could mean we need to ask the skyline team to help on improving the infrastructure and testing. >From my package maintainer point of view, the main problem I see is more then way things are layout (ie: the libs folder of skyline-apiserver that contains many projects). I don't really mind anymore the poetry thing. This shouldn't be so hard to address, right? > Instead of arriving to a standard testing, we embrace the difference and provide a possible new way of working. We could have two kinds of python projects. Instead of accepting that difference when a single project asking for it, maybe we can just tally who wanted this, and the TC decides at the end "when" it is worth doing. I don't think you understand the problem here. The issue is wiring Skyline to the OpenDev CI. With the way things are done, it's simply very hard to do, unless you're ready to spend *a lot* of time with the infra team, and if they have enough time to let you do that (probably they are already very busy). > In other words: Is it really better to change the skyline team to adapt our mold, or should we have multiple molds? > We agreed to go the way that skyline should adapt. I am simply raising the fact that some other people/projects won't do the effort to adapt to existing mold, because they won't necessarily see the value. > (I won't prevent my teams to work on non-openstack projects ;)) Same story: if the infra can handle the "new way" that you're proposing, why not. But we're not there yet at all. >> Let's take the current example point by point: >> >> * Skyline implements its own routines for things like >> configuration and logging rather than using Oslo libraries. >> Should all OpenStack projects "move forward" to this model, >> choose their own configuration and logging formats, implement >> their own solutions for documenting these, and so on? Does >> inconsistency here benefit operators in some way? > > Is Oslo a requirement to be an OpenStack project now? > AFAIK it wasn't (see swift). You wont make your case by pointing at the defects of other projects. You could also have pointed at Horizon's local_setting.py. IMO it's a much better example beacuse it's a very good example: it's a *REAL PAIN* for operators (for example: for puppet, that has no way of handling that configuration file using something else than a ERB template). > Side note: Bringing oslo projects for configuration and logging isn't enough to bring consistency from the user perspective. Probably, but it's a starting point. >> * Skyline defines the interdependencies between its components >> with a mix of "monorepo" (multiple packages from a single >> repository) and Git submodule references (pinning specific >> commit ids as virtual subtrees of another repository), rather >> than establishing a loose coupling with explicit version numbers >> and package relationships. This choice prevents them from taking >> advantage of features in Zuul like cross-project change >> dependencies. Is this a pattern the other OpenStack projects >> should follow? > > Which is why I am highlighting that _as of today_, the governance requires certain expectations, which needs clarifications (or exceptions) if we want to integrate Skyline. > > I know I am a bit pedantic in the following question: > Is (nowadays) the expectation for projects to be testable by Zuul (+release-able by zuul, ... list to be defined) or is the expectation for projects to be following the PTI guidelines (tested by zuul)? I'm not from the TC, but ... hopefully BOTH ! :) >> Does it benefit distribution package maintainers >> to need to track fundamentally different repository layouts and >> build processes for each of them? > > Isn't that exactly what the distros are doing right now with the packages they are maintaining? Not every software is like openstack. Yes, almost all openstack projects are similar to each other, so they can be packaged the same way. > > Yet, some official openstack projects aren't, as of today. So, some distros decide to not package them? Could you please be more specific? Maybe with some examples? I'm having a hard time trying to figure out what you have in mind here... :) > Keep in mind that we are _already doing exceptions_. Not all projects are similar to nova, glance, neutron, keystone, cinder. (Sorry for the many I missed). Which one are you thinking about? :) >> * Skyline performs its release versioning by committing the >> version strings to files in its repositories rather than >> assigning them at build time from Git tag metadata. This leads >> to ambiguity as to what exact state of the repository represents >> that version, as well as opening up the possibility of >> forgetting to merge the version update before tagging (a very >> common problem which drove OpenStack to rely on its current >> model). Wholly different tools are also needed in order to >> track versions for these projects as a result. Should the rest >> of OpenStack's projects follow suit there? Has something changed >> to make the idea of using Git to signal releases a bad one? > > You're raising again a valid point, for which the skyline team will have to find a solution. >From a distro perspective, it is easy to build multiple binary packages out of a single source tree. However, it's not possible to produce different versions of the binaries, if we have a single source package. So IMO, this should be addressed as a top priority. > For me, it boils down to defining that projects should follow a "project release interface", which explains the expected behaviour in regards of releasing for projects. We don't have such. I am not claiming we should, I am merely asking us to think about what we are doing. Yes, you should think about it! :) Would it be a huge effort to separte things into multiple projects instead of one big repo, and then start tagging properly? > Projects could indeed move to use pbr/setuptools. However, I am not sure many developers understand why. > Should we simply explain what is expected in terms of building the version from git? People might even come with interesting solutions. Promoting creativity will create tech debt, but might also reduce it over time. > What if we don't need pbr anymore? Will we even notice? >From my perspective, pbr/setuptools isn't the main problem. Single repo + not tagging is. > For me, tox is a test runner that provides many nice features thanks to its code (and integration with setuptools) > I would prefer if python projects using tox for testing interface indeed. > Should that be the standard? I don't know. > Does it make sense to rely on tox features for isolations/package management/other? I don't know. > You might be aware that it's possible to integrate tox with poetry. Does it provide all we _need_? I don't know: We aren't explicit in the requirements of what we need, we just talk about tools. > > If I were to rephrase: Is it easier to fit the existing mold or clarify what we need, why, and how we can deal with the tech debt? Probably the former. Should we do it? I don't know. The TC should however evaluate this seriously. What Jeremy wrote: it's important to be able to use the global requirements repository, so we can check your project works with the others. I don't think the TC will have any satisfying answer if you can't provide a way to check your projects works with the rest of the OpenStack world. >> * Skyline does not put its external Python package dependencies in >> central lists nor use tools which would allow it to rely on >> OpenStack's global constraints implementation, making it very >> hard for them to guarantee that their software can run with the >> same versions of shared dependencies as the rest of OpenStack. >> Is this unnecessary? Have we given up on the idea that two >> pieces of OpenStack software can be coinstalled into one >> environment, or that Linux distributions won't be stuck having >> to package many different versions of the same software because >> OpenStack projects don't agree on which versions work for them >> any longer? > > Now, we are hitting the nail, IMO :) > > This is a bit of the issue's core, isn't it? I think that's what we need to document. > While I (personally) don't believe it's important nowadays (cf. containers), I am sure distros will disagree with me. Indeed, I strongly disagree! :) > It's important to know who we are pleasing, as we can't please everyone ;) > Let's be clear about the audience. > > If we intend to please the distros, did we ask about RedHat/Canonical committments on skyline? Are they even remotely interested? I am *very* interested. :) If you didn't know, I've been packaging OpenStack in Debian since the very beginning (ie: in 2011 for me). > If they want to use it, wouldn't it make sense they do the work to make sure skyline is packageable for them, by improving upstream? Do you understand that *all* the distros are understaffed? Do you really expect package maintainers to address *ALL* of the issues they see in *ALL* of the OpenStack projects? That's not reasonable, and it wont ever happen, simply because we can't. > If they don't, why do we care? Why wouldn't you care about integration? Being package-able also means your project can be integrated in *any* environment, not only distro. If you don't care, why do you then care to have Skyline as an official project then? It doesn't make sense to me. Cheers, Thomas Goirand (zigo)
On 2021-12-10 10:43:10 +0100 (+0100), Jean-Philippe Evrard wrote:
On Thu, Dec 9, 2021, at 19:03, Jeremy Stanley wrote: [...]
* Skyline performs its release versioning by committing the version strings to files in its repositories rather than assigning them at build time from Git tag metadata. This leads to ambiguity as to what exact state of the repository represents that version, as well as opening up the possibility of forgetting to merge the version update before tagging (a very common problem which drove OpenStack to rely on its current model). Wholly different tools are also needed in order to track versions for these projects as a result. Should the rest of OpenStack's projects follow suit there? Has something changed to make the idea of using Git to signal releases a bad one?
You're raising again a valid point, for which the skyline team will have to find a solution.
For me, it boils down to defining that projects should follow a "project release interface", which explains the expected behaviour in regards of releasing for projects. We don't have such. I am not claiming we should, I am merely asking us to think about what we are doing.
Projects could indeed move to use pbr/setuptools. However, I am not sure many developers understand why. Should we simply explain what is expected in terms of building the version from git? People might even come with interesting solutions. Promoting creativity will create tech debt, but might also reduce it over time. What if we don't need pbr anymore? Will we even notice?
Isn't that exactly what I did? You'll notice here and in my review I didn't say "use PBR," I outlined the properties desired instead. It's the case that using PBR is likely the easiest way to do that since we already have a lot of examples, but like you I don't think that the implementation is important as long as we can find a suitable solution to the problems we face. That said, using a consistent implementation across as many projects as possible increases our familiarity and ability to address new issues with the fewest set of solutions.
* Skyline uses makefiles to script its build and test workflows rather than tox, adding a tox wrapper around the make targets in order to nominally comply with the OpenStack PTI. As a result, it's unable to rely on most of tox's features for environment isolation, package management, path overrides, et cetera. OpenStack projects have solved a lot of cross-project development workflow challenges through applying consistent changes across the tox.ini files of repositories, should those solutions be abandoned and recreated for make instead?
I didn't say so. I am only partially agreeing with you there, for once :)
For me, tox is a test runner that provides many nice features thanks to its code (and integration with setuptools) I would prefer if python projects using tox for testing interface indeed. Should that be the standard? I don't know. Does it make sense to rely on tox features for isolations/package management/other? I don't know. You might be aware that it's possible to integrate tox with poetry. Does it provide all we _need_? I don't know: We aren't explicit in the requirements of what we need, we just talk about tools.
If I were to rephrase: Is it easier to fit the existing mold or clarify what we need, why, and how we can deal with the tech debt? Probably the former. Should we do it? I don't know. The TC should however evaluate this seriously.
Let's be clear, I have nothing against make. In fact, I somewhat prefer it, since it's useful for projects written in all sorts of different programming languages beyond Python. At one point we even came very close to recommending OpenStack switch from tox to make during one of the more unfortunate backward-incompatible tox releases some years ago. Clark had a proof of concept recreating the fundamentals of tox itself in Makefiles, and it wasn't half bad. But having one project use make while the rest use tox isn't really great for anyone, it turns the make-based project into a second-class citizen within the whole. Its maintainers have to do a lot more work to support the same workflows with a different tool, and the community as a whole incurs some additional burden in working around that difference (additional CI jobs, separate functionality, different developer documentation). If we think make is a superior solution we should use it across all of OpenStack and not just in a few random projects.
* Skyline does not put its external Python package dependencies in central lists nor use tools which would allow it to rely on OpenStack's global constraints implementation, making it very hard for them to guarantee that their software can run with the same versions of shared dependencies as the rest of OpenStack. Is this unnecessary? Have we given up on the idea that two pieces of OpenStack software can be coinstalled into one environment, or that Linux distributions won't be stuck having to package many different versions of the same software because OpenStack projects don't agree on which versions work for them any longer?
Now, we are hitting the nail, IMO :)
This is a bit of the issue's core, isn't it? I think that's what we need to document. While I (personally) don't believe it's important nowadays (cf. containers), I am sure distros will disagree with me. It's important to know who we are pleasing, as we can't please everyone ;) Let's be clear about the audience. [...]
Just my opinion, but I think buying into container hype to the point where we make OpenStack only deployable through containerization would be a massive step backwards. -- Jeremy Stanley
Hello, On Fri, Dec 10, 2021, at 15:30, Jeremy Stanley wrote:
Isn't that exactly what I did? You'll notice here and in my review I didn't say "use PBR," I outlined the properties desired instead.
This is exactly what you did, and what I like in your answer. Everything you said, as wise as it is, isn't documented in the governance [1]. You have knowledge, and that knowledge on why certain things are done should be recorded for the future. The more we share, the more people will be able to challenge it or understand it.
That said, using a consistent implementation across as many projects as possible increases our familiarity and ability to address new issues with the fewest set of solutions.
Totally agree. I have so many points I want to raise. I realise asking what I have in mind will cause more trouble on this mailing list than the value it will bring. I just want the TC to think broader than the "we used to do it this way, and this is the only way". I am sure the topic raised some questions, and will give enough food for thoughts. Hence, I hope it will result in the improvement of OpenStack :)
Just my opinion, but I think buying into container hype to the point where we make OpenStack only deployable through containerization would be a massive step backwards.
I didn't say it was a goal or if I consider this important. I just wanted to mention that they are other ways to consume software than distribution packages. Side note: I have spent to much time on this, so I will just step out now. I hope the TC got enough details about what to make out of this in the future. If you need an extra $0.02, you know where to find me :) Regards, Jean-Philippe Evrard (evrardjp) [1]: https://governance.openstack.org/tc/reference/new-projects-requirements.html
On 12/10/21 5:29 PM, Jean-Philippe Evrard wrote:
Totally agree. I have so many points I want to raise. I realise asking what I have in mind will cause more trouble on this mailing list than the value it will bring. I just want the TC to think broader than the "we used to do it this way, and this is the only way".
Willing to change things within OpenStack is a completely different topic than "doing things my own way". I very welcome change if you have a cross-project attitude but I don't welcome "we don't do like the others, because we can" type of thinking. Packaging Telemetry, I've hated many times the design decision that was just a little bit different. It wasn't better or worse, just different. And it caused a lot of troubles just because I had to learn "their way" of doing things. I suppose the experience was the same for contributors. For example, aodh has it's build-dependency (well, "test requirements" if you prefer the Python wording than the distro ones) in setup.cfg instead of test-requirements.txt. Both works, but when I upgrade any component, I diff the test-requirements.txt between version, to see if there's new Build-Depends: to add to my package. But with Aodh, I have to remember to do it with setup.cfg. Its *VERY* annoying. Yes, I'm aware that this way, Aodh can have different test-requirements for mysql and postgresql for example. But this is still very annoying. I would have very much prefer if the Telemetry guys pushed so that *ALL* projects would move to push test-requirements.txt within setup.cfg, and *THEN* I would have accepted the change. Do you see where I'm going? :) I have other examples in mind, but I believe what's above is enough.
I am sure the topic raised some questions, and will give enough food for thoughts. Hence, I hope it will result in the improvement of OpenStack :)
If you don't do things on your own different than the others, and push every projects to move one direction at the same time then it's going to be an improvement. If you are the only one doing your own way, it's going to be a regression, and a pain for everyone.
Just my opinion, but I think buying into container hype to the point where we make OpenStack only deployable through containerization would be a massive step backwards.
I didn't say it was a goal or if I consider this important. I just wanted to mention that they are other ways to consume software than distribution packages.
Probably, but it's not up to you to decide for the others how they want to consume your deliverables.
Side note: I have spent to much time on this, so I will just step out now. I hope the TC got enough details about what to make out of this in the future. If you need an extra $0.02, you know where to find me :)
Regards, Jean-Philippe Evrard (evrardjp)
I very much hope that Skyline will be in a super nice shape in 6 months from now, so I can include in in Debian Bookworm! :) By the way, we discussed Python packaging a lot, but never JS. There's a huge discussing that sooner or later will have to get started too. NPM is a very dangerous thing that often leads to "npm install world" (ie: a JS orgy that is barely maintainable). I have to admit that I didn't look closely into how that's done in Skyline. But let's just take something else as the worst example: Grafana. I've heard someone from Red Hat attempted to package it, and discovered it included (directly or indirectly) 1600 JS packages, and then gave-up. How is Skyline in this regard? Cheers, Thomas Goirand (zigo)
---- On Thu, 09 Dec 2021 09:57:27 -0600 Jean-Philippe Evrard <openstack@a.spamming.party> wrote ----
Hello folks,
The current governance requires the projects to follow certain guidelines [1], some of which are already covered by skyline. One of such guidelines is the PTI [2], which is detailed in more details for python in [3].
I am interested by the case of Skyline. Skyline is a truely new project for OpenStack, bringing value to the ecosystem. It is forward looking. Yes, it is different than existing projects. Yes, it doesn't pass the pti for python [3].
If you check the PTI for golang [4], many of those commands could be easily wired on Skyline project using a Makefile (of course it wouldn't pass the dependency management part of that document, for which we require glide [5]... unmaintained since 2019, ahem... [6]).
So the question is: Is OpenStack ready to move forward and adapt to new kinds of packaging/software? If yes, let's modify the PTIs (or python PTI) to cover the case for skyline, and allow that team to move forward. It's not like it's doing something very different than some new python projects.
Yes, OpenStack is mature... I (simply) hope it's also a place for casual contributions, and where developers can have fun with tech. And yes, it won't please everyone from day one (especially distros), but I hope we can work this out together, rather than discard the project because it doesn't fit the mold.
Yes, those are the inconsistency we currently have in the Skyline project, but that is not what they will stick to. In Yoga PTG[1], the skyline team iterates through all these points and plans to improve those and be consistent with the other OpenStack projects. They said they will work on packaging, PTI, using Oslo code etc. Boxiang Zhu mentioned the same in the governance patch also[2]. If they deny moving Skyline towards the OpenStack way of packing, testing, using Oslo, etc., we should consider whether to reject the OpenStack official project or modify our new project application criteria. But this is not the case, and they agree to improve all those things. Now we have more clear direction from them in the governance patch also, so I feel this is good to go as an official project, and while they will be in OpenStack, we all can help them to improve the things in OpenStack way. Also, TC discussed in PTG the new concept of 'Tech pre-review' where we will monitor such new projects' health, contribution, and direction going in the direction of OpenStack way. This 'Tech pre-review' is not up yet, so let's see how we define it. [1] http://lists.openstack.org/pipermail/openstack-discuss/2021-October/025541.h... [2] https://review.opendev.org/c/openstack/governance/+/814037 [3] https://etherpad.opendev.org/p/tc-yoga-ptg#L263 -gmann
Regards, Jean-Philippe Evrard (evrardjp)
[1]: https://governance.openstack.org/tc/reference/new-projects-requirements.html [2]: https://governance.openstack.org/tc/reference/project-testing-interface.html [3]: https://governance.openstack.org/tc/reference/pti/python.html [4]: https://governance.openstack.org/tc/reference/pti/golang.html [5]: https://github.com/Masterminds/glide [6]: https://travis-ci.org/github/Masterminds/glide
Hello, On Fri, Dec 10, 2021, at 01:18, Ghanshyam Mann wrote:
Yes, those are the inconsistency we currently have in the Skyline project, but that is not what they will stick to. In Yoga PTG[1], the skyline team iterates through all these points and plans to improve those and be consistent with the other OpenStack projects. They said they will work on packaging, PTI, using Oslo code etc.
Boxiang Zhu mentioned the same in the governance patch also[2].
The questions I have still stand: Do we need to be more lax in the governance to not _require_ to follow the PTI (and/or have an intention to follow through)? Can the TC make an exception? When is the deadline for projects not following our testing to become compliant? What is the risk if the newly accepted project is not doing it? Moving it to a non-official project? Is that worth the effort?
If they deny moving Skyline towards the OpenStack way of packing, testing, using Oslo, etc., we should consider whether to reject the OpenStack official project or modify our new project application criteria. But this is not the case, and they agree to improve all those things.
I agree. Let's not talk hypothetics.
Now we have more clear direction from them in the governance patch also, so I feel this is good to go as an official project, and while they will be in OpenStack, we all can help them to improve the things in OpenStack way.
Perfect, great news.
Also, TC discussed in PTG the new concept of 'Tech pre-review' where we will monitor such new projects' health, contribution, and direction going in the direction of OpenStack way. This 'Tech pre-review' is not up yet, so let's see how we define it.
I am sorry I have missed that from the notes I read.
[1] http://lists.openstack.org/pipermail/openstack-discuss/2021-October/025541.h... [2] https://review.opendev.org/c/openstack/governance/+/814037 [3] https://etherpad.opendev.org/p/tc-yoga-ptg#L263
-gmann
Regards, Jean-Philippe Evrard (evrardjp)
[1]:
https://governance.openstack.org/tc/reference/new-projects-requirements.html
[2]: https://governance.openstack.org/tc/reference/project-testing-interface.html [3]: https://governance.openstack.org/tc/reference/pti/python.html [4]: https://governance.openstack.org/tc/reference/pti/golang.html [5]: https://github.com/Masterminds/glide [6]: https://travis-ci.org/github/Masterminds/glide
On 12/10/21 9:16 AM, Jean-Philippe Evrard wrote:
Hello,
On Fri, Dec 10, 2021, at 01:18, Ghanshyam Mann wrote:
Yes, those are the inconsistency we currently have in the Skyline project, but that is not what they will stick to. In Yoga PTG[1], the skyline team iterates through all these points and plans to improve those and be consistent with the other OpenStack projects. They said they will work on packaging, PTI, using Oslo code etc.
Boxiang Zhu mentioned the same in the governance patch also[2].
The questions I have still stand: Do we need to be more lax in the governance to not _require_ to follow the PTI (and/or have an intention to follow through)? Can the TC make an exception?
IMO, we shouldn't allow an exception. Otherwise, it's going to become super hard to contribute to OpenStack if every project is different. It is a pandora box that should not be opened. That being said, I don't think poetry is a problem for Debian anymore: it's been packaged already. :) Cheers, Thomas Goirand (zigo)
On 12/10/21 9:16 AM, Jean-Philippe Evrard wrote:
Hello,
On Fri, Dec 10, 2021, at 01:18, Ghanshyam Mann wrote:
Yes, those are the inconsistency we currently have in the Skyline project, but that is not what they will stick to. In Yoga PTG[1], the skyline team iterates through all these points and plans to improve those and be consistent with the other OpenStack projects. They said they will work on packaging, PTI, using Oslo code etc.
Boxiang Zhu mentioned the same in the governance patch also[2].
The questions I have still stand: Do we need to be more lax in the governance to not _require_ to follow the PTI (and/or have an intention to follow through)? Can the TC make an exception?
IMO, we shouldn't allow an exception. Otherwise, it's going to become super hard to contribute to OpenStack if every project is different. It is a pandora box that should not be opened. i would agree with this we should not make an excption however we could work on evolving the PTI over time.
That being said, I don't think poetry is a problem for Debian anymore: it's been packaged already. :)
On Fri, 2021-12-10 at 10:50 +0100, Thomas Goirand wrote: poetry i see as less of a porblem then using make honestly. as someone who has avoided using make for most of the last decade moderatly succesfully that is perhaps the most objectionable part of its build system. one concern i would have with using poetry is just makeing sure that upper-constratints.txt is used correctly based on https://github.com/python-poetry/poetry/pull/4005 and https://github.com/python-poetry/poetry/issues/3225 i dont think that poetry will ever actuly support upper-constarits so we would need to have some tooling to generate the equivalent for there format. i still think coinstallablity is an important requirement for being an offical python porject and moving to poetry or other packaging may not support that. i recently have been workign on a new python project which i was considering proposing to opendev/openstack at some point in the future i used https://pyscaffold.org/ to generate the skeloton of the project and do have a pyproject.toml https://github.com/SeanMooney/arbiterd/blob/master/pyproject.toml but have intentioally choose to continue to use setuptools so that it would be simple to support the pti in the future i think the main delta i have right now is i dont have my deps in requirement.txt but i can simply generate that in the future https://github.com/SeanMooney/arbiterd/blob/master/setup.cfg#L49-L77 that is just because pyscaffold put them in setup.cfg and i havent bothered to move them. i likely would have to add pbr for ChangeLog and AUTHORS generation and some minor other tweaks but you can have a pretty moddern an painless setup and be pti compliant at the same time.
Cheers,
Thomas Goirand (zigo)
On 2021-12-10 11:27:16 +0000 (+0000), Sean Mooney wrote: [...]
i likely would have to add pbr for ChangeLog and AUTHORS generation and some minor other tweaks but you can have a pretty moddern an painless setup and be pti compliant at the same time.
As an aside, most (if not all?) of the work needed to make PBR a functioning PEP 517 build backend is in 5.7.0, with improvements in master (not yet released) which eliminate use of some deprecated setuptools interfaces. A setup.py is still needed along with your pyproject.toml now, but there's no more calling into the setup.py install interface behind the scenes, and all the build requirements get correctly installed by pip now instead. -- Jeremy Stanley
---- On Fri, 10 Dec 2021 02:16:59 -0600 Jean-Philippe Evrard <openstack@a.spamming.party> wrote ----
Hello,
On Fri, Dec 10, 2021, at 01:18, Ghanshyam Mann wrote:
Yes, those are the inconsistency we currently have in the Skyline project, but that is not what they will stick to. In Yoga PTG[1], the skyline team iterates through all these points and plans to improve those and be consistent with the other OpenStack projects. They said they will work on packaging, PTI, using Oslo code etc.
Boxiang Zhu mentioned the same in the governance patch also[2].
The questions I have still stand: Do we need to be more lax in the governance to not _require_ to follow the PTI (and/or have an intention to follow through)? Can the TC make an exception? When is the deadline for projects not following our testing to become compliant? What is the risk if the newly accepted project is not doing it? Moving it to a non-official project? Is that worth the effort?
Those are all good points. We discussed around similar one in PTG and even considering if we should introduce different level of the official project like CNCF (Sandbox, graduate) or our old model or integrated vs incubated so that we can make it clear what projects are in-progress of becoming the OpenStack way and which are already. But these add more complications than solve the problem. We agreed to try a new concept with 'Tech Preview' to monitor the project health as well as the OpenStack consistency. Let's wait for that to be up and there we can review/improve the process. Or PTI change if needed. We will also define what is the deadline for projects to be in 'Tech Preview process. Let's wait for the Skyline team to improve the things which hey already mentioned to do and I think that can be done in a better way if they are in OpenStack community. If they do not then it can be dropped from OpenStack governance is nothing different from any existing projects do that. -gmann
If they deny moving Skyline towards the OpenStack way of packing, testing, using Oslo, etc., we should consider whether to reject the OpenStack official project or modify our new project application criteria. But this is not the case, and they agree to improve all those things.
I agree. Let's not talk hypothetics.
Now we have more clear direction from them in the governance patch also, so I feel this is good to go as an official project, and while they will be in OpenStack, we all can help them to improve the things in OpenStack way.
Perfect, great news.
Also, TC discussed in PTG the new concept of 'Tech pre-review' where we will monitor such new projects' health, contribution, and direction going in the direction of OpenStack way. This 'Tech pre-review' is not up yet, so let's see how we define it.
I am sorry I have missed that from the notes I read.
[1] http://lists.openstack.org/pipermail/openstack-discuss/2021-October/025541.h... [2] https://review.opendev.org/c/openstack/governance/+/814037 [3] https://etherpad.opendev.org/p/tc-yoga-ptg#L263
-gmann
Regards, Jean-Philippe Evrard (evrardjp)
[1]:
https://governance.openstack.org/tc/reference/new-projects-requirements.html
[2]: https://governance.openstack.org/tc/reference/project-testing-interface.html [3]: https://governance.openstack.org/tc/reference/pti/python.html [4]: https://governance.openstack.org/tc/reference/pti/golang.html [5]: https://github.com/Masterminds/glide [6]: https://travis-ci.org/github/Masterminds/glide
Ghanshyam Mann wrote:
[...] Yes, those are the inconsistency we currently have in the Skyline project, but that is not what they will stick to. In Yoga PTG[1], the skyline team iterates through all these points and plans to > improve those and be consistent with the other OpenStack projects. They said they will> work on packaging, PTI, using Oslo code etc. Ultimately there is a chicken and egg situation there -- they need to work on making it more like OpenStack, but it's difficult to justify the time to work on this until it actually is accepted as an OpenStack project.
We should communicate expectations clearly, make sure that promises are made, and then there is a leap of faith. But IMHO it is a very small leap as the TC can still control if it should be included in the next release, and the TC can also remove the project team if it never gets where it needs to be.
Boxiang Zhu mentioned the same in the governance patch also[2].
If they deny moving Skyline towards the OpenStack way of packing, testing, using Oslo, etc., we should consider whether to reject the OpenStack official project or modify our new project application criteria. But this is not the case, and they agree to improve all those things.
Now we have more clear direction from them in the governance patch also, so I feel this is good to go as an official project, and while they will be in OpenStack, we all can help them to improve the things in OpenStack way.
I agree. Expectations have been clearly communicated. It's time for us to give them a good chance to meet those. -- Thierry Carrez (ttx)
---- On Fri, 10 Dec 2021 11:32:32 -0600 Thierry Carrez <thierry@openstack.org> wrote ----
Ghanshyam Mann wrote:
[...] Yes, those are the inconsistency we currently have in the Skyline project, but that is not what they will stick to. In Yoga PTG[1], the skyline team iterates through all these points and plans to > improve those and be consistent with the other OpenStack projects. They said they will> work on packaging, PTI, using Oslo code etc. Ultimately there is a chicken and egg situation there -- they need to work on making it more like OpenStack, but it's difficult to justify the time to work on this until it actually is accepted as an OpenStack project.
We should communicate expectations clearly, make sure that promises are made, and then there is a leap of faith. But IMHO it is a very small leap as the TC can still control if it should be included in the next release, and the TC can also remove the project team if it never gets where it needs to be.
Boxiang Zhu mentioned the same in the governance patch also[2].
If they deny moving Skyline towards the OpenStack way of packing, testing, using Oslo, etc., we should consider whether to reject the OpenStack official project or modify our new project application criteria. But this is not the case, and they agree to improve all those things.
Now we have more clear direction from them in the governance patch also, so I feel this is good to go as an official project, and while they will be in OpenStack, we all can help them to improve the things in OpenStack way.
I agree. Expectations have been clearly communicated. It's time for us to give them a good chance to meet those.
Indeed. Also, I think we are making assumptions too early (I mentioned in this thread), if Skyline team deny doing OpenStack way then it is better to discuss all these points otherwise, we should just wait for them to improve the things and even better is that we (who need this project or think it is an interested one) should be part of their team and improve things. -gmann
-- Thierry Carrez (ttx)
participants (9)
-
Clark Boylan
-
Dmitry Tantsur
-
Ghanshyam Mann
-
Jean-Philippe Evrard
-
Jeremy Stanley
-
Sean Mooney
-
Thierry Carrez
-
Thomas Goirand
-
高瀚翔