Open Stack

Hi Jean-Philippe.

Please read what I'm writing keeping in mind that I'm *very* interested
in Skyline, and would love to have it fully packaged in Debian. I
probably will do that when I consider the project ready for it, which
unfortunately, in the current case, isn't the case (ie: it would need
too much effort from my side for it).

So I'm writing in the hope to get things improved... :)

On 12/10/21 10:43 AM, Jean-Philippe Evrard wrote:
> For the case of skyline, one could made a case to start building packages with a new way, using poetry, rather than setuptools. It could mean we need to ask the skyline team to help on improving the infrastructure and testing.

>From my package maintainer point of view, the main problem I see is more
then way things are layout (ie: the libs folder of skyline-apiserver
that contains many projects). I don't really mind anymore the poetry
thing. This shouldn't be so hard to address, right?

> Instead of arriving to a standard testing, we embrace the difference and provide a possible new way of working. We could have two kinds of python projects. Instead of accepting that difference when a single project asking for it, maybe we can just tally who wanted this, and the TC decides at the end "when" it is worth doing.

I don't think you understand the problem here. The issue is wiring
Skyline to the OpenDev CI. With the way things are done, it's simply
very hard to do, unless you're ready to spend *a lot* of time with the
infra team, and if they have enough time to let you do that (probably
they are already very busy).

> In other words:  Is it really better to change the skyline team to adapt our mold, or should we have multiple molds?
> We agreed to go the way that skyline should adapt. I am simply raising the fact that some other people/projects won't do the effort to adapt to existing mold, because they won't necessarily see the value.
> (I won't prevent my teams to work on non-openstack projects ;))

Same story: if the infra can handle the "new way" that you're proposing,
why not. But we're not there yet at all.

>> Let's take the current example point by point:
>>
>>   * Skyline implements its own routines for things like
>>     configuration and logging rather than using Oslo libraries.
>>     Should all OpenStack projects "move forward" to this model,
>>     choose their own configuration and logging formats, implement
>>     their own solutions for documenting these, and so on? Does
>>     inconsistency here benefit operators in some way?
> 
> Is Oslo a requirement to be an OpenStack project now?
> AFAIK it wasn't (see swift).

You wont make your case by pointing at the defects of other projects.

You could also have pointed at Horizon's local_setting.py. IMO it's a
much better example beacuse it's a very good example: it's a *REAL PAIN*
for operators (for example: for puppet, that has no way of handling that
configuration file using something else than a ERB template).

> Side note: Bringing oslo projects for configuration and logging isn't enough to bring consistency from the user perspective.

Probably, but it's a starting point.

>>   * Skyline defines the interdependencies between its components
>>     with a mix of "monorepo" (multiple packages from a single
>>     repository) and Git submodule references (pinning specific
>>     commit ids as virtual subtrees of another repository), rather
>>     than establishing a loose coupling with explicit version numbers
>>     and package relationships. This choice prevents them from taking
>>     advantage of features in Zuul like cross-project change
>>     dependencies. Is this a pattern the other OpenStack projects
>>     should follow?
> 
> Which is why I am highlighting that _as of today_, the governance requires certain expectations, which needs clarifications (or exceptions) if we want to integrate Skyline.
> 
> I know I am a bit pedantic in the following question:
> Is (nowadays) the expectation for projects to be testable by Zuul (+release-able by zuul, ... list to be defined) or is the expectation for projects to be following the PTI guidelines (tested by zuul)?

I'm not from the TC, but ... hopefully BOTH ! :)

>>  Does it benefit distribution package maintainers
>>     to need to track fundamentally different repository layouts and
>>     build processes for each of them?
> 
> Isn't that exactly what the distros are doing right now with the packages they are maintaining? Not every software is like openstack. Yes, almost all openstack projects are similar to each other, so they can be packaged the same way.
> 
> Yet, some official openstack projects aren't, as of today. So, some distros decide to not package them?

Could you please be more specific? Maybe with some examples? I'm having
a hard time trying to figure out what you have in mind here... :)

> Keep in mind that we are _already doing exceptions_. Not all projects are similar to nova, glance, neutron, keystone, cinder. (Sorry for the many I missed).

Which one are you thinking about? :)

>>   * Skyline performs its release versioning by committing the
>>     version strings to files in its repositories rather than
>>     assigning them at build time from Git tag metadata. This leads
>>     to ambiguity as to what exact state of the repository represents
>>     that version, as well as opening up the possibility of
>>     forgetting to merge the version update before tagging (a very
>>     common problem which drove OpenStack to rely on its current
>>     model). Wholly different tools are also needed in order to
>>     track versions for these projects as a result. Should the rest
>>     of OpenStack's projects follow suit there? Has something changed
>>     to make the idea of using Git to signal releases a bad one?
> 
> You're raising again a valid point, for which the skyline team will have to find a solution.

>From a distro perspective, it is easy to build multiple binary packages
out of a single source tree. However, it's not possible to produce
different versions of the binaries, if we have a single source package.
So IMO, this should be addressed as a top priority.

> For me, it boils down to defining that projects should follow a "project release interface", which explains the expected behaviour in regards of releasing for projects. We don't have such. I am not claiming we should, I am merely asking us to think about what we are doing.

Yes, you should think about it! :)
Would it be a huge effort to separte things into multiple projects
instead of one big repo, and then start tagging properly?

> Projects could indeed move to use pbr/setuptools. However, I am not sure many developers understand why.
> Should we simply explain what is expected in terms of building the version from git? People might even come with interesting solutions. Promoting creativity will create tech debt, but might also reduce it over time.
> What if we don't need pbr anymore? Will we even notice?

>From my perspective, pbr/setuptools isn't the main problem. Single repo
+ not tagging is.

> For me, tox is a test runner that provides many nice features thanks to its code (and integration with setuptools)
> I would prefer if python projects using tox for testing interface indeed.
> Should that be the standard? I don't know.
> Does it make sense to rely on tox features for isolations/package management/other? I don't know.
> You might be aware that it's possible to integrate tox with poetry. Does it provide all we _need_? I don't know: We aren't explicit in the requirements of what we need, we just talk about tools.
>
> If I were to rephrase: Is it easier to fit the existing mold or clarify what we need, why, and how we can deal with the tech debt? Probably the former. Should we do it? I don't know. The TC should however evaluate this seriously.

What Jeremy wrote: it's important to be able to use the global
requirements repository, so we can check your project works with the
others. I don't think the TC will have any satisfying answer if you
can't provide a way to check your projects works with the rest of the
OpenStack world.

>>   * Skyline does not put its external Python package dependencies in
>>     central lists nor use tools which would allow it to rely on
>>     OpenStack's global constraints implementation, making it very
>>     hard for them to guarantee that their software can run with the
>>     same versions of shared dependencies as the rest of OpenStack.
>>     Is this unnecessary? Have we given up on the idea that two
>>     pieces of OpenStack software can be coinstalled into one
>>     environment, or that Linux distributions won't be stuck having
>>     to package many different versions of the same software because
>>     OpenStack projects don't agree on which versions work for them
>>     any longer?
> 
> Now, we are hitting the nail, IMO :)
> 
> This is a bit of the issue's core, isn't it? I think that's what we need to document.
> While I (personally) don't believe it's important nowadays (cf. containers), I am sure distros will disagree with me.

Indeed, I strongly disagree! :)

> It's important to know who we are pleasing, as we can't please everyone ;)
> Let's be clear about the audience.
> 
> If we intend to please the distros, did we ask about RedHat/Canonical committments on skyline? Are they even remotely interested?

I am *very* interested. :)

If you didn't know, I've been packaging OpenStack in Debian since the
very beginning (ie: in 2011 for me).

> If they want to use it, wouldn't it make sense they do the work to make sure skyline is packageable for them, by improving upstream?

Do you understand that *all* the distros are understaffed? Do you really
expect package maintainers to address *ALL* of the issues they see in
*ALL* of the OpenStack projects? That's not reasonable, and it wont ever
happen, simply because we can't.

> If they don't, why do we care?

Why wouldn't you care about integration? Being package-able also means
your project can be integrated in *any* environment, not only distro. If
you don't care, why do you then care to have Skyline as an official
project then? It doesn't make sense to me.

Cheers,

Thomas Goirand (zigo)