Hey Artem, just a note on this:
On 11/9/25 12:36 AM, Artem Goncharov wrote:
> Another aspect is the spaghetti ecosystem that we have built. I can
> now say I spend more time fighting the dependencies and jobs that get
> broken without me doing anything before they broke than working on the
> features. Just for the sake of example, I got report of a mistake in
> Keystone Open API schema. The fix was fast: 2 minutes of work. But I
> have figured out that API-ref build job got broken. Investigation led
> to to the sphinx openapi extension. A direct dependency of it have a
> transitive dependency controlled by our constraints and was upgraded a
> week ago. Since my direct dependency does not have a vibrant community
> and it takes time to get update I am completely blocked with releasing
> the fix. I gave up and started simply getting rid (hoping for it to be
> a temporary issue) of building openapi to html as such just to be able
> to unblock fixes. Another example: I missed to control the commit sha
> of the releases change when making 2024.1 unmaintained while one
> change to (yet again) fix jobs in the branch was in the gate (I missed
> that because of amount of broken things distracting me). So the branch
> went unmaintained without the fix. Ok. I just cherry-pick it to the
> unmaintained. Good luck - the fix does not work there, come on, it is
> just a branch rename. I spent 2 days with different attempts and gave
> up, this is just unmaintained branch after all. But now I see Brian is
> messing with it and also "wastes" his time multiple days in a row. And
> the more you try to fix the more broken it becomes (even more jobs get
> broken on a daily basis). I think he will also give up.
I feel these frustrations greatly. I've spent a lot of my career in
OpenStack chasing down CI and transitive dependencies and the like --
but this is, to me, MORE of a reason to keep our technologies unified,
instead of separate. The integration issues you indicate aren't invented
by us, they are *experienced* by us due to the size of the ecosystem.
Separating from the ecosystem does have an immediate, positive effect on
that: you've reduced the size of YOUR ecosystem and concerns accordingly.
The problem is... the issues will still exist. Just now instead of
taking them on as an OpenStack group, they are pushed down to a small
collection of folks doing last-minute integration. These are lessons are
already learned as an ecosystem which is why, IMO, it's feature -- not a
bug -- that rot in the overall ecosystem slows down individual projects.
This seems to be the only way to ensure folks prioritize removing rot
from the ecosystem.
To be clear: I'm not pro or anti-rust in any way, in OpenStack or
otherwise, I just think this impression of speed being given by moving
out of the ecosystem or changing languages is merely work that's been
offloaded to others implicitly. To some extent, it's seems this may
already be happening regarding keystone-ng, based on the Keystone team
barely meeting deadlines to patch OSSA-2025-002 before it being
disclosed before-fix due to embargo timeout.
Thanks,
Jay Faulkner
G-Research OSS
P.S. I do not love that this is the reality. I've mentioned multiple
times that I wish you didn't need a PhD in devops to contribute to
OpenStack, but the only way to work out those complexities is by working
together, not by concentrating that knowledge further by passing the
buck on ecosystem improvements.