Before reading my answer: please keep in mind that I'm very enthusiastic about such move. Never the less, I have still a lot of questions for you, but none should be taken negatively.

Thank you very much for the very explicit clarification, it really helps a

lot. Same is valid for my answers.

On 11/6/25 4:26 PM, Artem Goncharov wrote:

...

- A nice coincidence is a freshly sent announcement from Debian that introduces a strong rust dependency in the packaging tooling, stating something similar to: either you work to comply with this requirement in the next 6 months or sunset the port [5].

Just like Jeremy wrote: this was *not* an announcement. It was a post written by a Canonical employee that has the mission (from Canonical) to rewrite stuff in Rust. He alone felt it was ok for him to impose a Rust rewrite of APT, even though some of the unofficial ports aren't ready yet with the Rust stack.

While I think it may be a nice idea to rewrite things in Rust, I don't think APT has any performance issues, and never had any memory issue. So I'm unsure what is Julian is trying to achieve.

got it, wasn't the best decision to mention it. But it shows that any change (or no change) in a community leads to opposition, just like it was with systemd, pulseaudio, you name it.

I am very much frustrated that you seem to be having the same approach: announcing what you want to do, or have already done, instead of discussing the mater with the wider community.

It is not the case. I announced my plans and discussed this already during the last PTG and even earlier within the Keystone team.

While I do like the idea of having a super-performing Keystone in Rust, I am sure there are better ways to move the OpenStack project in such direction. Why didn't you discuss this *before* writing the first line of code?

When you go to the Shark Tank/Dragon's Den show you need to prove your idea is working to get support. Nobody supports just an idea nowadays. Not in my surroundings. But really, since I see this type of statement not the first time (and please do not take it personally), is it so normal nowadays to "criticize" people for writing code to prove the idea? I want to solve the problem and not spend years in discussions on what could be "acceptable" by others. Anyway, I am a supporter of the "every tool has its own specific use case" philosophy, there is nothing universal out there.

I also remember the Swift project willing to start using Go. At the time, the Swift team was told that it would be ok if they spent a large amount of time making sure the infrastructure (ie: the CI, gating and all) would be ok for Go.

Are you willing to spend such time with the infra team?

We are not there yet. It is not even clear whether OpenStack is ever going to be there, so this is just a pure speculation at that time. But to answer your question - yes, if necessary and accepted by the community I am ready to spend time adding new jobs for Rust. Go example feels weird, to be honest, work has been done and there is not a single project using it now. Perhaps this is worth a separate analysis. As mentioned initially - we are not requesting Rust to be accepted now, but announce a work on a Rust project and the community may benefit from it being considered as official. But it is not a must now.

Also, you've blame some of the Python libs Keystone depends on. How can we make sure the same wont happen with the Rust things you'll depend on? Are all the Rust dependencies you depend on already in Debian, Ubuntu and Red Hat?

Valid points, you can never be sure, in nothing actually, not even in native libraries of the language itself. We have seen it also in python (some examples are crypt, setuptools support). I have tried to do wider analysis and rely on the libraries with the most healthy activity and wide adoption, used by big players (tokio, axum, sea-orm). I am not that familiar with the packaging requirements of Debian for Rust project, so I can't answer your questions precisely. While I believe I understand your points pretty well, I do not think this should be something that a regular maintainer of a certain random project should focus on. Every distro is coming with its own rules and processes and if one would want to keep everybody happy no development should ever happen. I personally think Rust makes the situation a bit easier (while perhaps requiring a certain number of initial changes in the processes) due to the static linking - it does not matter which dependency versions are installed at the user's OS, what matters is which versions were used during compilation. It is a dependency hell during the build, but not during the runtime. But you may have different valid opinions on this. No offence or blame!

...

- Keystone is a very sensitive component and in our eyes that fully justifies use of rust to provide a much stronger security on the language level.

There's been a few security issues with Keystone in the past. None of them were due to the language choice. I'm not buying into this at all. Performance, yes...

There were few, in my opinion, not directly in Keystone, but in the keystonemiddleware, but I do not want to bike-shed on this. I experienced (and fixed myself) plenty of flow and logic bugs in multiple projects that would have been prevented by Rust. Correct - performance improvement is big. Btw, that is also a reason for implementing before (or in parallel to) talking - you can't prove the performance improvement initially, now I can do this.

...

2. Do you deprecate python Keystone. - No - v3 is there, and we continue maintaining it

Does the Rust implementation already has *all* of the current v3 support? I wouldn't like having to deploy *2* Keystone. I'm also very much worried about the move to a v3, and what kind of migration effort this will push on all projects. I do remember that the switch from Keystone v2 to v3 was *very* painful and took really too long.

We tried to be as precise as possible, but apparently still not enough (I assume your mistyped v2 and v3 instead of v3 and v4). Few thoughts from me personally not representing the team anymore: - it was explicitly stated that KeystoneNG is an "extension" and not the replacement. Folks actually convinced me to state it this way not to be crucified immediately. - we discussed potential evolution and how to implement migration strategy with implementing (integrating) parts of existing v3 into NG to simplify migration. But the message was and stays: nobody is rewriting Keystone in Rust for the sake of rewriting, or fun, we start with a new project that CAN be considered as a successor WHEN there is an interest. Regards, Artem

Hey

IMO, that's a very common miss-understanding on how distros works. I'll speak for Debian, but Ubuntu and Red Hat have the same requirements.

In distros, even in Go or Rust, each and every library is packaged separately. We do not accept libs to be "vendored". Well, there are a few exceptions, but at least, that's the global effort.

This means that each and every library that you're going to (build) depend on will have to be in the distribution, so that at build time, your project can link against them.

The fact that all is, at the end, statically linked, changes nothing to what I wrote above. The only thing it changes, is for people wishing to run binaries built outside of the distribution.

So no, compared to the Python situation, it changes nothing (at least, from a package maintainer perspective). The only annoying thing for me: I will have to learn Rust packaging. Though this is probably a good thing! :)

I was exactly trying to explain that our points would have valid differences. You are right that it does not change much on the packager side. But in my eyes it should change a lot on the user side. When 2 different projects depend on a single library (i.e. libssl) of different versions you should not have a bigger problem when updating this lib, since Rust project is compiled with hopefully a certain supported version. From what I read about the debian Rust packaging process is that it is expected to have crates in many different versions available as packages exactly to address that. You, as a user, will not have a broken runtime system just because dependency was updated for the other project. From my point of view the constraints issue in runtime is addressed with that. Am I wrong here? The big difference between python and rust in transitive dependencies version handling is that rust allows you to have multiple versions of a single transitive dependency during the compilation. This is, however, exactly what makes my life in the python world a nightmare when updating transitive dep in constraints for the need of one project breaks some other dependency.

...

We tried to be as precise as possible, but apparently still not enough (I assume your mistyped v2 and v3 instead of v3 and v4). Few thoughts from me personally not representing the team anymore: - it was explicitly stated that KeystoneNG is an "extension" and not the replacement. Folks actually convinced me to state it this way not to be crucified immediately. - we discussed potential evolution and how to implement migration strategy with implementing (integrating) parts of existing v3 into NG to simplify migration. But the message was and stays: nobody is rewriting Keystone in Rust for the sake of rewriting, or fun, we start with a new project that CAN be considered as a successor WHEN there is an interest. Regards, Artem

I very much understand your point about not rewriting the v3 for the sake of it. It makes a lot of sense, and in fact, it's also probably avoiding some bugs where the implementation could have made small, undetected changes leading to hard to find bugs.

So, if I understand you correctly: there WILL be a time where we'll be running both keystone and keystone-ng in a single deployment, where the Python one would handle v3, and Rust one handling v4. In such case, do you see them sharing the same port? Or maybe the Rust one forwarding the traffic to the Python version? How will this work? Will there be a new OpenStack endpoint for the Rust version?

Valid points. Actually pretty much everything you have mentioned is valid on its own and makes sense. It can be implemented either way. It depends on the technical feedback and ideas we can gather from vendors and CSPs. It was removed from the summary email (not to give food for concerns), you can read details in the [PTG etherpad]( https://etherpad.opendev.org/p/oct2025-ptg-keystone) for options and potential strategy. You could route v3 to the left and v4 to the right, you can have only rust doing the proxying and internally forward requests to v3 (addressing the wsgi issues we have). Regarding the endpoint: current implementation that I was using during the demo in talk on summit is that version discovery is served by the Rust and shows v3 as stable and v4 as experimental (pointing at different url). Version discovery standard is there to address it without introducing new endpoint/service We explicitly gather feedback and ideas.

Last question: I did a git clone, and saw that all commits were from you. Is there others onboarding? Like Jeremy, I see Rust as possible blocker to get contributors (at last at first), and it's worrying if you're the only person pushing patches.

The reason for the message was to raise awareness. As also mentioned - there were provisional commitments from vendors and csps to contribute and we now wait to see how it evolves. We now also have problems with contributors in python, ...

Thanks for your work on this, and sharing your views, Cheers,

Thomas Goirand (zigo)

P.S: This project is one more new reason why I should learn Rust. :)

:) Using it on a daily basis I see so many problems we have in python that are conceptually not possible in rust. It is not an easy language, and from experience I can say here with absolute confidence - without knowing this language it is impossible to judge what is possible and what not and to estimate which kinds of problems would have been prevented. Regards, Artem

On 2025-11-07 18:17:14 +0100 (+0100), René Ribaud wrote: [...]

I don’t know what the Foundation or the Technical Committee’s current position is on Rust within OpenStack [...]

The OpenInfra Foundation expresses no opinion in its bylaws, and while https://board.openinfra.org/en/ProjectGuidelines has been used for determining project suitability it only concerns itself with licenses and practices not choice of programming language. The OpenStack Technical Committee does have published guidance about choice of programming language... https://governance.openstack.org/tc/resolutions/20150901-programming-languag... ...as well as how to qualify new languages: https://governance.openstack.org/tc/reference/new-language-requirements.html An example is this resolution from when Go was added: https://governance.openstack.org/tc/resolutions/20170329-golang-use-case.htm... Introducing Rust would in theory involve a similar resolution, and "just" needs discussion and a vote among the TC members. -- Jeremy Stanley

Thanks for the response, Andriy On Fri, 7 Nov 2025 at 19:52, Andriy Kurilin <andr.kurilin@gmail.com> wrote:

...

The work was done in Rust due to multiple factors: lack of necessary python libraries with trustable maintenance and/or licensing, necessity for enforcing security standards and practices using memory safe programming language

Just curious, did you consider creating a Rust "binding" layer for Keystone instead of a full rewrite? That might address the Python dependency and maintenance issues you mentioned.

As stated initially - yes, we have this possibility when using Rust. Due to the heavy plugin architecture (what you refer separately) bindings are not that trivial, but it depends on the "entry point" of the binding. To remind - we have a problem that certain things (saml) can be considered gone and we need to build the whole new federation logic around the oidc only giving users the control. So the reason is not the "rewrite", the reason is that features required an external work and the result of it looks very promising to be extended.

...

A side effect of it became immediately visible: performance. This allows us to achieve significant performance and maintainability improvements, eliminate the old ballast

From an operator’s perspective, Keystone performance has never been an issue for us. It just works. Thank you, folks! So if performance improvements are a side effect, great - but it has never been high on my list of concerns.

It is interesting how the situation differs from cloud to cloud. Just not that long time ago (less than a year) we had a bug ( https://review.opendev.org/c/openstack/keystone/+/880456 for the fix) that a token validation request for the token issued for application credentials took 5-10x longer than the one for password. The commit message states around 400ms (on devstack), but in our public cloud it was 5s. And this is compared to something around 100ms for the password one (measures under real load). The "fix" is just caching the data that Keystone fetches over and over from the DB due to the current architecture. Don't nail me on numbers, pls, it heavily depends on the environment and the use case, but token validation with rust takes without any caching 10-15ms on my laptop and on a VM with devstack

...

The new stack is fully async, uses parallel DB queries, and runs background tasks for periodic cleanups, synchronization, and similar.

This reminds me of the early async wave in Python. There were claims like “we built the fastest Postgres driver in the world,” but in practice, it did not noticeably change anything. I mean the problem quite often is not in the technological stack, but in the way people write software, which covers the dependencies and logical flow as well.

. The new stack makes it possible to execute queries in parallel. I do not know whether it is relevant for other services, but for Keystone it does. Current architecture does 60+ queries during a single request processing. Now we can control the data that we have already fetched (not based on the memcached) and send out some queries in parallel where the db schema requests multiple data sets.

Anyway, I would like to thank the Keystone team for the work you are doing and for opening this conversation. And to be fair, not all my colleagues share my skepticism - some of them can't wait for the new Keystone to be accepted so we can finally deploy it in our cloud :)

Please ask mentioned colleagues to step by and start participating, even if it is just in the discussions or issues regarding new features or the desired development direction. New features that I was presenting were focused on making operators and users happy. Having such feedback is making me feel like it was not a time waste.

At the same time, who knows - maybe part of the reason OpenStack struggles to attract new contributors is that most available Python developers are already here, and bringing in the Rust community could add some fresh air to the ecosystem.

Maybe, I was also thinking about that, but didn't want to explicitly mention it. Artem.

Hey, Let me please explain my view on the position that you and multiple others expressed (so don't take it personally) after drawing a long bold line separating it from the initial topic. If one wants to continue this particular discussion feel free to branch it out into a separate thread instead. ================================== The following is my point of view, yours would surely differ!!! I think the problem with a decline of maintainers is artificially created by ourselves. We are sitting behind the wall that we've built and continue making it higher and thicker. I believe we are wearing contributors out with the mentality of let's have everything done the same way so that others have a chance to take it over when (an example that I heard: you would be hit by the bus while crossing the street and there is nobody else in the project) the project is left unmaintained. We try to do everything possible to keep project or library alive beyond our capacities. Every time an OpenStack project/repo looses last maintainer we spend literally years discussing that and searching somebody who still has some capacity before reaching 200% load. Another aspect is the spaghetti ecosystem that we have built. I can now say I spend more time fighting the dependencies and jobs that get broken without me doing anything before they broke than working on the features. Just for the sake of example, I got report of a mistake in Keystone Open API schema. The fix was fast: 2 minutes of work. But I have figured out that API-ref build job got broken. Investigation led to to the sphinx openapi extension. A direct dependency of it have a transitive dependency controlled by our constraints and was upgraded a week ago. Since my direct dependency does not have a vibrant community and it takes time to get update I am completely blocked with releasing the fix. I gave up and started simply getting rid (hoping for it to be a temporary issue) of building openapi to html as such just to be able to unblock fixes. Another example: I missed to control the commit sha of the releases change when making 2024.1 unmaintained while one change to (yet again) fix jobs in the branch was in the gate (I missed that because of amount of broken things distracting me). So the branch went unmaintained without the fix. Ok. I just cherry-pick it to the unmaintained. Good luck - the fix does not work there, come on, it is just a branch rename. I spent 2 days with different attempts and gave up, this is just unmaintained branch after all. But now I see Brian is messing with it and also "wastes" his time multiple days in a row. And the more you try to fix the more broken it becomes (even more jobs get broken on a daily basis). I think he will also give up. What I am trying to say: - does it make sense how we have build our ecosystem: yes, it does - can it be done easier/better: almost certainly yes While, as said, there are reasons for it to be how it is, we must admit, that there is the other side of the coin that is a consequence of that approach: - OpenStack contribution is very complex - we wear contributors out with the mentality that everything is built similarly and so taking it over another abandoned thing (maybe for a reason) is relatively "easy" - this leads to more and more load and we spend our time in maintenance of everything around instead of a focus on a primary project - we loose concentration fixing multiple things in parallel - we cannot easily innovate (or even try to innovate) because of the mentioned rules and constraints. We even prevent using thing that would simplify the daily work with the rules and constraints of the ecosystem For me personally coding outside of OpenStack brings satisfaction, coding in OpenStack brings frustration every single day. Some know I burned myself down few years ago with that. Now I am again close to that. I thought multiple times already of just giving up on the OpenStack world and just move myself into a different area. I know personally maintainers who left OpenStack because of not seeing future in it and tired of artificial problems. Just to remind, mostly I do the work on a voluntary basis, I am not paid for working on OpenStack. I hope I was able to explain why I see the mentioned approach as dangerous. Another reminder: this is my personal opinion and is not representing the Keystone team Regards, Artem ---- typed from mobile, auto-correct typos assumed ---- On Sat, 8 Nov 2025, 02:21 Takashi Kajinami, <kajinamit@oss.nttdata.com> wrote:

On 11/8/25 4:36 AM, Artem Goncharov wrote:

...

At the same time, who knows - maybe part of the reason OpenStack

struggles to attract new contributors is that most available Python developers are already here, and bringing in the Rust community could add some fresh air to the ecosystem.

...

Maybe, I was also thinking about that, but didn't want to explicitly

mention it.

I have a different view about this point. Introducing a new language might be attractive to developers who are willing to learn new things and find python boring and legacy. However we should also be aware of potential increase of the complexity of the whole software stack and our tool chain. For example operators may need to learn how they debug software written in a new language. New developers joining the community may need to learn two sets of language and also different toolings (requirements management, the way to run tests, the way to run format checks, and so on). I have some concerns that these can increase user/developer effort and cause opposite effect.

I'm not saying that this is a hard blocker, but just want to reminds you of potential tradeoff we should consider. I hope that the plan to ease the additional complexity (maybe some documents for users and operators for initial learning) is also discussed as well as the software technology, for example.

...

Artem.

Thanks for your opinion Jay, For me it translates to: I should continue chasing dependencies instead of implementing features or addressing vulnerabilities, and, as you rightly pointed out, I should continue doing this alone. Or am I overseeing anything (no offense)? Actually I would like to disagree with your conclusions: everybody should deal with "own" responsibilities and during my Keystone work I should not take care of oslo.xxx, branching, requirements, zuul, ... - there should be people dealing with those. Right now I do not have time to work on Keystone exactly because I need to take care about everything else, keeping it just "running". The situation in the world in the last few years showed me that while globalization allows you to be more efficient it makes your system very vulnerable - a single point of failure. A balance of centralization and decentralization must be found to keep you having electricity and heating when a single rocket launched by a bad actor is enough to bring your whole electricity network down. I tried to make a point that disappearing contributors are perhaps caused by the complexity that we have and not just because nobody thinks OpenStack is relevant. And instead of accepting the state as it is, we may consider a change and try to find another way of solving the problem while helping people to actually enjoy the work. On Mon, 10 Nov 2025 at 17:43, Jay Faulkner <jay@gr-oss.io> wrote:

Hey Artem, just a note on this:

On 11/9/25 12:36 AM, Artem Goncharov wrote:

...

Another aspect is the spaghetti ecosystem that we have built. I can now say I spend more time fighting the dependencies and jobs that get broken without me doing anything before they broke than working on the features. Just for the sake of example, I got report of a mistake in Keystone Open API schema. The fix was fast: 2 minutes of work. But I have figured out that API-ref build job got broken. Investigation led to to the sphinx openapi extension. A direct dependency of it have a transitive dependency controlled by our constraints and was upgraded a week ago. Since my direct dependency does not have a vibrant community and it takes time to get update I am completely blocked with releasing the fix. I gave up and started simply getting rid (hoping for it to be a temporary issue) of building openapi to html as such just to be able to unblock fixes. Another example: I missed to control the commit sha of the releases change when making 2024.1 unmaintained while one change to (yet again) fix jobs in the branch was in the gate (I missed that because of amount of broken things distracting me). So the branch went unmaintained without the fix. Ok. I just cherry-pick it to the unmaintained. Good luck - the fix does not work there, come on, it is just a branch rename. I spent 2 days with different attempts and gave up, this is just unmaintained branch after all. But now I see Brian is messing with it and also "wastes" his time multiple days in a row. And the more you try to fix the more broken it becomes (even more jobs get broken on a daily basis). I think he will also give up.

I feel these frustrations greatly. I've spent a lot of my career in OpenStack chasing down CI and transitive dependencies and the like -- but this is, to me, MORE of a reason to keep our technologies unified, instead of separate. The integration issues you indicate aren't invented by us, they are *experienced* by us due to the size of the ecosystem. Separating from the ecosystem does have an immediate, positive effect on that: you've reduced the size of YOUR ecosystem and concerns accordingly.

The problem is... the issues will still exist. Just now instead of taking them on as an OpenStack group, they are pushed down to a small collection of folks doing last-minute integration. These are lessons are already learned as an ecosystem which is why, IMO, it's feature -- not a bug -- that rot in the overall ecosystem slows down individual projects. This seems to be the only way to ensure folks prioritize removing rot from the ecosystem.

To be clear: I'm not pro or anti-rust in any way, in OpenStack or otherwise, I just think this impression of speed being given by moving out of the ecosystem or changing languages is merely work that's been offloaded to others implicitly. To some extent, it's seems this may already be happening regarding keystone-ng, based on the Keystone team barely meeting deadlines to patch OSSA-2025-002 before it being disclosed before-fix due to embargo timeout.

Thanks, Jay Faulkner G-Research OSS

P.S. I do not love that this is the reality. I've mentioned multiple times that I wish you didn't need a PhD in devops to contribute to OpenStack, but the only way to work out those complexities is by working together, not by concentrating that knowledge further by passing the buck on ecosystem improvements.

On 10/11/2025 21:22, Jeremy Stanley wrote:

On 2025-11-10 18:12:54 +0100 (+0100), Artem Goncharov wrote: [...]

...

Actually I would like to disagree with your conclusions: everybody should deal with "own" responsibilities and during my Keystone work I should not take care of oslo.xxx, branching, requirements, zuul, ... - there should be people dealing with those. Right now I do not have time to work on Keystone exactly because I need to take care about everything else, keeping it just "running". [...]

As others have already suggested far more eloquently than I, maybe what we need isn't people working on Keystone (and Nova, and Cinder, and Neutron, and...), but people working on *OpenStack*. I tend to agree that even if you primarily contribute to one project that should not stop you contributing to other. that does not mean i believe it correct for companies to expect that they can ask you to work on any project withing openstack. that is a marital change in your employment terms if you are primary on project A and they tell you to work on project B but i do think working on other projects is good as it allow you to cross pollinate ideas and approaches form one to another.

I often come back to https://governance.openstack.org/tc/reference/principles.html#openstack-firs... as well. """ OpenStack First, Project Team Second, Company Third<https://governance.openstack.org/tc/reference/principles.html#openstack-first-project-team-second-company-third> OpenStack leaders are expected to put the needs of OpenStack first in their decision making, before the needs of any individual project team. They are also expected to put the needs of their project team before the needs of the organization they work for, if any. They can of course represent the needs of their project team or their company, but in case of conflicts of interest, they should be ready to put those needs aside and make the best call for OpenStack as a whole. """ I primarily gravitate around nova/neutron/placement (and now watcher/devstack) but i will happily work on almost any part of the python services when i have time and folks as for help if it help the community and projects i am most engaged with. keystone is one of those project that if it needed help and some of my time i would try and make room to contribute, there are certainly features i would love to implement in it like moving public keys to be associated with keystone users and allowing them to be used to login. The main issue i have is i don't know what support keystone needs form contributor and i don't really want to go bring more work to the keystone team by proposing feature extra if i know they are already over burdened. I know we can collaborate on things like this at the ptg or mailing list but i feel like this is one of the things we kind of lost form the in person events. it was a lot easier to hear about initiatives and gaps in the hallway track or over a drink in the evening then in "formal" sessions or virtual cross projects as you just don't have the time to follow up with folks 1:1 or in small groups when you hear about a problem or interesting idea. But yes +1 to working on openstack if your situation allows you too not just working on a specific project. reagards sean.

Hey there, Sorry for popping-up so much later after the discussion has happened, just got to it through my "TODO" list... But probably it's good to revive the discussion before the upcoming PTG, as I see that a big progress has been made on the topic. So what I wanted to add to this discussion, is that I think such effort of re-writing keystone should not be considered as an existing project replacement, but rather as a completely separate project. I think from potential governance perspective, if there is ever an intention to bring the project under OpenStack governance umbrella, that this should be registered/pulled in as a separate project of `identity` type, and should pass all criterias for emerging projects defined by governance [1] That being said, the emerging project process will definitely show if re-writing in a different language will help to bring new maintainers or contributors to the effort, as has been claimed multiple times above, or it will continue to be a single-person job. But also to the point about features that are being introduced and currently missing in Keystone - there is no reason not to implement them to Keystone directly, I don't see at all how using a completely heterogenous language for the existing ecosystem helps here, except complicate deployment and debugging process, as well as introduce complications for business (RUST developers are not cheap, and now we would have to hire one just to be able to debug and assist with one project in an established ecosystem). Coding language is just a tool that is being used to reach the goal. So nothing stopps community from bringing all these features to the existing Keystone project (Python one), even though implementing a new Keystone API version. [1] https://governance.openstack.org/tc/reference/emerging-technology-and-inacti... ср, 12 нояб. 2025 г. в 23:56, Jean-Philippe Jung <jjung@redhat.com>:

Hi,

I want to thank all the contributors to this discussion.

As a reminder, what Artem currently brings with keystone-ng extensions is:

- OIDC with self-service and CLI integration - Workload federation (Zuul, GitHub, Gitlab do not need to have secrets to access OpenStack anymore) - Service accounts (many use cases, one of the prominent ones: OpenStack on Kubernetes, GitHub workflow) - Passkeys (webauthn) - SCIM - Tight Open Policy Agent integration.

The OpenInfra sessions were full of people interested to hear about these features. The question is how to get these features in the hands of openstack operators. We have three options:

- Either embrace the codebase Artem developed (with all the rust learning, new codebase understanding, CI dependencies…) - Consider re-developing those features in python - Do nothing

Keystone maintainers are few, if we go with option “python”, will anyone step forward to help with the new feature development? Because we know who’s working in keystone, and it has not been growing in the last few years.

If we go the “rust” route, we get a working PoC but incomplete codebase, which might attract new contributors.

If the current approach is not working, maybe the approach has to change.

There is a demand for the modern features Artem developed. Either we can provide them in OpenStack, or maybe the solution will happen outside of OpenStack (e.g., k8s+kubevirt). The main risk is to achieve very little modernization in keystone due to lack of capacity.

We can argue about language, CI, splitting the community (all good arguments).

At the end of the day, what is the path of least resistance to modernize keystone & get new contributors to this modernization effort?

I find the keystone-ng experiment interesting, I hope it gets considered & interested maintainers will show up in the next 2-3 months mentioned in the initial email. Artem, I salute your determination in challenging the status-quo.

Best regards,

JP (Speaking in my own name and on my own behalf)

On 7 Nov 2025, at 12:17, René Ribaud wrote:

Hi,

Thank you very much for sharing this detailed update. That was a very interesting and inspiring read.

I’m probably a bit biased since I’m a big fan of Rust myself, but I truly appreciate how much it can help us build safer and more performant software. The direction you’re taking feels ambitious and forward-looking.

Unfortunately, I couldn’t attend your session at the OpenInfra Summit in October, but I’ll make sure to watch the replay soon. From what I read here, the work and vision behind this initiative are really impressive, and I’d like to express my full support for the effort.

I also realize that introducing a new language like Rust into the OpenStack ecosystem won’t be easy and will likely take time, both from a technical and a governance perspective. But it’s a fascinating step forward, and I think it’s also an interesting opportunity to see whether this approach can attract new contributors or a broader group of people willing to help sustain the project.

I don’t know what the Foundation or the Technical Committee’s current position is on Rust within OpenStack, but personally, I believe this kind of project should be encouraged under an *experimental* or *incubation* status. It’s exactly the kind of innovation that can keep the ecosystem vibrant and evolving.

Congrats again for pushing such an ambitious and technically exciting direction.

Best regards, René

Le jeu. 6 nov. 2025 à 16:27, Artem Goncharov <artem.goncharov@gmail.com> a écrit :

...

Hey guys

Let us give you an update of the OpenStack Keystone future following the great PTG week we just had.

Some news has already spread across the community, causing excitement and anxiety. It is not uncommon that changes cause fear. Some time ago, in the age of the Inquisition, people were even burned because of fear, misunderstanding, or questioning the rules. Let us clarify things.

Nearly 13 years ago, the Identity v3 API was introduced. Since then, the world has changed dramatically. Libraries we use got abandoned, standards went legacy and replaced with new ones, ecosystem components, certain functionality is based upon are being sunsetted by OS vendors (mod_auth_mellon is archived upstream and gone in CentOS and RHEL [6]), new security vulnerabilities (classes of vulnerabilities) appear like mushrooms without us being able to address them.

Cloud providers today face new challenges that were not present 10-15 years ago. Nowadays, a public cloud requires proper federation support with complete self-service. Users request passkey support, and security departments require additional hardening. These things cannot be covered in today's Keystone architecture and API version. As a result, work on an optional extension to Keystone [1] began some time ago, allowing CSPs to deploy it in parallel with the Python Keystone (of any v3 version) and access the new features.

To simplify parallel deployment, it has been decided to start introducing v4 with the new features (since those ARE for cloud end-users). The work was done in Rust due to multiple factors: lack of necessary python libraries with trustable maintenance and/or licensing, necessity for enforcing security standards and practices using memory safe programming language, demand to address python ecosystem madness of past years (I guess majority would agree, that in the past few years we were mostly only chasing the python ecosystem breaking us on setuptools, eventlet, passlib, …).

During the work, it became clear that implementing the mentioned features requires support of a given percentage of the existing functionality: users, groups, projects, roles, and tokens. And so it was implemented. A side effect of it became immediately visible: performance. Token check is THE most frequently called API in the OpenStack cloud. Right now, caching is used to reduce system stress. But in Rust, token decryption, without caching, is much faster than in python with caching (we expect a 2-3x faster order of magnitude).

As mentioned above, new vulnerability classes and attack vectors require us to disable caching, making such an improvement highly valuable. The new stack is fully async, uses parallel DB queries, and runs background tasks for periodic cleanups, synchronization, and similar. This allows us to achieve significant performance and maintainability improvements, eliminate the old ballast, and significantly reduce transitive dependency version conflicts.

Current measurements also indicate that we would likely be able to eliminate the need for caching in Keystone. For python, on the other hand, we do not have much to start with and would need to find and migrate to the new web framework, find maintained libraries with suitable licenses, and implement new features. We also need to rework the DB access patterns due to the number of bugs and the performance penalty caused by suboptimal DB access (many rows are selected multiple times during processing of a single request). With new authentication flows, we would need to do a wide security assessment of Keystone from scratch anyway.

With that work in mind and facing the prospect of having a major rework in python Keystone anyway (due to the abandoned complex dependencies), we, as the Keystone team, decided to try to focus the work on the Rust implementation, making it a proper evolution of the python Keystone and building a necessary community around it. This adoption is conditional on adding new contributors to support the project.

We believe this approach allows the best of both worlds: the trusted maturity of Keystone’s Python code-base, combined with the modern, high-safety, high-performance capabilities of Rust where they matter most.

Logically, this raises questions, so let's try to answer some of them:

1. Rust is not officially allowed in OpenStack. How are you dealing with that - The development is now going on on GitHub outside of OpenStack governance. - We would apply for OpenStack universe inclusion. - Once the TC policy allows Rust or the overall project inclusion requirement is reworked, the project will work toward becoming the official OpenStack project. - No one in the team is willing to separate from OpenStack, but with the current policies, we have no other way. Everybody and everything in the OpenStack ecosystem would benefit from that project being official to have first-class native support of the new functionality. - Multiple CSPs expressed support for such a move, desiring to see Rust in the ecosystem for better performance, maintainability, and resource utilization - A nice coincidence is a freshly sent announcement from Debian that introduces a strong rust dependency in the packaging tooling, stating something similar to: either you work to comply with this requirement in the next 6 months or sunset the port [5]. - Rust allows cross-integration with python with a high number of cryptography and typing libraries for python written directly in Rust. We are going to be able to reuse certain pieces of python in Rust during porting and Rust in python later in the future. - Keystone is a very sensitive component and in our eyes that fully justifies use of rust to provide a much stronger security on the language level.

2. Do you deprecate python Keystone. - No - v3 is there, and we continue maintaining it - We would unlikely accept major changes - Keystonemiddleware, keystoneauth, etc, are still there. They would likely be adopted for v4 support at some point in the future (in OpenStack services, we already communicate with “external” software, so there should be no problem). Those libraries are crucial for other services and thus are there to stay while necessary. - At some point, we will no longer be able to keep it alive when abandoned dependencies are physically gone. We are not going to invest our time in maintaining the abandoned software; we do not have enough time to do so.

3. What is the community around it? - As of now, there are provisional commitments from a few vendors and CSPs to collaborate on it. - Existing software is being transferred under the openstack-experimental organization to help build the community [3]. -The initial author (gtema) personally commits to continue working on that independently of others.

4. What are the /v3 features that would go away? - OIDC federation in its current form. v4 reimplements it from scratch natively without depending on external software (mod_auth_oidc, etc) since those make self-service impossible. - Saml2, this is a legacy protocol that is very unlikely to even get fixes for security vulnerabilities, and looks to be PQC unsafe. - Oauth1.

5. What new features are there in v4? - OIDC with self-service and CLI integration - Workload federation (Zuul, GitHub, Gitlab do not need to have secrets to access OpenStack anymore) - Service accounts (many use cases, one of the prominent ones: OpenStack on Kubernetes, GitHub workflow) - Passkeys (webauthn) - SCIM - Tight Open Policy Agent integration. We plan to consider the reintroduction of central policy storage in Keystone (not enforcing, but for central storage)[4]

We will use the next 2-3 months to gather interest in contributing to this approach and the new Rust-based code. There was a lot of interest in the new features listed right above during gtema’s sessions in OpenInfra Summit in October 2025. Still, it has to translate into active contributors for this effort to take off. If you have an interest in those new features, join us!

We hope this clarifies the situation. Feel free to reach out with other questions.

Thanks to everybody who participated in the discussions.

Regards, Keystone team

[1] https://github.com/gtema/keystone [2] https://www.youtube.com/watch?v=XOHYqE2HRw4&list=PLKqaoAnDyfgr91wN_12nwY3215... [3] https://github.com/openstack-experimental/keystone [4] https://www.youtube.com/watch?v=_B4Zsd8RG88&list=PLKqaoAnDyfgr91wN_12nwY3215... [5] https://lists.debian.org/debian-devel/2025/10/msg00285.html [6] https://access.redhat.com/solutions/2743391