[all][keystone] Keystone PTG summary
Hey guys Let us give you an update of the OpenStack Keystone future following the great PTG week we just had. Some news has already spread across the community, causing excitement and anxiety. It is not uncommon that changes cause fear. Some time ago, in the age of the Inquisition, people were even burned because of fear, misunderstanding, or questioning the rules. Let us clarify things. Nearly 13 years ago, the Identity v3 API was introduced. Since then, the world has changed dramatically. Libraries we use got abandoned, standards went legacy and replaced with new ones, ecosystem components, certain functionality is based upon are being sunsetted by OS vendors (mod_auth_mellon is archived upstream and gone in CentOS and RHEL [6]), new security vulnerabilities (classes of vulnerabilities) appear like mushrooms without us being able to address them. Cloud providers today face new challenges that were not present 10-15 years ago. Nowadays, a public cloud requires proper federation support with complete self-service. Users request passkey support, and security departments require additional hardening. These things cannot be covered in today's Keystone architecture and API version. As a result, work on an optional extension to Keystone [1] began some time ago, allowing CSPs to deploy it in parallel with the Python Keystone (of any v3 version) and access the new features. To simplify parallel deployment, it has been decided to start introducing v4 with the new features (since those ARE for cloud end-users). The work was done in Rust due to multiple factors: lack of necessary python libraries with trustable maintenance and/or licensing, necessity for enforcing security standards and practices using memory safe programming language, demand to address python ecosystem madness of past years (I guess majority would agree, that in the past few years we were mostly only chasing the python ecosystem breaking us on setuptools, eventlet, passlib, …). During the work, it became clear that implementing the mentioned features requires support of a given percentage of the existing functionality: users, groups, projects, roles, and tokens. And so it was implemented. A side effect of it became immediately visible: performance. Token check is THE most frequently called API in the OpenStack cloud. Right now, caching is used to reduce system stress. But in Rust, token decryption, without caching, is much faster than in python with caching (we expect a 2-3x faster order of magnitude). As mentioned above, new vulnerability classes and attack vectors require us to disable caching, making such an improvement highly valuable. The new stack is fully async, uses parallel DB queries, and runs background tasks for periodic cleanups, synchronization, and similar. This allows us to achieve significant performance and maintainability improvements, eliminate the old ballast, and significantly reduce transitive dependency version conflicts. Current measurements also indicate that we would likely be able to eliminate the need for caching in Keystone. For python, on the other hand, we do not have much to start with and would need to find and migrate to the new web framework, find maintained libraries with suitable licenses, and implement new features. We also need to rework the DB access patterns due to the number of bugs and the performance penalty caused by suboptimal DB access (many rows are selected multiple times during processing of a single request). With new authentication flows, we would need to do a wide security assessment of Keystone from scratch anyway. With that work in mind and facing the prospect of having a major rework in python Keystone anyway (due to the abandoned complex dependencies), we, as the Keystone team, decided to try to focus the work on the Rust implementation, making it a proper evolution of the python Keystone and building a necessary community around it. This adoption is conditional on adding new contributors to support the project. We believe this approach allows the best of both worlds: the trusted maturity of Keystone’s Python code-base, combined with the modern, high-safety, high-performance capabilities of Rust where they matter most. Logically, this raises questions, so let's try to answer some of them: 1. Rust is not officially allowed in OpenStack. How are you dealing with that - The development is now going on on GitHub outside of OpenStack governance. - We would apply for OpenStack universe inclusion. - Once the TC policy allows Rust or the overall project inclusion requirement is reworked, the project will work toward becoming the official OpenStack project. - No one in the team is willing to separate from OpenStack, but with the current policies, we have no other way. Everybody and everything in the OpenStack ecosystem would benefit from that project being official to have first-class native support of the new functionality. - Multiple CSPs expressed support for such a move, desiring to see Rust in the ecosystem for better performance, maintainability, and resource utilization - A nice coincidence is a freshly sent announcement from Debian that introduces a strong rust dependency in the packaging tooling, stating something similar to: either you work to comply with this requirement in the next 6 months or sunset the port [5]. - Rust allows cross-integration with python with a high number of cryptography and typing libraries for python written directly in Rust. We are going to be able to reuse certain pieces of python in Rust during porting and Rust in python later in the future. - Keystone is a very sensitive component and in our eyes that fully justifies use of rust to provide a much stronger security on the language level. 2. Do you deprecate python Keystone. - No - v3 is there, and we continue maintaining it - We would unlikely accept major changes - Keystonemiddleware, keystoneauth, etc, are still there. They would likely be adopted for v4 support at some point in the future (in OpenStack services, we already communicate with “external” software, so there should be no problem). Those libraries are crucial for other services and thus are there to stay while necessary. - At some point, we will no longer be able to keep it alive when abandoned dependencies are physically gone. We are not going to invest our time in maintaining the abandoned software; we do not have enough time to do so. 3. What is the community around it? - As of now, there are provisional commitments from a few vendors and CSPs to collaborate on it. - Existing software is being transferred under the openstack-experimental organization to help build the community [3]. -The initial author (gtema) personally commits to continue working on that independently of others. 4. What are the /v3 features that would go away? - OIDC federation in its current form. v4 reimplements it from scratch natively without depending on external software (mod_auth_oidc, etc) since those make self-service impossible. - Saml2, this is a legacy protocol that is very unlikely to even get fixes for security vulnerabilities, and looks to be PQC unsafe. - Oauth1. 5. What new features are there in v4? - OIDC with self-service and CLI integration - Workload federation (Zuul, GitHub, Gitlab do not need to have secrets to access OpenStack anymore) - Service accounts (many use cases, one of the prominent ones: OpenStack on Kubernetes, GitHub workflow) - Passkeys (webauthn) - SCIM - Tight Open Policy Agent integration. We plan to consider the reintroduction of central policy storage in Keystone (not enforcing, but for central storage)[4] We will use the next 2-3 months to gather interest in contributing to this approach and the new Rust-based code. There was a lot of interest in the new features listed right above during gtema’s sessions in OpenInfra Summit in October 2025. Still, it has to translate into active contributors for this effort to take off. If you have an interest in those new features, join us! We hope this clarifies the situation. Feel free to reach out with other questions. Thanks to everybody who participated in the discussions. Regards, Keystone team [1] https://github.com/gtema/keystone [2] https://www.youtube.com/watch?v=XOHYqE2HRw4&list=PLKqaoAnDyfgr91wN_12nwY321504Ctw1s&index=30 [3] https://github.com/openstack-experimental/keystone [4] https://www.youtube.com/watch?v=_B4Zsd8RG88&list=PLKqaoAnDyfgr91wN_12nwY321504Ctw1s&index=33 [5] https://lists.debian.org/debian-devel/2025/10/msg00285.html [6] https://access.redhat.com/solutions/2743391
On 2025-11-06 16:26:42 +0100 (+0100), Artem Goncharov wrote: [...]
A nice coincidence is a freshly sent announcement from Debian that introduces a strong rust dependency in the packaging tooling, stating something similar to: either you work to comply with this requirement in the next 6 months or sunset the port [5]. [...] [5] https://lists.debian.org/debian-devel/2025/10/msg00285.html [...]
I commend your bravery in linking to a discussion where developers are en masse condemning a unilateral decision by a single unchecked software maintainer to impose a Rust toolchain dependency on the entire community's collective work. I've been following it closely as a member of the Debian community since that thread started, and am not myself sure where it's going to end up. I'm personally wary of the choice to rewrite a key (no pun intended) component of OpenStack in a different programming language. We saw what happened with Swift Hummingbird in recent years. It's also true that Keystone was already rewritten once: the original implementation was in Java, but it was redone in Python in order to maintain consistency with other OpenStack services. Horizon and Skyline struggle to get contributors, to some extent because the bulk of OpenStack is in Python and to help in those projects you need a deep familiarity with other programming languages. Granted, Keystone is struggling to find maintainers as it is even though it's in the same language as the majority of OpenStack, but I worry this new decision will make the situation worse rather than better. You mention security as a selling point, but how many of the past vulnerabilities affecting Keystone have been because of the choice to write it in Python? (BTW, Python *is* a memory-safe language, it's not a type-safe language but that's a different matter altogether.) Keep in mind that our vulnerability management processes have been built over the past 15 years based on projects that produce source-only deliverables. What is your plan for the keystone-ng deliverables? Will they be pure Rust source, or compiled binary releases? Will you vendor/embed dependencies or incorporate them dynamically? What additional security responsibility is the OpenStack community assuming if it agrees to take on this project? -- Jeremy Stanley
Hi, Before reading my answer: please keep in mind that I'm very enthusiastic about such move. Never the less, I have still a lot of questions for you, but none should be taken negatively. On 11/6/25 4:26 PM, Artem Goncharov wrote:
- A nice coincidence is a freshly sent announcement from Debian that introduces a strong rust dependency in the packaging tooling, stating something similar to: either you work to comply with this requirement in the next 6 months or sunset the port [5].
Just like Jeremy wrote: this was *not* an announcement. It was a post written by a Canonical employee that has the mission (from Canonical) to rewrite stuff in Rust. He alone felt it was ok for him to impose a Rust rewrite of APT, even though some of the unofficial ports aren't ready yet with the Rust stack. While I think it may be a nice idea to rewrite things in Rust, I don't think APT has any performance issues, and never had any memory issue. So I'm unsure what is Julian is trying to achieve. I am very much frustrated that you seem to be having the same approach: announcing what you want to do, or have already done, instead of discussing the mater with the wider community. While I do like the idea of having a super-performing Keystone in Rust, I am sure there are better ways to move the OpenStack project in such direction. Why didn't you discuss this *before* writing the first line of code? I also remember the Swift project willing to start using Go. At the time, the Swift team was told that it would be ok if they spent a large amount of time making sure the infrastructure (ie: the CI, gating and all) would be ok for Go. Are you willing to spend such time with the infra team? Does the infra team has time for this? Also, you've blame some of the Python libs Keystone depends on. How can we make sure the same wont happen with the Rust things you'll depend on? Are all the Rust dependencies you depend on already in Debian, Ubuntu and Red Hat?
- Keystone is a very sensitive component and in our eyes that fully justifies use of rust to provide a much stronger security on the language level.
There's been a few security issues with Keystone in the past. None of them were due to the language choice. I'm not buying into this at all. Performance, yes...
2. Do you deprecate python Keystone. - No - v3 is there, and we continue maintaining it
Does the Rust implementation already has *all* of the current v3 support? I wouldn't like having to deploy *2* Keystone. I'm also very much worried about the move to a v3, and what kind of migration effort this will push on all projects. I do remember that the switch from Keystone v2 to v3 was *very* painful and took really too long. Cheers, Thomas Goirand (zigo)
Before reading my answer: please keep in mind that I'm very enthusiastic about such move. Never the less, I have still a lot of questions for you, but none should be taken negatively.
Thank you very much for the very explicit clarification, it really helps a
lot. Same is valid for my answers.
On 11/6/25 4:26 PM, Artem Goncharov wrote:
- A nice coincidence is a freshly sent announcement from Debian that introduces a strong rust dependency in the packaging tooling, stating something similar to: either you work to comply with this requirement in the next 6 months or sunset the port [5].
Just like Jeremy wrote: this was *not* an announcement. It was a post written by a Canonical employee that has the mission (from Canonical) to rewrite stuff in Rust. He alone felt it was ok for him to impose a Rust rewrite of APT, even though some of the unofficial ports aren't ready yet with the Rust stack.
While I think it may be a nice idea to rewrite things in Rust, I don't think APT has any performance issues, and never had any memory issue. So I'm unsure what is Julian is trying to achieve.
got it, wasn't the best decision to mention it. But it shows that any change (or no change) in a community leads to opposition, just like it was with systemd, pulseaudio, you name it.
I am very much frustrated that you seem to be having the same approach: announcing what you want to do, or have already done, instead of discussing the mater with the wider community.
It is not the case. I announced my plans and discussed this already during the last PTG and even earlier within the Keystone team.
While I do like the idea of having a super-performing Keystone in Rust, I am sure there are better ways to move the OpenStack project in such direction. Why didn't you discuss this *before* writing the first line of code?
When you go to the Shark Tank/Dragon's Den show you need to prove your idea is working to get support. Nobody supports just an idea nowadays. Not in my surroundings. But really, since I see this type of statement not the first time (and please do not take it personally), is it so normal nowadays to "criticize" people for writing code to prove the idea? I want to solve the problem and not spend years in discussions on what could be "acceptable" by others. Anyway, I am a supporter of the "every tool has its own specific use case" philosophy, there is nothing universal out there.
I also remember the Swift project willing to start using Go. At the time, the Swift team was told that it would be ok if they spent a large amount of time making sure the infrastructure (ie: the CI, gating and all) would be ok for Go.
Are you willing to spend such time with the infra team?
We are not there yet. It is not even clear whether OpenStack is ever going to be there, so this is just a pure speculation at that time. But to answer your question - yes, if necessary and accepted by the community I am ready to spend time adding new jobs for Rust. Go example feels weird, to be honest, work has been done and there is not a single project using it now. Perhaps this is worth a separate analysis. As mentioned initially - we are not requesting Rust to be accepted now, but announce a work on a Rust project and the community may benefit from it being considered as official. But it is not a must now.
Also, you've blame some of the Python libs Keystone depends on. How can we make sure the same wont happen with the Rust things you'll depend on? Are all the Rust dependencies you depend on already in Debian, Ubuntu and Red Hat?
Valid points, you can never be sure, in nothing actually, not even in native libraries of the language itself. We have seen it also in python (some examples are crypt, setuptools support). I have tried to do wider analysis and rely on the libraries with the most healthy activity and wide adoption, used by big players (tokio, axum, sea-orm). I am not that familiar with the packaging requirements of Debian for Rust project, so I can't answer your questions precisely. While I believe I understand your points pretty well, I do not think this should be something that a regular maintainer of a certain random project should focus on. Every distro is coming with its own rules and processes and if one would want to keep everybody happy no development should ever happen. I personally think Rust makes the situation a bit easier (while perhaps requiring a certain number of initial changes in the processes) due to the static linking - it does not matter which dependency versions are installed at the user's OS, what matters is which versions were used during compilation. It is a dependency hell during the build, but not during the runtime. But you may have different valid opinions on this. No offence or blame!
- Keystone is a very sensitive component and in our eyes that fully justifies use of rust to provide a much stronger security on the language level.
There's been a few security issues with Keystone in the past. None of them were due to the language choice. I'm not buying into this at all. Performance, yes...
There were few, in my opinion, not directly in Keystone, but in the keystonemiddleware, but I do not want to bike-shed on this. I experienced (and fixed myself) plenty of flow and logic bugs in multiple projects that would have been prevented by Rust. Correct - performance improvement is big. Btw, that is also a reason for implementing before (or in parallel to) talking - you can't prove the performance improvement initially, now I can do this.
2. Do you deprecate python Keystone. - No - v3 is there, and we continue maintaining it
Does the Rust implementation already has *all* of the current v3 support? I wouldn't like having to deploy *2* Keystone. I'm also very much worried about the move to a v3, and what kind of migration effort this will push on all projects. I do remember that the switch from Keystone v2 to v3 was *very* painful and took really too long.
We tried to be as precise as possible, but apparently still not enough (I assume your mistyped v2 and v3 instead of v3 and v4). Few thoughts from me personally not representing the team anymore: - it was explicitly stated that KeystoneNG is an "extension" and not the replacement. Folks actually convinced me to state it this way not to be crucified immediately. - we discussed potential evolution and how to implement migration strategy with implementing (integrating) parts of existing v3 into NG to simplify migration. But the message was and stays: nobody is rewriting Keystone in Rust for the sake of rewriting, or fun, we start with a new project that CAN be considered as a successor WHEN there is an interest. Regards, Artem
On 11/7/25 10:54 AM, Artem Goncharov wrote:
Also, you've blame some of the Python libs Keystone depends on. How can we make sure the same wont happen with the Rust things you'll depend on? Are all the Rust dependencies you depend on already in Debian, Ubuntu and Red Hat?
Valid points, you can never be sure, in nothing actually, not even in native libraries of the language itself. We have seen it also in python (some examples are crypt, setuptools support). I have tried to do wider analysis and rely on the libraries with the most healthy activity and wide adoption, used by big players (tokio, axum, sea-orm).
I am not that familiar with the packaging requirements of Debian for Rust project, so I can't answer your questions precisely. While I believe I understand your points pretty well, I do not think this should be something that a regular maintainer of a certain random project should focus on. Every distro is coming with its own rules and processes and if one would want to keep everybody happy no development should ever happen. I personally think Rust makes the situation a bit easier (while perhaps requiring a certain number of initial changes in the processes) due to the static linking - it does not matter which dependency versions are installed at the user's OS, what matters is which versions were used during compilation. It is a dependency hell during the build, but not during the runtime. But you may have different valid opinions on this. No offence or blame!
IMO, that's a very common miss-understanding on how distros works. I'll speak for Debian, but Ubuntu and Red Hat have the same requirements. In distros, even in Go or Rust, each and every library is packaged separately. We do not accept libs to be "vendored". Well, there are a few exceptions, but at least, that's the global effort. This means that each and every library that you're going to (build) depend on will have to be in the distribution, so that at build time, your project can link against them. The fact that all is, at the end, statically linked, changes nothing to what I wrote above. The only thing it changes, is for people wishing to run binaries built outside of the distribution. So no, compared to the Python situation, it changes nothing (at least, from a package maintainer perspective). The only annoying thing for me: I will have to learn Rust packaging. Though this is probably a good thing! :)
We tried to be as precise as possible, but apparently still not enough (I assume your mistyped v2 and v3 instead of v3 and v4). Few thoughts from me personally not representing the team anymore: - it was explicitly stated that KeystoneNG is an "extension" and not the replacement. Folks actually convinced me to state it this way not to be crucified immediately. - we discussed potential evolution and how to implement migration strategy with implementing (integrating) parts of existing v3 into NG to simplify migration. But the message was and stays: nobody is rewriting Keystone in Rust for the sake of rewriting, or fun, we start with a new project that CAN be considered as a successor WHEN there is an interest. Regards, Artem
I very much understand your point about not rewriting the v3 for the sake of it. It makes a lot of sense, and in fact, it's also probably avoiding some bugs where the implementation could have made small, undetected changes leading to hard to find bugs. So, if I understand you correctly: there WILL be a time where we'll be running both keystone and keystone-ng in a single deployment, where the Python one would handle v3, and Rust one handling v4. In such case, do you see them sharing the same port? Or maybe the Rust one forwarding the traffic to the Python version? How will this work? Will there be a new OpenStack endpoint for the Rust version? Last question: I did a git clone, and saw that all commits were from you. Is there others onboarding? Like Jeremy, I see Rust as possible blocker to get contributors (at last at first), and it's worrying if you're the only person pushing patches. Thanks for your work on this, and sharing your views, Cheers, Thomas Goirand (zigo) P.S: This project is one more new reason why I should learn Rust. :)
Hey
IMO, that's a very common miss-understanding on how distros works. I'll speak for Debian, but Ubuntu and Red Hat have the same requirements.
In distros, even in Go or Rust, each and every library is packaged separately. We do not accept libs to be "vendored". Well, there are a few exceptions, but at least, that's the global effort.
This means that each and every library that you're going to (build) depend on will have to be in the distribution, so that at build time, your project can link against them.
The fact that all is, at the end, statically linked, changes nothing to what I wrote above. The only thing it changes, is for people wishing to run binaries built outside of the distribution.
So no, compared to the Python situation, it changes nothing (at least, from a package maintainer perspective). The only annoying thing for me: I will have to learn Rust packaging. Though this is probably a good thing! :)
I was exactly trying to explain that our points would have valid differences. You are right that it does not change much on the packager side. But in my eyes it should change a lot on the user side. When 2 different projects depend on a single library (i.e. libssl) of different versions you should not have a bigger problem when updating this lib, since Rust project is compiled with hopefully a certain supported version. From what I read about the debian Rust packaging process is that it is expected to have crates in many different versions available as packages exactly to address that. You, as a user, will not have a broken runtime system just because dependency was updated for the other project. From my point of view the constraints issue in runtime is addressed with that. Am I wrong here? The big difference between python and rust in transitive dependencies version handling is that rust allows you to have multiple versions of a single transitive dependency during the compilation. This is, however, exactly what makes my life in the python world a nightmare when updating transitive dep in constraints for the need of one project breaks some other dependency.
We tried to be as precise as possible, but apparently still not enough (I assume your mistyped v2 and v3 instead of v3 and v4). Few thoughts from me personally not representing the team anymore: - it was explicitly stated that KeystoneNG is an "extension" and not the replacement. Folks actually convinced me to state it this way not to be crucified immediately. - we discussed potential evolution and how to implement migration strategy with implementing (integrating) parts of existing v3 into NG to simplify migration. But the message was and stays: nobody is rewriting Keystone in Rust for the sake of rewriting, or fun, we start with a new project that CAN be considered as a successor WHEN there is an interest. Regards, Artem
I very much understand your point about not rewriting the v3 for the sake of it. It makes a lot of sense, and in fact, it's also probably avoiding some bugs where the implementation could have made small, undetected changes leading to hard to find bugs.
So, if I understand you correctly: there WILL be a time where we'll be running both keystone and keystone-ng in a single deployment, where the Python one would handle v3, and Rust one handling v4. In such case, do you see them sharing the same port? Or maybe the Rust one forwarding the traffic to the Python version? How will this work? Will there be a new OpenStack endpoint for the Rust version?
Valid points. Actually pretty much everything you have mentioned is valid on its own and makes sense. It can be implemented either way. It depends on the technical feedback and ideas we can gather from vendors and CSPs. It was removed from the summary email (not to give food for concerns), you can read details in the [PTG etherpad]( https://etherpad.opendev.org/p/oct2025-ptg-keystone) for options and potential strategy. You could route v3 to the left and v4 to the right, you can have only rust doing the proxying and internally forward requests to v3 (addressing the wsgi issues we have). Regarding the endpoint: current implementation that I was using during the demo in talk on summit is that version discovery is served by the Rust and shows v3 as stable and v4 as experimental (pointing at different url). Version discovery standard is there to address it without introducing new endpoint/service We explicitly gather feedback and ideas.
Last question: I did a git clone, and saw that all commits were from you. Is there others onboarding? Like Jeremy, I see Rust as possible blocker to get contributors (at last at first), and it's worrying if you're the only person pushing patches.
The reason for the message was to raise awareness. As also mentioned - there were provisional commitments from vendors and csps to contribute and we now wait to see how it evolves. We now also have problems with contributors in python, ...
Thanks for your work on this, and sharing your views, Cheers,
Thomas Goirand (zigo)
P.S: This project is one more new reason why I should learn Rust. :)
:) Using it on a daily basis I see so many problems we have in python that are conceptually not possible in rust. It is not an easy language, and from experience I can say here with absolute confidence - without knowing this language it is impossible to judge what is possible and what not and to estimate which kinds of problems would have been prevented. Regards, Artem
On 2025-11-07 15:44:26 +0100 (+0100), Artem Goncharov wrote: [...]
When 2 different projects depend on a single library (i.e. libssl) of different versions you should not have a bigger problem when updating this lib, since Rust project is compiled with hopefully a certain supported version. From what I read about the debian Rust packaging process is that it is expected to have crates in many different versions available as packages exactly to address that. You, as a user, will not have a broken runtime system just because dependency was updated for the other project. From my point of view the constraints issue in runtime is addressed with that. Am I wrong here? [...]
The challenge they have with that is security support. If 10 different packaged applications all require different versions of the same dependency and there is a security vulnerability impacting all versions of that dependency, the security team needs to backport 10 different versions of the fix in one release of the distribution alone (then multiply that by the number of distribution releases they may be supporting at the same time). As a result, there is significant pressure within distributions to prefer carrying as few versions of a library in each release as possible, ideally just one. This problem is not unique to the Rust ecosystem though, and is one of the main reasons for distributions to simply not package otherwise popular applications written in Go or JavaScript too. What you view as a run-time "dependency hell" is often the preferable tradeoff to build-time "security hell." -- Jeremy Stanley
Hello, I agree with a lot that Thomas brings up here. I’m not negative or against a rewrite per se, what I’m most afraid of is community fragmentation and a 10 year migration effort similar to for example Python2 to Python3 and placing the maintenance burden on other projects to support v3 and keystone-ng (v4?) for the next 10 years or even more. I think it was inevitable that this kind of discussions and proposal for using “new" modern technology would start sooner rather than later and I think it’s only natural, evolution if you will, and if there are people willing to do the work we should overall embrace that moving forward. The primary win here seems to be performance, which is great, hopefully less energy consumed world-wide for all the gazillion of Keystone API requests – but it’s critical that we look at the history here, learn from it and not repeat some of the horrific migration stories we’ve had in the past. It’s not a silver bullet, it doesn’t really solve anything related to third-party dependencies, we need to follow the same process there and I suspect vetting dependencies in the Rust ecosystem is much harder due to more changes but that’s just a guess. Adding another language is hard; it’s not only about the code itself, we’ve had this discussion before around Go which didn’t solidify into anything in the ecosystem, in my view we’ve already started to see small scale fragmentation of projects moving in and out under the OpenStack umbrella, which in itself is one of the (although smaller) reasons I’m for more broader change, we don’t want stagnation we want innovation. I’m actually very positive moving the effort under the OpenStack umbrella but what will be done to be successful attracting Rust developers into the ecosystem and maintain all the tooling and infrastructure required to build and release it? I commend your entusiasm and willingness to bring this up with the community, I think we all want progress but progress without a plan is just a dream. /Tobias On 6 Nov 2025, at 16:26, Artem Goncharov <artem.goncharov@gmail.com> wrote: Hey guys Let us give you an update of the OpenStack Keystone future following the great PTG week we just had. Some news has already spread across the community, causing excitement and anxiety. It is not uncommon that changes cause fear. Some time ago, in the age of the Inquisition, people were even burned because of fear, misunderstanding, or questioning the rules. Let us clarify things. Nearly 13 years ago, the Identity v3 API was introduced. Since then, the world has changed dramatically. Libraries we use got abandoned, standards went legacy and replaced with new ones, ecosystem components, certain functionality is based upon are being sunsetted by OS vendors (mod_auth_mellon is archived upstream and gone in CentOS and RHEL [6]), new security vulnerabilities (classes of vulnerabilities) appear like mushrooms without us being able to address them. Cloud providers today face new challenges that were not present 10-15 years ago. Nowadays, a public cloud requires proper federation support with complete self-service. Users request passkey support, and security departments require additional hardening. These things cannot be covered in today's Keystone architecture and API version. As a result, work on an optional extension to Keystone [1] began some time ago, allowing CSPs to deploy it in parallel with the Python Keystone (of any v3 version) and access the new features. To simplify parallel deployment, it has been decided to start introducing v4 with the new features (since those ARE for cloud end-users). The work was done in Rust due to multiple factors: lack of necessary python libraries with trustable maintenance and/or licensing, necessity for enforcing security standards and practices using memory safe programming language, demand to address python ecosystem madness of past years (I guess majority would agree, that in the past few years we were mostly only chasing the python ecosystem breaking us on setuptools, eventlet, passlib, …). During the work, it became clear that implementing the mentioned features requires support of a given percentage of the existing functionality: users, groups, projects, roles, and tokens. And so it was implemented. A side effect of it became immediately visible: performance. Token check is THE most frequently called API in the OpenStack cloud. Right now, caching is used to reduce system stress. But in Rust, token decryption, without caching, is much faster than in python with caching (we expect a 2-3x faster order of magnitude). As mentioned above, new vulnerability classes and attack vectors require us to disable caching, making such an improvement highly valuable. The new stack is fully async, uses parallel DB queries, and runs background tasks for periodic cleanups, synchronization, and similar. This allows us to achieve significant performance and maintainability improvements, eliminate the old ballast, and significantly reduce transitive dependency version conflicts. Current measurements also indicate that we would likely be able to eliminate the need for caching in Keystone. For python, on the other hand, we do not have much to start with and would need to find and migrate to the new web framework, find maintained libraries with suitable licenses, and implement new features. We also need to rework the DB access patterns due to the number of bugs and the performance penalty caused by suboptimal DB access (many rows are selected multiple times during processing of a single request). With new authentication flows, we would need to do a wide security assessment of Keystone from scratch anyway. With that work in mind and facing the prospect of having a major rework in python Keystone anyway (due to the abandoned complex dependencies), we, as the Keystone team, decided to try to focus the work on the Rust implementation, making it a proper evolution of the python Keystone and building a necessary community around it. This adoption is conditional on adding new contributors to support the project. We believe this approach allows the best of both worlds: the trusted maturity of Keystone’s Python code-base, combined with the modern, high-safety, high-performance capabilities of Rust where they matter most. Logically, this raises questions, so let's try to answer some of them: 1. Rust is not officially allowed in OpenStack. How are you dealing with that - The development is now going on on GitHub outside of OpenStack governance. - We would apply for OpenStack universe inclusion. - Once the TC policy allows Rust or the overall project inclusion requirement is reworked, the project will work toward becoming the official OpenStack project. - No one in the team is willing to separate from OpenStack, but with the current policies, we have no other way. Everybody and everything in the OpenStack ecosystem would benefit from that project being official to have first-class native support of the new functionality. - Multiple CSPs expressed support for such a move, desiring to see Rust in the ecosystem for better performance, maintainability, and resource utilization - A nice coincidence is a freshly sent announcement from Debian that introduces a strong rust dependency in the packaging tooling, stating something similar to: either you work to comply with this requirement in the next 6 months or sunset the port [5]. - Rust allows cross-integration with python with a high number of cryptography and typing libraries for python written directly in Rust. We are going to be able to reuse certain pieces of python in Rust during porting and Rust in python later in the future. - Keystone is a very sensitive component and in our eyes that fully justifies use of rust to provide a much stronger security on the language level. 2. Do you deprecate python Keystone. - No - v3 is there, and we continue maintaining it - We would unlikely accept major changes - Keystonemiddleware, keystoneauth, etc, are still there. They would likely be adopted for v4 support at some point in the future (in OpenStack services, we already communicate with “external” software, so there should be no problem). Those libraries are crucial for other services and thus are there to stay while necessary. - At some point, we will no longer be able to keep it alive when abandoned dependencies are physically gone. We are not going to invest our time in maintaining the abandoned software; we do not have enough time to do so. 3. What is the community around it? - As of now, there are provisional commitments from a few vendors and CSPs to collaborate on it. - Existing software is being transferred under the openstack-experimental organization to help build the community [3]. -The initial author (gtema) personally commits to continue working on that independently of others. 4. What are the /v3 features that would go away? - OIDC federation in its current form. v4 reimplements it from scratch natively without depending on external software (mod_auth_oidc, etc) since those make self-service impossible. - Saml2, this is a legacy protocol that is very unlikely to even get fixes for security vulnerabilities, and looks to be PQC unsafe. - Oauth1. 5. What new features are there in v4? - OIDC with self-service and CLI integration - Workload federation (Zuul, GitHub, Gitlab do not need to have secrets to access OpenStack anymore) - Service accounts (many use cases, one of the prominent ones: OpenStack on Kubernetes, GitHub workflow) - Passkeys (webauthn) - SCIM - Tight Open Policy Agent integration. We plan to consider the reintroduction of central policy storage in Keystone (not enforcing, but for central storage)[4] We will use the next 2-3 months to gather interest in contributing to this approach and the new Rust-based code. There was a lot of interest in the new features listed right above during gtema’s sessions in OpenInfra Summit in October 2025. Still, it has to translate into active contributors for this effort to take off. If you have an interest in those new features, join us! We hope this clarifies the situation. Feel free to reach out with other questions. Thanks to everybody who participated in the discussions. Regards, Keystone team [1] https://github.com/gtema/keystone [2] https://www.youtube.com/watch?v=XOHYqE2HRw4&list=PLKqaoAnDyfgr91wN_12nwY321504Ctw1s&index=30 [3] https://github.com/openstack-experimental/keystone [4] https://www.youtube.com/watch?v=_B4Zsd8RG88&list=PLKqaoAnDyfgr91wN_12nwY321504Ctw1s&index=33 [5] https://lists.debian.org/debian-devel/2025/10/msg00285.html [6] https://access.redhat.com/solutions/2743391
Thanks a lot, Tobias I’m not negative or against a rewrite per se, what I’m most afraid of is
community fragmentation and a 10 year migration effort similar to for example Python2 to Python3 and placing the maintenance burden on other projects to support v3 and keystone-ng (v4?) for the next 10 years or even more.
I guess one of the points of misunderstanding is that it is not only about Rust per se, it is about the fact that due to the changes in the modern standards we can't cope with it within v3 without breaking it dramatically. So one of the things - we have no choice but to start work on v4. With my SDK and client tooling hat I can say definitely - microversions make it only worse. But back to the point - it does not make a difference whether v4 is implemented in Rust or python from the normal api version lifecycle point of view. It gives us, however, the possibility to do foundational changes while doing that.
I think it was inevitable that this kind of discussions and proposal for using “new" modern technology would start sooner rather than later and I think it’s only natural, evolution if you will, and if there are people willing to do the work we should overall embrace that moving forward.
The primary win here seems to be performance, which is great, hopefully less energy consumed world-wide for all the gazillion of Keystone API requests – but it’s critical that we look at the history here, learn from it and not repeat some of the horrific migration stories we’ve had in the past. It’s not a silver bullet, it doesn’t really solve anything related to third-party dependencies, we need to follow the same process there and I suspect vetting dependencies in the Rust ecosystem is much harder due to more changes but that’s just a guess.
Adding another language is hard; it’s not only about the code itself, we’ve had this discussion before around Go which didn’t solidify into anything in the ecosystem, in my view we’ve already started to see small scale fragmentation of projects moving in and out under the OpenStack umbrella, which in itself is one of the (although smaller) reasons I’m for more broader change, we don’t want stagnation we want innovation.
I’m actually very positive moving the effort under the OpenStack umbrella but what will be done to be successful attracting Rust developers into the ecosystem and maintain all the tooling and infrastructure required to build and release it?
I see your point. Right now I see a traction for people working at major vendors who are getting those skills. On the other side - when currently the project is maintained by 1-2 people it is not really that different.
I commend your entusiasm and willingness to bring this up with the community, I think we all want progress but progress without a plan is just a dream.
It was mentioned in the message, currently we make an announcement and call people to participate. Perhaps ones who see the point of being interested in getting Rust experience would be interested. Or CSPs who need the features come on board and start supporting it more. In the meanwhile I continue development as I was doing before. The plan exists, but it assumes the idea would have followers. Without them either nobody wants that or nobody needs that. Then the plan stays a dream. Right.
Hi, Thank you very much for sharing this detailed update. That was a very interesting and inspiring read. I’m probably a bit biased since I’m a big fan of Rust myself, but I truly appreciate how much it can help us build safer and more performant software. The direction you’re taking feels ambitious and forward-looking. Unfortunately, I couldn’t attend your session at the OpenInfra Summit in October, but I’ll make sure to watch the replay soon. From what I read here, the work and vision behind this initiative are really impressive, and I’d like to express my full support for the effort. I also realize that introducing a new language like Rust into the OpenStack ecosystem won’t be easy and will likely take time, both from a technical and a governance perspective. But it’s a fascinating step forward, and I think it’s also an interesting opportunity to see whether this approach can attract new contributors or a broader group of people willing to help sustain the project. I don’t know what the Foundation or the Technical Committee’s current position is on Rust within OpenStack, but personally, I believe this kind of project should be encouraged under an *experimental* or *incubation* status. It’s exactly the kind of innovation that can keep the ecosystem vibrant and evolving. Congrats again for pushing such an ambitious and technically exciting direction. Best regards, René Le jeu. 6 nov. 2025 à 16:27, Artem Goncharov <artem.goncharov@gmail.com> a écrit :
Hey guys
Let us give you an update of the OpenStack Keystone future following the great PTG week we just had.
Some news has already spread across the community, causing excitement and anxiety. It is not uncommon that changes cause fear. Some time ago, in the age of the Inquisition, people were even burned because of fear, misunderstanding, or questioning the rules. Let us clarify things.
Nearly 13 years ago, the Identity v3 API was introduced. Since then, the world has changed dramatically. Libraries we use got abandoned, standards went legacy and replaced with new ones, ecosystem components, certain functionality is based upon are being sunsetted by OS vendors (mod_auth_mellon is archived upstream and gone in CentOS and RHEL [6]), new security vulnerabilities (classes of vulnerabilities) appear like mushrooms without us being able to address them.
Cloud providers today face new challenges that were not present 10-15 years ago. Nowadays, a public cloud requires proper federation support with complete self-service. Users request passkey support, and security departments require additional hardening. These things cannot be covered in today's Keystone architecture and API version. As a result, work on an optional extension to Keystone [1] began some time ago, allowing CSPs to deploy it in parallel with the Python Keystone (of any v3 version) and access the new features.
To simplify parallel deployment, it has been decided to start introducing v4 with the new features (since those ARE for cloud end-users). The work was done in Rust due to multiple factors: lack of necessary python libraries with trustable maintenance and/or licensing, necessity for enforcing security standards and practices using memory safe programming language, demand to address python ecosystem madness of past years (I guess majority would agree, that in the past few years we were mostly only chasing the python ecosystem breaking us on setuptools, eventlet, passlib, …).
During the work, it became clear that implementing the mentioned features requires support of a given percentage of the existing functionality: users, groups, projects, roles, and tokens. And so it was implemented. A side effect of it became immediately visible: performance. Token check is THE most frequently called API in the OpenStack cloud. Right now, caching is used to reduce system stress. But in Rust, token decryption, without caching, is much faster than in python with caching (we expect a 2-3x faster order of magnitude).
As mentioned above, new vulnerability classes and attack vectors require us to disable caching, making such an improvement highly valuable. The new stack is fully async, uses parallel DB queries, and runs background tasks for periodic cleanups, synchronization, and similar. This allows us to achieve significant performance and maintainability improvements, eliminate the old ballast, and significantly reduce transitive dependency version conflicts.
Current measurements also indicate that we would likely be able to eliminate the need for caching in Keystone. For python, on the other hand, we do not have much to start with and would need to find and migrate to the new web framework, find maintained libraries with suitable licenses, and implement new features. We also need to rework the DB access patterns due to the number of bugs and the performance penalty caused by suboptimal DB access (many rows are selected multiple times during processing of a single request). With new authentication flows, we would need to do a wide security assessment of Keystone from scratch anyway.
With that work in mind and facing the prospect of having a major rework in python Keystone anyway (due to the abandoned complex dependencies), we, as the Keystone team, decided to try to focus the work on the Rust implementation, making it a proper evolution of the python Keystone and building a necessary community around it. This adoption is conditional on adding new contributors to support the project.
We believe this approach allows the best of both worlds: the trusted maturity of Keystone’s Python code-base, combined with the modern, high-safety, high-performance capabilities of Rust where they matter most.
Logically, this raises questions, so let's try to answer some of them:
1. Rust is not officially allowed in OpenStack. How are you dealing with that - The development is now going on on GitHub outside of OpenStack governance. - We would apply for OpenStack universe inclusion. - Once the TC policy allows Rust or the overall project inclusion requirement is reworked, the project will work toward becoming the official OpenStack project. - No one in the team is willing to separate from OpenStack, but with the current policies, we have no other way. Everybody and everything in the OpenStack ecosystem would benefit from that project being official to have first-class native support of the new functionality. - Multiple CSPs expressed support for such a move, desiring to see Rust in the ecosystem for better performance, maintainability, and resource utilization - A nice coincidence is a freshly sent announcement from Debian that introduces a strong rust dependency in the packaging tooling, stating something similar to: either you work to comply with this requirement in the next 6 months or sunset the port [5]. - Rust allows cross-integration with python with a high number of cryptography and typing libraries for python written directly in Rust. We are going to be able to reuse certain pieces of python in Rust during porting and Rust in python later in the future. - Keystone is a very sensitive component and in our eyes that fully justifies use of rust to provide a much stronger security on the language level.
2. Do you deprecate python Keystone. - No - v3 is there, and we continue maintaining it - We would unlikely accept major changes - Keystonemiddleware, keystoneauth, etc, are still there. They would likely be adopted for v4 support at some point in the future (in OpenStack services, we already communicate with “external” software, so there should be no problem). Those libraries are crucial for other services and thus are there to stay while necessary. - At some point, we will no longer be able to keep it alive when abandoned dependencies are physically gone. We are not going to invest our time in maintaining the abandoned software; we do not have enough time to do so.
3. What is the community around it? - As of now, there are provisional commitments from a few vendors and CSPs to collaborate on it. - Existing software is being transferred under the openstack-experimental organization to help build the community [3]. -The initial author (gtema) personally commits to continue working on that independently of others.
4. What are the /v3 features that would go away? - OIDC federation in its current form. v4 reimplements it from scratch natively without depending on external software (mod_auth_oidc, etc) since those make self-service impossible. - Saml2, this is a legacy protocol that is very unlikely to even get fixes for security vulnerabilities, and looks to be PQC unsafe. - Oauth1.
5. What new features are there in v4? - OIDC with self-service and CLI integration - Workload federation (Zuul, GitHub, Gitlab do not need to have secrets to access OpenStack anymore) - Service accounts (many use cases, one of the prominent ones: OpenStack on Kubernetes, GitHub workflow) - Passkeys (webauthn) - SCIM - Tight Open Policy Agent integration. We plan to consider the reintroduction of central policy storage in Keystone (not enforcing, but for central storage)[4]
We will use the next 2-3 months to gather interest in contributing to this approach and the new Rust-based code. There was a lot of interest in the new features listed right above during gtema’s sessions in OpenInfra Summit in October 2025. Still, it has to translate into active contributors for this effort to take off. If you have an interest in those new features, join us!
We hope this clarifies the situation. Feel free to reach out with other questions.
Thanks to everybody who participated in the discussions.
Regards, Keystone team
[1] https://github.com/gtema/keystone [2] https://www.youtube.com/watch?v=XOHYqE2HRw4&list=PLKqaoAnDyfgr91wN_12nwY321504Ctw1s&index=30 [3] https://github.com/openstack-experimental/keystone [4] https://www.youtube.com/watch?v=_B4Zsd8RG88&list=PLKqaoAnDyfgr91wN_12nwY321504Ctw1s&index=33 [5] https://lists.debian.org/debian-devel/2025/10/msg00285.html [6] https://access.redhat.com/solutions/2743391
Thank you very much, Rene, for the words. Artem On Fri, 7 Nov 2025 at 18:17, René Ribaud <rene.ribaud@gmail.com> wrote:
Hi,
Thank you very much for sharing this detailed update. That was a very interesting and inspiring read.
I’m probably a bit biased since I’m a big fan of Rust myself, but I truly appreciate how much it can help us build safer and more performant software. The direction you’re taking feels ambitious and forward-looking.
Unfortunately, I couldn’t attend your session at the OpenInfra Summit in October, but I’ll make sure to watch the replay soon. From what I read here, the work and vision behind this initiative are really impressive, and I’d like to express my full support for the effort.
I also realize that introducing a new language like Rust into the OpenStack ecosystem won’t be easy and will likely take time, both from a technical and a governance perspective. But it’s a fascinating step forward, and I think it’s also an interesting opportunity to see whether this approach can attract new contributors or a broader group of people willing to help sustain the project.
I don’t know what the Foundation or the Technical Committee’s current position is on Rust within OpenStack, but personally, I believe this kind of project should be encouraged under an *experimental* or *incubation* status. It’s exactly the kind of innovation that can keep the ecosystem vibrant and evolving.
Congrats again for pushing such an ambitious and technically exciting direction.
Best regards, René
Le jeu. 6 nov. 2025 à 16:27, Artem Goncharov <artem.goncharov@gmail.com> a écrit :
Hey guys
Let us give you an update of the OpenStack Keystone future following the great PTG week we just had.
Some news has already spread across the community, causing excitement and anxiety. It is not uncommon that changes cause fear. Some time ago, in the age of the Inquisition, people were even burned because of fear, misunderstanding, or questioning the rules. Let us clarify things.
Nearly 13 years ago, the Identity v3 API was introduced. Since then, the world has changed dramatically. Libraries we use got abandoned, standards went legacy and replaced with new ones, ecosystem components, certain functionality is based upon are being sunsetted by OS vendors (mod_auth_mellon is archived upstream and gone in CentOS and RHEL [6]), new security vulnerabilities (classes of vulnerabilities) appear like mushrooms without us being able to address them.
Cloud providers today face new challenges that were not present 10-15 years ago. Nowadays, a public cloud requires proper federation support with complete self-service. Users request passkey support, and security departments require additional hardening. These things cannot be covered in today's Keystone architecture and API version. As a result, work on an optional extension to Keystone [1] began some time ago, allowing CSPs to deploy it in parallel with the Python Keystone (of any v3 version) and access the new features.
To simplify parallel deployment, it has been decided to start introducing v4 with the new features (since those ARE for cloud end-users). The work was done in Rust due to multiple factors: lack of necessary python libraries with trustable maintenance and/or licensing, necessity for enforcing security standards and practices using memory safe programming language, demand to address python ecosystem madness of past years (I guess majority would agree, that in the past few years we were mostly only chasing the python ecosystem breaking us on setuptools, eventlet, passlib, …).
During the work, it became clear that implementing the mentioned features requires support of a given percentage of the existing functionality: users, groups, projects, roles, and tokens. And so it was implemented. A side effect of it became immediately visible: performance. Token check is THE most frequently called API in the OpenStack cloud. Right now, caching is used to reduce system stress. But in Rust, token decryption, without caching, is much faster than in python with caching (we expect a 2-3x faster order of magnitude).
As mentioned above, new vulnerability classes and attack vectors require us to disable caching, making such an improvement highly valuable. The new stack is fully async, uses parallel DB queries, and runs background tasks for periodic cleanups, synchronization, and similar. This allows us to achieve significant performance and maintainability improvements, eliminate the old ballast, and significantly reduce transitive dependency version conflicts.
Current measurements also indicate that we would likely be able to eliminate the need for caching in Keystone. For python, on the other hand, we do not have much to start with and would need to find and migrate to the new web framework, find maintained libraries with suitable licenses, and implement new features. We also need to rework the DB access patterns due to the number of bugs and the performance penalty caused by suboptimal DB access (many rows are selected multiple times during processing of a single request). With new authentication flows, we would need to do a wide security assessment of Keystone from scratch anyway.
With that work in mind and facing the prospect of having a major rework in python Keystone anyway (due to the abandoned complex dependencies), we, as the Keystone team, decided to try to focus the work on the Rust implementation, making it a proper evolution of the python Keystone and building a necessary community around it. This adoption is conditional on adding new contributors to support the project.
We believe this approach allows the best of both worlds: the trusted maturity of Keystone’s Python code-base, combined with the modern, high-safety, high-performance capabilities of Rust where they matter most.
Logically, this raises questions, so let's try to answer some of them:
1. Rust is not officially allowed in OpenStack. How are you dealing with that - The development is now going on on GitHub outside of OpenStack governance. - We would apply for OpenStack universe inclusion. - Once the TC policy allows Rust or the overall project inclusion requirement is reworked, the project will work toward becoming the official OpenStack project. - No one in the team is willing to separate from OpenStack, but with the current policies, we have no other way. Everybody and everything in the OpenStack ecosystem would benefit from that project being official to have first-class native support of the new functionality. - Multiple CSPs expressed support for such a move, desiring to see Rust in the ecosystem for better performance, maintainability, and resource utilization - A nice coincidence is a freshly sent announcement from Debian that introduces a strong rust dependency in the packaging tooling, stating something similar to: either you work to comply with this requirement in the next 6 months or sunset the port [5]. - Rust allows cross-integration with python with a high number of cryptography and typing libraries for python written directly in Rust. We are going to be able to reuse certain pieces of python in Rust during porting and Rust in python later in the future. - Keystone is a very sensitive component and in our eyes that fully justifies use of rust to provide a much stronger security on the language level.
2. Do you deprecate python Keystone. - No - v3 is there, and we continue maintaining it - We would unlikely accept major changes - Keystonemiddleware, keystoneauth, etc, are still there. They would likely be adopted for v4 support at some point in the future (in OpenStack services, we already communicate with “external” software, so there should be no problem). Those libraries are crucial for other services and thus are there to stay while necessary. - At some point, we will no longer be able to keep it alive when abandoned dependencies are physically gone. We are not going to invest our time in maintaining the abandoned software; we do not have enough time to do so.
3. What is the community around it? - As of now, there are provisional commitments from a few vendors and CSPs to collaborate on it. - Existing software is being transferred under the openstack-experimental organization to help build the community [3]. -The initial author (gtema) personally commits to continue working on that independently of others.
4. What are the /v3 features that would go away? - OIDC federation in its current form. v4 reimplements it from scratch natively without depending on external software (mod_auth_oidc, etc) since those make self-service impossible. - Saml2, this is a legacy protocol that is very unlikely to even get fixes for security vulnerabilities, and looks to be PQC unsafe. - Oauth1.
5. What new features are there in v4? - OIDC with self-service and CLI integration - Workload federation (Zuul, GitHub, Gitlab do not need to have secrets to access OpenStack anymore) - Service accounts (many use cases, one of the prominent ones: OpenStack on Kubernetes, GitHub workflow) - Passkeys (webauthn) - SCIM - Tight Open Policy Agent integration. We plan to consider the reintroduction of central policy storage in Keystone (not enforcing, but for central storage)[4]
We will use the next 2-3 months to gather interest in contributing to this approach and the new Rust-based code. There was a lot of interest in the new features listed right above during gtema’s sessions in OpenInfra Summit in October 2025. Still, it has to translate into active contributors for this effort to take off. If you have an interest in those new features, join us!
We hope this clarifies the situation. Feel free to reach out with other questions.
Thanks to everybody who participated in the discussions.
Regards, Keystone team
[1] https://github.com/gtema/keystone [2] https://www.youtube.com/watch?v=XOHYqE2HRw4&list=PLKqaoAnDyfgr91wN_12nwY321504Ctw1s&index=30 [3] https://github.com/openstack-experimental/keystone [4] https://www.youtube.com/watch?v=_B4Zsd8RG88&list=PLKqaoAnDyfgr91wN_12nwY321504Ctw1s&index=33 [5] https://lists.debian.org/debian-devel/2025/10/msg00285.html [6] https://access.redhat.com/solutions/2743391
On 2025-11-07 18:17:14 +0100 (+0100), René Ribaud wrote: [...]
I don’t know what the Foundation or the Technical Committee’s current position is on Rust within OpenStack [...]
The OpenInfra Foundation expresses no opinion in its bylaws, and while https://board.openinfra.org/en/ProjectGuidelines has been used for determining project suitability it only concerns itself with licenses and practices not choice of programming language. The OpenStack Technical Committee does have published guidance about choice of programming language... https://governance.openstack.org/tc/resolutions/20150901-programming-languag... ...as well as how to qualify new languages: https://governance.openstack.org/tc/reference/new-language-requirements.html An example is this resolution from when Go was added: https://governance.openstack.org/tc/resolutions/20170329-golang-use-case.htm... Introducing Rust would in theory involve a similar resolution, and "just" needs discussion and a vote among the TC members. -- Jeremy Stanley
The work was done in Rust due to multiple factors: lack of necessary
Hi all, As a developer and the single maintainer of one of the OpenStack projects, I often feel bad when project owners propose something, try to move their service forward, and then hit a wall of community resistance. I believe maintainers/core developers should have the space to implement their vision. Without that, it is easy to lose interest, and then we end up retiring yet another project. So, from a developer's point of view: if the Keystone team believes Rust is the right direction, they should be able to move forward with it. I do not contribute to Keystone, so my personal opinion about Rust should not matter here :) But things change when I put on an Operator's hat. As an operator, I have a different view, and I feel bad that I'm leaning toward skepticism. I share Jeremy's concerns about the Swift and Skyline cases. I would also bring up the Glare project, which spun out of Glance with support from one or two original core developers (as far as I remember). We all know where Glance is today (it is here) and where Glare ended up (some people in this thread may not have even heard of it). We can also remember Nova V3 story. My main concern as an operator is simple: while Keystone may have only a couple of active core developers today, a very large number of people have contributed to the current Python codebase over the years. If something breaks (not a new feature), it can be relatively quick fixed because many people can still dig into the existing code in an emergency. A smaller concern: I like the current pluggable architecture. We use a small plugin to integrate Keystone with our internal environment, no patching required. I understand Keystone v4 is still early and may introduce a new plugin mechanism, but that uncertainty still raises some worry for me :) python libraries with trustable maintenance and/or licensing, necessity for enforcing security standards and practices using memory safe programming language Just curious, did you consider creating a Rust "binding" layer for Keystone instead of a full rewrite? That might address the Python dependency and maintenance issues you mentioned.
A side effect of it became immediately visible: performance. This allows us to achieve significant performance and maintainability improvements, eliminate the old ballast
From an operator’s perspective, Keystone performance has never been an issue for us. It just works. Thank you, folks! So if performance improvements are a side effect, great - but it has never been high on my list of concerns.
The new stack is fully async, uses parallel DB queries, and runs background tasks for periodic cleanups, synchronization, and similar.
This reminds me of the early async wave in Python. There were claims like “we built the fastest Postgres driver in the world,” but in practice, it did not noticeably change anything. I mean the problem quite often is not in the technological stack, but in the way people write software, which covers the dependencies and logical flow as well. Anyway, I would like to thank the Keystone team for the work you are doing and for opening this conversation. And to be fair, not all my colleagues share my skepticism - some of them can't wait for the new Keystone to be accepted so we can finally deploy it in our cloud :) At the same time, who knows - maybe part of the reason OpenStack struggles to attract new contributors is that most available Python developers are already here, and bringing in the Rust community could add some fresh air to the ecosystem. пт, 7 лист. 2025 р. о 18:55 Jeremy Stanley <fungi@yuggoth.org> пише:
On 2025-11-07 18:17:14 +0100 (+0100), René Ribaud wrote: [...]
I don’t know what the Foundation or the Technical Committee’s current position is on Rust within OpenStack [...]
The OpenInfra Foundation expresses no opinion in its bylaws, and while https://board.openinfra.org/en/ProjectGuidelines has been used for determining project suitability it only concerns itself with licenses and practices not choice of programming language.
The OpenStack Technical Committee does have published guidance about choice of programming language...
https://governance.openstack.org/tc/resolutions/20150901-programming-languag...
...as well as how to qualify new languages:
https://governance.openstack.org/tc/reference/new-language-requirements.html
An example is this resolution from when Go was added:
https://governance.openstack.org/tc/resolutions/20170329-golang-use-case.htm...
Introducing Rust would in theory involve a similar resolution, and "just" needs discussion and a vote among the TC members. -- Jeremy Stanley
-- Best regards, Andriy Kurilin.
Thanks for the response, Andriy On Fri, 7 Nov 2025 at 19:52, Andriy Kurilin <andr.kurilin@gmail.com> wrote:
The work was done in Rust due to multiple factors: lack of necessary python libraries with trustable maintenance and/or licensing, necessity for enforcing security standards and practices using memory safe programming language
Just curious, did you consider creating a Rust "binding" layer for Keystone instead of a full rewrite? That might address the Python dependency and maintenance issues you mentioned.
As stated initially - yes, we have this possibility when using Rust. Due to the heavy plugin architecture (what you refer separately) bindings are not that trivial, but it depends on the "entry point" of the binding. To remind - we have a problem that certain things (saml) can be considered gone and we need to build the whole new federation logic around the oidc only giving users the control. So the reason is not the "rewrite", the reason is that features required an external work and the result of it looks very promising to be extended.
A side effect of it became immediately visible: performance. This allows us to achieve significant performance and maintainability improvements, eliminate the old ballast
From an operator’s perspective, Keystone performance has never been an issue for us. It just works. Thank you, folks! So if performance improvements are a side effect, great - but it has never been high on my list of concerns.
It is interesting how the situation differs from cloud to cloud. Just not that long time ago (less than a year) we had a bug ( https://review.opendev.org/c/openstack/keystone/+/880456 for the fix) that a token validation request for the token issued for application credentials took 5-10x longer than the one for password. The commit message states around 400ms (on devstack), but in our public cloud it was 5s. And this is compared to something around 100ms for the password one (measures under real load). The "fix" is just caching the data that Keystone fetches over and over from the DB due to the current architecture. Don't nail me on numbers, pls, it heavily depends on the environment and the use case, but token validation with rust takes without any caching 10-15ms on my laptop and on a VM with devstack
The new stack is fully async, uses parallel DB queries, and runs background tasks for periodic cleanups, synchronization, and similar.
This reminds me of the early async wave in Python. There were claims like “we built the fastest Postgres driver in the world,” but in practice, it did not noticeably change anything. I mean the problem quite often is not in the technological stack, but in the way people write software, which covers the dependencies and logical flow as well.
. The new stack makes it possible to execute queries in parallel. I do not know whether it is relevant for other services, but for Keystone it does. Current architecture does 60+ queries during a single request processing. Now we can control the data that we have already fetched (not based on the memcached) and send out some queries in parallel where the db schema requests multiple data sets.
Anyway, I would like to thank the Keystone team for the work you are doing and for opening this conversation. And to be fair, not all my colleagues share my skepticism - some of them can't wait for the new Keystone to be accepted so we can finally deploy it in our cloud :)
Please ask mentioned colleagues to step by and start participating, even if it is just in the discussions or issues regarding new features or the desired development direction. New features that I was presenting were focused on making operators and users happy. Having such feedback is making me feel like it was not a time waste.
At the same time, who knows - maybe part of the reason OpenStack struggles to attract new contributors is that most available Python developers are already here, and bringing in the Rust community could add some fresh air to the ecosystem.
Maybe, I was also thinking about that, but didn't want to explicitly mention it. Artem.
On 11/8/25 4:36 AM, Artem Goncharov wrote:
At the same time, who knows - maybe part of the reason OpenStack struggles to attract new contributors is that most available Python developers are already here, and bringing in the Rust community could add some fresh air to the ecosystem.
Maybe, I was also thinking about that, but didn't want to explicitly mention it.
I have a different view about this point. Introducing a new language might be attractive to developers who are willing to learn new things and find python boring and legacy. However we should also be aware of potential increase of the complexity of the whole software stack and our tool chain. For example operators may need to learn how they debug software written in a new language. New developers joining the community may need to learn two sets of language and also different toolings (requirements management, the way to run tests, the way to run format checks, and so on). I have some concerns that these can increase user/developer effort and cause opposite effect. I'm not saying that this is a hard blocker, but just want to reminds you of potential tradeoff we should consider. I hope that the plan to ease the additional complexity (maybe some documents for users and operators for initial learning) is also discussed as well as the software technology, for example.
Artem.
Hey, Let me please explain my view on the position that you and multiple others expressed (so don't take it personally) after drawing a long bold line separating it from the initial topic. If one wants to continue this particular discussion feel free to branch it out into a separate thread instead. ================================== The following is my point of view, yours would surely differ!!! I think the problem with a decline of maintainers is artificially created by ourselves. We are sitting behind the wall that we've built and continue making it higher and thicker. I believe we are wearing contributors out with the mentality of let's have everything done the same way so that others have a chance to take it over when (an example that I heard: you would be hit by the bus while crossing the street and there is nobody else in the project) the project is left unmaintained. We try to do everything possible to keep project or library alive beyond our capacities. Every time an OpenStack project/repo looses last maintainer we spend literally years discussing that and searching somebody who still has some capacity before reaching 200% load. Another aspect is the spaghetti ecosystem that we have built. I can now say I spend more time fighting the dependencies and jobs that get broken without me doing anything before they broke than working on the features. Just for the sake of example, I got report of a mistake in Keystone Open API schema. The fix was fast: 2 minutes of work. But I have figured out that API-ref build job got broken. Investigation led to to the sphinx openapi extension. A direct dependency of it have a transitive dependency controlled by our constraints and was upgraded a week ago. Since my direct dependency does not have a vibrant community and it takes time to get update I am completely blocked with releasing the fix. I gave up and started simply getting rid (hoping for it to be a temporary issue) of building openapi to html as such just to be able to unblock fixes. Another example: I missed to control the commit sha of the releases change when making 2024.1 unmaintained while one change to (yet again) fix jobs in the branch was in the gate (I missed that because of amount of broken things distracting me). So the branch went unmaintained without the fix. Ok. I just cherry-pick it to the unmaintained. Good luck - the fix does not work there, come on, it is just a branch rename. I spent 2 days with different attempts and gave up, this is just unmaintained branch after all. But now I see Brian is messing with it and also "wastes" his time multiple days in a row. And the more you try to fix the more broken it becomes (even more jobs get broken on a daily basis). I think he will also give up. What I am trying to say: - does it make sense how we have build our ecosystem: yes, it does - can it be done easier/better: almost certainly yes While, as said, there are reasons for it to be how it is, we must admit, that there is the other side of the coin that is a consequence of that approach: - OpenStack contribution is very complex - we wear contributors out with the mentality that everything is built similarly and so taking it over another abandoned thing (maybe for a reason) is relatively "easy" - this leads to more and more load and we spend our time in maintenance of everything around instead of a focus on a primary project - we loose concentration fixing multiple things in parallel - we cannot easily innovate (or even try to innovate) because of the mentioned rules and constraints. We even prevent using thing that would simplify the daily work with the rules and constraints of the ecosystem For me personally coding outside of OpenStack brings satisfaction, coding in OpenStack brings frustration every single day. Some know I burned myself down few years ago with that. Now I am again close to that. I thought multiple times already of just giving up on the OpenStack world and just move myself into a different area. I know personally maintainers who left OpenStack because of not seeing future in it and tired of artificial problems. Just to remind, mostly I do the work on a voluntary basis, I am not paid for working on OpenStack. I hope I was able to explain why I see the mentioned approach as dangerous. Another reminder: this is my personal opinion and is not representing the Keystone team Regards, Artem ---- typed from mobile, auto-correct typos assumed ---- On Sat, 8 Nov 2025, 02:21 Takashi Kajinami, <kajinamit@oss.nttdata.com> wrote:
On 11/8/25 4:36 AM, Artem Goncharov wrote:
At the same time, who knows - maybe part of the reason OpenStack
struggles to attract new contributors is that most available Python developers are already here, and bringing in the Rust community could add some fresh air to the ecosystem.
Maybe, I was also thinking about that, but didn't want to explicitly
mention it.
I have a different view about this point. Introducing a new language might be attractive to developers who are willing to learn new things and find python boring and legacy. However we should also be aware of potential increase of the complexity of the whole software stack and our tool chain. For example operators may need to learn how they debug software written in a new language. New developers joining the community may need to learn two sets of language and also different toolings (requirements management, the way to run tests, the way to run format checks, and so on). I have some concerns that these can increase user/developer effort and cause opposite effect.
I'm not saying that this is a hard blocker, but just want to reminds you of potential tradeoff we should consider. I hope that the plan to ease the additional complexity (maybe some documents for users and operators for initial learning) is also discussed as well as the software technology, for example.
Artem.
Hey Artem, just a note on this: On 11/9/25 12:36 AM, Artem Goncharov wrote:
Another aspect is the spaghetti ecosystem that we have built. I can now say I spend more time fighting the dependencies and jobs that get broken without me doing anything before they broke than working on the features. Just for the sake of example, I got report of a mistake in Keystone Open API schema. The fix was fast: 2 minutes of work. But I have figured out that API-ref build job got broken. Investigation led to to the sphinx openapi extension. A direct dependency of it have a transitive dependency controlled by our constraints and was upgraded a week ago. Since my direct dependency does not have a vibrant community and it takes time to get update I am completely blocked with releasing the fix. I gave up and started simply getting rid (hoping for it to be a temporary issue) of building openapi to html as such just to be able to unblock fixes. Another example: I missed to control the commit sha of the releases change when making 2024.1 unmaintained while one change to (yet again) fix jobs in the branch was in the gate (I missed that because of amount of broken things distracting me). So the branch went unmaintained without the fix. Ok. I just cherry-pick it to the unmaintained. Good luck - the fix does not work there, come on, it is just a branch rename. I spent 2 days with different attempts and gave up, this is just unmaintained branch after all. But now I see Brian is messing with it and also "wastes" his time multiple days in a row. And the more you try to fix the more broken it becomes (even more jobs get broken on a daily basis). I think he will also give up.
I feel these frustrations greatly. I've spent a lot of my career in OpenStack chasing down CI and transitive dependencies and the like -- but this is, to me, MORE of a reason to keep our technologies unified, instead of separate. The integration issues you indicate aren't invented by us, they are *experienced* by us due to the size of the ecosystem. Separating from the ecosystem does have an immediate, positive effect on that: you've reduced the size of YOUR ecosystem and concerns accordingly. The problem is... the issues will still exist. Just now instead of taking them on as an OpenStack group, they are pushed down to a small collection of folks doing last-minute integration. These are lessons are already learned as an ecosystem which is why, IMO, it's feature -- not a bug -- that rot in the overall ecosystem slows down individual projects. This seems to be the only way to ensure folks prioritize removing rot from the ecosystem. To be clear: I'm not pro or anti-rust in any way, in OpenStack or otherwise, I just think this impression of speed being given by moving out of the ecosystem or changing languages is merely work that's been offloaded to others implicitly. To some extent, it's seems this may already be happening regarding keystone-ng, based on the Keystone team barely meeting deadlines to patch OSSA-2025-002 before it being disclosed before-fix due to embargo timeout. Thanks, Jay Faulkner G-Research OSS P.S. I do not love that this is the reality. I've mentioned multiple times that I wish you didn't need a PhD in devops to contribute to OpenStack, but the only way to work out those complexities is by working together, not by concentrating that knowledge further by passing the buck on ecosystem improvements.
Thanks for your opinion Jay, For me it translates to: I should continue chasing dependencies instead of implementing features or addressing vulnerabilities, and, as you rightly pointed out, I should continue doing this alone. Or am I overseeing anything (no offense)? Actually I would like to disagree with your conclusions: everybody should deal with "own" responsibilities and during my Keystone work I should not take care of oslo.xxx, branching, requirements, zuul, ... - there should be people dealing with those. Right now I do not have time to work on Keystone exactly because I need to take care about everything else, keeping it just "running". The situation in the world in the last few years showed me that while globalization allows you to be more efficient it makes your system very vulnerable - a single point of failure. A balance of centralization and decentralization must be found to keep you having electricity and heating when a single rocket launched by a bad actor is enough to bring your whole electricity network down. I tried to make a point that disappearing contributors are perhaps caused by the complexity that we have and not just because nobody thinks OpenStack is relevant. And instead of accepting the state as it is, we may consider a change and try to find another way of solving the problem while helping people to actually enjoy the work. On Mon, 10 Nov 2025 at 17:43, Jay Faulkner <jay@gr-oss.io> wrote:
Hey Artem, just a note on this:
On 11/9/25 12:36 AM, Artem Goncharov wrote:
Another aspect is the spaghetti ecosystem that we have built. I can now say I spend more time fighting the dependencies and jobs that get broken without me doing anything before they broke than working on the features. Just for the sake of example, I got report of a mistake in Keystone Open API schema. The fix was fast: 2 minutes of work. But I have figured out that API-ref build job got broken. Investigation led to to the sphinx openapi extension. A direct dependency of it have a transitive dependency controlled by our constraints and was upgraded a week ago. Since my direct dependency does not have a vibrant community and it takes time to get update I am completely blocked with releasing the fix. I gave up and started simply getting rid (hoping for it to be a temporary issue) of building openapi to html as such just to be able to unblock fixes. Another example: I missed to control the commit sha of the releases change when making 2024.1 unmaintained while one change to (yet again) fix jobs in the branch was in the gate (I missed that because of amount of broken things distracting me). So the branch went unmaintained without the fix. Ok. I just cherry-pick it to the unmaintained. Good luck - the fix does not work there, come on, it is just a branch rename. I spent 2 days with different attempts and gave up, this is just unmaintained branch after all. But now I see Brian is messing with it and also "wastes" his time multiple days in a row. And the more you try to fix the more broken it becomes (even more jobs get broken on a daily basis). I think he will also give up.
I feel these frustrations greatly. I've spent a lot of my career in OpenStack chasing down CI and transitive dependencies and the like -- but this is, to me, MORE of a reason to keep our technologies unified, instead of separate. The integration issues you indicate aren't invented by us, they are *experienced* by us due to the size of the ecosystem. Separating from the ecosystem does have an immediate, positive effect on that: you've reduced the size of YOUR ecosystem and concerns accordingly.
The problem is... the issues will still exist. Just now instead of taking them on as an OpenStack group, they are pushed down to a small collection of folks doing last-minute integration. These are lessons are already learned as an ecosystem which is why, IMO, it's feature -- not a bug -- that rot in the overall ecosystem slows down individual projects. This seems to be the only way to ensure folks prioritize removing rot from the ecosystem.
To be clear: I'm not pro or anti-rust in any way, in OpenStack or otherwise, I just think this impression of speed being given by moving out of the ecosystem or changing languages is merely work that's been offloaded to others implicitly. To some extent, it's seems this may already be happening regarding keystone-ng, based on the Keystone team barely meeting deadlines to patch OSSA-2025-002 before it being disclosed before-fix due to embargo timeout.
Thanks, Jay Faulkner G-Research OSS
P.S. I do not love that this is the reality. I've mentioned multiple times that I wish you didn't need a PhD in devops to contribute to OpenStack, but the only way to work out those complexities is by working together, not by concentrating that knowledge further by passing the buck on ecosystem improvements.
On 2025-11-10 18:12:54 +0100 (+0100), Artem Goncharov wrote: [...]
Actually I would like to disagree with your conclusions: everybody should deal with "own" responsibilities and during my Keystone work I should not take care of oslo.xxx, branching, requirements, zuul, ... - there should be people dealing with those. Right now I do not have time to work on Keystone exactly because I need to take care about everything else, keeping it just "running". [...]
As others have already suggested far more eloquently than I, maybe what we need isn't people working on Keystone (and Nova, and Cinder, and Neutron, and...), but people working on *OpenStack*. -- Jeremy Stanley
On 2025-11-10 18:12:54 +0100 (+0100), Artem Goncharov wrote: [...]
Actually I would like to disagree with your conclusions: everybody should deal with "own" responsibilities and during my Keystone work I should not take care of oslo.xxx, branching, requirements, zuul, ... - there should be people dealing with those. Right now I do not have time to work on Keystone exactly because I need to take care about everything else, keeping it just "running". [...]
As others have already suggested far more eloquently than I, maybe what we need isn't people working on Keystone (and Nova, and Cinder, and Neutron, and...), but people working on *OpenStack*. I tend to agree that even if you primarily contribute to one project
On 10/11/2025 21:22, Jeremy Stanley wrote: that should not stop you contributing to other. that does not mean i believe it correct for companies to expect that they can ask you to work on any project withing openstack. that is a marital change in your employment terms if you are primary on project A and they tell you to work on project B but i do think working on other projects is good as it allow you to cross pollinate ideas and approaches form one to another. I often come back to https://governance.openstack.org/tc/reference/principles.html#openstack-firs... as well. """ OpenStack First, Project Team Second, Company Third<https://governance.openstack.org/tc/reference/principles.html#openstack-first-project-team-second-company-third> OpenStack leaders are expected to put the needs of OpenStack first in their decision making, before the needs of any individual project team. They are also expected to put the needs of their project team before the needs of the organization they work for, if any. They can of course represent the needs of their project team or their company, but in case of conflicts of interest, they should be ready to put those needs aside and make the best call for OpenStack as a whole. """ I primarily gravitate around nova/neutron/placement (and now watcher/devstack) but i will happily work on almost any part of the python services when i have time and folks as for help if it help the community and projects i am most engaged with. keystone is one of those project that if it needed help and some of my time i would try and make room to contribute, there are certainly features i would love to implement in it like moving public keys to be associated with keystone users and allowing them to be used to login. The main issue i have is i don't know what support keystone needs form contributor and i don't really want to go bring more work to the keystone team by proposing feature extra if i know they are already over burdened. I know we can collaborate on things like this at the ptg or mailing list but i feel like this is one of the things we kind of lost form the in person events. it was a lot easier to hear about initiatives and gaps in the hallway track or over a drink in the evening then in "formal" sessions or virtual cross projects as you just don't have the time to follow up with folks 1:1 or in small groups when you hear about a problem or interesting idea. But yes +1 to working on openstack if your situation allows you too not just working on a specific project. reagards sean.
Hi, I want to thank all the contributors to this discussion. As a reminder, what Artem currently brings with keystone-ng extensions is: * OIDC with self-service and CLI integration * Workload federation (Zuul, GitHub, Gitlab do not need to have secrets to access OpenStack anymore) * Service accounts (many use cases, one of the prominent ones: OpenStack on Kubernetes, GitHub workflow) * Passkeys (webauthn) * SCIM * Tight Open Policy Agent integration. The OpenInfra sessions were full of people interested to hear about these features. The question is how to get these features in the hands of openstack operators. We have three options: * Either embrace the codebase Artem developed (with all the rust learning, new codebase understanding, CI dependencies…) * Consider re-developing those features in python * Do nothing Keystone maintainers are few, if we go with option “python”, will anyone step forward to help with the new feature development? Because we know who’s working in keystone, and it has not been growing in the last few years. If we go the “rust” route, we get a working PoC but incomplete codebase, which might attract new contributors. If the current approach is not working, maybe the approach has to change. There is a demand for the modern features Artem developed. Either we can provide them in OpenStack, or maybe the solution will happen outside of OpenStack (e.g., k8s+kubevirt). The main risk is to achieve very little modernization in keystone due to lack of capacity. We can argue about language, CI, splitting the community (all good arguments). At the end of the day, what is the path of least resistance to modernize keystone & get new contributors to this modernization effort? I find the keystone-ng experiment interesting, I hope it gets considered & interested maintainers will show up in the next 2-3 months mentioned in the initial email. Artem, I salute your determination in challenging the status-quo. Best regards, JP (Speaking in my own name and on my own behalf) On 7 Nov 2025, at 12:17, René Ribaud wrote:
Hi,
Thank you very much for sharing this detailed update. That was a very interesting and inspiring read.
I’m probably a bit biased since I’m a big fan of Rust myself, but I truly appreciate how much it can help us build safer and more performant software. The direction you’re taking feels ambitious and forward-looking.
Unfortunately, I couldn’t attend your session at the OpenInfra Summit in October, but I’ll make sure to watch the replay soon. From what I read here, the work and vision behind this initiative are really impressive, and I’d like to express my full support for the effort.
I also realize that introducing a new language like Rust into the OpenStack ecosystem won’t be easy and will likely take time, both from a technical and a governance perspective. But it’s a fascinating step forward, and I think it’s also an interesting opportunity to see whether this approach can attract new contributors or a broader group of people willing to help sustain the project.
I don’t know what the Foundation or the Technical Committee’s current position is on Rust within OpenStack, but personally, I believe this kind of project should be encouraged under an *experimental* or *incubation* status. It’s exactly the kind of innovation that can keep the ecosystem vibrant and evolving.
Congrats again for pushing such an ambitious and technically exciting direction.
Best regards, René
Le jeu. 6 nov. 2025 à 16:27, Artem Goncharov <artem.goncharov@gmail.com> a écrit :
Hey guys
Let us give you an update of the OpenStack Keystone future following the great PTG week we just had.
Some news has already spread across the community, causing excitement and anxiety. It is not uncommon that changes cause fear. Some time ago, in the age of the Inquisition, people were even burned because of fear, misunderstanding, or questioning the rules. Let us clarify things.
Nearly 13 years ago, the Identity v3 API was introduced. Since then, the world has changed dramatically. Libraries we use got abandoned, standards went legacy and replaced with new ones, ecosystem components, certain functionality is based upon are being sunsetted by OS vendors (mod_auth_mellon is archived upstream and gone in CentOS and RHEL [6]), new security vulnerabilities (classes of vulnerabilities) appear like mushrooms without us being able to address them.
Cloud providers today face new challenges that were not present 10-15 years ago. Nowadays, a public cloud requires proper federation support with complete self-service. Users request passkey support, and security departments require additional hardening. These things cannot be covered in today's Keystone architecture and API version. As a result, work on an optional extension to Keystone [1] began some time ago, allowing CSPs to deploy it in parallel with the Python Keystone (of any v3 version) and access the new features.
To simplify parallel deployment, it has been decided to start introducing v4 with the new features (since those ARE for cloud end-users). The work was done in Rust due to multiple factors: lack of necessary python libraries with trustable maintenance and/or licensing, necessity for enforcing security standards and practices using memory safe programming language, demand to address python ecosystem madness of past years (I guess majority would agree, that in the past few years we were mostly only chasing the python ecosystem breaking us on setuptools, eventlet, passlib, …).
During the work, it became clear that implementing the mentioned features requires support of a given percentage of the existing functionality: users, groups, projects, roles, and tokens. And so it was implemented. A side effect of it became immediately visible: performance. Token check is THE most frequently called API in the OpenStack cloud. Right now, caching is used to reduce system stress. But in Rust, token decryption, without caching, is much faster than in python with caching (we expect a 2-3x faster order of magnitude).
As mentioned above, new vulnerability classes and attack vectors require us to disable caching, making such an improvement highly valuable. The new stack is fully async, uses parallel DB queries, and runs background tasks for periodic cleanups, synchronization, and similar. This allows us to achieve significant performance and maintainability improvements, eliminate the old ballast, and significantly reduce transitive dependency version conflicts.
Current measurements also indicate that we would likely be able to eliminate the need for caching in Keystone. For python, on the other hand, we do not have much to start with and would need to find and migrate to the new web framework, find maintained libraries with suitable licenses, and implement new features. We also need to rework the DB access patterns due to the number of bugs and the performance penalty caused by suboptimal DB access (many rows are selected multiple times during processing of a single request). With new authentication flows, we would need to do a wide security assessment of Keystone from scratch anyway.
With that work in mind and facing the prospect of having a major rework in python Keystone anyway (due to the abandoned complex dependencies), we, as the Keystone team, decided to try to focus the work on the Rust implementation, making it a proper evolution of the python Keystone and building a necessary community around it. This adoption is conditional on adding new contributors to support the project.
We believe this approach allows the best of both worlds: the trusted maturity of Keystone’s Python code-base, combined with the modern, high-safety, high-performance capabilities of Rust where they matter most.
Logically, this raises questions, so let's try to answer some of them:
1. Rust is not officially allowed in OpenStack. How are you dealing with that - The development is now going on on GitHub outside of OpenStack governance. - We would apply for OpenStack universe inclusion. - Once the TC policy allows Rust or the overall project inclusion requirement is reworked, the project will work toward becoming the official OpenStack project. - No one in the team is willing to separate from OpenStack, but with the current policies, we have no other way. Everybody and everything in the OpenStack ecosystem would benefit from that project being official to have first-class native support of the new functionality. - Multiple CSPs expressed support for such a move, desiring to see Rust in the ecosystem for better performance, maintainability, and resource utilization - A nice coincidence is a freshly sent announcement from Debian that introduces a strong rust dependency in the packaging tooling, stating something similar to: either you work to comply with this requirement in the next 6 months or sunset the port [5]. - Rust allows cross-integration with python with a high number of cryptography and typing libraries for python written directly in Rust. We are going to be able to reuse certain pieces of python in Rust during porting and Rust in python later in the future. - Keystone is a very sensitive component and in our eyes that fully justifies use of rust to provide a much stronger security on the language level.
2. Do you deprecate python Keystone. - No - v3 is there, and we continue maintaining it - We would unlikely accept major changes - Keystonemiddleware, keystoneauth, etc, are still there. They would likely be adopted for v4 support at some point in the future (in OpenStack services, we already communicate with “external” software, so there should be no problem). Those libraries are crucial for other services and thus are there to stay while necessary. - At some point, we will no longer be able to keep it alive when abandoned dependencies are physically gone. We are not going to invest our time in maintaining the abandoned software; we do not have enough time to do so.
3. What is the community around it? - As of now, there are provisional commitments from a few vendors and CSPs to collaborate on it. - Existing software is being transferred under the openstack-experimental organization to help build the community [3]. -The initial author (gtema) personally commits to continue working on that independently of others.
4. What are the /v3 features that would go away? - OIDC federation in its current form. v4 reimplements it from scratch natively without depending on external software (mod_auth_oidc, etc) since those make self-service impossible. - Saml2, this is a legacy protocol that is very unlikely to even get fixes for security vulnerabilities, and looks to be PQC unsafe. - Oauth1.
5. What new features are there in v4? - OIDC with self-service and CLI integration - Workload federation (Zuul, GitHub, Gitlab do not need to have secrets to access OpenStack anymore) - Service accounts (many use cases, one of the prominent ones: OpenStack on Kubernetes, GitHub workflow) - Passkeys (webauthn) - SCIM - Tight Open Policy Agent integration. We plan to consider the reintroduction of central policy storage in Keystone (not enforcing, but for central storage)[4]
We will use the next 2-3 months to gather interest in contributing to this approach and the new Rust-based code. There was a lot of interest in the new features listed right above during gtema’s sessions in OpenInfra Summit in October 2025. Still, it has to translate into active contributors for this effort to take off. If you have an interest in those new features, join us!
We hope this clarifies the situation. Feel free to reach out with other questions.
Thanks to everybody who participated in the discussions.
Regards, Keystone team
[1] https://github.com/gtema/keystone [2] https://www.youtube.com/watch?v=XOHYqE2HRw4&list=PLKqaoAnDyfgr91wN_12nwY321504Ctw1s&index=30 [3] https://github.com/openstack-experimental/keystone [4] https://www.youtube.com/watch?v=_B4Zsd8RG88&list=PLKqaoAnDyfgr91wN_12nwY321504Ctw1s&index=33 [5] https://lists.debian.org/debian-devel/2025/10/msg00285.html [6] https://access.redhat.com/solutions/2743391
participants (10)
-
Andriy Kurilin
-
Artem Goncharov
-
Jay Faulkner
-
Jean-Philippe Jung
-
Jeremy Stanley
-
René Ribaud
-
Sean Mooney
-
Takashi Kajinami
-
Thomas Goirand
-
Tobias Urdin - Binero