[all][oslo][ops][tc] Future of PostgreSQL support
Hello, PostgreSQL support in OpenStack was introduced to oslo.db long time ago and has been there for multiple OpenStack services. However maintaining this backend support has caused multiple challenges (due to limited test coverage and also differences between DBMS which ware not totally hidden by ORM), and neutron eventually decided to remove its PostgreSQL support during 2025.1 cycle[1]. I'm sending this email to gather inputs about current status and needs to maintain the PostgreSQL support, so that we can discuss whether we can globally remove the feature. As I mentioned earlier, Neutron no longer supports PostgreSQL. This means we can no longer use PostgreSQL as a single database backend in OpenStack deployments. Does anyone still want to continue using PostgreSQL for only specific services ? Note that sqlalchemy 2.1 switched the default dialect for PostgreSQL from psycopg2 to psycopg3[2]. This was earlier caught in oslo.db[3] in its test coverage with the latest sqlalchemy. This switch hasn't yet appeared globally because of sqlalchemy still pinned to <2.1 in upper constraints. The switch likely requires adoption efforts (replace psycopg2 libraries in test requirements or update multiple tests to explicitly request psycopg2). Thank you, Takashi [1] https://review.opendev.org/c/openstack/neutron/+/934171 [2] https://docs.sqlalchemy.org/en/21/changelog/changelog_21.html#change-bb4fc90... [3] https://review.opendev.org/c/openstack/oslo.db/+/977543 -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit
On 7/11/26 9:51 AM, Takashi Kajinami wrote:
Hello,
PostgreSQL support in OpenStack was introduced to oslo.db long time ago and has been there for multiple OpenStack services. However maintaining this backend support has caused multiple challenges (due to limited test coverage and also differences between DBMS which ware not totally hidden by ORM), and neutron eventually decided to remove its PostgreSQL support during 2025.1 cycle[1].
I'm sending this email to gather inputs about current status and needs to maintain the PostgreSQL support, so that we can discuss whether we can globally remove the feature. As I mentioned earlier, Neutron no longer supports PostgreSQL. This means we can no longer use PostgreSQL as a single database backend in OpenStack deployments. Does anyone still want to continue using PostgreSQL for only specific services ?
Note that sqlalchemy 2.1 switched the default dialect for PostgreSQL from psycopg2 to psycopg3[2]. This was earlier caught in oslo.db[3] in its test coverage with the latest sqlalchemy. This switch hasn't yet appeared globally because of sqlalchemy still pinned to <2.1 in upper constraints. The switch likely requires adoption efforts (replace psycopg2 libraries in test requirements or update multiple tests to explicitly request psycopg2).
Thank you Takashi. We have lacked support to adequately test postgresql since 2017: https://governance.openstack.org/tc/resolutions/20170613-postgresql-status.h... We're aware of deployments using Postgresql; so this call for attention is warranted.
Thank you, Takashi
[1] https://review.opendev.org/c/openstack/neutron/+/934171 [2] https://docs.sqlalchemy.org/en/21/changelog/ changelog_21.html#change-bb4fc905d5e7a23adcb3dd455617983d [3] https://review.opendev.org/c/openstack/oslo.db/+/977543
Hi Takashi, Thanks for starting the discussion on this. I thought I'd give my personal opinion on support for PostgreSQL in OpenStack. At Catalyst Cloud we currently use a shared MariaDB cluster for most of our OpenStack services (all of which are containerised). Since I was not around when we got our cloud started I asked around as to why we went with MySQL/MariaDB and not PostgreSQL originally; apparently it was because one of the services we needed only supported MySQL at the time (I am unsure which, unfortunately). We are currently in the process of moving all of our OpenStack services into Kubernetes. One of the main motivations for doing this was to more easily run per-service database clusters in segregated namespaces; and one of the major advantages we were hoping to take advantage of was the ability to switch from MariaDB to PostgreSQL for some of these services. In my opinion PostgreSQL has a number of major advantages over MySQL/MariaDB, including but not limited to: * Support for CHECK constraints, not just foreign key constraints, removing the need for complicated database triggers to enforc constraints at the database level in some cases. * Support for custom collations, allowing more fine-grained handling of comparison and ordering in string columns. * Availability of column types such as JSONB, allowing more flexibility in how data is stored and queried. * Closer API compatibility with SQLite (which is extensively used for unit testing in not just our internal services, but a lot of upsteam OpenStack services). We often find ourselves having to implement branching handling for MySQL/MariaDB and then everything else. Containerisation and particular Kubernetes allows for increased separation between OpenStack services without a corresponding increase in maintenance workload (in fact, the services using their own undercloud services reduces the impact of upgrades and downtime, making services *easier* to manage). Given this context, I believe that now more than ever OpenStack services should have the right to choose which databases they'd like to support (like Neutron choosing to support MariaDB in your example). Even with oslo.db, SQLAlchemy and Django reducing some of the cost of supporting multiple database types, it does not eliminate it after all. So I have no problem with Neutron (or other srevices) only supporting MariaDB, or PostgreSQL or whatever they choose to support. However, I believe that oslo.db must continue to support MariaDB and PostgreSQL equally, in order to provide this choice. Looking at the oslo.db code I see that PostgreSQL is only mentioned outside of testing code only in a handful of places, and this PostgreSQL-specific code does not appear to have been touched in a number of years. SQLAlchemy does most of the work here with regards to database support, and therefore, I don't believe the burden of supporting PostgreSQL in oslo.db is particularly large. Regarding upgrading to psycopg3, the main reason why the package name changed from psycopg2 to psycopg was to allow a dependency interface change that cleaned up the situation where psycopg2 was forced to split the source and binary packages [1]. This kind of paradigm shift is unlikely to happen again, so I think in this case it is worth the effort to go ahead with the upgrade and future-proof OpenStack to allow PostgreSQL to remain an option for services to use. Internally at Catalyst Cloud we have OpenStack-centric services (mainly billing related) that also interface with databases using oslo.db. We were planning on more seriously considering switching those to PostgreSQL as in our case they are easier to switch, and would like to maintain this option in the future without having to carry patches for oslo.db to support it (or switching to using SQLAlchemy directly and going without the convenience features oslo.db provides). To sum up the above: * Given the way OpenStack services are now being deployed, we believe that OpenStack services should have the right to choose which database they support without worrying about the fact that more than one database type would be used in a deployment. * oslo.db must continue supporting both MariaDB and PostgreSQL, in order to facilitate this choice. Thank you again for starting the conversation on this, Takashi. Let me know if you have any questions. Kind regards, Callum [1]: https://www.psycopg.org/psycopg3/docs/basic/from_pg2.html
+1 to Callum. I would like to see it continue to be there and happy to help fix gaps if needed. However, I'm curious as to why we're in a spot where we have compatibility issues, since we're relying on an ORM that should in theory make this disappear from us? ________________________________ From: callum.dickinson@catalystcloud.nz <callum.dickinson@catalystcloud.nz> Sent: July 15, 2026 17:30 To: openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org> Subject: Re: [all][oslo][ops][tc] Future of PostgreSQL support [You don't often get email from callum.dickinson@catalystcloud.nz. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] Hi Takashi, Thanks for starting the discussion on this. I thought I'd give my personal opinion on support for PostgreSQL in OpenStack. At Catalyst Cloud we currently use a shared MariaDB cluster for most of our OpenStack services (all of which are containerised). Since I was not around when we got our cloud started I asked around as to why we went with MySQL/MariaDB and not PostgreSQL originally; apparently it was because one of the services we needed only supported MySQL at the time (I am unsure which, unfortunately). We are currently in the process of moving all of our OpenStack services into Kubernetes. One of the main motivations for doing this was to more easily run per-service database clusters in segregated namespaces; and one of the major advantages we were hoping to take advantage of was the ability to switch from MariaDB to PostgreSQL for some of these services. In my opinion PostgreSQL has a number of major advantages over MySQL/MariaDB, including but not limited to: * Support for CHECK constraints, not just foreign key constraints, removing the need for complicated database triggers to enforc constraints at the database level in some cases. * Support for custom collations, allowing more fine-grained handling of comparison and ordering in string columns. * Availability of column types such as JSONB, allowing more flexibility in how data is stored and queried. * Closer API compatibility with SQLite (which is extensively used for unit testing in not just our internal services, but a lot of upsteam OpenStack services). We often find ourselves having to implement branching handling for MySQL/MariaDB and then everything else. Containerisation and particular Kubernetes allows for increased separation between OpenStack services without a corresponding increase in maintenance workload (in fact, the services using their own undercloud services reduces the impact of upgrades and downtime, making services *easier* to manage). Given this context, I believe that now more than ever OpenStack services should have the right to choose which databases they'd like to support (like Neutron choosing to support MariaDB in your example). Even with oslo.db, SQLAlchemy and Django reducing some of the cost of supporting multiple database types, it does not eliminate it after all. So I have no problem with Neutron (or other srevices) only supporting MariaDB, or PostgreSQL or whatever they choose to support. However, I believe that oslo.db must continue to support MariaDB and PostgreSQL equally, in order to provide this choice. Looking at the oslo.db code I see that PostgreSQL is only mentioned outside of testing code only in a handful of places, and this PostgreSQL-specific code does not appear to have been touched in a number of years. SQLAlchemy does most of the work here with regards to database support, and therefore, I don't believe the burden of supporting PostgreSQL in oslo.db is particularly large. Regarding upgrading to psycopg3, the main reason why the package name changed from psycopg2 to psycopg was to allow a dependency interface change that cleaned up the situation where psycopg2 was forced to split the source and binary packages [1]. This kind of paradigm shift is unlikely to happen again, so I think in this case it is worth the effort to go ahead with the upgrade and future-proof OpenStack to allow PostgreSQL to remain an option for services to use. Internally at Catalyst Cloud we have OpenStack-centric services (mainly billing related) that also interface with databases using oslo.db. We were planning on more seriously considering switching those to PostgreSQL as in our case they are easier to switch, and would like to maintain this option in the future without having to carry patches for oslo.db to support it (or switching to using SQLAlchemy directly and going without the convenience features oslo.db provides). To sum up the above: * Given the way OpenStack services are now being deployed, we believe that OpenStack services should have the right to choose which database they support without worrying about the fact that more than one database type would be used in a deployment. * oslo.db must continue supporting both MariaDB and PostgreSQL, in order to facilitate this choice. Thank you again for starting the conversation on this, Takashi. Let me know if you have any questions. Kind regards, Callum [1]: https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.psycopg.org%2Fpsycopg3%2Fdocs%2Fbasic%2Ffrom_pg2.html&data=05%7C02%7Cmnaser%40vexxhost.com%7C836aeedfa4014a9a23f308dee3471e2a%7C54e2b12264054dafa35bf65edc45c621%7C0%7C0%7C639198091759837629%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C80000%7C%7C%7C&sdata=c1apFgPhU0KGcl4UQvrNEHJ6ATsDOxjvb6EEwyCCRQM%3D&reserved=0<https://www.psycopg.org/psycopg3/docs/basic/from_pg2.html>
On 16/07/2026 15:39, Mohammed Naser wrote:
+1 to Callum.
I would like to see it continue to be there and happy to help fix gaps if needed.
However, I'm curious as to why we're in a spot where we have compatibility issues, since we're relying on an ORM that should in theory make this disappear from us? there are some dialect specific delta between what postgress supprot and mysql/mariadb however i think for the most part your giht tthat you shoudl expect this to "just work" the thing is that there has been liggel investment in testing postgress for a very long time
nova never removed supprot offically so if we were to do that it would need to be deprecated in 2027.1 before it could then be drop based on our slurp rules since i have been involed in reviving cybrog and watcher devleopment we have only tested with mysql in devatack sqlite in unit tests. we may have some basic usage of the opertuneisti test fixutre form oslo.db but i woudl assume both serivce may or may not work as if they were ever tested with postgress it was before we mvoed form SQLAlchmey 1.x to 2.x and if you cahnge the major version of your ORM and then rip out eventlet since it was last tested there are proably bugs. with my downstram hat on redhat openstack distibution deploy openstack on k8s (really openshift) using galera/maraidb using a golgan operator to manage that. so its certenlly possibel to run mariadb in produciton in a kubernetes env. PostgreSQL may work as well or better but one of the things you have to be very careful of is the consitency gaurentees provided by your db. openstack requries that if you finish a write transaction then the next read shoudl see it if you read form a differnt node in the cluster so you can you acitive active galeara for example with multiple writers a modification we had to do is have only one of the galarea ednpoitn be active in the k8s service to accomidate that or very bad things will happen in placment and elsewhere that depend on write transaction commit semantics and this is one of the things tha tthe ORM does not entrily hide. and its why we dont actully supprot active active galaera culsters today. you have to run them in active passive mode so those kind of issues are the reason we focused on oen db in teh past for testign byond just maintianing the devstack job which isnt hard but debuging why a tempest test fails only with postgresql is hard.
------------------------------------------------------------------------ *From:* callum.dickinson@catalystcloud.nz <callum.dickinson@catalystcloud.nz> *Sent:* July 15, 2026 17:30 *To:* openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org> *Subject:* Re: [all][oslo][ops][tc] Future of PostgreSQL support [You don't often get email from callum.dickinson@catalystcloud.nz. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
Hi Takashi,
Thanks for starting the discussion on this. I thought I'd give my personal opinion on support for PostgreSQL in OpenStack.
At Catalyst Cloud we currently use a shared MariaDB cluster for most of our OpenStack services (all of which are containerised). Since I was not around when we got our cloud started I asked around as to why we went with MySQL/MariaDB and not PostgreSQL originally; apparently it was because one of the services we needed only supported MySQL at the time (I am unsure which, unfortunately).
We are currently in the process of moving all of our OpenStack services into Kubernetes. One of the main motivations for doing this was to more easily run per-service database clusters in segregated namespaces; and one of the major advantages we were hoping to take advantage of was the ability to switch from MariaDB to PostgreSQL for some of these services.
In my opinion PostgreSQL has a number of major advantages over MySQL/MariaDB, including but not limited to:
* Support for CHECK constraints, not just foreign key constraints, removing the need for complicated database triggers to enforc constraints at the database level in some cases. * Support for custom collations, allowing more fine-grained handling of comparison and ordering in string columns. * Availability of column types such as JSONB, allowing more flexibility in how data is stored and queried. * Closer API compatibility with SQLite (which is extensively used for unit testing in not just our internal services, but a lot of upsteam OpenStack services). We often find ourselves having to implement branching handling for MySQL/MariaDB and then everything else.
Containerisation and particular Kubernetes allows for increased separation between OpenStack services without a corresponding increase in maintenance workload (in fact, the services using their own undercloud services reduces the impact of upgrades and downtime, making services *easier* to manage).
Given this context, I believe that now more than ever OpenStack services should have the right to choose which databases they'd like to support (like Neutron choosing to support MariaDB in your example). Even with oslo.db, SQLAlchemy and Django reducing some of the cost of supporting multiple database types, it does not eliminate it after all. So I have no problem with Neutron (or other srevices) only supporting MariaDB, or PostgreSQL or whatever they choose to support.
However, I believe that oslo.db must continue to support MariaDB and PostgreSQL equally, in order to provide this choice.
Looking at the oslo.db code I see that PostgreSQL is only mentioned outside of testing code only in a handful of places, and this PostgreSQL-specific code does not appear to have been touched in a number of years. SQLAlchemy does most of the work here with regards to database support, and therefore, I don't believe the burden of supporting PostgreSQL in oslo.db is particularly large.
Regarding upgrading to psycopg3, the main reason why the package name changed from psycopg2 to psycopg was to allow a dependency interface change that cleaned up the situation where psycopg2 was forced to split the source and binary packages [1]. This kind of paradigm shift is unlikely to happen again, so I think in this case it is worth the effort to go ahead with the upgrade and future-proof OpenStack to allow PostgreSQL to remain an option for services to use.
Internally at Catalyst Cloud we have OpenStack-centric services (mainly billing related) that also interface with databases using oslo.db. We were planning on more seriously considering switching those to PostgreSQL as in our case they are easier to switch, and would like to maintain this option in the future without having to carry patches for oslo.db to support it (or switching to using SQLAlchemy directly and going without the convenience features oslo.db provides).
To sum up the above:
* Given the way OpenStack services are now being deployed, we believe that OpenStack services should have the right to choose which database they support without worrying about the fact that more than one database type would be used in a deployment. * oslo.db must continue supporting both MariaDB and PostgreSQL, in order to facilitate this choice.
Thank you again for starting the conversation on this, Takashi. Let me know if you have any questions.
Kind regards, Callum
[1]: https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.psycop... <https://www.psycopg.org/psycopg3/docs/basic/from_pg2.html>
On 7/16/26 11:39 PM, Mohammed Naser wrote:
+1 to Callum.
I would like to see it continue to be there and happy to help fix gaps if needed. Yes please :-)
However, I'm curious as to why we're in a spot where we have compatibility issues, since we're relying on an ORM that should in theory make this disappear from us?
That's what we want in "ideal ORM" but the reality doesn't follow. For example SQLAlchemy does not abstract all database errors. If you want to detect FK constraint error then you have to examine the origin exceptions, which are specific to underlying libraries. Also there are a few types which are not commonly implemented for all backends but are used by some services. We sometimes have to explicitly select the correct one according to the backend used. We've implemented such additional abstraction handling in oslo.db, while there are also some additional handling in each services, mainly in alembic migration scripts which does not use oslo.db .
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ *From:* callum.dickinson@catalystcloud.nz <callum.dickinson@catalystcloud.nz> *Sent:* July 15, 2026 17:30 *To:* openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org> *Subject:* Re: [all][oslo][ops][tc] Future of PostgreSQL support [You don't often get email from callum.dickinson@catalystcloud.nz. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification <https://aka.ms/LearnAboutSenderIdentification> ]
Hi Takashi,
Thanks for starting the discussion on this. I thought I'd give my personal opinion on support for PostgreSQL in OpenStack.
At Catalyst Cloud we currently use a shared MariaDB cluster for most of our OpenStack services (all of which are containerised). Since I was not around when we got our cloud started I asked around as to why we went with MySQL/MariaDB and not PostgreSQL originally; apparently it was because one of the services we needed only supported MySQL at the time (I am unsure which, unfortunately).
We are currently in the process of moving all of our OpenStack services into Kubernetes. One of the main motivations for doing this was to more easily run per-service database clusters in segregated namespaces; and one of the major advantages we were hoping to take advantage of was the ability to switch from MariaDB to PostgreSQL for some of these services.
In my opinion PostgreSQL has a number of major advantages over MySQL/MariaDB, including but not limited to:
* Support for CHECK constraints, not just foreign key constraints, removing the need for complicated database triggers to enforc constraints at the database level in some cases. * Support for custom collations, allowing more fine-grained handling of comparison and ordering in string columns. * Availability of column types such as JSONB, allowing more flexibility in how data is stored and queried. * Closer API compatibility with SQLite (which is extensively used for unit testing in not just our internal services, but a lot of upsteam OpenStack services). We often find ourselves having to implement branching handling for MySQL/MariaDB and then everything else.
Containerisation and particular Kubernetes allows for increased separation between OpenStack services without a corresponding increase in maintenance workload (in fact, the services using their own undercloud services reduces the impact of upgrades and downtime, making services *easier* to manage).
Given this context, I believe that now more than ever OpenStack services should have the right to choose which databases they'd like to support (like Neutron choosing to support MariaDB in your example). Even with oslo.db, SQLAlchemy and Django reducing some of the cost of supporting multiple database types, it does not eliminate it after all. So I have no problem with Neutron (or other srevices) only supporting MariaDB, or PostgreSQL or whatever they choose to support.
However, I believe that oslo.db must continue to support MariaDB and PostgreSQL equally, in order to provide this choice.
Looking at the oslo.db code I see that PostgreSQL is only mentioned outside of testing code only in a handful of places, and this PostgreSQL-specific code does not appear to have been touched in a number of years. SQLAlchemy does most of the work here with regards to database support, and therefore, I don't believe the burden of supporting PostgreSQL in oslo.db is particularly large.
Regarding upgrading to psycopg3, the main reason why the package name changed from psycopg2 to psycopg was to allow a dependency interface change that cleaned up the situation where psycopg2 was forced to split the source and binary packages [1]. This kind of paradigm shift is unlikely to happen again, so I think in this case it is worth the effort to go ahead with the upgrade and future-proof OpenStack to allow PostgreSQL to remain an option for services to use.
Internally at Catalyst Cloud we have OpenStack-centric services (mainly billing related) that also interface with databases using oslo.db. We were planning on more seriously considering switching those to PostgreSQL as in our case they are easier to switch, and would like to maintain this option in the future without having to carry patches for oslo.db to support it (or switching to using SQLAlchemy directly and going without the convenience features oslo.db provides).
To sum up the above:
* Given the way OpenStack services are now being deployed, we believe that OpenStack services should have the right to choose which database they support without worrying about the fact that more than one database type would be used in a deployment. * oslo.db must continue supporting both MariaDB and PostgreSQL, in order to facilitate this choice.
Thank you again for starting the conversation on this, Takashi. Let me know if you have any questions.
Kind regards, Callum
[1]: https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.psycop... <https://www.psycopg.org/psycopg3/docs/basic/from_pg2.html>
Thanks Callum, for joining the discussion. NOTE: I might sound denying what you wrote, but that's not my core intention. However I'd like to make sure that people are fully aware of the challenge in this feature support, which has never been actually improved for long time. On 7/16/26 6:30 AM, callum.dickinson@catalystcloud.nz wrote:
Hi Takashi,
Thanks for starting the discussion on this. I thought I'd give my personal opinion on support for PostgreSQL in OpenStack.
At Catalyst Cloud we currently use a shared MariaDB cluster for most of our OpenStack services (all of which are containerised). Since I was not around when we got our cloud started I asked around as to why we went with MySQL/MariaDB and not PostgreSQL originally; apparently it was because one of the services we needed only supported MySQL at the time (I am unsure which, unfortunately).
We are currently in the process of moving all of our OpenStack services into Kubernetes. One of the main motivations for doing this was to more easily run per-service database clusters in segregated namespaces; and one of the major advantages we were hoping to take advantage of was the ability to switch from MariaDB to PostgreSQL for some of these services.
In my opinion PostgreSQL has a number of major advantages over MySQL/MariaDB, including but not limited to:
* Support for CHECK constraints, not just foreign key constraints, removing the need for complicated database triggers to enforc constraints at the database level in some cases. * Support for custom collations, allowing more fine-grained handling of comparison and ordering in string columns. * Availability of column types such as JSONB, allowing more flexibility in how data is stored and queried.
Although these are very nice feature, using these feature also requires us to maintain compatibility later, because these are not available in the other database backends, namely MySQL and MariaDB.
* Closer API compatibility with SQLite (which is extensively used for unit testing in not just our internal services, but a lot of upsteam OpenStack services). We often find ourselves having to implement branching handling for MySQL/MariaDB and then everything else.
Containerisation and particular Kubernetes allows for increased separation between OpenStack services without a corresponding increase in maintenance workload (in fact, the services using their own undercloud services reduces the impact of upgrades and downtime, making services *easier* to manage).
Given this context, I believe that now more than ever OpenStack services should have the right to choose which databases they'd like to support (like Neutron choosing to support MariaDB in your example). Even with oslo.db, SQLAlchemy and Django reducing some of the cost of supporting multiple database types, it does not eliminate it after all. So I have no problem with Neutron (or other srevices) only supporting MariaDB, or PostgreSQL or whatever they choose to support.
I'd be more careful about the wording here. I don't want neutron supporting only mariadb while keystone supporting only postgresql (NOTE: this is just an example showcase and doesn't describe the real plan), which requires me to deploy two database backends in a single OpenStack cluster. There should be a primary common default backend for all, which would be MySQL/MariaDB, while I agree that it'd be on project's decision to support additional backends. I'm not an actual operator of OpenStack now, but I don't know how many operators would prefer maintaining two database backends, different monitoring or management mechanisms for these, different upgrade operations, in a single OpenStack cluster. Finally the most important point is that we've lacked anyone who are interested in using different backends and also provide actual effort in testing, development. For example since neutron removed support for MySQL, the common postgresql devstack job has been broken. Unfortunately no project has ever launched a job to deploy multiple database backends so that they can test their service with PostgreSQL. Such thing may need to be addressed if a project aims to continue PostgreSQL support.
However, I believe that oslo.db must continue to support MariaDB and PostgreSQL equally, in order to provide this choice.
Looking at the oslo.db code I see that PostgreSQL is only mentioned outside of testing code only in a handful of places, and this PostgreSQL-specific code does not appear to have been touched in a number of years. SQLAlchemy does most of the work here with regards to database support, and therefore, I don't believe the burden of supporting PostgreSQL in oslo.db is particularly large.
Having these code untouched doesn't mean these are stable. We've seen number of problems especially in migration script specific to PostgreSQL in the past and these have been handled outside of oslo.db for years. We could probably fix some of these in oslo.db but unfortunately nobody really spent time to do so. Also we lost actual devstack test coverage several years, so there might be some problems we've not just discovered.
Regarding upgrading to psycopg3, the main reason why the package name changed from psycopg2 to psycopg was to allow a dependency interface change that cleaned up the situation where psycopg2 was forced to split the source and binary packages [1]. This kind of paradigm shift is unlikely to happen again, so I think in this case it is worth the effort to go ahead with the upgrade and future-proof OpenStack to allow PostgreSQL to remain an option for services to use.
I agree such changes would not happen quite frequently but I'm unsure if it doesn't really happen. As long as the library is used extensively we may see such desire popping up again.
Internally at Catalyst Cloud we have OpenStack-centric services (mainly billing related) that also interface with databases using oslo.db. We were planning on more seriously considering switching those to PostgreSQL as in our case they are easier to switch, and would like to maintain this option in the future without having to carry patches for oslo.db to support it (or switching to using SQLAlchemy directly and going without the convenience features oslo.db provides).
To sum up the above:
* Given the way OpenStack services are now being deployed, we believe that OpenStack services should have the right to choose which database they support without worrying about the fact that more than one database type would be used in a deployment. * oslo.db must continue supporting both MariaDB and PostgreSQL, in order to facilitate this choice.
I'm hesitant to say "must" here. If we want oslo.db to support PostgreSQL then we need someone maintaining that support in oslo.db and also in real use in services. That's the pain point which hasn't been improved for long time. If no one really pay effort for it, then we can't maintain it really.
Thank you again for starting the conversation on this, Takashi. Let me know if you have any questions.
Kind regards, Callum
[1]: https://www.psycopg.org/psycopg3/docs/basic/from_pg2.html
participants (5)
-
callum.dickinson@catalystcloud.nz
-
Goutham Pacha Ravi
-
Mohammed Naser
-
Sean Mooney
-
Takashi Kajinami