[tc][all][self-healing-sig] Service-side health checks community goal for Train cycle
Hello everyone,
As you might have seen on the ML, two of the 3 top contenders for the Train cycle community got some traction. Let's here talk about the last one: The service-side health checks.
While people were interested in this goal previously, nobody really came forward on the pre-work.
Last week, I met a few of my colleagues to see what we can do together. Matt (irc: mattoliverau), Adam (irc: aspiers), and I discussed about the different ways to implement this new API, with the help of many in #openstack-sdk.
Long story short, the current framework might be "good enough" for extension already, as we could have extra "backends" (basically "tests"), to increase the coverage of this healthcheck endpoint.
While the immediate next step would be to work on the v2 prototype that Graham started (see link [1], anyone is welcome to help there!), the next step would be far easier if it was crowd sourced: We need to know which service is already using that oslo middleware, which service doesn't want to use it, and which service is already ready for healtchecks.
When we'll have a lay of the land, we'll know where the energy will be spent in this community goal: Would that be bringing oslo.middleware to <insert amount here> services or bringing common "backends" that can be used by each service (like DB/MQ/cache checks).
I would be very happy if you could have a look at this ethercal [2], and add/edit your project capabilities there.
Thank you in advance. Jean-Philippe Evrard (evrardjp)
[1]: https://review.openstack.org/#/c/617924/ [2]: https://ethercalc.openstack.org/di0mxkiepll8
On Fri, Jan 25, 2019, at 11:44 PM, Jean-Philippe Evrard wrote:
Hello everyone,
As you might have seen on the ML, two of the 3 top contenders for the Train cycle community got some traction. Let's here talk about the last one: The service-side health checks.
While people were interested in this goal previously, nobody really came forward on the pre-work.
Last week, I met a few of my colleagues to see what we can do together. Matt (irc: mattoliverau), Adam (irc: aspiers), and I discussed about the different ways to implement this new API, with the help of many in #openstack-sdk.
Long story short, the current framework might be "good enough" for extension already, as we could have extra "backends" (basically "tests"), to increase the coverage of this healthcheck endpoint.
While the immediate next step would be to work on the v2 prototype that Graham started (see link [1], anyone is welcome to help there!), the next step would be far easier if it was crowd sourced: We need to know which service is already using that oslo middleware, which service doesn't want to use it, and which service is already ready for healtchecks.
When we'll have a lay of the land, we'll know where the energy will be spent in this community goal: Would that be bringing oslo.middleware to <insert amount here> services or bringing common "backends" that can be used by each service (like DB/MQ/cache checks).
I would be very happy if you could have a look at this ethercal [2], and add/edit your project capabilities there.
Thank you in advance. Jean-Philippe Evrard (evrardjp)
I noticed the ethercalc has a column "Project has paste with healthcheck in pipelines in paste.ini". Is using Paste a requirement for this goal? If so, I think that's a non-starter.
Keystone just removed Paste, but does already have support for healthchecks via oslo.middleware:
https://docs.openstack.org/keystone/latest/admin/health-check-middleware.htm...
Colleen
On Mon, 2019-01-28 at 11:29 +0100, Colleen Murphy wrote:
On Fri, Jan 25, 2019, at 11:44 PM, Jean-Philippe Evrard wrote:
Hello everyone,
As you might have seen on the ML, two of the 3 top contenders for the Train cycle community got some traction. Let's here talk about the last one: The service-side health checks.
While people were interested in this goal previously, nobody really came forward on the pre-work.
Last week, I met a few of my colleagues to see what we can do together. Matt (irc: mattoliverau), Adam (irc: aspiers), and I discussed about the different ways to implement this new API, with the help of many in #openstack-sdk.
Long story short, the current framework might be "good enough" for extension already, as we could have extra "backends" (basically "tests"), to increase the coverage of this healthcheck endpoint.
While the immediate next step would be to work on the v2 prototype that Graham started (see link [1], anyone is welcome to help there!), the next step would be far easier if it was crowd sourced: We need to know which service is already using that oslo middleware, which service doesn't want to use it, and which service is already ready for healtchecks.
When we'll have a lay of the land, we'll know where the energy will be spent in this community goal: Would that be bringing oslo.middleware to <insert amount here> services or bringing common "backends" that can be used by each service (like DB/MQ/cache checks).
I would be very happy if you could have a look at this ethercal [2], and add/edit your project capabilities there.
Thank you in advance. Jean-Philippe Evrard (evrardjp)
I noticed the ethercalc has a column "Project has paste with healthcheck in pipelines in paste.ini". Is using Paste a requirement for this goal? If so, I think that's a non-starter.
Keystone just removed Paste, but does already have support for healthchecks via oslo.middleware:
https://docs.openstack.org/keystone/latest/admin/health-check-middleware.htm...
Colleen
It is not a non-starter. I knew this would show up :) It's fine that some projects do differently (for example swift has different middleware, keystone is not using paste).
I think it's also too big of a change to move everyone to one single technology in a cycle :) Instead, I want to focus on the real use case for people (bringing a common healthcheck "api" itself), which doesn't matter on the technology.
But I still would like to crowdsource the information about how projects are doing things, as it would help understand the complexity better.
Jean-Philippe Evrard (evrardjp)
On Mon, 28 Jan 2019, Jean-Philippe Evrard wrote:
It is not a non-starter. I knew this would show up :) It's fine that some projects do differently (for example swift has different middleware, keystone is not using paste).
Tangent so that people are clear on the state of Paste and PasteDeploy.
I recommend projects move away from using either.
Until recently both were abandonware, not receiving updates, and had issues working with Python3.
I managed to locate maintainers from a few years ago, and negotiated to bring them under some level of maintenance, but in both cases the people involved are only interested in doing limited management to keep the projects barely alive.
pastedeploy (the thing that is more often used in OpenStack, and is usually used to load the paste.ini file and doesn't have to have a dependency on paste itself) is now under the Pylons project: https://github.com/Pylons/pastedeploy
Paste itself is with me: https://github.com/cdent/paste
I think it's also too big of a change to move everyone to one single technology in a cycle :) Instead, I want to focus on the real use case for people (bringing a common healthcheck "api" itself), which doesn't matter on the technology.
I agree that the healthcheck change can and should be completely separate from any question of what is used to load middleware. That's the great thing about WSGI.
As long as the healthcheck tooling presents are "normal" WSGI interface it ought to either "just work" or be wrappable by other tooling, so I wouldn't spend too much time making a survey of how people are doing middleware.
The tricky part (but not that tricky) will be with managing how the "tests" are provided to the middleware.
On 1/28/19 5:34 AM, Chris Dent wrote:
On Mon, 28 Jan 2019, Jean-Philippe Evrard wrote:
It is not a non-starter. I knew this would show up :) It's fine that some projects do differently (for example swift has different middleware, keystone is not using paste).
Tangent so that people are clear on the state of Paste and PasteDeploy.
I recommend projects move away from using either.
Until recently both were abandonware, not receiving updates, and had issues working with Python3.
I managed to locate maintainers from a few years ago, and negotiated to bring them under some level of maintenance, but in both cases the people involved are only interested in doing limited management to keep the projects barely alive.
pastedeploy (the thing that is more often used in OpenStack, and is usually used to load the paste.ini file and doesn't have to have a dependency on paste itself) is now under the Pylons project: https://github.com/Pylons/pastedeploy
Paste itself is with me: https://github.com/cdent/paste
I think it's also too big of a change to move everyone to one single technology in a cycle :) Instead, I want to focus on the real use case for people (bringing a common healthcheck "api" itself), which doesn't matter on the technology.
I agree that the healthcheck change can and should be completely separate from any question of what is used to load middleware. That's the great thing about WSGI.
As long as the healthcheck tooling presents are "normal" WSGI interface it ought to either "just work" or be wrappable by other tooling, so I wouldn't spend too much time making a survey of how people are doing middleware.
So should that question be re-worded? The current Keystone answer is accurate but unhelpful, given that I believe Keystone does enable the healthcheck middleware by default: https://docs.openstack.org/keystone/latest/admin/health-check-middleware.htm...
Since what we care about isn't the WSGI implementation but the availability of the feature, shouldn't that question be more like "Project enables healthcheck middleware by default"? In which case Keystone's answer becomes a simple "yes" and Manila's a simple "no".
The tricky part (but not that tricky) will be with managing how the "tests" are provided to the middleware.
On Mon, 2019-02-11 at 16:28 -0600, Ben Nemec wrote:
On 1/28/19 5:34 AM, Chris Dent wrote:
On Mon, 28 Jan 2019, Jean-Philippe Evrard wrote:
It is not a non-starter. I knew this would show up :) It's fine that some projects do differently (for example swift has different middleware, keystone is not using paste).
Tangent so that people are clear on the state of Paste and PasteDeploy.
I recommend projects move away from using either.
Until recently both were abandonware, not receiving updates, and had issues working with Python3.
I managed to locate maintainers from a few years ago, and negotiated to bring them under some level of maintenance, but in both cases the people involved are only interested in doing limited management to keep the projects barely alive.
pastedeploy (the thing that is more often used in OpenStack, and is usually used to load the paste.ini file and doesn't have to have a dependency on paste itself) is now under the Pylons project: https://github.com/Pylons/pastedeploy
Paste itself is with me: https://github.com/cdent/paste
I think it's also too big of a change to move everyone to one single technology in a cycle :) Instead, I want to focus on the real use case for people (bringing a common healthcheck "api" itself), which doesn't matter on the technology.
I agree that the healthcheck change can and should be completely separate from any question of what is used to load middleware. That's the great thing about WSGI.
As long as the healthcheck tooling presents are "normal" WSGI interface it ought to either "just work" or be wrappable by other tooling, so I wouldn't spend too much time making a survey of how people are doing middleware.
So should that question be re-worded? The current Keystone answer is accurate but unhelpful, given that I believe Keystone does enable the healthcheck middleware by default: https://docs.openstack.org/keystone/latest/admin/health-check-middleware.htm...
Since what we care about isn't the WSGI implementation but the availability of the feature, shouldn't that question be more like "Project enables healthcheck middleware by default"? In which case Keystone's answer becomes a simple "yes" and Manila's a simple "no".
The tricky part (but not that tricky) will be with managing how the "tests" are provided to the middleware.
Totally fair, to me.
participants (4)
-
Ben Nemec
-
Chris Dent
-
Colleen Murphy
-
Jean-Philippe Evrard