[tc][all] Wallaby Cycle Community Goals
Hi All
It is that time of year / release again - and we need to choose the community goals for Wallaby.
Myself and Nate looked over the list of goals [1][2][3], and we are suggesting one of the following:
- Finish moving legacy python-*client CLIs to python-openstackclient - Move from oslo.rootwrap to oslo.privsep - Implement the API reference guide changes - All API to provide a /healthcheck URL like Keystone (and others) provide
Some of these goals have champions signed up already, but we need to make sure they are still available to do them. If you are interested in helping drive any of the goals, please speak up!
We need to select goals in time for the new release cycle - so please reply if there is goals you think should be included in this list, or not included.
Next steps after this will be helping people write a proposed goal and then the TC selecting the ones we will pursue during Wallaby.
Additionally, we have traditionally selected 2 goals per cycle - however with the people available to do the work across projects Nate and I briefly discussed reducing that to one for this cycle.
What does the community think about this?
Thanks,
Graham
1 - https://etherpad.opendev.org/p/community-goals 2 - https://governance.openstack.org/tc/goals/proposed/index.html 3 - https://etherpad.opendev.org/p/community-w-series-goals 4 - https://governance.openstack.org/tc/goals/index.html#goal-selection-schedule
Hi, I would like to selectively respond to the number of goals per cycle question.
A possible direction could be to forget the "one cycle goal" thing and allow to finish the goals in a longer time frame. From "management" perspective the important is to have a fix number of goals per cycle to avoid overallocation of people.
Another approach could be to attach a number, a difficulty feeling or similar to the proposed goals to make it easier to select them, and avoid to choose 2 hard-to-do goal for one cycle. This numbering can be done by project teams/PTLs whoever has the insight for the projects. Example: zuulv3 migration can be a hard to do goal as affects the whole gating of a project with hard coordination between projects. Add healthcheck API is much simpler as can be done without affecting the life of a whole project, or the community.
Regards Lajos Katona (lajoskatona)
Graham Hayes gr@ham.ie ezt írta (időpont: 2020. szept. 21., H, 19:54):
Hi All
It is that time of year / release again - and we need to choose the community goals for Wallaby.
Myself and Nate looked over the list of goals [1][2][3], and we are suggesting one of the following:
- Finish moving legacy python-*client CLIs to python-openstackclient
- Move from oslo.rootwrap to oslo.privsep
- Implement the API reference guide changes
- All API to provide a /healthcheck URL like Keystone (and others)
provide
Some of these goals have champions signed up already, but we need to make sure they are still available to do them. If you are interested in helping drive any of the goals, please speak up!
We need to select goals in time for the new release cycle - so please reply if there is goals you think should be included in this list, or not included.
Next steps after this will be helping people write a proposed goal and then the TC selecting the ones we will pursue during Wallaby.
Additionally, we have traditionally selected 2 goals per cycle - however with the people available to do the work across projects Nate and I briefly discussed reducing that to one for this cycle.
What does the community think about this?
Thanks,
Graham
1 - https://etherpad.opendev.org/p/community-goals 2 - https://governance.openstack.org/tc/goals/proposed/index.html 3 - https://etherpad.opendev.org/p/community-w-series-goals 4 -
https://governance.openstack.org/tc/goals/index.html#goal-selection-schedule
---- On Thu, 24 Sep 2020 03:45:38 -0500 Lajos Katona katonalala@gmail.com wrote ----
Hi,I would like to selectively respond to the number of goals per cycle question. A possible direction could be to forget the "one cycle goal" thing and allow to finish the goals in a longer time frame. From "management" perspective the important is to have a fix number of goals per cycle to avoid overallocation of people. Another approach could be to attach a number, a difficulty feeling or similar to the proposed goals to make it easier to select them, and avoid to choose 2 hard-to-do goal for one cycle.This numbering can be done by project teams/PTLs whoever has the insight for the projects.Example: zuulv3 migration can be a hard to do goal as affects the whole gating of a project with hard coordination between projects.Add healthcheck API is much simpler as can be done without affecting the life of a whole project, or the community.
+1 on these feedback.
A possible direction could be to forget the "one cycle goal" thing
This is much needed for complex goal or need more work like OSC one, we have not done yet but we discussed the possibility of multi-cycle goal in last cycle. We can add the framwork/guidance for multi-cycle goal like it has to be devided by work to be done or by projects(like target set of projects per cycle).
-gmann
RegardsLajos Katona (lajoskatona)
Graham Hayes gr@ham.ie ezt írta (időpont: 2020. szept. 21., H, 19:54): Hi All
It is that time of year / release again - and we need to choose the community goals for Wallaby.
Myself and Nate looked over the list of goals [1][2][3], and we are suggesting one of the following:
- Finish moving legacy python-*client CLIs to python-openstackclient
- Move from oslo.rootwrap to oslo.privsep
- Implement the API reference guide changes
- All API to provide a /healthcheck URL like Keystone (and others) provide
Some of these goals have champions signed up already, but we need to make sure they are still available to do them. If you are interested in helping drive any of the goals, please speak up!
We need to select goals in time for the new release cycle - so please reply if there is goals you think should be included in this list, or not included.
Next steps after this will be helping people write a proposed goal and then the TC selecting the ones we will pursue during Wallaby.
Additionally, we have traditionally selected 2 goals per cycle - however with the people available to do the work across projects Nate and I briefly discussed reducing that to one for this cycle.
What does the community think about this?
Thanks,
Graham
1 - https://etherpad.opendev.org/p/community-goals 2 - https://governance.openstack.org/tc/goals/proposed/index.html 3 - https://etherpad.opendev.org/p/community-w-series-goals 4 - https://governance.openstack.org/tc/goals/index.html#goal-selection-schedule
---- On Mon, 21 Sep 2020 12:53:17 -0500 Graham Hayes gr@ham.ie wrote ----
Hi All
It is that time of year / release again - and we need to choose the community goals for Wallaby.
Myself and Nate looked over the list of goals [1][2][3], and we are suggesting one of the following:
Thanks Graham, Nate for starting this.
- Finish moving legacy python-*client CLIs to python-openstackclient
Are not we going with popup team first for osc work? I am fine with goal also but we should do this as multi-cycle goal with no other goal in parallel so that we actually finish this on time.
- Move from oslo.rootwrap to oslo.privsep
+1, this is already proposed goal since last cycle.
-gmann
- Implement the API reference guide changes
- All API to provide a /healthcheck URL like Keystone (and others) provide
Some of these goals have champions signed up already, but we need to make sure they are still available to do them. If you are interested in helping drive any of the goals, please speak up!
We need to select goals in time for the new release cycle - so please reply if there is goals you think should be included in this list, or not included.
Next steps after this will be helping people write a proposed goal and then the TC selecting the ones we will pursue during Wallaby.
Additionally, we have traditionally selected 2 goals per cycle - however with the people available to do the work across projects Nate and I briefly discussed reducing that to one for this cycle.
What does the community think about this?
Thanks,
Graham
1 - https://etherpad.opendev.org/p/community-goals 2 - https://governance.openstack.org/tc/goals/proposed/index.html 3 - https://etherpad.opendev.org/p/community-w-series-goals 4 - https://governance.openstack.org/tc/goals/index.html#goal-selection-schedule
On 24/09/2020 16:03, Ghanshyam Mann wrote:
---- On Mon, 21 Sep 2020 12:53:17 -0500 Graham Hayes gr@ham.ie wrote ----
Hi All
It is that time of year / release again - and we need to choose the community goals for Wallaby.
Myself and Nate looked over the list of goals [1][2][3], and we are suggesting one of the following:
Thanks Graham, Nate for starting this.
- Finish moving legacy python-*client CLIs to python-openstackclient
Are not we going with popup team first for osc work? I am fine with goal also but we should do this as multi-cycle goal with no other goal in parallel so that we actually finish this on time.
Yeah - this was just one of the goals we thought might have some discussion, and we didn't know where the popup team was in their work.
If that work is still on going, we should leave the goal for another cycle or two.
- Move from oslo.rootwrap to oslo.privsep
+1, this is already proposed goal since last cycle.
-gmann
- Implement the API reference guide changes
- All API to provide a /healthcheck URL like Keystone (and others) provide
Some of these goals have champions signed up already, but we need to make sure they are still available to do them. If you are interested in helping drive any of the goals, please speak up!
We need to select goals in time for the new release cycle - so please reply if there is goals you think should be included in this list, or not included.
Next steps after this will be helping people write a proposed goal and then the TC selecting the ones we will pursue during Wallaby.
Additionally, we have traditionally selected 2 goals per cycle - however with the people available to do the work across projects Nate and I briefly discussed reducing that to one for this cycle.
What does the community think about this?
Thanks,
Graham
1 - https://etherpad.opendev.org/p/community-goals 2 - https://governance.openstack.org/tc/goals/proposed/index.html 3 - https://etherpad.opendev.org/p/community-w-series-goals 4 - https://governance.openstack.org/tc/goals/index.html#goal-selection-schedule
On Thu, Sep 24, 2020 at 9:36 AM Graham Hayes gr@ham.ie wrote:
On 24/09/2020 16:03, Ghanshyam Mann wrote:
---- On Mon, 21 Sep 2020 12:53:17 -0500 Graham Hayes gr@ham.ie
wrote ----
Hi All
It is that time of year / release again - and we need to choose the community goals for Wallaby.
Myself and Nate looked over the list of goals [1][2][3], and we are suggesting one of the following:
Thanks Graham, Nate for starting this.
- Finish moving legacy python-*client CLIs to
python-openstackclient
Are not we going with popup team first for osc work? I am fine with goal
also but
we should do this as multi-cycle goal with no other goal in parallel so
that we actually
finish this on time.
Yeah - this was just one of the goals we thought might have some discussion, and we didn't know where the popup team was in their work.
If that work is still on going, we should leave the goal for another cycle or two.
I don't think a *ton* of progress was made this last release, but I could be wrong. I am guessing we will want to wait one more cycle before making it a goal. I am 100% behind this being a goal at some point though.
- Move from oslo.rootwrap to oslo.privsep
+1, this is already proposed goal since last cycle.
-gmann
- Implement the API reference guide changes
- All API to provide a /healthcheck URL like Keystone (and others)
provide
Some of these goals have champions signed up already, but we need to make sure they are still available to do them. If you are interested
in
helping drive any of the goals, please speak up!
We need to select goals in time for the new release cycle - so please reply if there is goals you think should be included in this list, or not included.
Next steps after this will be helping people write a proposed goal and then the TC selecting the ones we will pursue during Wallaby.
Additionally, we have traditionally selected 2 goals per cycle - however with the people available to do the work across projects Nate and I briefly discussed reducing that to one for this cycle.
What does the community think about this?
Thanks,
Graham
1 - https://etherpad.opendev.org/p/community-goals 2 - https://governance.openstack.org/tc/goals/proposed/index.html 3 - https://etherpad.opendev.org/p/community-w-series-goals 4 -
https://governance.openstack.org/tc/goals/index.html#goal-selection-schedule
-Kendall (diablo_rojo)
On 25/09/2020 19:22, Kendall Nelson wrote:
On Thu, Sep 24, 2020 at 9:36 AM Graham Hayes <gr@ham.ie mailto:gr@ham.ie> wrote:
On 24/09/2020 16:03, Ghanshyam Mann wrote: > ---- On Mon, 21 Sep 2020 12:53:17 -0500 Graham Hayes <gr@ham.ie <mailto:gr@ham.ie>> wrote ---- > > Hi All > > > > It is that time of year / release again - and we need to choose the > > community goals for Wallaby. > > > > Myself and Nate looked over the list of goals [1][2][3], and we are > > suggesting one of the following: > > > > > > Thanks Graham, Nate for starting this. > > > - Finish moving legacy python-*client CLIs to python-openstackclient > > Are not we going with popup team first for osc work? I am fine with goal also but > we should do this as multi-cycle goal with no other goal in parallel so that we actually > finish this on time. Yeah - this was just one of the goals we thought might have some discussion, and we didn't know where the popup team was in their work. If that work is still on going, we should leave the goal for another cycle or two.I don't think a *ton* of progress was made this last release, but I could be wrong. I am guessing we will want to wait one more cycle before making it a goal. I am 100% behind this being a goal at some point though.
Yeah, that makes sense to postpone then.
> > - Move from oslo.rootwrap to oslo.privsep > > +1, this is already proposed goal since last cycle. > > -gmann > > > - Implement the API reference guide changes > > - All API to provide a /healthcheck URL like Keystone (and others) provide > > > > Some of these goals have champions signed up already, but we need to > > make sure they are still available to do them. If you are interested in > > helping drive any of the goals, please speak up! > > > > We need to select goals in time for the new release cycle - so please > > reply if there is goals you think should be included in this list, or > > not included. > > > > Next steps after this will be helping people write a proposed goal > > and then the TC selecting the ones we will pursue during Wallaby. > > > > Additionally, we have traditionally selected 2 goals per cycle - > > however with the people available to do the work across projects > > Nate and I briefly discussed reducing that to one for this cycle. > > > > What does the community think about this? > > > > Thanks, > > > > Graham > > > > 1 - https://etherpad.opendev.org/p/community-goals <https://etherpad.opendev.org/p/community-goals> > > 2 - https://governance.openstack.org/tc/goals/proposed/index.html <https://governance.openstack.org/tc/goals/proposed/index.html> > > 3 - https://etherpad.opendev.org/p/community-w-series-goals <https://etherpad.opendev.org/p/community-w-series-goals> > > 4 - > > https://governance.openstack.org/tc/goals/index.html#goal-selection-schedule <https://governance.openstack.org/tc/goals/index.html#goal-selection-schedule> > > > > >-Kendall (diablo_rojo)
On 9/21/20 7:53 PM, Graham Hayes wrote:
Hi All
It is that time of year / release again - and we need to choose the community goals for Wallaby.
Myself and Nate looked over the list of goals [1][2][3], and we are suggesting one of the following:
- Finish moving legacy python-*client CLIs to python-openstackclient
Go go go !!! :)
- Move from oslo.rootwrap to oslo.privsep
Dito. Rootwrap is painfully slow (because it takes too long to spawn a python process...).
- Implement the API reference guide changes - All API to provide a /healthcheck URL like Keystone (and others) provide
What about an "openstack purge <project-name>" that would call all projects? We once had a "/purge" goal, I'm not sure how far it went... What I know, is that purging all resources of a project is currently still a big painpoint.
Some of these goals have champions signed up already, but we need to make sure they are still available to do them. If you are interested in helping drive any of the goals, please speak up!
I'm still available to attempt the /healthcheck thingy, I kind of succeed in all major project but ... nova. Unfortunately, it was decided in the project that we should put this on hold until the /healthcheck can implement more check than just to know if the API is alive. 5 months forward, I believe my original patch [1] should have been approved first as a first approach. Nova team: any reaction? Any progress on your super-nice-health-check? Can this be implemented elsewhere using what you've done? Maybe that work should go in oslo.middleware too?
Cheers,
Thomas Goirand (zigo)
[1] https://review.opendev.org/#/c/724684/
Additionally, we have traditionally selected 2 goals per cycle - however with the people available to do the work across projects Nate and I briefly discussed reducing that to one for this cycle.
What does the community think about this?
The /healthcheck is super-easy to implement for any project using oslo.middleware, so please select that one (and others). It's also mostly done...
Cheers,
Thomas Goirand (zigo)
Hi
On 24. Sep 2020, at 22:39, Thomas Goirand zigo@debian.org wrote:
- Finish moving legacy python-*client CLIs to python-openstackclient
Go go go !!! :)
Cool, that somebody needs that. What we actually miss here is the people doing that. Amount of work is really big (while in the long run it will definitely help reducing overall efforts). My suggestion is to focus for this cycle only on nova (targeting more is simply useless). I have started switching some parts in OSC to use SDK (https://review.opendev.org/#/q/topic:osc-first https://review.opendev.org/#/q/topic:osc-first). Any help (even simply reviews) is welcome. I don’t like summer dev cycle, since the work is really close to 0 due to vacations season. But in autumn/winter I will try do more if there is acceptance.
Honestly speaking I do not believe in a community goal for that any more. It was tried, but it feels there is a split in the community. My personal opinion is that we should simply start doing that from SDK/CLI side closely working with one service at a time (since nova team expressed desire I started supporting them. Hope it will bring its fruits). Teams willing that will be able to achieve the target, teams not willing are free to stay. TC might think differently and act correspondingly, this is just my personal opinion. Other approaches seem a bit of utopia to me.
What about an "openstack purge <project-name>" that would call all projects? We once had a "/purge" goal, I'm not sure how far it went... What I know, is that purging all resources of a project is currently still a big painpoint.
Please have a look at https://review.opendev.org/#/c/734485/ https://review.opendev.org/#/c/734485/ It implements an “alternative” from the OSC point of view by invoking relatively newly introduced project cleanup functionality in SDK. This supports more that original purge (except identity parts), tries to do in parallel as much as possible and even now supports filtering (i.e. drop all resources created or updated before DD.MM.YYYY). I am not sure here whether it should replace purge or come as alternative so far. Here as well - reviews and comments are welcome.
For reference, back in Denver it was agreed to go this way, since no other approach seemed to be really achievable.
Regards, Artem
Hey Artem!
On Fri, Sep 25, 2020 at 2:56 AM Artem Goncharov artem.goncharov@gmail.com wrote:
Hi
On 24. Sep 2020, at 22:39, Thomas Goirand zigo@debian.org wrote:
- Finish moving legacy python-*client CLIs to python-openstackclient
Go go go !!! :)
Cool, that somebody needs that. What we actually miss here is the people doing that. Amount of work is really big (while in the long run it will definitely help reducing overall efforts). My suggestion is to focus for this cycle only on nova (targeting more is simply useless). I have started switching some parts in OSC to use SDK ( https://review.opendev.org/#/q/topic:osc-first). Any help (even simply reviews) is welcome. I don’t like summer dev cycle, since the work is really close to 0 due to vacations season. But in autumn/winter I will try do more if there is acceptance.
I'm resolving now to be better at reviews over the upcoming months! I have been hovering around on the peripherie and I think this release I will be able to get more involved. I plan on attending your PTG sessions and am trying to rally more help for you as well.
Honestly speaking I do not believe in a community goal for that any more. It was tried, but it feels there is a split in the community. My personal opinion is that we should simply start doing that from SDK/CLI side closely working with one service at a time (since nova team expressed desire I started supporting them. Hope it will bring its fruits). Teams willing that will be able to achieve the target, teams not willing are free to stay. TC might think differently and act correspondingly, this is just my personal opinion. Other approaches seem a bit of utopia to me.
What about an "openstack purge <project-name>" that would call all projects? We once had a "/purge" goal, I'm not sure how far it went... What I know, is that purging all resources of a project is currently still a big painpoint.
Please have a look at https://review.opendev.org/#/c/734485/ It implements an “alternative” from the OSC point of view by invoking relatively newly introduced project cleanup functionality in SDK. This supports more that original purge (except identity parts), tries to do in parallel as much as possible and even now supports filtering (i.e. drop all resources created or updated before DD.MM.YYYY). I am not sure here whether it should replace purge or come as alternative so far. Here as well - reviews and comments are welcome.
For reference, back in Denver it was agreed to go this way, since no other approach seemed to be really achievable.
Regards, Artem
Sounds cool, Kendall
---- typed from mobile, auto-correct typos assumed ----
On Fri, 25 Sep 2020, 20:25 Kendall Nelson, kennelson11@gmail.com wrote:
Hey Artem!
On Fri, Sep 25, 2020 at 2:56 AM Artem Goncharov artem.goncharov@gmail.com wrote:
Hi
On 24. Sep 2020, at 22:39, Thomas Goirand zigo@debian.org wrote:
- Finish moving legacy python-*client CLIs to python-openstackclient
Go go go !!! :)
Cool, that somebody needs that. What we actually miss here is the people doing that. Amount of work is really big (while in the long run it will definitely help reducing overall efforts). My suggestion is to focus for this cycle only on nova (targeting more is simply useless). I have started switching some parts in OSC to use SDK ( https://review.opendev.org/#/q/topic:osc-first). Any help (even simply reviews) is welcome. I don’t like summer dev cycle, since the work is really close to 0 due to vacations season. But in autumn/winter I will try do more if there is acceptance.
I'm resolving now to be better at reviews over the upcoming months! I have been hovering around on the peripherie and I think this release I will be able to get more involved. I plan on attending your PTG sessions and am trying to rally more help for you as well.
Honestly speaking I do not believe in a community goal for that any more. It was tried, but it feels there is a split in the community. My personal opinion is that we should simply start doing that from SDK/CLI side closely working with one service at a time (since nova team expressed desire I started supporting them. Hope it will bring its fruits). Teams willing that will be able to achieve the target, teams not willing are free to stay. TC might think differently and act correspondingly, this is just my personal opinion. Other approaches seem a bit of utopia to me.
What about an "openstack purge <project-name>" that would call all projects? We once had a "/purge" goal, I'm not sure how far it went... What I know, is that purging all resources of a project is currently still a big painpoint.
Please have a look at https://review.opendev.org/#/c/734485/ It implements an “alternative” from the OSC point of view by invoking relatively newly introduced project cleanup functionality in SDK. This supports more that original purge (except identity parts), tries to do in parallel as much as possible and even now supports filtering (i.e. drop all resources created or updated before DD.MM.YYYY). I am not sure here whether it should replace purge or come as alternative so far. Here as well - reviews and comments are welcome.
For reference, back in Denver it was agreed to go this way, since no other approach seemed to be really achievable.
Regards, Artem
On Thu, 2020-09-24 at 22:39 +0200, Thomas Goirand wrote:
On 9/21/20 7:53 PM, Graham Hayes wrote:
Hi All
It is that time of year / release again - and we need to choose the community goals for Wallaby.
Myself and Nate looked over the list of goals [1][2][3], and we are suggesting one of the following:
- Finish moving legacy python-*client CLIs to python-openstackclient
Go go go !!! :)
- Move from oslo.rootwrap to oslo.privsep
Dito. Rootwrap is painfully slow (because it takes too long to spawn a python process...).
- Implement the API reference guide changes
- All API to provide a /healthcheck URL like Keystone (and others) provide
What about an "openstack purge <project-name>" that would call all projects? We once had a "/purge" goal, I'm not sure how far it went... What I know, is that purging all resources of a project is currently still a big painpoint.
Some of these goals have champions signed up already, but we need to make sure they are still available to do them. If you are interested in helping drive any of the goals, please speak up!
I'm still available to attempt the /healthcheck thingy, I kind of succeed in all major project but ... nova. Unfortunately, it was decided in the project that we should put this on hold until the /healthcheck can implement more check than just to know if the API is alive. 5 months forward, I believe my original patch [1] should have been approved first as a first approach. Nova team: any reaction?
i actually dont think the healthcheck enpoint is userful in it current from for any porject that has distibuted process like nova, neutron or cinder. that was not the only concern raised either as the default content of the detail responce wich include package infomation was considerd a possible security vulnerablity so with out agreeing on what kindo fo info can be retruned, its format and wether this would be a admin only endpoint or a public endpoint tenant can check potentially without auth i dont think we should be procedding with this as a goal.
https://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/hea... ^ is concerning from a security point of view.
nova support configurable middelway still through the api-paste.ini file so untill we have a useful health check im not sure we should merge anything since operators can just enable it themselves if they want too.
Any progress on your super-nice-health-check? Can this be implemented elsewhere using what you've done? Maybe that work should go in oslo.middleware too?
Cheers,
Thomas Goirand (zigo)
[1] https://review.opendev.org/#/c/724684/
Additionally, we have traditionally selected 2 goals per cycle - however with the people available to do the work across projects Nate and I briefly discussed reducing that to one for this cycle.
What does the community think about this?
The /healthcheck is super-easy to implement for any project using oslo.middleware, so please select that one (and others). It's also mostly done...
Cheers,
Thomas Goirand (zigo)
Hi Sean,
Thanks for your reply and sharing your concerns. I still don't agree with you, and here's why.
On 9/30/20 12:29 PM, Sean Mooney wrote:
On Thu, 2020-09-24 at 22:39 +0200, Thomas Goirand wrote:
I'm still available to attempt the /healthcheck thingy, I kind of succeed in all major project but ... nova. Unfortunately, it was decided in the project that we should put this on hold until the /healthcheck can implement more check than just to know if the API is alive. 5 months forward, I believe my original patch [1] should have been approved first as a first approach. Nova team: any reaction?
i actually dont think the healthcheck enpoint is userful in it current from for any porject that has distibuted process like nova, neutron or cinder.
With my operator's hat on, I assure you that it is very useful to wire-up haproxy. This is btw exactly what it was designed for.
The way haproxy works, is that it actually has to perform an http connection to check if the web service is alive. Without a specific URL, we can't filter-out that /healthcheck URL from the saved logs in Elastic search, which is very annoying.
Not doing a real HTTP check means one falls back to a TCP check, which means that your logs are polluted with so many "client disconnected unexpectedly" (that's not the actual message, but that's what it is doing) since haproxy otherwise does a TCP connection, then closes it before what a normal HTTP query would do.
I've been told to use other URLs, which isn't the way to go. I very much think the /healthcheck URL was well designed, and should be used.
that was not the only concern raised either as the default content of the detail responce wich include package infomation was considerd a possible security vulnerablity so with out agreeing on what kindo fo info can be retruned, its format and wether this would be a admin only endpoint or a public endpoint tenant can check potentially without auth i dont think we should be procedding with this as a goal.
Are you aware that one could simply use "/" of most projects without auth, and get the version of the project? Example with Nova:
curl -s https://api.example.com/compute/ | jq .
{ "versions": [ { "id": "v2.0", "status": "SUPPORTED", "version": "", "min_version": "", "updated": "2011-01-21T11:33:21Z", "links": [ { "rel": "self", "href": "https://api.example.com/v2/" } ] }, { "id": "v2.1", "status": "CURRENT", "version": "2.79", "min_version": "2.1", "updated": "2013-07-23T11:33:21Z", "links": [ { "rel": "self", "href": "https://clint1-api.cloud.infomaniak.ch/v2.1/" } ] } ] }
I believe "version": "2.79" is the microversion of the Nova API, which therefore, exposes what version of Nova (here: Train). Am I correct? I believe we also must leave this, because clients must be able to discover the micro-version of the API, right?
Then, if I'm right, isn't this a much more annoying problem than having a /healtcheck URL which could anyway be filtered by an HAProxy in front? (note: I don't think the above is really a problem, since the micro-version doesn't tell if Nova has been patched for security or not...)
https://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/hea... ^ is concerning from a security point of view.
This needs to be explicitly enabled in the api-paste.ini, and is otherwise not displayed. Here's what I get in my Train deployment with the query suggested in the example you gave:
{ "detailed": false, "reasons": [] }
The code in here: https://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/hea...
shows I'm right. That's also well documented here: https://docs.openstack.org/oslo.middleware/latest/reference/healthcheck_plug...
See where it says this:
# set to True to enable detailed output, False is the default detailed = False
in the api-paste.ini example.
So here, the point you are making isn't IMO valid.
nova support configurable middelway still through the api-paste.ini file so untill we have a useful health check
As we discussed earlier, what is *not* useful, is any healthcheck that would do more than what oslo.middleware does right now without caching things, because that /healthcheck URL is typically called multiple times per seconds (in our deployment: at least 6 times per seconds), so it needs to reply fast. So I hope that the super-nice-healthcheck thingy wont fry my server CPUs ... :)
What is *not* useful as well, is delaying such a trivial patch for more than 6 months, just in the hope that in a distant future, we may have something better.
Sure, take your time, get something implemented that does a nice healtcheck with db access and rabbitmq connectivity checks. But that should in no way get in the path of having a configuration which works for everyone by default.
Cheers,
Thomas Goirand (zigo)
On Wed, 2020-09-30 at 22:08 +0200, Thomas Goirand wrote:
Hi Sean,
Thanks for your reply and sharing your concerns. I still don't agree with you, and here's why.
On 9/30/20 12:29 PM, Sean Mooney wrote:
On Thu, 2020-09-24 at 22:39 +0200, Thomas Goirand wrote:
I'm still available to attempt the /healthcheck thingy, I kind of succeed in all major project but ... nova. Unfortunately, it was decided in the project that we should put this on hold until the /healthcheck can implement more check than just to know if the API is alive. 5 months forward, I believe my original patch [1] should have been approved first as a first approach. Nova team: any reaction?
i actually dont think the healthcheck enpoint is userful in it current from for any porject that has distibuted process like nova, neutron or cinder.
With my operator's hat on, I assure you that it is very useful to wire-up haproxy. This is btw exactly what it was designed for.
The way haproxy works, is that it actually has to perform an http connection to check if the web service is alive. Without a specific URL, we can't filter-out that /healthcheck URL from the saved logs in Elastic search, which is very annoying.
cant you just hit the root of the service? that is unauthenticated for microversion version discovery so haproxy could simple use / for a http check if its just bing used to test if the rest api is running.
Not doing a real HTTP check means one falls back to a TCP check, which means that your logs are polluted with so many "client disconnected unexpectedly" (that's not the actual message, but that's what it is doing) since haproxy otherwise does a TCP connection, then closes it before what a normal HTTP query would do.
I've been told to use other URLs, which isn't the way to go. I very much think the /healthcheck URL was well designed, and should be used.
that was not the only concern raised either as the default content of the detail responce wich include package infomation was considerd a possible security vulnerablity so with out agreeing on what kindo fo info can be retruned, its format and wether this would be a admin only endpoint or a public endpoint tenant can check potentially without auth i dont think we should be procedding with this as a goal.
Are you aware that one could simply use "/" of most projects without auth, and get the version of the project? Example with Nova:
curl -s https://api.example.com/compute/ | jq .
{ "versions": [ { "id": "v2.0", "status": "SUPPORTED", "version": "", "min_version": "", "updated": "2011-01-21T11:33:21Z", "links": [ { "rel": "self", "href": "https://api.example.com/v2/" } ] }, { "id": "v2.1", "status": "CURRENT", "version": "2.79", "min_version": "2.1", "updated": "2013-07-23T11:33:21Z", "links": [ { "rel": "self", "href": "https://clint1-api.cloud.infomaniak.ch/v2.1/" } ] } ] }
yes so this is the endpoint i would expect peopel to use as an alternitive to /healtcheck. im not suggesting we do not
I believe "version": "2.79" is the microversion of the Nova API, which therefore, exposes what version of Nova (here: Train). Am I correct?
no you are not. it does not expose the package infomation it tells you the make microversion the api support but that is a different thing. we dont always bump the microversion in a release. ussuri and victoria but share the same microversion https://docs.openstack.org/nova/latest/reference/api-microversion-history.ht...
the microverion also wont change on a stable branch no matter what bugs exist or have been patched.
I believe we also must leave this, because clients must be able to discover the micro-version of the API, right?
yes without this no client can determin what api version is supported by a specific cloud. this is intened to be a public endpoint with no auth for that reason.
Then, if I'm right, isn't this a much more annoying problem than having a /healtcheck URL which could anyway be filtered by an HAProxy in front?
im not following this was intended to be public form the start and does not ahve the same issue as the health check api
(note: I don't think the above is really a problem, since the micro-version doesn't tell if Nova has been patched for security or not...)
correct its not and it is the endpoint that i would suggest using in the absence of a /healtcheck endpoint until we ca develop one that actuly reports the healt of the service and not just the api.
https://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/hea... ^ is concerning from a security point of view.
This needs to be explicitly enabled in the api-paste.ini, and is otherwise not displayed. Here's what I get in my Train deployment with the query suggested in the example you gave:
{ "detailed": false, "reasons": [] }
The code in here: https://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/hea...
shows I'm right. That's also well documented here: https://docs.openstack.org/oslo.middleware/latest/reference/healthcheck_plug...
See where it says this:
# set to True to enable detailed output, False is the default detailed = False
in the api-paste.ini example.
So here, the point you are making isn't IMO valid.
nova support configurable middelway still through the api-paste.ini file so untill we have a useful health check
As we discussed earlier, what is *not* useful, is any healthcheck that would do more than what oslo.middleware does right now without caching things, because that /healthcheck URL is typically called multiple times per seconds (in our deployment: at least 6 times per seconds), so it needs to reply fast. So I hope that the super-nice-healthcheck thingy wont fry my server CPUs ... :)
what we were thinking was basically checking that the from the api that services that is handeling the request we would confirm the db and message bus were accessable, and that api instance could reach the conductor and maybe the schedler services or assert tehy were active via a db check.
if the generic oslo ping rpc was added we coudl use that but i think dansmith had a simpler proposal for caching it based on if we were able to connect during normal operation and jsut have the api check look at teh in memeory value. i.e. if the last attempt to read form the db failed to connect we would set a global variable e.g. DB_ACCESSIBLE=FALSE and then the next time it succeded we set it to True. the health check woudl just read the global so there should be little to no overhead vs what oslo does
this would basically cache the last knon state and the health check is just doing the equivalent of return DB_ACCESSIBLE and RPC_ACCESSIBLE
if detail=true was set it could do a more advanced check and check the service status exctra which would be a dbquery but detail=false is just a memory read of two variables.
What is *not* useful as well, is delaying such a trivial patch for more than 6 months, just in the hope that in a distant future, we may have something better.
but as you yourself pointed out almost every service has a / enpoint that is used for microverion discovery that is public so not implementing /healtcheck in nova does not block you using / as the healthcheck url and you can enable the oslo endpoint if you chose too by enable the middle ware in your local deployment.
Sure, take your time, get something implemented that does a nice healtcheck with db access and rabbitmq connectivity checks. But that should in no way get in the path of having a configuration which works for everyone by default.
there is nothing stopping install tools providing that experience by default today. at least as long as nova support configurable middleware they can enable or even enable the /healthcheck endpoint by default without requiring nova code change. i have looked at enough customer bug to know that network partions are common in real envionments where someone trying to use the /healthcheck endpoint to know if nova is healty would be severly disapointed when it says its healty and they cant boot any vms because rabbitmq is not reachable. for usecause outside fo haproxy failover a bad health check is arguable worse then no healthcheck.
im not unsympathic to your request but with what oslo does by default we would basically have to document that this should not be used to monitor the healthy of the nova service to prempt the bug reports we would get from customers related to this. we have already got several bug reports to the status of vm not matching reality when connectivity to the cell is down. e.g. when we cant connect to the cell database if the vm is stoped say via a power off vis ssh then its state will not be reflected in a nova show.
if we were willing to add a big warning and clearly call out that this is just saying the api is accesable but not necessarily functional then i would be more ok with what olso provides but it does not tell you anything about the health of nova or if any other api request will actually work.
i would suggest adding this to the nova ptg etherpad if you want to move this forward in nova in particular.
Cheers,
Thomas Goirand (zigo)
On 10/1/20 12:13 AM, Sean Mooney wrote:
cant you just hit the root of the service? that is unauthenticated for microversion version discovery so haproxy could simple use / for a http check if its just bing used to test if the rest api is running.
How will I make the difference, in my logs, between a client hitting / for microversion discovery, and haproxy doing an healthcheck?
I believe "version": "2.79" is the microversion of the Nova API, which therefore, exposes what version of Nova (here: Train). Am I correct?
no you are not. it does not expose the package infomation it tells you the make microversion the api support but that is a different thing. we dont always bump the microversion in a release. ussuri and victoria but share the same microversion https://docs.openstack.org/nova/latest/reference/api-microversion-history.ht...
the microverion also wont change on a stable branch no matter what bugs exist or have been patched.
Right, but this still tells what version of OpenStack is installed. Maybe Nova hasn't bumped it, but one could check multiple services, and find out what version of OpenStack is there. This is still a problem, at least more than what you've described with the /healthcheck URL.
believe we also must leave this, because clients must be able to discover the micro-version of the API, right?
yes without this no client can determin what api version is supported by a specific cloud. this is intened to be a public endpoint with no auth for that reason.
That part I don't understand. Couldn't this be an authenticated thing?
if the generic oslo ping rpc was added we coudl use that but i think dansmith had a simpler proposal for caching it based on if we were able to connect during normal operation and jsut have the api check look at teh in memeory value. i.e. if the last attempt to read form the db failed to connect we would set a global variable e.g. DB_ACCESSIBLE=FALSE and then the next time it succeded we set it to True. the health check woudl just read the global so there should be little to no overhead vs what oslo does
this would basically cache the last knon state and the health check is just doing the equivalent of return DB_ACCESSIBLE and RPC_ACCESSIBLE
That's a good idea, but my patch has been sitting as rejected for the last 5 months. It could be in Victoria already...
What is *not* useful as well, is delaying such a trivial patch for more than 6 months, just in the hope that in a distant future, we may have something better.
but as you yourself pointed out almost every service has a / enpoint that is used for microverion discovery that is public so not implementing /healtcheck in nova does not block you using / as the healthcheck url and you can enable the oslo endpoint if you chose too by enable the middle ware in your local deployment.
As I wrote above: that's *not* a good idea.
Sure, take your time, get something implemented that does a nice healtcheck with db access and rabbitmq connectivity checks. But that should in no way get in the path of having a configuration which works for everyone by default.
there is nothing stopping install tools providing that experience by default today.
Indeed, and I'm doing it. However, it's very annoying to rebase such a patch on every single release of OpenStack, just like with every other patch that the Debian packages are carying.
at least as long as nova support configurable middleware they can enable or even enable the /healthcheck endpoint by default without requiring nova code change. i have looked at enough customer bug to know that network partions are common in real envionments where someone trying to use the /healthcheck endpoint to know if nova is healty would be severly disapointed when it says its healty and they cant boot any vms because rabbitmq is not reachable.
We discussed this already, and we all agree that the name of the URL was a bad choice. It could have been called /reverse-proxycheck instead. But that's too late and changing that name would break users, unfortunately. If we want to rename it, then we must follow a cycle of deprecation.
However, I don't really care if some are disappointed because they can't read the doc or the code, and just miss-guess because of the URL name. We need this thing for HA setup, it is easy to get installed, so why not do it? The URL name mistake cannot be a blocker.
usecause outside fo haproxy failover a bad health check is arguable worse then no healthcheck.
I never wrote that I don't want a better health check. Just that the one we have is already useful, and that it should be on by default.
im not unsympathic to your request but with what oslo does by default we would basically have to document that this should not be used to monitor the healthy of the nova service
What it does is already documented. If you want to add more in the documentation, please do so.
we have already got several bug reports to the status of vm not matching reality when connectivity to the cell is down. e.g. when we cant connect to the cell database if the vm is stoped say via a power off vis ssh then its state will not be reflected in a nova show.
This has nothing to do with what we're currently discussing, which is having an URL to wire-in haproxy checks.
if we were willing to add a big warning and clearly call out that this is just saying the api is accesable but not necessarily functional then i would be more ok with what olso provides but it does not tell you anything about the health of nova or if any other api request will actually work.
https://review.opendev.org/755433
i would suggest adding this to the nova ptg etherpad if you want to move this forward in nova in particular.
I would suggest not discussing the mater too much, and actually doing something about it. :)
It has been discussed already for a way too long.
Cheers,
Thomas Goirand (zigo)
So, we have had a single goal proposed that does not have push back, or limitations - the privsep migration.
As this may be a challenging goal, I think doing this one on its own for the cycle is a good idea.
Rodolfo has agreed to continue championing this goal, so I have proposed moving the goal from the proposed to selected folder[1]
1 - https://review.opendev.org/755590
Thanks all!
- Graham
On 21/09/2020 18:53, Graham Hayes wrote:
Hi All
It is that time of year / release again - and we need to choose the community goals for Wallaby.
Myself and Nate looked over the list of goals [1][2][3], and we are suggesting one of the following:
- Finish moving legacy python-*client CLIs to python-openstackclient - Move from oslo.rootwrap to oslo.privsep - Implement the API reference guide changes - All API to provide a /healthcheck URL like Keystone (and others) provide
Some of these goals have champions signed up already, but we need to make sure they are still available to do them. If you are interested in helping drive any of the goals, please speak up!
We need to select goals in time for the new release cycle - so please reply if there is goals you think should be included in this list, or not included.
Next steps after this will be helping people write a proposed goal and then the TC selecting the ones we will pursue during Wallaby.
Additionally, we have traditionally selected 2 goals per cycle - however with the people available to do the work across projects Nate and I briefly discussed reducing that to one for this cycle.
What does the community think about this?
Thanks,
Graham
1 - https://etherpad.opendev.org/p/community-goals 2 - https://governance.openstack.org/tc/goals/proposed/index.html 3 - https://etherpad.opendev.org/p/community-w-series-goals 4 - https://governance.openstack.org/tc/goals/index.html#goal-selection-schedule
Hi Everybody,
Sorry for the top post, and resurrecting an old thread :)
We have had a second proposal for a goal for the Wallaby cycle[1] - and the TC is likely to accept and add it as a second goal for the cycle.
Looking at the work required, it was felt it is a smaller burden than other goals we have had, and enables us to move forward with some of the important policy work, and deprecate the JSON format for policy definition.
We wanted to let teams know about the upcoming change, and if there is any objection, please chime in here, or on the review.
Thanks for reading!
- Graham
1 - https://review.opendev.org/#/c/759881
On 21/09/2020 18:53, Graham Hayes wrote:
Hi All
It is that time of year / release again - and we need to choose the community goals for Wallaby.
Myself and Nate looked over the list of goals [1][2][3], and we are suggesting one of the following:
- Finish moving legacy python-*client CLIs to python-openstackclient - Move from oslo.rootwrap to oslo.privsep - Implement the API reference guide changes - All API to provide a /healthcheck URL like Keystone (and others) provide
Some of these goals have champions signed up already, but we need to make sure they are still available to do them. If you are interested in helping drive any of the goals, please speak up!
We need to select goals in time for the new release cycle - so please reply if there is goals you think should be included in this list, or not included.
Next steps after this will be helping people write a proposed goal and then the TC selecting the ones we will pursue during Wallaby.
Additionally, we have traditionally selected 2 goals per cycle - however with the people available to do the work across projects Nate and I briefly discussed reducing that to one for this cycle.
What does the community think about this?
Thanks,
Graham
1 - https://etherpad.opendev.org/p/community-goals 2 - https://governance.openstack.org/tc/goals/proposed/index.html 3 - https://etherpad.opendev.org/p/community-w-series-goals 4 - https://governance.openstack.org/tc/goals/index.html#goal-selection-schedule
participants (7)
-
Artem Goncharov
-
Ghanshyam Mann
-
Graham Hayes
-
Kendall Nelson
-
Lajos Katona
-
Sean Mooney
-
Thomas Goirand