---- On Fri, 04 Oct 2024 09:12:13 -0700 Artem Goncharov wrote ---

Hey Stephen

On Friday, 4 October 2024 15:15:08 GMT+2 Stephen Finucane wrote:

...

o/

I've proposed a series of patches to "clean up" the service catalog that DevStack creates in a deployment. Currently, DevStack creates two endpoints for Cinder: one called 'cinderv3' of type 'volumev3' and another called 'cinder' of type 'block-storage'. For example:

{ "services": [ { "name": "cinderv3", "description": "Cinder Volume Service V3", "id": "65460706a9864e71926f13042f526aeb", "type": "volumev3", "enabled": true, "links": { "self": "http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb" } }, { "name": "cinder", "description": "Cinder Volume Service", "id": "cd406a0e296d47dca1e481e992851bb2", "type": "block-storage", "enabled": true, "links": { "self": "http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2" } }, // ... ], "links": { "next": null, "self": "http://10.0.108.50/identity/v3/services", "previous": null } }

They both ultimately point to the same API, the Cinder v3 API, meaning one of these endpoints is redundant. My assumption was 'block-storage' was the correct one, since that's the official service type [1], and that 'cinder' was the appropriate name since it mirrors what we use for other services (nova, neutron, keystone, glance, ...). However, tkajinam raised the point [2] that the Install Guide currently recommends use of the 'volumev3' type and that projects like Nova and Glance have an '[cinder] catalog_info' that defaults to 'volumev3'. If so, is 'block-storage' really the official service type, or is it 'volumev3'. If it is, should we be changing the Install Guide and the various config option defaults?

Previously we had volumev2 and volumev3. Those are both versioned aliases to the block-storage as the official service type name (that is what structure of the service-types-authority claims). Now there is no volumev2 and thus the confusion grows. I suggest looking at the rendered json at [2] which states: ``` "cinder": { "aliases": [ "volumev3", "volumev2", "volume", "block-store" ], "api_reference": "https://docs.openstack.org/api-ref/block-storage/", "project": "cinder", "service_type": "block-storage" }, ```

Catalog consumption process is in detail described at [3]. It is insanely complex and confusing but that is what all tools that look at the service catalog do.

Overall: - official and unique service_type of the cinder is block-storage - volumev3 is an alias pointing to explicit v3 api of block-storage - there was an "idea" that once a service exposing multiple API versions (and those are exposed independently with version-suffixed service types) there is an unversioned one pointing to the "latest" version (therefore typically you also find `volume` service pointing to the v3 - technically install docs should stop suggesting registering volumev3 including project_id suffix. This is in detail addressed by the service catalog consumption procedure and in addition blosk-storage v3 api has a series of APIs without project_id. Continuing appending project_id in the service catalog is heavily harmful. - install docs may switch to registering `block-storage` or `volume` instead of volumev3 and even stop pointing it to the v3 api explicitly since version discovery is there to figure out all the aspects.

With the API version changing and just to keep backward compatibility of devstack among tooling, we kept modifying the catalog name in this way and ended up with more confusion. I agree to make the cinder catalogue also consistent with others and name them only 'blosk-storage'. Tempest still uses 'volumev3' as the default cinder service catalog, but that is something we should change as first: - https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee8... -gmann

...

[1] https://specs.openstack.org/openstack/service-types-authority/

[2] https://service-types.openstack.org/service-types.json [3]

IMHO changing the default value now is not a good option here. If we change the default value in tempest, nova, glance (and so on, if exists) it means that deployment would break during upgrade unless users explicitly set these options to the legacy value or update keystone endpoints to use the new type. This sounds like a quite breaking change which makes upgrade of real deployments difficult. May I know if there is any clear downside of using 'volumev3' ? To be honest based on the fact that "volumev3" has been used as default and mentioned in docs for very long time, I find it little valuable while it has big use impact. On 10/5/24 01:50, Ghanshyam Mann wrote:

---- On Fri, 04 Oct 2024 09:48:25 -0700 Ghanshyam Mann wrote ---

...

---- On Fri, 04 Oct 2024 09:12:13 -0700 Artem Goncharov wrote ---

...

Hey Stephen

On Friday, 4 October 2024 15:15:08 GMT+2 Stephen Finucane wrote:

...

o/

I've proposed a series of patches to "clean up" the service catalog that DevStack creates in a deployment. Currently, DevStack creates two endpoints for Cinder: one called 'cinderv3' of type 'volumev3' and another called 'cinder' of type 'block-storage'. For example:

{ "services": [ { "name": "cinderv3", "description": "Cinder Volume Service V3", "id": "65460706a9864e71926f13042f526aeb", "type": "volumev3", "enabled": true, "links": { "self": "http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb" } }, { "name": "cinder", "description": "Cinder Volume Service", "id": "cd406a0e296d47dca1e481e992851bb2", "type": "block-storage", "enabled": true, "links": { "self": "http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2" } }, // ... ], "links": { "next": null, "self": "http://10.0.108.50/identity/v3/services", "previous": null } }

They both ultimately point to the same API, the Cinder v3 API, meaning one of these endpoints is redundant. My assumption was 'block-storage' was the correct one, since that's the official service type [1], and that 'cinder' was the appropriate name since it mirrors what we use for other services (nova, neutron, keystone, glance, ...). However, tkajinam raised the point [2] that the Install Guide currently recommends use of the 'volumev3' type and that projects like Nova and Glance have an '[cinder] catalog_info' that defaults to 'volumev3'. If so, is 'block-storage' really the official service type, or is it 'volumev3'. If it is, should we be changing the Install Guide and the various config option defaults?

Previously we had volumev2 and volumev3. Those are both versioned aliases to the block-storage as the official service type name (that is what structure of the service-types-authority claims). Now there is no volumev2 and thus the confusion grows. I suggest looking at the rendered json at [2] which states: ``` "cinder": { "aliases": [ "volumev3", "volumev2", "volume", "block-store" ], "api_reference": "https://docs.openstack.org/api-ref/block-storage/", "project": "cinder", "service_type": "block-storage" }, ```

Catalog consumption process is in detail described at [3]. It is insanely complex and confusing but that is what all tools that look at the service catalog do.

Overall: - official and unique service_type of the cinder is block-storage - volumev3 is an alias pointing to explicit v3 api of block-storage - there was an "idea" that once a service exposing multiple API versions (and those are exposed independently with version-suffixed service types) there is an unversioned one pointing to the "latest" version (therefore typically you also find `volume` service pointing to the v3 - technically install docs should stop suggesting registering volumev3 including project_id suffix. This is in detail addressed by the service catalog consumption procedure and in addition blosk-storage v3 api has a series of APIs without project_id. Continuing appending project_id in the service catalog is heavily harmful. - install docs may switch to registering `block-storage` or `volume` instead of volumev3 and even stop pointing it to the v3 api explicitly since version discovery is there to figure out all the aspects.

With the API version changing and just to keep backward compatibility of devstack among tooling, we kept modifying the catalog name in this way and ended up with more confusion.

I agree to make the cinder catalogue also consistent with others and name them only 'blosk-storage'.

Tempest still uses 'volumev3' as the default cinder service catalog, but that is something we should change as first:

- https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee8...

Just saw that you already proposed that. thanks ++

https://review.opendev.org/c/openstack/tempest/+/930296

-gmann

...

-gmann

...

...

[1] https://specs.openstack.org/openstack/service-types-authority/

[2] https://service-types.openstack.org/service-types.json [3]

On 10/5/24 02:03, Artem Goncharov wrote:

On Fri, Oct 4, 2024, 18:56 Takashi Kajinami <kajinamit@oss.nttdata.com <mailto:kajinamit@oss.nttdata.com>> wrote:

IMHO changing the default value now is not a good option here. If we change the default value in tempest, nova, glance (and so on, if exists) it means that deployment would break during upgrade unless users explicitly set these options to the legacy value or update keystone endpoints to use the new type. This sounds like a quite breaking change which makes upgrade of real deployments difficult.

May I know if there is any clear downside of using 'volumev3' ? To be honest based on the fact that "volumev3" has been used as default and mentioned in docs for very long time, I find it little valuable while it has big use impact.

There should be no impact at all as long as keystoneauth lib is used which implements service/version discovery. If services hard coded service type of cinder without implementing discovery it is clearly a bug and must be fixed instead of sticking to keeping bugs forever what we do so often.

I'm not following this. Even if we use keystoneauth, we should know the service type to look up the correct endpoint from the list. Or are you saying that we can determine the endpoint for cinder without service_type field ? There are number of implementations which hard-code volumev3 or use the volumev3 as default. https://codesearch.opendev.org/?q=volumev3&i=nope&literal=nope&files=&excludeFiles=&repos= examples: https://opendev.org/openstack/horizon/src/branch/master/openstack_dashboard/... https://opendev.org/openstack/heat/src/branch/master/heat/engine/clients/os/... https://opendev.org/openstack/ceilometer/src/branch/master/ceilometer/volume... We may be need to make these at least configurable.

In addition to that a split between upgrade vs. new install should be implemented (install docs should explicitly recommend proper usage while existing installations may stick to their current configuration). Moreover nothing prevents adding multiple values to the catalog and drop older ones once all consumers are updated to follow the discovery procedure.

I agree we can add multiple service types and endpoints for the same service. However my main concern is that not all users may be aware of the required steps to avoid inconsistency caused by such change of default behavior. I can easily imagine that users upgrade their deployment and now their services require block-storage service type without noticing they should create that "new" service type.

On 10/5/24 01:50, Ghanshyam Mann wrote: >   ---- On Fri, 04 Oct 2024 09:48:25 -0700  Ghanshyam Mann  wrote --- >   >  ---- On Fri, 04 Oct 2024 09:12:13 -0700  Artem Goncharov  wrote --- >   >  > Hey Stephen >   >  > >   >  > On Friday, 4 October 2024 15:15:08 GMT+2 Stephen Finucane wrote: >   >  > > o/ >   >  > > >   >  > > I've proposed a series of patches to "clean up" the service catalog that >   >  > > DevStack creates in a deployment. Currently, DevStack creates two endpoints >   >  > > for Cinder: one called 'cinderv3' of type 'volumev3' and another called >   >  > > 'cinder' of type 'block-storage'. For example: >   >  > > >   >  > >     { >   >  > >         "services": >   >  > >         [ >   >  > >             { >   >  > >                 "name": "cinderv3", >   >  > >                 "description": "Cinder Volume Service V3", >   >  > >                 "id": "65460706a9864e71926f13042f526aeb", >   >  > >                 "type": "volumev3", >   >  > >                 "enabled": true, >   >  > >                 "links": >   >  > >                 { >   >  > >                     "self": >   >  > > "http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb <http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb>" >   >  > > } >   >  > >             }, >   >  > >             { >   >  > >                 "name": "cinder", >   >  > >                 "description": "Cinder Volume Service", >   >  > >                 "id": "cd406a0e296d47dca1e481e992851bb2", >   >  > >                 "type": "block-storage", >   >  > >                 "enabled": true, >   >  > >                 "links": >   >  > >                 { >   >  > >                     "self": >   >  > > "http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2 <http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2>" >   >  > > } >   >  > >             }, >   >  > >             // ... >   >  > >         ], >   >  > >         "links": >   >  > >         { >   >  > >             "next": null, >   >  > >             "self": "http://10.0.108.50/identity/v3/services <http://10.0.108.50/identity/v3/services>", >   >  > >             "previous": null >   >  > >         } >   >  > >     } >   >  > > >   >  > > They both ultimately point to the same API, the Cinder v3 API, meaning one >   >  > > of these endpoints is redundant. My assumption was 'block-storage' was the >   >  > > correct one, since that's the official service type [1], and that 'cinder' >   >  > > was the appropriate name since it mirrors what we use for other services >   >  > > (nova, neutron, keystone, glance, ...). However, tkajinam raised the point >   >  > > [2] that the Install Guide currently recommends use of the 'volumev3' type >   >  > > and that projects like Nova and Glance have an '[cinder] catalog_info' that >   >  > > defaults to 'volumev3'. If so, is 'block-storage' really the official >   >  > > service type, or is it 'volumev3'. If it is, should we be changing the >   >  > > Install Guide and the various config option defaults? >   >  > > >   >  > >   >  > Previously we had volumev2 and volumev3. Those are both versioned aliases to >   >  > the block-storage as the official service type name (that is what structure of >   >  > the service-types-authority claims). Now there is no volumev2 and thus the >   >  > confusion grows. >   >  > I suggest looking at the rendered json at [2] which states: >   >  > ``` >   >  >     "cinder": { >   >  >       "aliases": [ >   >  >         "volumev3", >   >  >         "volumev2", >   >  >         "volume", >   >  >         "block-store" >   >  >       ], >   >  >       "api_reference": "https://docs.openstack.org/api-ref/block-storage/ <https://docs.openstack.org/api-ref/block-storage/>", >   >  >       "project": "cinder", >   >  >       "service_type": "block-storage" >   >  >     }, >   >  > ``` >   >  > >   >  > Catalog consumption process is in detail described at [3]. It is insanely >   >  > complex and confusing but that is what all tools that look at the service >   >  > catalog do. >   >  > >   >  > Overall: >   >  > - official and unique service_type of the cinder is block-storage >   >  > - volumev3 is an alias pointing to explicit v3 api of block-storage >   >  > - there was an "idea" that once a service exposing multiple API versions (and >   >  > those are exposed independently with version-suffixed service types) there is an >   >  > unversioned one pointing to the "latest" version (therefore typically you also >   >  > find `volume` service pointing to the v3 >   >  > - technically install docs should stop suggesting registering volumev3 >   >  > including project_id suffix. This is in detail addressed by the service catalog >   >  > consumption procedure and in addition blosk-storage v3 api has a series of >   >  > APIs without project_id. Continuing appending project_id in the service >   >  > catalog is heavily harmful. >   >  > - install docs may switch to registering `block-storage` or `volume` instead >   >  > of volumev3 and even stop pointing it to the v3 api explicitly since version >   >  > discovery is there to figure out all the aspects. >   > >   > With the API version changing and just to keep backward compatibility of devstack >   > among tooling, we kept modifying the catalog name in this way and ended up with more >   > confusion. >   > >   > I agree to make the cinder catalogue also consistent with others and name them only >   > 'blosk-storage'. >   > >   > Tempest still uses 'volumev3' as the default cinder service catalog, but that is something we >   > should change as first: >   > >   > - https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee8... <https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee8cffe39/tempest/config.py#L981> > > Just saw that you already proposed that. thanks ++ > > https://review.opendev.org/c/openstack/tempest/+/930296 <https://review.opendev.org/c/openstack/tempest/+/930296> > > -gmann > >   > >   > -gmann >   >  > >   >  > > >   >  > > [1]

This procedure is followed in keystoneauth, openstacksdk, rust SDK/cli so no users should be affected unless they explicitly ignored official rules. ---- typed from mobile, auto-correct typos assumed ---- On Fri, Oct 4, 2024, 19:23 Artem Goncharov <artem.goncharov@gmail.com> wrote:

Official procedure is described at https://specs.openstack.org/openstack/api-sig/guidelines/consuming-catalog.h...

You know the service that you want to talk to and there is a list of service types that the service supports. You follow the procedure to find the one set by your cloud.

---- typed from mobile, auto-correct typos assumed ----

On Fri, Oct 4, 2024, 19:19 Takashi Kajinami <kajinamit@oss.nttdata.com> wrote:

...

On 10/5/24 02:03, Artem Goncharov wrote:

...

On Fri, Oct 4, 2024, 18:56 Takashi Kajinami <kajinamit@oss.nttdata.com

<mailto:kajinamit@oss.nttdata.com>> wrote:

...

IMHO changing the default value now is not a good option here. If

we change the default value

...

in tempest, nova, glance (and so on, if exists) it means that

deployment would break during upgrade

...

unless users explicitly set these options to the legacy value or

update keystone endpoints to use

...

the new type. This sounds like a quite breaking change which makes upgrade of

real deployments difficult.

...

May I know if there is any clear downside of using 'volumev3' ? To

be honest based on the fact that

...

"volumev3" has been used as default and mentioned in docs for very

long time, I find it little valuable

...

while it has big use impact.

There should be no impact at all as long as keystoneauth lib is used

which implements service/version discovery.

...

If services hard coded service type of cinder without implementing discovery it is clearly a bug and must be fixed instead of sticking to keeping bugs forever what we do so often.

I'm not following this. Even if we use keystoneauth, we should know the service type to look up the correct endpoint from the list. Or are you saying that we can determine the endpoint for cinder without service_type field ?

There are number of implementations which hard-code volumev3 or use the volumev3 as default.

https://codesearch.opendev.org/?q=volumev3&i=nope&literal=nope&files=&excludeFiles=&repos=

examples:

https://opendev.org/openstack/horizon/src/branch/master/openstack_dashboard/...

https://opendev.org/openstack/heat/src/branch/master/heat/engine/clients/os/...

https://opendev.org/openstack/ceilometer/src/branch/master/ceilometer/volume...

We may be need to make these at least configurable.

...

In addition to that a split between upgrade vs. new install should be implemented (install docs should explicitly recommend proper usage while existing installations may stick to their current configuration). Moreover nothing prevents adding multiple values to the catalog and drop older ones once all consumers are updated to follow the discovery procedure.

I agree we can add multiple service types and endpoints for the same service. However my main concern is that not all users may be aware of the required steps to avoid inconsistency caused by such change of default behavior.

I can easily imagine that users upgrade their deployment and now their services require block-storage service type without noticing they should create that "new" service type.

...

On 10/5/24 01:50, Ghanshyam Mann wrote: > ---- On Fri, 04 Oct 2024 09:48:25 -0700 Ghanshyam Mann wrote

---

...

> > ---- On Fri, 04 Oct 2024 09:12:13 -0700 Artem Goncharov

wrote ---

...

> > > Hey Stephen > > > > > > On Friday, 4 October 2024 15:15:08 GMT+2 Stephen Finucane

wrote:

...

> > > > o/ > > > > > > > > I've proposed a series of patches to "clean up" the

service catalog that

...

> > > > DevStack creates in a deployment. Currently, DevStack

creates two endpoints

...

> > > > for Cinder: one called 'cinderv3' of type 'volumev3'

and another called

...

> > > > 'cinder' of type 'block-storage'. For example: > > > > > > > > { > > > > "services": > > > > [ > > > > { > > > > "name": "cinderv3", > > > > "description": "Cinder Volume Service

V3",

...

> > > > "id":

"65460706a9864e71926f13042f526aeb",

...

> > > > "type": "volumev3", > > > > "enabled": true, > > > > "links": > > > > { > > > > "self": > > > > "

...

" > > > > } > > > > }, > > > > { > > > > "name": "cinder", > > > > "description": "Cinder Volume Service", > > > > "id": "cd406a0e296d47dca1e481e992851bb2", > > > > "type": "block-storage", > > > > "enabled": true, > > > > "links": > > > > { > > > > "self": > > > > " http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2 <http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2 " > > > > } > > > > }, > > > > // ... > > > > ], > > > > "links": > > > > { > > > > "next": null, > > > > "self": " http://10.0.108.50/identity/v3/services < http://10.0.108.50/identity/v3/services>", > > > > "previous": null > > > > } > > > > } > > > > > > > > They both ultimately point to the same API, the Cinder v3 API, meaning one > > > > of these endpoints is redundant. My assumption was 'block-storage' was the > > > > correct one, since that's the official service type [1], and that 'cinder' > > > > was the appropriate name since it mirrors what we use for other services > > > > (nova, neutron, keystone, glance, ...). However, tkajinam raised the point > > > > [2] that the Install Guide currently recommends use of

http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb <http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb the 'volumev3' type

...

> > > > and that projects like Nova and Glance have an

'[cinder] catalog_info' that

...

> > > > defaults to 'volumev3'. If so, is 'block-storage'

really the official

...

> > > > service type, or is it 'volumev3'. If it is, should we

be changing the

...

> > > > Install Guide and the various config option defaults? > > > > > > > > > > Previously we had volumev2 and volumev3. Those are both

versioned aliases to

...

> > > the block-storage as the official service type name (that

is what structure of

...

> > > the service-types-authority claims). Now there is no

volumev2 and thus the

...

> > > confusion grows. > > > I suggest looking at the rendered json at [2] which

states:

...

> > > ``` > > > "cinder": { > > > "aliases": [ > > > "volumev3", > > > "volumev2", > > > "volume", > > > "block-store" > > > ], > > > "api_reference": "

https://docs.openstack.org/api-ref/block-storage/ < https://docs.openstack.org/api-ref/block-storage/>",

...

> > > "project": "cinder", > > > "service_type": "block-storage" > > > }, > > > ``` > > > > > > Catalog consumption process is in detail described at

[3]. It is insanely

...

> > > complex and confusing but that is what all tools that

look at the service

...

> > > catalog do. > > > > > > Overall: > > > - official and unique service_type of the cinder is

block-storage

...

> > > - volumev3 is an alias pointing to explicit v3 api of

block-storage

...

> > > - there was an "idea" that once a service exposing

multiple API versions (and

...

> > > those are exposed independently with version-suffixed

service types) there is an

...

> > > unversioned one pointing to the "latest" version

(therefore typically you also

...

> > > find `volume` service pointing to the v3 > > > - technically install docs should stop suggesting

registering volumev3

...

> > > including project_id suffix. This is in detail addressed

by the service catalog

...

> > > consumption procedure and in addition blosk-storage v3

api has a series of

...

> > > APIs without project_id. Continuing appending project_id

in the service

...

> > > catalog is heavily harmful. > > > - install docs may switch to registering `block-storage`

or `volume` instead

...

> > > of volumev3 and even stop pointing it to the v3 api

explicitly since version

...

> > > discovery is there to figure out all the aspects. > > > > With the API version changing and just to keep backward

compatibility of devstack

...

> > among tooling, we kept modifying the catalog name in this

way and ended up with more

...

> > confusion. > > > > I agree to make the cinder catalogue also consistent with

others and name them only

...

> > 'blosk-storage'. > > > > Tempest still uses 'volumev3' as the default cinder service

catalog, but that is something we

...

> > should change as first: > > > > -

https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee8... < https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee8...

...

> > Just saw that you already proposed that. thanks ++ > > https://review.opendev.org/c/openstack/tempest/+/930296 <

https://review.opendev.org/c/openstack/tempest/+/930296>

...

> > -gmann > > > > > -gmann > > > > > > > > > > > [1]

OK. Thanks for the explanation. I then probably misunderstood the impact of the change. It sounds like as long as we use appropriate SDK in a proper manner, we can look up the endpoint with official service type by legacy service type and vise versa, because of internal translation which I was not aware of. If that is true, I agree the change has no impact unless something too hacky is made in services. On 10/5/24 02:26, Artem Goncharov wrote:

This procedure is followed in keystoneauth, openstacksdk, rust SDK/cli so no users should be affected unless they explicitly ignored official rules.

---- typed from mobile, auto-correct typos assumed ----

On Fri, Oct 4, 2024, 19:23 Artem Goncharov <artem.goncharov@gmail.com <mailto:artem.goncharov@gmail.com>> wrote:

Official procedure is described at https://specs.openstack.org/openstack/api-sig/guidelines/consuming-catalog.h... <https://specs.openstack.org/openstack/api-sig/guidelines/consuming-catalog.html>

You know the service that you want to talk to and there is a list of service types that the service supports. You follow the procedure to find the one set by your cloud.

---- typed from mobile, auto-correct typos assumed ----

On Fri, Oct 4, 2024, 19:19 Takashi Kajinami <kajinamit@oss.nttdata.com <mailto:kajinamit@oss.nttdata.com>> wrote:

On 10/5/24 02:03, Artem Goncharov wrote: > > > > On Fri, Oct 4, 2024, 18:56 Takashi Kajinami <kajinamit@oss.nttdata.com <mailto:kajinamit@oss.nttdata.com> <mailto:kajinamit@oss.nttdata.com <mailto:kajinamit@oss.nttdata.com>>> wrote: > >     IMHO changing the default value now is not a good option here. If we change the default value >     in tempest, nova, glance (and so on, if exists) it means that deployment would break during upgrade >     unless users explicitly set these options to the legacy value or update keystone endpoints to use >     the new type. >     This sounds like a quite breaking change which makes upgrade of real deployments difficult. > >     May I know if there is any clear downside of using 'volumev3' ? To be honest based on the fact that >     "volumev3" has been used as default and mentioned in docs for very long time, I find it little valuable >     while it has big use impact. > > > > There should be no impact at all as long as keystoneauth lib is used which implements service/version discovery. > If services hard coded service type of cinder without implementing discovery it is clearly a bug and must be fixed instead of sticking to keeping bugs forever what we do so often.

I'm not following this. Even if we use keystoneauth, we should know the service type to look up the correct endpoint from the list. Or are you saying that we can determine the endpoint for cinder without service_type field ?

There are number of implementations which hard-code volumev3 or use the volumev3 as default.

https://codesearch.opendev.org/?q=volumev3&i=nope&literal=nope&files=&excludeFiles=&repos= <https://codesearch.opendev.org/?q=volumev3&i=nope&literal=nope&files=&excludeFiles=&repos=>

examples: https://opendev.org/openstack/horizon/src/branch/master/openstack_dashboard/... <https://opendev.org/openstack/horizon/src/branch/master/openstack_dashboard/api/cinder.py#L218> https://opendev.org/openstack/heat/src/branch/master/heat/engine/clients/os/... <https://opendev.org/openstack/heat/src/branch/master/heat/engine/clients/os/cinder.py#L40> https://opendev.org/openstack/ceilometer/src/branch/master/ceilometer/volume... <https://opendev.org/openstack/ceilometer/src/branch/master/ceilometer/volume/discovery.py#L22>

We may be need to make these at least configurable.

> In addition to that a split between upgrade vs. new install should be implemented (install docs should explicitly recommend proper usage while existing installations may stick to their current configuration). Moreover nothing prevents adding multiple values to the catalog and drop older ones once all consumers are updated to follow the discovery procedure.

I agree we can add multiple service types and endpoints for the same service. However my main concern is that not all users may be aware of the required steps to avoid inconsistency caused by such change of default behavior.

I can easily imagine that users upgrade their deployment and now their services require block-storage service type without noticing they should create that "new" service type.

> > > >     On 10/5/24 01:50, Ghanshyam Mann wrote: >      >   ---- On Fri, 04 Oct 2024 09:48:25 -0700  Ghanshyam Mann  wrote --- >      >   >  ---- On Fri, 04 Oct 2024 09:12:13 -0700  Artem Goncharov  wrote --- >      >   >  > Hey Stephen >      >   >  > >      >   >  > On Friday, 4 October 2024 15:15:08 GMT+2 Stephen Finucane wrote: >      >   >  > > o/ >      >   >  > > >      >   >  > > I've proposed a series of patches to "clean up" the service catalog that >      >   >  > > DevStack creates in a deployment. Currently, DevStack creates two endpoints >      >   >  > > for Cinder: one called 'cinderv3' of type 'volumev3' and another called >      >   >  > > 'cinder' of type 'block-storage'. For example: >      >   >  > > >      >   >  > >     { >      >   >  > >         "services": >      >   >  > >         [ >      >   >  > >             { >      >   >  > >                 "name": "cinderv3", >      >   >  > >                 "description": "Cinder Volume Service V3", >      >   >  > >                 "id": "65460706a9864e71926f13042f526aeb", >      >   >  > >                 "type": "volumev3", >      >   >  > >                 "enabled": true, >      >   >  > >                 "links": >      >   >  > >                 { >      >   >  > >                     "self": >      >   >  > > "http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb <http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb> <http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb <http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb>>" >      >   >  > > } >      >   >  > >             }, >      >   >  > >             { >      >   >  > >                 "name": "cinder", >      >   >  > >                 "description": "Cinder Volume Service", >      >   >  > >                 "id": "cd406a0e296d47dca1e481e992851bb2", >      >   >  > >                 "type": "block-storage", >      >   >  > >                 "enabled": true, >      >   >  > >                 "links": >      >   >  > >                 { >      >   >  > >                     "self": >      >   >  > > "http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2 <http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2> <http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2 <http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2>>" >      >   >  > > } >      >   >  > >             }, >      >   >  > >             // ... >      >   >  > >         ], >      >   >  > >         "links": >      >   >  > >         { >      >   >  > >             "next": null, >      >   >  > >             "self": "http://10.0.108.50/identity/v3/services <http://10.0.108.50/identity/v3/services> <http://10.0.108.50/identity/v3/services <http://10.0.108.50/identity/v3/services>>", >      >   >  > >             "previous": null >      >   >  > >         } >      >   >  > >     } >      >   >  > > >      >   >  > > They both ultimately point to the same API, the Cinder v3 API, meaning one >      >   >  > > of these endpoints is redundant. My assumption was 'block-storage' was the >      >   >  > > correct one, since that's the official service type [1], and that 'cinder' >      >   >  > > was the appropriate name since it mirrors what we use for other services >      >   >  > > (nova, neutron, keystone, glance, ...). However, tkajinam raised the point >      >   >  > > [2] that the Install Guide currently recommends use of the 'volumev3' type >      >   >  > > and that projects like Nova and Glance have an '[cinder] catalog_info' that >      >   >  > > defaults to 'volumev3'. If so, is 'block-storage' really the official >      >   >  > > service type, or is it 'volumev3'. If it is, should we be changing the >      >   >  > > Install Guide and the various config option defaults? >      >   >  > > >      >   >  > >      >   >  > Previously we had volumev2 and volumev3. Those are both versioned aliases to >      >   >  > the block-storage as the official service type name (that is what structure of >      >   >  > the service-types-authority claims). Now there is no volumev2 and thus the >      >   >  > confusion grows. >      >   >  > I suggest looking at the rendered json at [2] which states: >      >   >  > ``` >      >   >  >     "cinder": { >      >   >  >       "aliases": [ >      >   >  >         "volumev3", >      >   >  >         "volumev2", >      >   >  >         "volume", >      >   >  >         "block-store" >      >   >  >       ], >      >   >  >       "api_reference": "https://docs.openstack.org/api-ref/block-storage/ <https://docs.openstack.org/api-ref/block-storage/> <https://docs.openstack.org/api-ref/block-storage/ <https://docs.openstack.org/api-ref/block-storage/>>", >      >   >  >       "project": "cinder", >      >   >  >       "service_type": "block-storage" >      >   >  >     }, >      >   >  > ``` >      >   >  > >      >   >  > Catalog consumption process is in detail described at [3]. It is insanely >      >   >  > complex and confusing but that is what all tools that look at the service >      >   >  > catalog do. >      >   >  > >      >   >  > Overall: >      >   >  > - official and unique service_type of the cinder is block-storage >      >   >  > - volumev3 is an alias pointing to explicit v3 api of block-storage >      >   >  > - there was an "idea" that once a service exposing multiple API versions (and >      >   >  > those are exposed independently with version-suffixed service types) there is an >      >   >  > unversioned one pointing to the "latest" version (therefore typically you also >      >   >  > find `volume` service pointing to the v3 >      >   >  > - technically install docs should stop suggesting registering volumev3 >      >   >  > including project_id suffix. This is in detail addressed by the service catalog >      >   >  > consumption procedure and in addition blosk-storage v3 api has a series of >      >   >  > APIs without project_id. Continuing appending project_id in the service >      >   >  > catalog is heavily harmful. >      >   >  > - install docs may switch to registering `block-storage` or `volume` instead >      >   >  > of volumev3 and even stop pointing it to the v3 api explicitly since version >      >   >  > discovery is there to figure out all the aspects. >      >   > >      >   > With the API version changing and just to keep backward compatibility of devstack >      >   > among tooling, we kept modifying the catalog name in this way and ended up with more >      >   > confusion. >      >   > >      >   > I agree to make the cinder catalogue also consistent with others and name them only >      >   > 'blosk-storage'. >      >   > >      >   > Tempest still uses 'volumev3' as the default cinder service catalog, but that is something we >      >   > should change as first: >      >   > >      >   > - https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee8... <https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee8cffe39/tempest/config.py#L981> <https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee8... <https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee8cffe39/tempest/config.py#L981>> >      > >      > Just saw that you already proposed that. thanks ++ >      > >      > https://review.opendev.org/c/openstack/tempest/+/930296 <https://review.opendev.org/c/openstack/tempest/+/930296> <https://review.opendev.org/c/openstack/tempest/+/930296 <https://review.opendev.org/c/openstack/tempest/+/930296>> >      > >      > -gmann >      > >      >   > >      >   > -gmann >      >   >  > >      >   >  > > >      >   >  > > [1] >

---- On Mon, 07 Oct 2024 07:35:09 -0700 Rajat Dhasmana wrote ---

Hi, Summarizing the history of endpoints to make it easier to understand the timeline.Looking at the history of changes in devstack, initially we used the convention volume, volumev2 and volumev3 for endpoints pointing to different cinder API versions v1, v2 and v3[1] respectively.Overtime we removed v1 and v2[2] as their support came to end of life.The endpoint for block-storage was added around 1 year after the addition of volumev3 endpoint and both have existed in the catalog for ~7 years. From an official project name perspective, block-storage seems to be the right choice and is consistent with other OpenStack services in the endpoint catalog.From a legacy point of view, volumev have been the convention followed and adopted by Cinder and other OpenStack projects. My preference would be to use block-storage endpoint if we can identify/fix all the issues associated with the change.One question which I'm unsure about is, if we plan a Cinder v4 version, would it again create this conflict of a new volumev4 endpoint?

In that case, we should keep ' block-storage' always pointing to the latest and recommended version, and if we have an old one still supported, then change that to something else, for example, ' block-storage-legacy' or 'block-storage-v3'. This is how we did for nova v2 ('compute-legacy') vs v2.1 ('compute') API. -gmann

[1] https://opendev.org/openstack/devstack/commit/06c7a4404edf25b9a4d913e77d12a2f61a608053[2] https://opendev.org/openstack/devstack/commit/35cec0d7c0857d76d3ea0b52b97f2a166c04c13e[3] https://opendev.org/openstack/devstack/commit/cda2cb557f7176c431d151b32bc44e... ThanksRajat Dhasmana On Fri, Oct 4, 2024 at 11:19 PM Takashi Kajinami kajinamit@oss.nttdata.com> wrote: OK. Thanks for the explanation. I then probably misunderstood the impact of the change. It sounds like as long as we use appropriate SDK in a proper manner, we can look up the endpoint with official service type by legacy service type and vise versa, because of internal translation which I was not aware of.

If that is true, I agree the change has no impact unless something too hacky is made in services.

On 10/5/24 02:26, Artem Goncharov wrote:

...

This procedure is followed in keystoneauth, openstacksdk, rust SDK/cli so no users should be affected unless they explicitly ignored official rules.

---- typed from mobile, auto-correct typos assumed ----

On Fri, Oct 4, 2024, 19:23 Artem Goncharov artem.goncharov@gmail.com artem.goncharov@gmail.com>> wrote:

     Official procedure is described at https://specs.openstack.org/openstack/api-sig/guidelines/consuming-catalog.h... https://specs.openstack.org/openstack/api-sig/guidelines/consuming-catalog.html>

     You know the service that you want to talk to and there is a list of service types that the service supports. You follow the procedure to find the one set by your cloud.

     ----      typed from mobile, auto-correct typos assumed      ----

     On Fri, Oct 4, 2024, 19:19 Takashi Kajinami kajinamit@oss.nttdata.com kajinamit@oss.nttdata.com>> wrote:

         On 10/5/24 02:03, Artem Goncharov wrote:           >           >           >           > On Fri, Oct 4, 2024, 18:56 Takashi Kajinami kajinamit@oss.nttdata.com kajinamit@oss.nttdata.com> kajinamit@oss.nttdata.com kajinamit@oss.nttdata.com>>> wrote:           >           >     IMHO changing the default value now is not a good option here. If we change the default value           >     in tempest, nova, glance (and so on, if exists) it means that deployment would break during upgrade           >     unless users explicitly set these options to the legacy value or update keystone endpoints to use           >     the new type.           >     This sounds like a quite breaking change which makes upgrade of real deployments difficult.           >           >     May I know if there is any clear downside of using 'volumev3' ? To be honest based on the fact that           >     "volumev3" has been used as default and mentioned in docs for very long time, I find it little valuable           >     while it has big use impact.           >           >           >           > There should be no impact at all as long as keystoneauth lib is used which implements service/version discovery.           > If services hard coded service type of cinder without implementing discovery it is clearly a bug and must be fixed instead of sticking to keeping bugs forever what we do so often.

         I'm not following this. Even if we use keystoneauth, we should know the service type          to look up the correct endpoint from the list. Or are you saying that we can determine          the endpoint for cinder without service_type field ?

         There are number of implementations which hard-code volumev3 or use the volumev3 as default.

         https://codesearch.opendev.org/?q=volumev3&i=nope&literal=nope&files=&excludeFiles=&repos= https://codesearch.opendev.org/?q=volumev3&i=nope&literal=nope&files=&excludeFiles=&repos=>

         examples:          https://opendev.org/openstack/horizon/src/branch/master/openstack_dashboard/... https://opendev.org/openstack/horizon/src/branch/master/openstack_dashboard/api/cinder.py#L218>          https://opendev.org/openstack/heat/src/branch/master/heat/engine/clients/os/... https://opendev.org/openstack/heat/src/branch/master/heat/engine/clients/os/cinder.py#L40>          https://opendev.org/openstack/ceilometer/src/branch/master/ceilometer/volume... https://opendev.org/openstack/ceilometer/src/branch/master/ceilometer/volume/discovery.py#L22>

         We may be need to make these at least configurable.

          > In addition to that a split between upgrade vs. new install should be implemented (install docs should explicitly recommend proper usage while existing installations may stick to their current configuration). Moreover nothing prevents adding multiple values to the catalog and drop older ones once all consumers are updated to follow the discovery procedure.

         I agree we can add multiple service types and endpoints for the same service. However my main          concern is that not all users may be aware of the required steps to avoid inconsistency caused by          such change of default behavior.

         I can easily imagine that users upgrade their deployment and now their services require block-storage          service type without noticing they should create that "new" service type.

          >           >           >           >     On 10/5/24 01:50, Ghanshyam Mann wrote:           >      >   ---- On Fri, 04 Oct 2024 09:48:25 -0700  Ghanshyam Mann  wrote ---           >      >   >  ---- On Fri, 04 Oct 2024 09:12:13 -0700  Artem Goncharov  wrote ---           >      >   >  > Hey Stephen           >      >   >  >           >      >   >  > On Friday, 4 October 2024 15:15:08 GMT+2 Stephen Finucane wrote:           >      >   >  > > o/           >      >   >  > >           >      >   >  > > I've proposed a series of patches to "clean up" the service catalog that           >      >   >  > > DevStack creates in a deployment. Currently, DevStack creates two endpoints           >      >   >  > > for Cinder: one called 'cinderv3' of type 'volumev3' and another called           >      >   >  > > 'cinder' of type 'block-storage'. For example:           >      >   >  > >           >      >   >  > >     {           >      >   >  > >         "services":           >      >   >  > >         [           >      >   >  > >             {           >      >   >  > >                 "name": "cinderv3",           >      >   >  > >                 "description": "Cinder Volume Service V3",           >      >   >  > >                 "id": "65460706a9864e71926f13042f526aeb",           >      >   >  > >                 "type": "volumev3",           >      >   >  > >                 "enabled": true,           >      >   >  > >                 "links":           >      >   >  > >                 {           >      >   >  > >                     "self":           >      >   >  > > "http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb> http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb>>"           >      >   >  > > }           >      >   >  > >             },           >      >   >  > >             {           >      >   >  > >                 "name": "cinder",           >      >   >  > >                 "description": "Cinder Volume Service",           >      >   >  > >                 "id": "cd406a0e296d47dca1e481e992851bb2",           >      >   >  > >                 "type": "block-storage",           >      >   >  > >                 "enabled": true,           >      >   >  > >                 "links":           >      >   >  > >                 {           >      >   >  > >                     "self":           >      >   >  > > "http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2 http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2> http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2 http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2>>"           >      >   >  > > }           >      >   >  > >             },           >      >   >  > >             // ...           >      >   >  > >         ],           >      >   >  > >         "links":           >      >   >  > >         {           >      >   >  > >             "next": null,           >      >   >  > >             "self": "http://10.0.108.50/identity/v3/services http://10.0.108.50/identity/v3/services> http://10.0.108.50/identity/v3/services http://10.0.108.50/identity/v3/services>>",           >      >   >  > >             "previous": null           >      >   >  > >         }           >      >   >  > >     }           >      >   >  > >           >      >   >  > > They both ultimately point to the same API, the Cinder v3 API, meaning one           >      >   >  > > of these endpoints is redundant. My assumption was 'block-storage' was the           >      >   >  > > correct one, since that's the official service type [1], and that 'cinder'           >      >   >  > > was the appropriate name since it mirrors what we use for other services           >      >   >  > > (nova, neutron, keystone, glance, ...). However, tkajinam raised the point           >      >   >  > > [2] that the Install Guide currently recommends use of the 'volumev3' type           >      >   >  > > and that projects like Nova and Glance have an '[cinder] catalog_info' that           >      >   >  > > defaults to 'volumev3'. If so, is 'block-storage' really the official           >      >   >  > > service type, or is it 'volumev3'. If it is, should we be changing the           >      >   >  > > Install Guide and the various config option defaults?           >      >   >  > >           >      >   >  >           >      >   >  > Previously we had volumev2 and volumev3. Those are both versioned aliases to           >      >   >  > the block-storage as the official service type name (that is what structure of           >      >   >  > the service-types-authority claims). Now there is no volumev2 and thus the           >      >   >  > confusion grows.           >      >   >  > I suggest looking at the rendered json at [2] which states:           >      >   >  > ```           >      >   >  >     "cinder": {           >      >   >  >       "aliases": [           >      >   >  >         "volumev3",           >      >   >  >         "volumev2",           >      >   >  >         "volume",           >      >   >  >         "block-store"           >      >   >  >       ],           >      >   >  >       "api_reference": "https://docs.openstack.org/api-ref/block-storage/ https://docs.openstack.org/api-ref/block-storage/> https://docs.openstack.org/api-ref/block-storage/ https://docs.openstack.org/api-ref/block-storage/>>",           >      >   >  >       "project": "cinder",           >      >   >  >       "service_type": "block-storage"           >      >   >  >     },           >      >   >  > ```           >      >   >  >           >      >   >  > Catalog consumption process is in detail described at [3]. It is insanely           >      >   >  > complex and confusing but that is what all tools that look at the service           >      >   >  > catalog do.           >      >   >  >           >      >   >  > Overall:           >      >   >  > - official and unique service_type of the cinder is block-storage           >      >   >  > - volumev3 is an alias pointing to explicit v3 api of block-storage           >      >   >  > - there was an "idea" that once a service exposing multiple API versions (and           >      >   >  > those are exposed independently with version-suffixed service types) there is an           >      >   >  > unversioned one pointing to the "latest" version (therefore typically you also           >      >   >  > find `volume` service pointing to the v3           >      >   >  > - technically install docs should stop suggesting registering volumev3           >      >   >  > including project_id suffix. This is in detail addressed by the service catalog           >      >   >  > consumption procedure and in addition blosk-storage v3 api has a series of           >      >   >  > APIs without project_id. Continuing appending project_id in the service           >      >   >  > catalog is heavily harmful.           >      >   >  > - install docs may switch to registering `block-storage` or `volume` instead           >      >   >  > of volumev3 and even stop pointing it to the v3 api explicitly since version           >      >   >  > discovery is there to figure out all the aspects.           >      >   >           >      >   > With the API version changing and just to keep backward compatibility of devstack           >      >   > among tooling, we kept modifying the catalog name in this way and ended up with more           >      >   > confusion.           >      >   >           >      >   > I agree to make the cinder catalogue also consistent with others and name them only           >      >   > 'blosk-storage'.           >      >   >           >      >   > Tempest still uses 'volumev3' as the default cinder service catalog, but that is something we           >      >   > should change as first:           >      >   >           >      >   > - https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee8... https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee8cffe39/tempest/config.py#L981> https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee8... https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee8cffe39/tempest/config.py#L981>>           >      >           >      > Just saw that you already proposed that. thanks ++           >      >           >      > https://review.opendev.org/c/openstack/tempest/+/930296 https://review.opendev.org/c/openstack/tempest/+/930296> https://review.opendev.org/c/openstack/tempest/+/930296 https://review.opendev.org/c/openstack/tempest/+/930296>>           >      >           >      > -gmann           >      >           >      >   >           >      >   > -gmann           >      >   >  >           >      >   >  > >           >      >   >  > > [1]           >