October 2024 - openstack-discuss - lists.openstack.org

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-25)
by Goutham Pacha Ravi 08 Oct '24

08 Oct '24

Hello Stackers, This week, we begin our 26-week endeavor towards the next SLURP release, 2025.1 ("Epoxy") [1]. OpenStack Project Teams will meet virtually at the Project Teams Gathering (PTG) in two weeks, starting on 2024-10-21 [2]. The OpenStack TC plans to host cross project meetings during the following time slots: - 2024-10-21 (Monday): 1400 UTC - 1700 UTC - 2024-10-25 (Friday): 1500 UTC - 1700 UTC You'll find the proposed topics on the PTG Etherpad [3]; please add your IRC nickname if you'd like to attend or be notified when discussions begin. Last week, a few community leads presented at OpenInfra Live, recapping the 2024.2 release [4]. I encourage you to watch the presentation and follow the themes each team is pursuing in the "Epoxy" release cycle. I'm excited to share that the organizers of the upcoming OpenInfra Days North America (Oct 15-16) have made it a hybrid event. Please register if you plan to attend virtually [5]. === Weekly Meeting === The last weekly meeting of the OpenStack Technical Committee was held simultaneously on IRC [6] and video [7]. We discussed meeting times, and the current time (Tuesdays at 1800 UTC) was retained due to a lack of consensus on better alternatives. Sylvain Bauza (bauzas) volunteered to be an Election Official for the 2025.2 elections, which will be announced around February 2025. We also discussed "leaderless" projects for the 2025.1 release and appointed leaders for the OpenStack Mistral, OpenStack Watcher, and OpenStack Swift projects. Additionally, we created a TC tracker for the 2025.1 release cycle [8] to monitor the progress of community goals and other governance initiatives. The next OpenStack Technical Committee meeting is today (2024-10-08) at 1800 UTC on the #openstack-tc IRC channel on OFTC. You can find the agenda on the weekly meeting wiki page [9]. I hope you can join us! Below is a list of governance changes that have merged in the past week and those still pending community review. === Governance Proposals === ==== Merged ==== - Appoint Tim Burke as PTL for Swift | https://review.opendev.org/c/openstack/governance/+/928881 ==== Open for Review ==== - Mark kuryr-kubernetes and kuryr-tempest-plugin inactive | https://review.opendev.org/c/openstack/governance/+/929698 - Add Axel Vanzaghi as PTL for Mistral | https://review.opendev.org/c/openstack/governance/+/927962 - Propose the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/931254 === Upcoming Events === - 2024-10-08: OpenInfra Monthly Board Meeting: https://board.openinfra.dev/ - 2024-10-15: OpenInfra Days NA, Indianapolis: https://ittraining.iu.edu/explore-topics/titles/oid-iu/ - 2024-10-21: OpenInfra Project Teams Gathering: https://openinfra.dev/ptg/ Thank you for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] "Epoxy" PTG Schedule: https://ptg.opendev.org/ptg.html [3] Technical Committee PTG Etherpad: https://etherpad.opendev.org/p/oct2024-ptg-os-tc [4] "Introducing OpenStack Dalmatian 2024.2": https://youtu.be/6igJNIJ9yFE [5] OpenInfra Days NA: https://ittraining.iu.edu/explore-topics/titles/oid-iu/index.html#register [6] TC Meeting IRC Log, 2024-10-01: https://meetings.opendev.org/meetings/tc/2024/tc.2024-10-01-18.00.log.html [7] TC Meeting Video Recording, 2024-10-01: https://youtu.be/6RXE1LfEv7w [8] 2025.1 TC Tracker: https://etherpad.opendev.org/p/tc-2025.1-tracker [9] TC Meeting Agenda, 2024-10-08: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

L2 gateway
by engineer2024 07 Oct '24

07 Oct '24

I am trying to understand the L2-GW concept in openstack neutron. I understand that its primary purpose is to bridge two l2 networks into one single network. We have tenant and provider networks in neutron. 1. Why would anyone want to bridge two l2 networks in the first place ? 2. Can these two l2 networks be, two unroutable tenant networks ? Please bear with my ignorance here. I am trying to grasp and learn these concepts. Any help in this regard is much appreciated... Thanks elinux

2 2

[neutron][nova][oslo][all] Common middleware to log requests
by Takashi Kajinami 07 Oct '24

07 Oct '24

Hello, (I'm adding tags from teams which may be directly involved with this discussion but also add all tag because this topic is actually a global topic) I recently spent some time to try neutron deployments with api run by httpd and rcp-server, and I noticed that I no longer see request details logged in neutron logs. It used to be logged by WSGI server implementation in oslo.service (which heavily depends on eventlet and being removed). Based on my past experience, the information is quite important when we have to debug any problems triggered by API request (API returned 500, Tried to create a resource but it failed, etc...). WSGI server (uwsgi or apache) may have their own request loggings but the problem with these is that these do not log context details (especially request id) so analyzing logs is still difficult. I initially created https://bugs.launchpad.net/neutron/+bug/2083760 to discuss this problem in neutron. However, when I looked at a few other repos, I noticed that current implementation is quite inconsistent across projects and now I think this needs to be discussed as a more global topic. IMO the current inconsistency makes debugging quite difficult and we need more consistent logging to help operators get more consistent and detailed information. 1. Some projects (eg. aodh, designate, octavia) does not log requests 2. Other projects (eg. cinder, glance, heat) log requests in their logic but does not log some items such as remote addr and response time. 3. nova[1] and placement[2] have own request_log middleware with more complete information (but one in nova does not have debug log dumped when api starts processing the request) I'm wondering if it makes sense to import the existing request_log middleware implementation from nova/placement and use it globally (and drop the existing logging with less incomplete information to avoid redundant logs). Thank you, Takashi [1] https://github.com/openstack/nova/blob/master/nova/api/openstack/requestlog… [2] https://github.com/openstack/placement/blob/master/placement/requestlog.py -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit

3 4

[neutron] [ovs] OVS agent constant bridge recreate flow
by Max Payne 07 Oct '24

07 Oct '24

Hi, Team! Pretty long time we’re experiencing some troubles with auto-restoration of ovs after some storms. We’re using neutron-wallaby and openvswitch 2.15. The problem is the next: After some storms we’re receiving situation, when ovs agent constantly recreating bridges and doing full syncs, which affects our network stability. All versions (incl. 2024.2), containing some parts of code that I want to highlight: 1. Non-empty `idl_monitor.bridges_added` triggers 2. `_do_reconfigure_physical_bridges`, then it triggers 3. `setup_physical_bridges(bridge_mappings)` then it contains 4. br.create() I’m seeing note in code «The bridge already exists, so create won't recreate it», BUT at the same time we’re receiving in logs: ``` Matched CREATE: BridgeCreateEvent(events=('create',), table='Bridge', conditions=None, old_conditions=None) to row=Bridge(name=br-tun ``` and ``` BridgeCreateEvent, bridge name: br-tun ``` Constantly – by hours and for all our bridges (we have four: br-tun, br-ex, br-int, br-rmi) while we will not restart ovs agent. So this event fills idl_monitor.bridges_added then ovs reacting on it by _do_reconfigure_physical_bridges, then it triggers br.create() and we’re receiving event once again – we’re in the loop. Want to add that our datapath-id NOT changing between these runs – keeps same. Additional debugging shows that bridges `_uuid` are not changing in tables, which means that rows were not added => but for some reason, we’re receiving an event about it. Any ideas or ways how to fix this problem and never experience it again?

2 1

[neutron] Bug report 40th week (Sep 30th - Oct 6th)
by Rodolfo Alonso Hernandez 07 Oct '24

07 Oct '24

Hello Neutrinos: This is the bug report of the 40th week (Sep 30th - Oct 6th). High: * https://bugs.launchpad.net/neutron/+bug/2083237: Initial router state is not set correctly. Assigned. Patch proposed: https://review.opendev.org/c/openstack/neutron/+/929406. * https://bugs.launchpad.net/neutron/+bug/2083287: test_add_router_interfaces_on_overlapping_subnets_returns_400 failed when retrying router interface removal after API timeout; (pymysql.err.OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') in neutron log. *Not assigned*. * https://bugs.launchpad.net/neutron/+bug/2083482: "neutron-fullstack-with-uwsgi-fips" failing in stable releases (Zed, 2023.1). Assigned. Patches proposed: https://review.opendev.org/q/topic:%22bug/2083482%22. * https://bugs.launchpad.net/neutron/+bug/2083483: [ovn-octavia-provider] [neutron]/endpoint_type is not respected. Assigned. Patch proposed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/931141. * https://bugs.launchpad.net/neutron/+bug/2083609: neutron.tests.fullstack.test_l3_agent.TestHAL3Agent.test_ha_router_failover_host_failure fails with "MismatchError: 2 != 1: More than one active HA routers". *Not assigned*. * https://bugs.launchpad.net/neutron/+bug/2083682: Slowness of security groups list API. *Not assigned*. Medium: * https://bugs.launchpad.net/neutron/+bug/2083226: [scale] Adding a public external network to a router is killing database. Assigned. Patch proposed: https://review.opendev.org/c/openstack/neutron/+/930888. * https://bugs.launchpad.net/neutron/+bug/2083456: Unable to detect from OVN DB if Neutron uses distributed floating IPs or not. Assigned. Patch proposed: https://review.opendev.org/c/openstack/neutron/+/931067. * https://bugs.launchpad.net/neutron/+bug/2083570: Neutron times out handling a request to create a network. *Not assigned*. To be discussed/undecided: * https://bugs.launchpad.net/neutron/+bug/2083527: [OVN] dnat_and_snat rules are setting GATEWAY_PORT when LR pinned to chassis. * https://bugs.launchpad.net/neutron/+bug/2083760: API request is not logged when neutron is run by httpd+mod_wsgi. Regards.

1 0

[Openstack] devstack won't startup nova-api
by Maxim Sava 07 Oct '24

07 Oct '24

Hi, During new devstack install get nova-api did not start. I appreciate any help you can provide. # Logs: +:: sleep 1 ++:: curl -g -k --noproxy '*' -s -o /dev/null -w '%{http_code}' http://192.168.122.157/compute/v2.1/ +:: [[ 503 == 503 ]] +:: sleep 1 ++:: curl -g -k --noproxy '*' -s -o /dev/null -w '%{http_code}' http://192.168.122.157/compute/v2.1/ +:: [[ 503 == 503 ]] +:: sleep 1 +functions:wait_for_service:472 rval=124 +functions:wait_for_service:477 time_stop wait_for_service +functions-common:time_stop:2421 local name +functions-common:time_stop:2422 local end_time +functions-common:time_stop:2423 local elapsed_time +functions-common:time_stop:2424 local total +functions-common:time_stop:2425 local start_time +functions-common:time_stop:2427 name=wait_for_service +functions-common:time_stop:2428 start_time=1728285798601 +functions-common:time_stop:2430 [[ -z 1728285798601 ]] ++functions-common:time_stop:2433 date +%s%3N +functions-common:time_stop:2433 end_time=1728285858734 +functions-common:time_stop:2434 elapsed_time=60133 +functions-common:time_stop:2435 total=2287 +functions-common:time_stop:2437 _TIME_START[$name]= +functions-common:time_stop:2438 _TIME_TOTAL[$name]=62420 +functions:wait_for_service:478 return 124 +lib/nova:start_nova_api:1015 die 1015 'nova-api did not start' +functions-common:die:285 local exitcode=0 [Call Trace] ./stack.sh:1299:start_nova_api /opt/stack/devstack/lib/nova:1015:die [ERROR] /opt/stack/devstack/lib/nova:1015 nova-api did not start exit_trap: cleaning up child processes Error on exit # Trying to connect [stack@devstack devstack]$ curl http://192.168.122.157/compute/v2.1 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>503 Service Unavailable</title> </head><body> <h1>Service Unavailable</h1> <p>The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.</p> <hr> <address>Apache/2.4.52 (Ubuntu) Server at 192.168.122.157 Port 80</address> </body></html>

2 1

How To Create Snapshot Via Python Module
by Saeed 06 Oct '24

06 Oct '24

Hello everyone, this is my python code to find a specific server: import openstack openstack.enable_logging(debug=False) servers = [] for region in ['region1', 'region2', 'region3']: openstack_connection = openstack.connect(cloud=region) # clouds.yaml for server in openstack_connection.compute.servers(details=True, hostname='my_specific_hostname'): servers.append(server.to_dict()) servers = servers[0]['id'] Now I'm going to create a snapshot from this ID only. How can I do that? I read the documents but was unable to find proper way in Python. Regards, Saeed

1 0

Re: openstack-discuss Digest, Vol 72, Issue 23
by BEDDA Fadhel 06 Oct '24

06 Oct '24

Good morning, I created a debian (12) instance on openstack. But I don't know what the username and password are. Please help m Le ven. 4 oct. 2024 à 19:48, <openstack-discuss-request(a)lists.openstack.org> a écrit : > Send openstack-discuss mailing list submissions to > openstack-discuss(a)lists.openstack.org > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > openstack-discuss-request(a)lists.openstack.org > > You can reach the person managing the list at > openstack-discuss-owner(a)lists.openstack.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of openstack-discuss digest..." > > Today's Topics: > > 1. Re: [all][qa][infra] Restore unit tests for intermediate python > versions > (Takashi Kajinami) > 2. Re: [all][qa][infra] Restore unit tests for intermediate python > versions > (Takashi Kajinami) > 3. Re: [cinder] What is the "official" name for Cinder in the service > catalog? > (Ghanshyam Mann) > 4. Re: [cinder] What is the "official" name for Cinder in the service > catalog? > (Takashi Kajinami) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 5 Oct 2024 02:26:59 +0900 > From: Takashi Kajinami <kajinamit(a)oss.nttdata.com> > Subject: Re: [all][qa][infra] Restore unit tests for intermediate > python versions > To: Clark Boylan <cboylan(a)sapwetik.org>, > openstack-discuss(a)lists.openstack.org > Message-ID: <9aec1a2b-0f30-4a47-879f-76ada4412057(a)oss.nttdata.com> > Content-Type: text/plain; charset=UTF-8; format=flowed > > > > On 10/5/24 01:47, Clark Boylan wrote: > > On Fri, Oct 4, 2024, at 9:30 AM, Takashi Kajinami wrote: > >> Hello, > >> > >> > >> As you may know, current master runs unit tests with only Python 3.9 > >> and Python 3.12. > >> My understanding about this set up is that we test the minimum boundary > >> and the maximum > >> boundary and assume the change may work with all the versions between > >> these two. > >> > >> However in some of my recent work I had to introduce the switch in the > >> middle of these > >> two versions. For example > >> https://review.opendev.org/c/openstack/neutron/+/931271 introduces > >> the logic to distinguish Python < 3.10 from Python >= 3.10 , and we > >> don't specifically > >> test that boundary. I'm afraid that this may potentially cause uncaught > >> problems. > > > > One approach that wasn't discussed in the change is to continue to use > the existing code (pkg_resources) until python3.12 then only change the > behavior for 3.12. That approach would generally work if we are only > testing the boundaries since we're flipping the behavior on the boundary. > > For this case we need python 3.11 job to test the boundary. Also this > leaves > noisy deprecation warnings in < Python 3.11 . > > > > >> > >> So my question here is ... Can we test all supported python versions, > >> not only min/max, > >> at least in check jobs ? This would helps us catch any regressions > >> caused by such > >> implementation. > > > > We can. The main limitation is that we may not have a test platform for > every version of python depending on how the distro releases align with > python releases. Currently I think we have a platform for every release > though (centos/rocky 3.9, jammy 3.10, bookworm 3.11, noble 3.12). If we end > up without a platform for a version of python jobs may need to be updated > to install python from source or some other method. > > acknowledged > > > >> > >> I know there could be tradeoffs between test coverage and infra > >> resources, but IMO > >> this is the test coverage we should fill. > > > > Yes, the main concern was simply to avoid unnecessary effort. In the > case of unittest jobs the vast majority of those are much cheaper in terms > of resources than tempest jobs. I think if we stuck to this primarily for > unittests the impact would be minimal. If we tried to expand the tempest > matrix that would become unwieldy. > > Yes. I don't intend to run all tests in all versions but at least having > unit test jobs, > which are supposed to be cheap, may avoid 0 test coverage and would be > enough now. > > In case we find any real problems then we can expand the scope to other > jobs. > > > > >> > >> Thank you, > >> > >> -- > >> Takashi Kajinami > >> irc: tkajinam > >> github: https://github.com/kajinamit > >> launchpad: https://launchpad.net/~kajinamit > > ------------------------------ > > Message: 2 > Date: Sat, 5 Oct 2024 02:29:31 +0900 > From: Takashi Kajinami <kajinamit(a)oss.nttdata.com> > Subject: Re: [all][qa][infra] Restore unit tests for intermediate > python versions > To: Jeremy Stanley <fungi(a)yuggoth.org>, > openstack-discuss(a)lists.openstack.org > Message-ID: <6b5f06e2-1c2f-4e62-be6f-fb289922bb4d(a)oss.nttdata.com> > Content-Type: text/plain; charset=UTF-8; format=flowed > > > > On 10/5/24 02:16, Jeremy Stanley wrote: > > On 2024-10-05 01:30:30 +0900 (+0900), Takashi Kajinami wrote: > > [...] > >> to distinguish Python < 3.10 from Python >= 3.10 > > [...] > > > > This is a questionable choice of implementation, given some distros > > in the past have backported features to older releases or reverted > > removals they saw as regressions. When possible, it makes more sense > > to test for the existence/absence of the feature you want to use > > rather than blindly assuming the (C)Python version will serve as a > > faithful proxy. An up-side to this approach is that you don't need > > to worry as much about version boundaries, as long as you test > > versions both with and without the feature/behavior in question. > > The super annoying thing with that change is that > importlib.metadata.entry_point > has been existing since python 3.8 but it changed the behavior in python > 3.10 . > So checking existance of modules/attributes didn't work and I avoided > implementing > hanky inspect check to analyze accepted arguments. > > An alternative solution may be relying on importlib_metadata for all > versions because > it works for Python 3.12, but I was unsure if requiring unnecessary 3rd > party dependency > to make the code older version would be a good choice. > > ------------------------------ > > Message: 3 > Date: Fri, 04 Oct 2024 10:38:36 -0700 > From: Ghanshyam Mann <gmann(a)ghanshyammann.com> > Subject: Re: [cinder] What is the "official" name for Cinder in the > service catalog? > To: "Artem Goncharov" <artem.goncharov(a)gmail.com> > Cc: openstack-discuss <openstack-discuss(a)lists.openstack.org> > Message-ID: > <192589cb074.be2d62a51024073.7540045177503817038(a)ghanshyammann.com > > > Content-Type: text/plain; charset="UTF-8" > > ---- On Fri, 04 Oct 2024 10:26:33 -0700 Artem Goncharov wrote --- > > This procedure is followed in keystoneauth, openstacksdk, rust SDK/cli > so no users should be affected unless they explicitly ignored official > rules. > > ---- > > typed from mobile, auto-correct typos assumed > > ---- > > On Fri, Oct 4, 2024, 19:23 Artem Goncharov artem.goncharov(a)gmail.com> > wrote: > > Official procedure is described at > https://specs.openstack.org/openstack/api-sig/guidelines/consuming-catalog.… > > You know the service that you want to talk to and there is a list of > service types that the service supports. You follow the procedure to find > the one set by your cloud. > > We have been trying to standardize the use of service catalog type for > many years (I think ~7-8 years :) or maybe more), and > if we still expect/support users to use them the wrong way, then there is > no point in the whole effort on these guidelines. > In the first step, we should fix our implementation in all the > services/tools you mentioned. This is what devstack change > is trying to impose. > > -gmann > > > ---- > > typed from mobile, auto-correct typos assumed > > ---- > > On Fri, Oct 4, 2024, 19:19 Takashi Kajinami kajinamit(a)oss.nttdata.com> > wrote: > > > > > > On 10/5/24 02:03, Artem Goncharov wrote: > > > > > > > > > > > > On Fri, Oct 4, 2024, 18:56 Takashi Kajinami kajinamit(a)oss.nttdata.com > kajinamit(a)oss.nttdata.com>> wrote: > > > > > > IMHO changing the default value now is not a good option here. If > we change the default value > > > in tempest, nova, glance (and so on, if exists) it means that > deployment would break during upgrade > > > unless users explicitly set these options to the legacy value or > update keystone endpoints to use > > > the new type. > > > This sounds like a quite breaking change which makes upgrade of > real deployments difficult. > > > > > > May I know if there is any clear downside of using 'volumev3' ? > To be honest based on the fact that > > > "volumev3" has been used as default and mentioned in docs for > very long time, I find it little valuable > > > while it has big use impact. > > > > > > > > > > > > There should be no impact at all as long as keystoneauth lib is used > which implements service/version discovery. > > > If services hard coded service type of cinder without implementing > discovery it is clearly a bug and must be fixed instead of sticking to > keeping bugs forever what we do so often. > > > > I'm not following this. Even if we use keystoneauth, we should know the > service type > > to look up the correct endpoint from the list. Or are you saying that > we can determine > > the endpoint for cinder without service_type field ? > > > > There are number of implementations which hard-code volumev3 or use the > volumev3 as default. > > > > > https://codesearch.opendev.org/?q=volumev3&i=nope&literal=nope&files=&exclu… > > > > examples: > > > https://opendev.org/openstack/horizon/src/branch/master/openstack_dashboard… > > > https://opendev.org/openstack/heat/src/branch/master/heat/engine/clients/os… > > > https://opendev.org/openstack/ceilometer/src/branch/master/ceilometer/volum… > > > > We may be need to make these at least configurable. > > > > > > > In addition to that a split between upgrade vs. new install should be > implemented (install docs should explicitly recommend proper usage while > existing installations may stick to their current configuration). Moreover > nothing prevents adding multiple values to the catalog and drop older ones > once all consumers are updated to follow the discovery procedure. > > > > I agree we can add multiple service types and endpoints for the same > service. However my main > > concern is that not all users may be aware of the required steps to > avoid inconsistency caused by > > such change of default behavior. > > > > I can easily imagine that users upgrade their deployment and now their > services require block-storage > > service type without noticing they should create that "new" service > type. > > > > > > > > > > > > > > On 10/5/24 01:50, Ghanshyam Mann wrote: > > > > ---- On Fri, 04 Oct 2024 09:48:25 -0700 Ghanshyam Mann > wrote --- > > > > > ---- On Fri, 04 Oct 2024 09:12:13 -0700 Artem Goncharov > wrote --- > > > > > > Hey Stephen > > > > > > > > > > > > On Friday, 4 October 2024 15:15:08 GMT+2 Stephen > Finucane wrote: > > > > > > > o/ > > > > > > > > > > > > > > I've proposed a series of patches to "clean up" the > service catalog that > > > > > > > DevStack creates in a deployment. Currently, DevStack > creates two endpoints > > > > > > > for Cinder: one called 'cinderv3' of type 'volumev3' > and another called > > > > > > > 'cinder' of type 'block-storage'. For example: > > > > > > > > > > > > > > { > > > > > > > "services": > > > > > > > [ > > > > > > > { > > > > > > > "name": "cinderv3", > > > > > > > "description": "Cinder Volume Service > V3", > > > > > > > "id": > "65460706a9864e71926f13042f526aeb", > > > > > > > "type": "volumev3", > > > > > > > "enabled": true, > > > > > > > "links": > > > > > > > { > > > > > > > "self": > > > > > > > " > http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb > http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb>" > > > > > > > } > > > > > > > }, > > > > > > > { > > > > > > > "name": "cinder", > > > > > > > "description": "Cinder Volume > Service", > > > > > > > "id": > "cd406a0e296d47dca1e481e992851bb2", > > > > > > > "type": "block-storage", > > > > > > > "enabled": true, > > > > > > > "links": > > > > > > > { > > > > > > > "self": > > > > > > > " > http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2 > http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2>" > > > > > > > } > > > > > > > }, > > > > > > > // ... > > > > > > > ], > > > > > > > "links": > > > > > > > { > > > > > > > "next": null, > > > > > > > "self": " > http://10.0.108.50/identity/v3/services > http://10.0.108.50/identity/v3/services>", > > > > > > > "previous": null > > > > > > > } > > > > > > > } > > > > > > > > > > > > > > They both ultimately point to the same API, the > Cinder v3 API, meaning one > > > > > > > of these endpoints is redundant. My assumption was > 'block-storage' was the > > > > > > > correct one, since that's the official service type > [1], and that 'cinder' > > > > > > > was the appropriate name since it mirrors what we use > for other services > > > > > > > (nova, neutron, keystone, glance, ...). However, > tkajinam raised the point > > > > > > > [2] that the Install Guide currently recommends use > of the 'volumev3' type > > > > > > > and that projects like Nova and Glance have an > '[cinder] catalog_info' that > > > > > > > defaults to 'volumev3'. If so, is 'block-storage' > really the official > > > > > > > service type, or is it 'volumev3'. If it is, should > we be changing the > > > > > > > Install Guide and the various config option defaults? > > > > > > > > > > > > > > > > > > > Previously we had volumev2 and volumev3. Those are both > versioned aliases to > > > > > > the block-storage as the official service type name > (that is what structure of > > > > > > the service-types-authority claims). Now there is no > volumev2 and thus the > > > > > > confusion grows. > > > > > > I suggest looking at the rendered json at [2] which > states: > > > > > > ``` > > > > > > "cinder": { > > > > > > "aliases": [ > > > > > > "volumev3", > > > > > > "volumev2", > > > > > > "volume", > > > > > > "block-store" > > > > > > ], > > > > > > "api_reference": " > https://docs.openstack.org/api-ref/block-storage/ > https://docs.openstack.org/api-ref/block-storage/>", > > > > > > "project": "cinder", > > > > > > "service_type": "block-storage" > > > > > > }, > > > > > > ``` > > > > > > > > > > > > Catalog consumption process is in detail described at > [3]. It is insanely > > > > > > complex and confusing but that is what all tools that > look at the service > > > > > > catalog do. > > > > > > > > > > > > Overall: > > > > > > - official and unique service_type of the cinder is > block-storage > > > > > > - volumev3 is an alias pointing to explicit v3 api of > block-storage > > > > > > - there was an "idea" that once a service exposing > multiple API versions (and > > > > > > those are exposed independently with version-suffixed > service types) there is an > > > > > > unversioned one pointing to the "latest" version > (therefore typically you also > > > > > > find `volume` service pointing to the v3 > > > > > > - technically install docs should stop suggesting > registering volumev3 > > > > > > including project_id suffix. This is in detail > addressed by the service catalog > > > > > > consumption procedure and in addition blosk-storage v3 > api has a series of > > > > > > APIs without project_id. Continuing appending > project_id in the service > > > > > > catalog is heavily harmful. > > > > > > - install docs may switch to registering > `block-storage` or `volume` instead > > > > > > of volumev3 and even stop pointing it to the v3 api > explicitly since version > > > > > > discovery is there to figure out all the aspects. > > > > > > > > > > With the API version changing and just to keep backward > compatibility of devstack > > > > > among tooling, we kept modifying the catalog name in this > way and ended up with more > > > > > confusion. > > > > > > > > > > I agree to make the cinder catalogue also consistent with > others and name them only > > > > > 'blosk-storage'. > > > > > > > > > > Tempest still uses 'volumev3' as the default cinder > service catalog, but that is something we > > > > > should change as first: > > > > > > > > > > - > https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee… > https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee… > > > > > > > > > > Just saw that you already proposed that. thanks ++ > > > > > > > > https://review.opendev.org/c/openstack/tempest/+/930296 > https://review.opendev.org/c/openstack/tempest/+/930296> > > > > > > > > -gmann > > > > > > > > > > > > > > -gmann > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > ------------------------------ > > Message: 4 > Date: Sat, 5 Oct 2024 02:48:13 +0900 > From: Takashi Kajinami <kajinamit(a)oss.nttdata.com> > Subject: Re: [cinder] What is the "official" name for Cinder in the > service catalog? > To: Artem Goncharov <artem.goncharov(a)gmail.com> > Cc: openstack-discuss <openstack-discuss(a)lists.openstack.org> > Message-ID: <e3731ee2-6ffb-4827-b03e-a0008625a8b5(a)oss.nttdata.com> > Content-Type: text/plain; charset=UTF-8; format=flowed > > OK. Thanks for the explanation. > I then probably misunderstood the impact of the change. It sounds like as > long as we use appropriate SDK > in a proper manner, we can look up the endpoint with official service type > by legacy service type and > vise versa, because of internal translation which I was not aware of. > > If that is true, I agree the change has no impact unless something too > hacky is made in services. > > On 10/5/24 02:26, Artem Goncharov wrote: > > This procedure is followed in keystoneauth, openstacksdk, rust SDK/cli > so no users should be affected unless they explicitly ignored official > rules. > > > > ---- > > typed from mobile, auto-correct typos assumed > > ---- > > > > On Fri, Oct 4, 2024, 19:23 Artem Goncharov <artem.goncharov(a)gmail.com > <mailto:artem.goncharov@gmail.com>> wrote: > > > > Official procedure is described at > https://specs.openstack.org/openstack/api-sig/guidelines/consuming-catalog.… > < > https://specs.openstack.org/openstack/api-sig/guidelines/consuming-catalog.… > > > > > > You know the service that you want to talk to and there is a list of > service types that the service supports. You follow the procedure to find > the one set by your cloud. > > > > ---- > > typed from mobile, auto-correct typos assumed > > ---- > > > > On Fri, Oct 4, 2024, 19:19 Takashi Kajinami < > kajinamit(a)oss.nttdata.com <mailto:kajinamit@oss.nttdata.com>> wrote: > > > > > > > > On 10/5/24 02:03, Artem Goncharov wrote: > > > > > > > > > > > > On Fri, Oct 4, 2024, 18:56 Takashi Kajinami < > kajinamit(a)oss.nttdata.com <mailto:kajinamit@oss.nttdata.com> <mailto: > kajinamit(a)oss.nttdata.com <mailto:kajinamit@oss.nttdata.com>>> wrote: > > > > > > IMHO changing the default value now is not a good option > here. If we change the default value > > > in tempest, nova, glance (and so on, if exists) it means > that deployment would break during upgrade > > > unless users explicitly set these options to the legacy > value or update keystone endpoints to use > > > the new type. > > > This sounds like a quite breaking change which makes > upgrade of real deployments difficult. > > > > > > May I know if there is any clear downside of using > 'volumev3' ? To be honest based on the fact that > > > "volumev3" has been used as default and mentioned in docs > for very long time, I find it little valuable > > > while it has big use impact. > > > > > > > > > > > > There should be no impact at all as long as keystoneauth lib > is used which implements service/version discovery. > > > If services hard coded service type of cinder without > implementing discovery it is clearly a bug and must be fixed instead of > sticking to keeping bugs forever what we do so often. > > > > I'm not following this. Even if we use keystoneauth, we should > know the service type > > to look up the correct endpoint from the list. Or are you saying > that we can determine > > the endpoint for cinder without service_type field ? > > > > There are number of implementations which hard-code volumev3 or > use the volumev3 as default. > > > > > https://codesearch.opendev.org/?q=volumev3&i=nope&literal=nope&files=&exclu… > < > https://codesearch.opendev.org/?q=volumev3&i=nope&literal=nope&files=&exclu… > > > > > > examples: > > > https://opendev.org/openstack/horizon/src/branch/master/openstack_dashboard… > < > https://opendev.org/openstack/horizon/src/branch/master/openstack_dashboard… > > > > > https://opendev.org/openstack/heat/src/branch/master/heat/engine/clients/os… > < > https://opendev.org/openstack/heat/src/branch/master/heat/engine/clients/os… > > > > > https://opendev.org/openstack/ceilometer/src/branch/master/ceilometer/volum… > < > https://opendev.org/openstack/ceilometer/src/branch/master/ceilometer/volum… > > > > > > We may be need to make these at least configurable. > > > > > > > In addition to that a split between upgrade vs. new install > should be implemented (install docs should explicitly recommend proper > usage while existing installations may stick to their current > configuration). Moreover nothing prevents adding multiple values to the > catalog and drop older ones once all consumers are updated to follow the > discovery procedure. > > > > I agree we can add multiple service types and endpoints for the > same service. However my main > > concern is that not all users may be aware of the required steps > to avoid inconsistency caused by > > such change of default behavior. > > > > I can easily imagine that users upgrade their deployment and now > their services require block-storage > > service type without noticing they should create that "new" > service type. > > > > > > > > > > > > > > On 10/5/24 01:50, Ghanshyam Mann wrote: > > > > ---- On Fri, 04 Oct 2024 09:48:25 -0700 Ghanshyam > Mann wrote --- > > > > > ---- On Fri, 04 Oct 2024 09:12:13 -0700 Artem > Goncharov wrote --- > > > > > > Hey Stephen > > > > > > > > > > > > On Friday, 4 October 2024 15:15:08 GMT+2 > Stephen Finucane wrote: > > > > > > > o/ > > > > > > > > > > > > > > I've proposed a series of patches to "clean > up" the service catalog that > > > > > > > DevStack creates in a deployment. Currently, > DevStack creates two endpoints > > > > > > > for Cinder: one called 'cinderv3' of type > 'volumev3' and another called > > > > > > > 'cinder' of type 'block-storage'. For example: > > > > > > > > > > > > > > { > > > > > > > "services": > > > > > > > [ > > > > > > > { > > > > > > > "name": "cinderv3", > > > > > > > "description": "Cinder Volume > Service V3", > > > > > > > "id": > "65460706a9864e71926f13042f526aeb", > > > > > > > "type": "volumev3", > > > > > > > "enabled": true, > > > > > > > "links": > > > > > > > { > > > > > > > "self": > > > > > > > " > http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb < > http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb> > <http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb > <http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb > >>" > > > > > > > } > > > > > > > }, > > > > > > > { > > > > > > > "name": "cinder", > > > > > > > "description": "Cinder Volume > Service", > > > > > > > "id": > "cd406a0e296d47dca1e481e992851bb2", > > > > > > > "type": "block-storage", > > > > > > > "enabled": true, > > > > > > > "links": > > > > > > > { > > > > > > > "self": > > > > > > > " > http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2 < > http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2> > <http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2 > <http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2 > >>" > > > > > > > } > > > > > > > }, > > > > > > > // ... > > > > > > > ], > > > > > > > "links": > > > > > > > { > > > > > > > "next": null, > > > > > > > "self": " > http://10.0.108.50/identity/v3/services < > http://10.0.108.50/identity/v3/services> < > http://10.0.108.50/identity/v3/services < > http://10.0.108.50/identity/v3/services>>", > > > > > > > "previous": null > > > > > > > } > > > > > > > } > > > > > > > > > > > > > > They both ultimately point to the same API, > the Cinder v3 API, meaning one > > > > > > > of these endpoints is redundant. My > assumption was 'block-storage' was the > > > > > > > correct one, since that's the official > service type [1], and that 'cinder' > > > > > > > was the appropriate name since it mirrors > what we use for other services > > > > > > > (nova, neutron, keystone, glance, ...). > However, tkajinam raised the point > > > > > > > [2] that the Install Guide currently > recommends use of the 'volumev3' type > > > > > > > and that projects like Nova and Glance have > an '[cinder] catalog_info' that > > > > > > > defaults to 'volumev3'. If so, is > 'block-storage' really the official > > > > > > > service type, or is it 'volumev3'. If it is, > should we be changing the > > > > > > > Install Guide and the various config option > defaults? > > > > > > > > > > > > > > > > > > > Previously we had volumev2 and volumev3. Those > are both versioned aliases to > > > > > > the block-storage as the official service type > name (that is what structure of > > > > > > the service-types-authority claims). Now there > is no volumev2 and thus the > > > > > > confusion grows. > > > > > > I suggest looking at the rendered json at [2] > which states: > > > > > > ``` > > > > > > "cinder": { > > > > > > "aliases": [ > > > > > > "volumev3", > > > > > > "volumev2", > > > > > > "volume", > > > > > > "block-store" > > > > > > ], > > > > > > "api_reference": " > https://docs.openstack.org/api-ref/block-storage/ < > https://docs.openstack.org/api-ref/block-storage/> < > https://docs.openstack.org/api-ref/block-storage/ < > https://docs.openstack.org/api-ref/block-storage/>>", > > > > > > "project": "cinder", > > > > > > "service_type": "block-storage" > > > > > > }, > > > > > > ``` > > > > > > > > > > > > Catalog consumption process is in detail > described at [3]. It is insanely > > > > > > complex and confusing but that is what all > tools that look at the service > > > > > > catalog do. > > > > > > > > > > > > Overall: > > > > > > - official and unique service_type of the > cinder is block-storage > > > > > > - volumev3 is an alias pointing to explicit v3 > api of block-storage > > > > > > - there was an "idea" that once a service > exposing multiple API versions (and > > > > > > those are exposed independently with > version-suffixed service types) there is an > > > > > > unversioned one pointing to the "latest" > version (therefore typically you also > > > > > > find `volume` service pointing to the v3 > > > > > > - technically install docs should stop > suggesting registering volumev3 > > > > > > including project_id suffix. This is in detail > addressed by the service catalog > > > > > > consumption procedure and in addition > blosk-storage v3 api has a series of > > > > > > APIs without project_id. Continuing appending > project_id in the service > > > > > > catalog is heavily harmful. > > > > > > - install docs may switch to registering > `block-storage` or `volume` instead > > > > > > of volumev3 and even stop pointing it to the v3 > api explicitly since version > > > > > > discovery is there to figure out all the > aspects. > > > > > > > > > > With the API version changing and just to keep > backward compatibility of devstack > > > > > among tooling, we kept modifying the catalog name > in this way and ended up with more > > > > > confusion. > > > > > > > > > > I agree to make the cinder catalogue also > consistent with others and name them only > > > > > 'blosk-storage'. > > > > > > > > > > Tempest still uses 'volumev3' as the default > cinder service catalog, but that is something we > > > > > should change as first: > > > > > > > > > > - > https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee… > < > https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee…> > < > https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee… > < > https://github.com/openstack/tempest/blob/da14e2972243e4890e3ef6889bf5a85ee… > >> > > > > > > > > Just saw that you already proposed that. thanks ++ > > > > > > > > > https://review.opendev.org/c/openstack/tempest/+/930296 < > https://review.opendev.org/c/openstack/tempest/+/930296> < > https://review.opendev.org/c/openstack/tempest/+/930296 < > https://review.opendev.org/c/openstack/tempest/+/930296>> > > > > > > > > -gmann > > > > > > > > > > > > > > -gmann > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > openstack-discuss mailing list -- openstack-discuss(a)lists.openstack.org > To unsubscribe send an email to > openstack-discuss-leave(a)lists.openstack.org > > > ------------------------------ > > End of openstack-discuss Digest, Vol 72, Issue 23 > ************************************************* >

2 1

Re: openstack-discuss Digest, Vol 72, Issue 24
by BEDDA Fadhel 05 Oct '24

05 Oct '24

Good morning, I'm testing openstack (devstack). I downloaded an ubuntu qcow2 image, I created the image from this image, but when I access the console => I find the message: booting form hard disk and nothing happens after Le sam. 5 oct. 2024 à 12:50, <openstack-discuss-request(a)lists.openstack.org> a écrit : > Send openstack-discuss mailing list submissions to > openstack-discuss(a)lists.openstack.org > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > openstack-discuss-request(a)lists.openstack.org > > You can reach the person managing the list at > openstack-discuss-owner(a)lists.openstack.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of openstack-discuss digest..." > > Today's Topics: > > 1. Re: [all][qa][infra] Restore unit tests for intermediate python > versions > (Clark Boylan) > 2. Re: Stop bumping major versions when dropping EOL Python versions > (Elõd Illés) > 3. Re: [oslo.messaging] Heartbeat in pthread (Herve Beraud) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 04 Oct 2024 11:22:17 -0700 > From: "Clark Boylan" <cboylan(a)sapwetik.org> > Subject: Re: [all][qa][infra] Restore unit tests for intermediate > python versions > To: "Takashi Kajinami" <kajinamit(a)oss.nttdata.com>, > openstack-discuss(a)lists.openstack.org > Message-ID: <99b5ae0b-2d69-4040-9d3b-8b28491969a6(a)app.fastmail.com> > Content-Type: text/plain > > On Fri, Oct 4, 2024, at 10:26 AM, Takashi Kajinami wrote: > > On 10/5/24 01:47, Clark Boylan wrote: > >> On Fri, Oct 4, 2024, at 9:30 AM, Takashi Kajinami wrote: > >>> Hello, > >>> > >>> > >>> As you may know, current master runs unit tests with only Python 3.9 > >>> and Python 3.12. > >>> My understanding about this set up is that we test the minimum boundary > >>> and the maximum > >>> boundary and assume the change may work with all the versions between > >>> these two. > >>> > >>> However in some of my recent work I had to introduce the switch in the > >>> middle of these > >>> two versions. For example > >>> https://review.opendev.org/c/openstack/neutron/+/931271 introduces > >>> the logic to distinguish Python < 3.10 from Python >= 3.10 , and we > >>> don't specifically > >>> test that boundary. I'm afraid that this may potentially cause uncaught > >>> problems. > >> > >> One approach that wasn't discussed in the change is to continue to use > the existing code (pkg_resources) until python3.12 then only change the > behavior for 3.12. That approach would generally work if we are only > testing the boundaries since we're flipping the behavior on the boundary. > > > > For this case we need python 3.11 job to test the boundary. Also this > leaves > > noisy deprecation warnings in < Python 3.11 . > > The lower bound (currently 3.9) would continue to test the old behavior > (pkg_resources) and the upper bound (python3.12) would test the new > behavior (importlib). It would be covered as well as all of the other upper > and lower bound testing we've done. Python3.10 and 3.11 would be "covered" > by the 3.9 test ensuring that pkg_resources continues to work. When 3.9 is > dropped 3.10 would take over this responsibility. Eventually we'll only > have the importlib behavior and python3.12/3.13/etc would cover it. > > > > >> > >>> > >>> So my question here is ... Can we test all supported python versions, > >>> not only min/max, > >>> at least in check jobs ? This would helps us catch any regressions > >>> caused by such > >>> implementation. > >> > >> We can. The main limitation is that we may not have a test platform for > every version of python depending on how the distro releases align with > python releases. Currently I think we have a platform for every release > though (centos/rocky 3.9, jammy 3.10, bookworm 3.11, noble 3.12). If we end > up without a platform for a version of python jobs may need to be updated > to install python from source or some other method. > > > > acknowledged > >> > >>> > >>> I know there could be tradeoffs between test coverage and infra > >>> resources, but IMO > >>> this is the test coverage we should fill. > >> > >> Yes, the main concern was simply to avoid unnecessary effort. In the > case of unittest jobs the vast majority of those are much cheaper in terms > of resources than tempest jobs. I think if we stuck to this primarily for > unittests the impact would be minimal. If we tried to expand the tempest > matrix that would become unwieldy. > > > > Yes. I don't intend to run all tests in all versions but at least > > having unit test jobs, > > which are supposed to be cheap, may avoid 0 test coverage and would be > > enough now. > > > > In case we find any real problems then we can expand the scope to other > jobs. > > > >> > >>> > >>> Thank you, > >>> > >>> -- > >>> Takashi Kajinami > >>> irc: tkajinam > >>> github: https://github.com/kajinamit > >>> launchpad: https://launchpad.net/~kajinamit > > ------------------------------ > > Message: 2 > Date: Fri, 4 Oct 2024 20:05:02 +0000 > From: Elõd Illés <elod.illes(a)est.tech> > Subject: Re: Stop bumping major versions when dropping EOL Python > versions > To: Jeremy Stanley <fungi(a)yuggoth.org>, > "openstack-discuss(a)lists.openstack.org" > <openstack-discuss(a)lists.openstack.org> > Message-ID: <VI1P18901MB07518D7DBD7CA7D11AF91E18FF722@VI1P18901MB0751 > .EURP189.PROD.OUTLOOK.COM> > Content-Type: text/plain; charset="utf-8" > > Indeed, we started bumping MAJOR version due to dropping py27 support. > Though, the rule is, that a change in required package's version, only > warrants a MINOR version bump. But python is not just a simple dependency, > and dropping support of a python version is more than just bumping any > python package's version. Anyway, if the community wants to stop > bumping major versions in case of dropping python version support, > then i'm probably OK with that. > > Cheers, > Előd > > ________________________________________ > From: Jeremy Stanley > Sent: Monday, September 23, 2024 15:22 > To: openstack-discuss(a)lists.openstack.org > Subject: Re: Stop bumping major versions when dropping EOL Python versions > > > On 2024-09-23 12:25:09 +0100 (+0100), Stephen Finucane wrote: > [...] > > I'd be in favour of getting rid of this particular constraint, and > > instead focusing on API changes when deciding whether to bump the > > major version. Is this a reasonable suggestion, and if so what > > would we need to do to tweak our policy to allow it? > > It makes sense to me. Dropping Python 2.x support was a more > significant event, but these days removing support for minor Python > versions after the corresponding CPython interpreter reaches EOL > upstream is fairly unremarkable. We don't perform major version > increments just for raising the minimum required versions of any > other dependencies, after all. > -- > Jeremy Stanley > > > > ------------------------------ > > Message: 3 > Date: Sat, 5 Oct 2024 12:50:04 +0200 > From: Herve Beraud <hberaud(a)redhat.com> > Subject: Re: [oslo.messaging] Heartbeat in pthread > To: Michel Jouvin <michel.jouvin(a)ijclab.in2p3.fr> > Cc: openstack-discuss(a)lists.openstack.org > Message-ID: > < > CAFDq9gXbkoiRj18zTuqbWAzhg5yhf00JPn6kXh0R83i2CdpyRw(a)mail.gmail.com> > Content-Type: multipart/alternative; > boundary="0000000000008792280623b88de9" > > Voici la version mise à jour avec la solution de **retry backoff** et les > liens supplémentaires : > > Hey folks, > > The major concern behind this thread is that RabbitMQ connection drops due > to the absence of a reliable heartbeat. > While `heartbeat_in_pthread=True` aimed to fix this, it introduced other > bugs. > > Indeed, the Greenlet documentation is pretty clear, the limitations between > Python threads and greenlets lead to issues. > As eventlet is itself based on greenlet it leads to recurring issues in our > stacks. > The heartbeat_in_pthread bugs are living examples of this kind of issue. > > For this reason, we support keeping `heartbeat_in_pthread` disabled by > default. > > As a workaround, adjusting the RabbitMQ `heartbeat_timeout` and > `rabbit_heartbeat_timeout_threshold` can mitigate connection drops. > > Additionally, oslo.messaging offers the `connection_retry_interval` and > `connection_retry_backoff` parameters, > which implement retry backoff strategies to better handle connection drops. > This ensures that the system can manage reconnections more efficiently. > > We encourage investigating these paths to mitigate the connection problems. > > For more details please read: > - https://greenlet.readthedocs.io/en/latest/python_threads.html > - > > https://docs.openstack.org/oslo.messaging/xena/configuration/opts.html#oslo… > - https://www.rabbitmq.com/docs/heartbeats > - > > https://docs.openstack.org/oslo.messaging/xena/configuration/opts.html#oslo… > - > > https://docs.openstack.org/oslo.messaging/xena/configuration/opts.html#oslo… > > > Le jeu. 3 oct. 2024 à 22:46, Michel Jouvin <michel.jouvin(a)ijclab.in2p3.fr> > a écrit : > > > Hi Sean, > > > > Not sure why we misunderstood eachother but we agree! I understood your > > sentence as "people should avoid changing this option from its default > > (False) to True." but I understand now you mean the opposite and I > > totally agree based on our experience. Heat seems to be another service > > that will be in trouble if it is changed. > > > > Michel > > > > Le 02/10/2024 à 14:17, smooney(a)redhat.com a écrit : > > > On Wed, 2024-10-02 at 13:06 +0200, Michel Jouvin wrote: > > >> Hi Sean, > > >> > > >> As for the situation in our cloud after reverting to > > >> heartbeat_in_pthread=false, it was a "black and white situation": > > >> creating a cluster was impossible (because of the Heat issue > mentioned) > > >> since we changed to heartbeat_in_pthread=true (but we didn't realize > > >> immediately as we don't create clusters everyday) and restarted to > work > > >> properly immediately after reverting to heartbeat_in_pthread=false. > > >> There is a clear link between this parameter and Heat behaviour > (Caracal > > >> version in our case, so Oslo client 24.0). > > >> > > >> As for your last sentence "people should avoid cahnging this option > form > > >> tis default fo False.", I think/hope you wanted to say the opposite : > > >> "people should avoid changing this option form its default to > True."... > > > no at least for nova we default to false in our downstream product > > > > > > > > > https://github.com/openstack-k8s-operators/nova-operator/blob/main/template… > > > > > > we had signifcant ci usee when we had it set to true orgianly because > of > > the oslo.log issue > > > but we didn not revert to enabling this for nova-api after that was > > backported because > > > we have not seen any sideefct from seting it to false. > > > > > > we have never set this to true in OSP to my knolage for nova-comptue > > > > > > puppet-nova considered it experimental > > > > > > https://opendev.org/openstack/puppet-nova/src/commit/17bd61e042591305e461e5… > > > > > > in tripleo we disabled it in many services including heat and nova > > > > > > https://github.com/openstack-archive/tripleo-heat-templates/commit/cf4d4f88… > > > > > > in kolla it also default to false for nova-compute and other eventlet > > services > > > > > > https://github.com/openstack/kolla-ansible/blob/2218b7852fda94d0f498d5140f7… > > > although it is enabel for nova-api and some heat compoents > > > > > > https://github.com/openstack/kolla-ansible/blob/2218b7852fda94d0f498d5140f7… > > > > > > https://github.com/openstack/kolla-ansible/blob/2218b7852fda94d0f498d5140f7… > > > > > > at least for nova i would not recommend using > > > > > > heartbeat_in_pthread = True for any serrvice > > > nova-api runnign under uwsgi or mod_wsgi is the only possibel excpetion > > and even then i woudl discurage it. > > > > > > i cant really speak to other services but i think `heartbeat_in_pthread > > = false` is generally the correct default. > > > > > > > > >> Michel > > >> > > >> Le 02/10/2024 à 11:44, smooney(a)redhat.com a écrit : > > >>> On Tue, 2024-10-01 at 22:32 +0200, Michel Jouvin wrote: > > >>>> Hi, > > >>>> > > >>>> I am not an expert in these matters but we recently suffered the > > problem > > >>>> of client deconnection in RabbitMQ due to the heartbeat timeout and > I > > >>>> confirm it was a disaster for the cloud usage with many things not > > >>>> working properly (we are running Antelope, except > Barbican/Heat/Magnum > > >>>> where we run Caracal). The reason is still not clear for me, it was > > >>>> fixed by increasing the heartbeat timeout but at the same time, my > > >>>> colleague who implemented the change also defined > > >>>> heartbeat_in_pthread=true for all the services, something normally > > >>>> unnecessary as we configure uwsgi or Apache to use only one thread > > (and > > >>>> several processes). Initially we didn't see any bad impact of this > > >>>> setting but a few days ago users started to report that Magnum > cluster > > >>>> creation was failing due a "response timeout" in Heat during the > > master > > >>>> software deployment. > > >>>> > > >>>> Reading this thread this morning I had the idea if could be the > source > > >>>> of the problem (as the service was running properly a couple of > weeks > > >>>> ago, before the change). We reverted the change and defined > > >>>> heartbeat_in_pthread=false and it restored the normal behaviour of > > Heat. > > >>>> We have not seen a negative impact on other services so far. So I > > >>>> confirm that setting this parameter to false by default seems a good > > >>>> idea and that setting it to true can break some services like Heat. > > >>> thank you for the data point, im sure you will monitor the situration > > in your > > >>> clodu but please let use know in a week or two if the heat/magnum > > issues you > > >>> obsevered retrun or if the could continue to fucntion normally, > > >>> i expect it to but again it would be a good data point. > > >>>> Cheers, > > >>>> > > >>>> Michel > > >>>> > > >>>> Le 01/10/2024 à 16:31, Arnaud Morin a écrit : > > >>>>> Hey, > > >>>>> > > >>>>> I totally agree about the fact that heartbeat_in_pthread and the > > >>>>> oslo.log PipeMutex are technical debt that we need to get rid of, > > >>>>> as well as eventlet. > > >>>>> > > >>>>> However, despite the fact that it seems purely cosmetic on your > side, > > >>>>> we believe it's not. > > >>>>> I can't prove / reproduce the issue on a small infra, but > definetely, > > >>>>> at large scale, having those tcp connections to be dropped by > > rabbitmq > > >>>>> and recreated in a loop by agents is affecting the cluster. > > >>>>> > > >>>>> I know all the pain that these settings introduced in the past, but > > now > > >>>>> I feel we are in a stable situation regarding this, that's why I am > > >>>>> surprised about deprecating heartbeat_in_pthread now. > > >>> deprecateign a config option requires the deprecation to be > advertised > > in a slrup > > >>> before it can then be removed in a follwoing release. > > >>> Given the deprecation was done in dalmaition 2024.2 which is not a > > slurp release the removal > > >>> cannot take effect in 2025.1, 2025.2 is the earliest release we could > > remove this option. > > >>> > > >>> as a result i think maintaining the deprecation is correct here. > > >>> we may decied not to remove this until 2026.1 or later but i think > its > > correct to send the > > >>> message that people should avoid cahnging this option form tis > default > > fo False. > > >>> we coudl even tag this option as advanced to make that more clear > > >>> > > > https://docs.openstack.org/oslo.config/latest/reference/defining.html#advan… > > >>> > > >>>>> Can we, as least, make sure we keep all of this until we switch off > > >>>>> eventlet? > > >>>>> In other words, can we get rid of eventlet, then remove this > params? > > >>>>> and not the opposite? > > >>>>> > > >>>>> Regards, > > >>>>> > > >>>>> Arnaud > > > > > > -- > Hervé Beraud > Senior Software Engineer at Red Hat > irc: hberaud > https://github.com/4383/ >

1 1

[oslo.messaging] Heartbeat in pthread
by Arnaud Morin 05 Oct '24

05 Oct '24

Hello, I completely miss the deprecation of heartbeat_in_pthread in oslo.messaging [1]. We heavily rely on this parameter downstream and our opinion is that it should be set to True by default. We use it for both wsgi services and agents (nova-compute, neutron agents, etc.). I understand that eventlet will be dropped in the future, but should we set heartbeat_in_pthread to True by default until then? Regards, Arnaud. [1] https://review.opendev.org/c/openstack/oslo.messaging/+/925778

5 12