- openstack-discuss - lists.openstack.org

[watcher][tc] watcher-drivers launchpad membership
by Sean Mooney 29 Nov '24

29 Nov '24

Hi folks o/ Following up form the PTG discussion https://etherpad.opendev.org/p/r.79644f8f3cd04686c4f45c10aa35d8ac#L65 and the recently merged governance patch https://github.com/openstack/governance/commit/1e547ae39f9913dc9cfd9c755b24… I wanted to raise the topic of the watcher-drivers team https://launchpad.net/~watcher-drivers membership and ownership. during the TC sessions we discussed that that the <project>-coresec teams should be owned by the https://launchpad.net/~openstack-admins team https://etherpad.opendev.org/p/r.0f25532f564fb786a89cfe1de39b004b#L123 watcher does not currently have a sperate watcher-coresec team in launchpadd only a a watcher-drivers team i also think that the watcher driver team should be owned by the https://launchpad.net/~openstack-adminsteam. https://launchpad.net/~watcher-drivers is currently owned by https://launchpad.net/~david-tardivel and the only other member with admin rights is https://launchpad.net/~li-canwei2 Can i ask that one of the existing admin either grant my membership request and delegate admin rights to me or add the https://launchpad.net/~openstack-admins team as admins and make them the owner of the team? licanwei, david-tardivel, would either of ye be willing and able to make those changes in launchpad? i also think it would be prudent to add other active member (the current liaisons) as admins/moderators of the watcher-driver team but i can do that once i have access or https://launchpad.net/~openstack-adminscan. having all launchpad teams owned by https://launchpad.net/~openstack-adminsteam mitigates the risk of loosing access by the current admins becoming inactive, but it would be nice to not need to escalation to them for simple tasks like moderation of the group. that is why i want to add the other liaisons to the admins of the team. I am asking for this access for 2 reasons: Fist to add https://launchpad.net/~openstack-adminsas admins of the team and the launchpad project, Second so that i can create new launchpad series for current releases to do blueprint and bug tracking. at present because we do not have launchpad series fro 2025.1 we cannot properly track bugs/blueprints for the current cycle. In the future if there are no objections form the existing contributors i would also like to create a separate open bug team with less project wide permission so that anyone can contribute to bug triage without needing access to the drivers team. this is in lien with how other project like nova work for example https://launchpad.net/~nova-bugs i also think we should be creating a separate watcher-coresec team to allow dan as the security liaison and the vulnerability management team to properly manage security reports. once we have access to the watcher-drivers team we should be able to update the launchpad project to make that possible. Regards sean.

3 4

[watcher][tc][tempest] watcher gerrit group membership
by Sean Mooney 28 Nov '24

28 Nov '24

Hi everyone o/ As many will be aware, over the last month or two, I and some others have been looking to help maintain and extend the Watcher project. At the ptg, I proposed several topics to cover the administrivia of reviving the project. To bring it to a healthy state where features, bugs and general maintenance could be performed on an ongoing basis and avoid its retirement at the end of the 2025.1 cycle. One of the follow-up items to enable that was the extension of the watcher Gerrit groups to allow others to supplement the largely inactive core groups. https://etherpad.opendev.org/p/r.79644f8f3cd04686c4f45c10aa35d8ac#L47 <https://etherpad.opendev.org/p/r.79644f8f3cd04686c4f45c10aa35d8ac#L47> To that end, I would like to ask the existing watcher core team and TC (or other community members :) ) to provide input on the following proposal. 0.) I want to state upfront that I do not want to take away review rights or diminish the experience and efforts of any of the past or present members of the watcher team. I am sending this proposal to enable the watcher project to function effectively and I am welcome to feedback or objections. 1.) To facilitate the changes in the rest of the proposal I would like to formally ask to be added to all of the existing watcher Gerrit groups * watcher-core https://review.opendev.org/admin/groups/77cd139bd57514e98e5e694a82f7e261b47… <https://review.opendev.org/admin/groups/77cd139bd57514e98e5e694a82f7e261b47…> * watcher-release https://review.opendev.org/admin/groups/b265e6b6457de71c7ecfc2a3c70f0c52cde… <https://review.opendev.org/admin/groups/b265e6b6457de71c7ecfc2a3c70f0c52cde…> * watcher-dashboard-core https://review.opendev.org/admin/groups/77cd139bd57514e98e5e694a82f7e261b47… <https://review.opendev.org/admin/groups/77cd139bd57514e98e5e694a82f7e261b47…> * watcher-dashboard-release https://review.opendev.org/admin/groups/904be58ec9fa0e811166ce6144d6314cc7b… <https://review.opendev.org/admin/groups/904be58ec9fa0e811166ce6144d6314cc7b…> * watcher-specs-core https://review.opendev.org/admin/groups/747af616eb8748aee5b17d9493f9ebbc49e… <https://review.opendev.org/admin/groups/747af616eb8748aee5b17d9493f9ebbc49e…> * watcher-tempest-core https://review.opendev.org/admin/groups/09a91d8e24af9ce44b80062c4851a1d2fa3… <https://review.opendev.org/admin/groups/09a91d8e24af9ce44b80062c4851a1d2fa3…> 2.) watcher-release and watcher-dashboard-release Historically, there has been a benefit to having separate core teams for the master and the stable branches. As the velocity of projects has decreased and projects have become more mature the delineation between the maintainers of the master branch and the stable branches has lessened. I believe there is still merit in having the flexibility to add people directly to the release team independently from the core team so I am not proposing replacing the *-release teams with *-core. To that end, I would like to make the following changes to the membership of watcher-release and watcher-dashboard-release - add stable-maint to both release groups - add watcher-core to watcher-release, as we have done in other projects the ask to the watcher-core team is, if you do not have time to review stable branches, or do not feel comfortable with applying stable policy, then there is no expectation to review on the stable branches. - add watcher-dashboard-core to watcher-dashboard-release - me (sean-k-mooney as release-liaison) to facilitate backports and release activities and Gerrit membership admin. 3.) watcher-core and watcher-dashboard-core As it currently stands the core groups are relatively inactive https://tinyurl.com/bdz9k47n full url: https://openstack.biterg.io/app/dashboards?security_tenant=openstack#/view/…:'Gerrit%20Approvals%20panel%20by%20Bitergia',filters:!(('$state':(store:appState),meta:(alias:Bots,disabled:!f,index:gerrit,key:author_bot,negate:!t,params:(query:!t),type:phrase),query:(match:(author_bot:(query:!t,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:gerrit,key:project,negate:!f,params:(query:watcher),type:phrase),query:(match_phrase:(project:watcher)))),fullScreenMode:!f,options:(darkTheme:!f,hidePanelTitles:!f,useMargins:!t),query:(language:lucene,query:''),timeRestore:!f,title:'Gerrit%20Approvals',viewMode:view <https://openstack.biterg.io/app/dashboards?security_tenant=openstack#/view/…:'Gerrit%20Approvals%20panel%20by%20Bitergia',filters:!(('$state':(store:appState),meta:(alias:Bots,disabled:!f,index:gerrit,key:author_bot,negate:!t,params:(query:!t),type:phrase),query:(match:(author_bot:(query:!t,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:gerrit,key:project,negate:!f,params:(query:watcher),type:phrase),query:(match_phrase:(project:watcher)))),fullScreenMode:!f,options:(darkTheme:!f,hidePanelTitles:!f,useMargins:!t),query:(language:lucene,query:''),timeRestore:!f,title:'Gerrit%20Approvals',viewMode:view>) Dan Smith and Slawek Kaplonski were added to help with the retirement/gate issues on behalf of the TC While Dan has offered to help bootstrap the core team we need to be cognizant that both Dan and Slawek have other commitments. Chenker has previously expressed that they intend to review when time allows but they have limited time to do so. To that end, I would like to propose that I be added to both core teams, To have a health core team we will eventually need to have at least 3 active reviewers. Watcher will still be 1-2 reviews short of that, but I hope that can be addressed over the coming months. As others become active over the 2025.1 cycles I would like to revisit the membership and extend the core team after milestone 2 or more realistically after milestone 3 if others demonstrate interest and capacity to review. 4.) watcher-tempest-core This is a newly created group added by https://review.opendev.org/c/openstack/project-config/+/934357 <https://review.opendev.org/c/openstack/project-config/+/934357> This group intends to allow the review team for the tempest plugin to diverge from the main watcher core group. To enable effective review and maintenance of the tempest plugin I would like to propose the following membership changes. - tempest-core should be added as a fallback, to guide best practices, and review if they have time. - gmann has volunteered to help with this too so I propose they be added as a direct member - I would like to be added to have a second active reviewer available. Note, while I can provide some review input I have less knowledge of tempest in general as such I will rely on Gmann's guidance in this area. 5.) watcher-specs Several years ago Nova expanded the nova-specs core group to include nova-core there were a number of motivating factors but one of them was the large overlap between the two groups. another factor was the role of a spec core (focusing on the design constraints, upgrade impact, API impact ectra) is different from the implementation considerations(testing, code quality, performance) often the line between the two is subtle and blurred. To that end, I would like to propose the following changes. - first, I would like to be directly added to the specs core group - second, I would like to add the watcher-core group to watcher-specs. If this proposal is accepted I would like to try and move forward with this in the next week or so to unblock the pending review backlog. Regards sean PS many of the open reviews are cleanup i have submitted so this does not directly address review bandwidth for my personal patches but i think it still has value in general.

3 3

Re: [kolla][oslo] RabbitMQ issue Adding/Removing a Controller Node in an OpenStack Cluster
by Ishan Shanware (ishanwar) 28 Nov '24

28 Nov '24

For everyone’s reference From: Ishan Shanware (ishanwar) <ishanwar(a)cisco.com> Date: Wednesday, 27 November 2024 at 10:01 PM To: Sven Kieske <kieske(a)osism.tech> Subject: Re: [kolla][oslo] RabbitMQ issue Adding/Removing a Controller Node in an OpenStack Cluster Hi Sven, Thanks for getting back to me. We have configured kolla to use 3 controllers in each cluster. Each controller has one rabbitmq container created by kolla. The process we follow move a controller out of rotation is as follows: 1. We first failover all the l3 and dhcp agents running on the controller to the 2 other controllers. 2. We then turn of the server and then delete any neutron agents on it. Currently the rabbitmq cluster only consists of only 2 containers. The problem specifically occurs during the reinstall of the rabbitmq container in the kolla deploy stage. Specifically, after it executes the checking rabbitmq containers role. It triggers a restart for the contains on all the blades. After this stage completed we observe that the masakari-engine disables all the nova-computes. Let me know if you need any other information for discussion Thanks Ishan From: Sven Kieske <kieske(a)osism.tech> Date: Tuesday, 26 November 2024 at 11:36 PM To: Ishan Shanware (ishanwar) <ishanwar(a)cisco.com>, openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [kolla][oslo] RabbitMQ issue Adding/Removing a Controller Node in an OpenStack Cluster Hi Ishan, my first question would be, how many controller nodes and specifically, rabbitmq nodes you are running inside your Openstack cluster? You should be following the general guidelines for running any raft consensus based distributed system and only run an odd number of systems, e.g. 3 or 5 control nodes. Can you confirm that this is the case in your setup? If you e.g. run a two node setup such errors are indeed expected. See also our production architecture guide: https://docs.openstack.org/kolla-ansible/latest/admin/production-architectu… > Control - Cloud controller nodes which host control services like APIs and databases. This group should have odd number of nodes for quorum. If you are running an odd number of control nodes and you're still facing this issue, I would be curious to know the rabbitmq cluster state before you add or remove a node, because this should theoretically just work. But maybe there is another issue with your cluster. HTH -- Sven Kieske Senior Cloud Engineer Mail: kieske(a)osism.tech Web: https://osism.tech OSISM GmbH / Talweg 8 / 75417 Mühlacker / Deutschland Geschäftsführer: Christian Berendt Unternehmenssitz: Mühlacker Amtsgericht: Stuttgart, HRB 756139

1 1

[watcher] Canceling Dec 5th weekly meeting
by Douglas Viroel 28 Nov '24

28 Nov '24

Hi all o/ Since most of our audience will be out next week, we agreed[1] on canceling watcher's weekly meeting[2], on Dec 5th. We will meet again in two weeks, on Dec 12th. [1] https://meetings.opendev.org/meetings/watcher/2024/watcher.2024-11-28-12.00… [2] https://meetings.opendev.org/#Watcher_Team_Meeting BR, -- Douglas Viroel - dviroel

1 0

[kolla-ansible] nova-cell can't not find nova.keyring file when deploy
by daniel890723＠gmail.com 28 Nov '24

28 Nov '24

hello Daniel here Previously i was testing openstack-helm but it doesn't work so i switch to kolla-ansible with all-in-one mode i was following the official guide and put the keyring file i generate to the config folder: (venv) root:~# nano /etc/kolla/config/nova/ ceph.client.nova.keyring ceph.conf localhost/ nova-compute.conf But when i try to deploy it show the error below TASK [nova-cell : Check nova keyring file] ***************************************************************************************************************** fatal: [localhost]: FAILED! => {"msg": "No file was found when using first_found."} Just in case i also put the keyring file in /config/localhost/nova Still shows the same error But when i put the cinder keyring file in nova folder like this: (venv) root@:~# nano /etc/kolla/config/nova/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf localhost/ nova-compute.conf It can find the cinder keyring file but this time the nova-compute container show error below: unable to find a keyring on /etc/ceph/ceph.client.nova.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Did i do anything wrong or something makes it not able to detect the nova.keyring file P.S. Cinder and Glance works fine and Ceph can be used normally

2 5

[manila] Cancelling Nov 28th upstream weekly meeting
by Carlos Silva 28 Nov '24

28 Nov '24

Hello Zorillas, At least half of our audience tomorrow will not be online tomorrow, as it is thanksgiving in the US, so let's cancel the meeting. We'll resume the weekly meetings on Dec 5th. Regards, carloss

1 0

[ops][kolla-ansible][openstack-ansible] Deploying OpenStack using Ansible to LXC while project is used to grab for Kolla-Ansible
by just.me.on.reddit＠protonmail.com 27 Nov '24

27 Nov '24

Hi, I am learning OS because my imminent mission uses a cloud-based solution (cloud-app) which runs in OpenStack cloud. How do you estimate the probability for a cloud-app to feature built-in Docker dependency (runtime) in situation where the project of mentioned app appears to be used to apply Kolla Ansible in order to deploy OpenStack cloud? It means the cloud-app to depend at runtime on Docker-specific artefacts. 3rd-party cloud-app going to be in use will play a major role in my project. It is in responsibility of cloud-app operator to do provisioning of OpenStack cloud before mentioned app can be deployed to cloud. Recommendations found in cloud-app's documentation read to use Kolla Ansible in order to conduct mentioned provision. Myself learned that Kolla uses Docker. Docker use is that component which triggers me to ask this question. My own preference would be rather to deploy OpenStack services to LXC-containers. One can find in OpenStack web the project Ansible-based deployment to LXC containers. I wonder following if cloud-app developers could had placed in it any built-in Docker (eco-system) runtime dependency and they achieved it by the use of Ansible Kolla for OpenStack deployment. Considering my very flat knowledge of Docker (only familiarized with its purpose and very few exercises made in far past) there exists just hypothetical risk for cloud project in my focus to feature built-in docker eco-system dependency. Should such dependency be a fact (not just a risk) my aim to use OpenStack services in LXC-containers has good chance to fail. It is hard to contact 3rd-party cloud project developer regarding my questions. Question 2 Any common conclusions had been drawn so far if one of the two approaches be of remarkably higher suitableness to host OpenStack-based cloud services? * placing OpenStack-based cloud services to Docker-based containers * placing services of OpenStack-based cloud to Linux containers (LXC/LXD/...)

1 0

[os-vif][nova][neutron] Proposing Rodolfo as an os-vif core
by Sean Mooney 27 Nov '24

27 Nov '24

hi folks o/ I intended to send this last week but didn't find time till now. I'm sure many are familiar with Rodolfo form the work that he has done in neutron over the years. I have worked with them for a long time and over the years they have also contributed too and reviewed changes too os-vif. I think having a neutron core on the os-vif team would certainly help bubble up pain points form them and address such issue faster in os-vif going forward. Rodolfo is familiar with the code and with the upstream review process so i think they would be a good fit. If there are no objection to this ill bring this up again in next weeks nova team meeting and add rodoflo to the core list. reagrds sean.

2 2

[Octavia] Cancelling today's meeting
by Gregory Thiemonge 27 Nov '24

27 Nov '24

Hi Folks, Due to personal reasons, I have to cancel this week's meeting, sorry for the short notice, See you next week, Greg

1 0

Openstack ansible variables
by engineer2024 27 Nov '24

27 Nov '24

I want to print the values of all the ansible variables at the playbook and task level. If we execute os_glance.yml , I want to see the variable values of all the tasks/playbooks involved in this glance playbook. Is it also possible doing it from any one file ? Thanks elinux

1 0

[largescale-sig] Next meeting: November 27, 9utc
by Thierry Carrez 27 Nov '24

27 Nov '24

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 9UTC. You can doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20241127T09 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[devstack][neutron] DNS Resolution Issue After Server Reboot with DevStack
by xuq.fnstxz＠fujitsu.com 27 Nov '24

27 Nov '24

Hi everyone, I am reaching out to seek your assistance regarding an issue I am experiencing with my DevStack installation. After successfully installing DevStack stable/2024.2 on ubuntu 22.04, I configured the network and set up DNS for private subnet. stack@controller:~$ openstack subnet list +--------------------------------------+---------------------+--------------------------------------+---------------------+ | ID | Name | Network | Subnet | +--------------------------------------+---------------------+--------------------------------------+---------------------+ | 2b6411d5-9c19-4a45-bd47-58dc748f5e00 | public-subnet | 45502b6a-773b-40cf-866a-e8b18397ffcb | 192.168.1.0/24 | | 69629170-2e0e-49c0-b741-61560558001c | ipv6-public-subnet | 45502b6a-773b-40cf-866a-e8b18397ffcb | 2001:db8::/64 | | c74bf062-b67a-45cb-8a50-73583498dfcd | ipv6-private-subnet | 8fb37e61-bd59-4624-a4be-1fc1841987d7 | fd3a:2f86:ca98::/64 | | f2e10d3d-f785-4cf9-8804-1969269623c2 | private-subnet | 8fb37e61-bd59-4624-a4be-1fc1841987d7 | 10.0.0.0/26 | +--------------------------------------+---------------------+--------------------------------------+---------------------+ stack@controller:~$ openstack subnet set --dns-nameserver 10.12.255.254 --dns-nameserver 192.168.1.95 f2e10d3d-f785-4cf9-8804-1969269623c2 10.12.255.254 is the real DNS server address and 192.168.1.95 is my server IP address. Initially, when I launched a VM instance, it was able to resolve the proxy and connect to the external network without any issues. jenkins@np0000000531:~$ curl opendev.org <a href="https://opendev.org/">Temporary Redirect</a>. However, after rebooting the server and launching the instance again, the instance is unable to resolve the proxy. jenkins@np0000000531:~$ curl opendev.org curl: (5) Could not resolve proxy: my.proxy.com I have verified that the systemd-resolved service is in an active state, but the logs show a large number of entries with the following message: Nov 13 11:53:51 np0000000531 systemd-resolved[568]: Using degraded feature set UDP instead of TCP for DNS server 10.12.255.254. Nov 13 11:54:01 np0000000531 systemd-resolved[568]: Using degraded feature set UDP instead of TCP for DNS server 192.168.1.95. Nov 13 11:54:01 np0000000531 systemd-resolved[568]: Using degraded feature set TCP instead of UDP for DNS server 10.12.255.254. Nov 13 11:54:11 np0000000531 systemd-resolved[568]: Using degraded feature set TCP instead of UDP for DNS server 192.168.1.95. Nov 13 11:54:21 np0000000531 systemd-resolved[568]: Using degraded feature set UDP instead of TCP for DNS server 10.12.255.254. I have checked the network and routing table of the VM instance, as well as the bridge settings on the server. Everything appears to be the same as it was before the reboot. Could you please help me identify the cause of this issue and suggest any potential solutions? If you need any additional information, please let me know, and I will be happy to provide it. Thank you for your time and assistance. Best regards, Inori.

2 3

[all] 2025 Open Infrastructure Foundation Individual Director nominations are open
by Jeremy Stanley 27 Nov '24

27 Nov '24

To those who aren't following the foundation mailing list, please be aware that the nomination period for the 2025 Open Infrastructure Foundation Board of Director Individual Member elections is open, and will close at 23:59 UTC on December 13. If you've considered taking on some additional leadership responsibility, this could be your opportunity! See Wes's post for more information: https://lists.openinfra.dev/archives/list/foundation@lists.openinfra.dev/th… -- Jeremy Stanley

1 0

[kolla] RabbitMQ issue Adding/Removing a Controller Node in an OpenStack Cluster
by Ishan Shanware (ishanwar) 26 Nov '24

26 Nov '24

Hi Kolla Community, I would like to understand the current guidelines for adding or removing a controller node from an OpenStack cluster. With the introduction of quorum queues, we’ve observed that adding a new controller node causes a loss of quorum within the cluster. Consequently, during the reinstallation of a controller node, Masakari disables all the nova-compute services. Is this behaviour expected with quorum queues, or could this be addressed differently? Additionally, I’ve noticed that as soon as the Kolla containers are restarted in the cluster, clients are unable to connect, causing the cluster to go down. Could you please share insights or recommendations on how to handle this scenario effectively? Looking forward to hearing your thoughts. Thanks, Ishan Cisco Systems

2 2

[keystone] Weekly meeting canceled 27-Nov-2024
by Dave Wilde 26 Nov '24

26 Nov '24

Hello, Due to the Thanksgiving holiday in the US I'm canceling the weekly meeting and reviewathon this week. Please reach out here or on IRC if you need anything. Thanks! /Dave

1 0

[kolla] [openeuler] CI jobs status
by Michał Nasiadka 26 Nov '24

26 Nov '24

Hello Koalas, OpenEuler jobs in master (to be Antelope) are failing for a long time. If no one is interested in maintaining OpenEuler support in Kolla-Ansible - I will propose dropping it’s support on Wednesday Kolla meeting. Best regards, Michal

4 3

[openstack helm] [Ansible] Openstack-helm Ansible inventory.yaml missing some value
by daniel890723＠gmail.com 26 Nov '24

26 Nov '24

Hello today i'm testing installation with completely new enviroment i was following the guide from Openstack-helm document and install k8s cluster using ansible and deploy-env playbook but i encounter this error when setup ssh cluster TASK [deploy-env : Generate ssh key pair] ****************************************************************************************************************** skipping: [node-1] fatal: [primary]: FAILED! => {"msg": "Failed to change ownership of the temporary files Ansible (via chmod nor setfacl) needs to create despite connecting as a privileged user. Unprivileged become user would be unable to read the file."} and the log with -vvv output: debug2: mux_client_hello_exchange: master version 4 debug3: mux_client_forwards: request forwardings: 0 local, 0 remote debug3: mux_client_request_session: entering debug3: mux_client_request_alive: entering debug3: mux_client_request_alive: done pid = 114038 debug3: mux_client_request_session: session request sent debug1: mux_client_request_session: master session id: 2 chown: invalid user: ‘zuul’ debug3: mux_client_read_packet: read header failed: Broken pipe debug2: Received exit status from master 1 And by checking the playbook we can see the playbook is using "{{ client_ssh_user }}" on primary node and "{{ cluster_ssh_user }}" with cluster node which is not defined in inventory.yaml Causing it using default user zuul by adding client_ssh_user & cluster_ssh_user in ansible inventory it works normally Is this meant to be like this? or the document missing some variables in inventory.yaml

3 3

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-18)
by Goutham Pacha Ravi 26 Nov '24

26 Nov '24

Hello Stackers, We're 18 weeks away from the coordinated release for 2025.1 ("Epoxy") [1]. We're beginning a period of low activity due to year-end holidays across the world. Last week, the TC didn't merge any new governance proposals; however, several are currently in progress. === Weekly Meeting === The TC met in OFTC's #openstack-tc IRC channel on 2024-11-19 [2]. We discussed a series of global requirements upper constraint bumps that were committed and the ensuing CI flakiness. We appreciated the effort to proactively update dependencies to their latest versions earlier in the release cycle so we can have more time to test and stabilize OpenStack software. Sometimes, these failures may interrupt other project priorities. The best way we can help is if project teams identify a relatively quick-running and stable integration job to run against changes to the global requirements repository. The TC and OpenDev administrators worked with Launchpad administrators to regain control of trackers belonging to the Watcher project team. It was noted that Launchpad trackers for any project should not be owned/administered by individuals. Instead, there must be a Launchpad team driving all trackers associated with an OpenStack initiative, and "openstack-admins" must own this team. This approach ensures the sustainability of project teams and protects projects from the impact of changing individual priorities. We also briefly discussed the "eventlet-removal" goal proposal. The goal document is being rewritten [3] to simplify what is being done and the timelines we're aiming for. I'd love for the community to take a look at this change and review it. The next meeting of the OpenStack Technical Committee is today, 2024-11-25, at 1800 UTC in OFTC's #openstack-tc channel. The agenda for this meeting is in our meeting wiki [4]. I hope you're able to join us. === Governance Proposals === ==== Open for Review ==== Add Cinder Huawei charm | https://review.opendev.org/c/openstack/governance/+/867588 Rework the initial goal proposal as suggested by people | https://review.opendev.org/c/openstack/governance/+/931254 Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/934936 Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 Retire Murano/Senlin/Sahara OpenStack-Ansible roles | https://review.opendev.org/c/openstack/governance/+/935677 Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 ==== Merged ==== Fix doc job for pillow 11.0.0 | https://review.opendev.org/c/openstack/governance/+/935967 Add Ubuntu Noble migration goal tracking etherpad | https://review.opendev.org/c/openstack/governance/+/935461 === How to Contact the TC === You can reach the TC in several ways: - Email: Send an email with the tag [tc] on this mailing list. - IRC: Ping us using the 'tc-members' keyword on the #openstack-tc IRC channel on OFTC. - Weekly Meetings: Join us at our weekly meeting. The Technical Committee meets every week on Tuesdays at 1800 UTC [4]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] TC Meeting IRC Log 2024-11-19: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-19-18.01.log.html [3] Eventlet removal Community goal: https://review.opendev.org/c/openstack/governance/+/931254 [4] TC Meeting Agenda, 2024-11-26: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

1 0

[docs][dev] replace_global_vars not working in project-deploy-guide
by Jakub Darmach 26 Nov '24

26 Nov '24

Hello everyone, I've been updating documentation in kolla-ansible and noticed that while replacing variable content works in https://docs.openstack.org/kolla-ansible it fails to replace the variable with proper content in https://docs.openstack.org/project-deploy-guide/kolla-ansible. If I understand correctly, it's being substituted with var having proper content in here: https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/co… Later on used in https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/co… And now for the example - |KOLLA_BRANCH_NAME| variable: Working substitution: https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html#t… https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/us… Failed to render content: https://docs.openstack.org/project-deploy-guide/kolla-ansible/latest/quicks… https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/us… The var seems to be used the same in both examples, I suspect slightly different configuration on project end of the things. Can anyone advise where should I start looking? Best regards, Jakub

2 1

[neutron] Bug deputy report for week of November 18th
by Brian Haley 26 Nov '24

26 Nov '24

Hi, I was the Neutron bug deputy last week. Below is a short summary about the reported bugs. -Brian Critical bugs ------------- * https://bugs.launchpad.net/neutron/+bug/2088423 - [functional test] test_ovn_nb_sync* tests fails randomly with Internal Server Error - https://review.opendev.org/c/openstack/neutron/+/935804 High bugs --------- * https://bugs.launchpad.net/neutron/+bug/2088453 - [UT] Neutron tests failing with eventlet 0.37.0 - https://review.opendev.org/c/openstack/neutron/+/935524 * https://bugs.launchpad.net/neutron/+bug/2089157 - [neutron-specs] CI job "openstack-tox-docs" broken - https://review.opendev.org/c/openstack/neutron-specs/+/935836 * https://bugs.launchpad.net/neutron/+bug/2089250 - neutron-vpnaas: Automatic VPN Agent failover not working with OVN driver - needs owner * https://bugs.launchpad.net/neutron/+bug/2089252 - neutron-vpnaas: LibreSwan 4 connections not listed as ACTIVE - Submitter said he would look this week Medium ------ * https://bugs.launchpad.net/neutron/+bug/2089169 - [OVN] Test ``TestCreateNeutronPgDrop.test_non_existing`` is randomly failing - https://review.opendev.org/c/openstack/neutron/+/935802

1 0

[cinder] Meeting canceled this week, 27 Nov
by Jon Bernard 25 Nov '24

25 Nov '24

Hello all, schools are closed this week so I won't be able to host the weekly meeting due to the holidays. I'll see you all the following week. Take care, -- Jon

1 0

[horizon][ceph] object storage container storage policy missing
by Satish Patel 25 Nov '24

25 Nov '24

Folks, Openstack Release: Zed Recently I have integrated ceph RGW service with openstack/keystone and testing with command line tool "openstack container create foo1" and it works but when I go to Horizon UI I am seeing it doesn't have storage policy dropdown list and because of that I am not able to create container or upload file from Horizon GUI https://ibb.co/KhYW6Rd It works with commands like but not horizon. $ openstack container create foo1 +---------------------------------------+-----------+------------------------------------------------+ | account | container | x-trans-id | +---------------------------------------+-----------+------------------------------------------------+ | AUTH_d4d82849885e4126aef402c408ad2c5c | foo1 | tx00000c057454841b13497-0065b558b4-42ff2eb-ahm | +---------------------------------------+-----------+------------------------------------------------+ $ openstack container list +------+ | Name | +------+ | foo1 | +------+

2 2

[Neutron] - Unable to ping my instance from my Openstack controller node
by ABATCHA OLLOH 23 Nov '24

23 Nov '24

Hello Everyone, Could you give me any advice and help me to better understand neutron. On my VirtualBox VM I properly installed Openstack using packstack (all-in-one installation). I have access to the horizon dashboard. I'm able to launch an instance and associate floating IP but from the controller node I cannot reach my instance. - Here are my interfaces config: - enp0s3: for mgmt - enp0s8: for openstack services - enp0s9: for neutron_external_interface (used as a public interface) - enp0s10: interface on my controller to be connected to the neutron_external_interface - VirtualBox Promiscuous mode : Allow All [root@packstack ~(keystone_admin)]# ip -br -c a lo UNKNOWN 127.0.0.1/8 ::1/128 enp0s3 UP 9.10.93.8/24 fe80::a00:27ff:fe2e:150a/64 enp0s8 UP 9.11.93.8/24 fe80::a00:27ff:fec7:56ab/64 enp0s9 UP fe80::a00:27ff:fef9:3cc7/64 enp0s10 UP 9.12.93.15/24 fe80::a00:27ff:feff:3641/64 ovs-system DOWN br-tun DOWN br-int DOWN br-ex UNKNOWN 9.12.93.8/24 fe80::b021:85ff:fe8a:9d44/64 qbr9eefea66-89 UP qvo9eefea66-89@qvb9eefea66-89 UP fe80::1409:2aff:feb4:e37d/64 qvb9eefea66-89@qvo9eefea66-89 UP fe80::8c84:15ff:fe7d:8896/64 tap9eefea66-89 UNKNOWN fe80::fc16:3eff:fec6:5f5c/64 - Security Group Rules associated to my instance - Egress IPv4 Any Any 0.0.0.0/0 - Ingress IPv4 ICMP Any 0.0.0.0/0 - Ingress IPv4 TCP 22 (SSH) 0.0.0.0/0 - Ingress IPv4 TCP 80 (HTTP) 0.0.0.0/0 - Instances status [root@packstack ~(keystone_admin)]# openstack server list +--------------------------------------+-----------+--------+----------------------------------+--------------------------+-----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------+--------+----------------------------------+--------------------------+-----------+ | 8d2e2f04-8080-44df-923f-9728ebabe9e5 | testrocky | ACTIVE | private1=10.0.0.202, 9.12.93.203 | N/A (booted from volume) | m1.devops | +--------------------------------------+-----------+--------+----------------------------------+--------------------------+-----------+ - From the router inside my openstack I can ping the floating ip [root@packstack ~(keystone_admin)]# ip netns list qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e (id: 2) qdhcp-2a02741e-35f0-4a61-81b0-abd4b5a09f36 (id: 0) qdhcp-bcc1c132-074f-45d5-a715-a2d371cdb5be (id: 1) [root@packstack ~(keystone_admin)]# ip netns exec qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e ping PING 9.12.93.203 (9.12.93.203) 56(84) bytes of data. 64 bytes from 9.12.93.203: icmp_seq=1 ttl=64 time=6.67 ms 64 bytes from 9.12.93.203: icmp_seq=2 ttl=64 time=1.51 ms9.12.93.203 [root@packstack ~(keystone_admin)]# ip netns qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e (id: 2) qdhcp-2a02741e-35f0-4a61-81b0-abd4b5a09f36 (id: 0) qdhcp-bcc1c132-074f-45d5-a715-a2d371cdb5be (id: 1) [root@packstack ~(keystone_admin)]# ip netns exec qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e ip route default via 9.12.93.1 dev qg-68351942-28 proto static 9.12.93.0/24 dev qg-68351942-28 proto kernel scope link src 9.12.93.201 10.0.0.0/24 dev qr-5ba4fd5b-b5 proto kernel scope link src 10.0.0.1 - Trying to reach the floating IP from the controller node [root@packstack ~(keystone_admin)]# ping PING 9.12.93.203 (9.12.93.203) 56(84) bytes of data. From icmp_seq=1 Destination Host Unreachable ...9.12.93.2039.12.93.8 - Check floating ip status [root@packstack ~(keystone_admin)]# openstack port list --network private1 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ | 5ba4fd5b-b5e1-4f56-b681-963cf2c9ac8a | | fa:16:3e:14:0b:4a | ip_address='10.0.0.1', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | ACTIVE | | 9eefea66-897b-4682-883c-b61e22c3c09a | | fa:16:3e:c6:5f:5c | ip_address='10.0.0.202', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | ACTIVE | | a73cd16c-8480-4461-99f2-7837e10f27b1 | | fa:16:3e:91:5f:98 | ip_address='10.0.0.200', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | ACTIVE | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ [root@packstack ~(keystone_admin)]# openstack port show 9eefea66-897b-4682-883c-b61e22c3c09a +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | packstack.example.local | | binding_profile | | | binding_vif_details | bound_drivers.0='openvswitch', bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='True', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | normal | | created_at | 2024-11-15T11:38:07Z | | data_plane_status | None | | description | | | device_id | 8d2e2f04-8080-44df-923f-9728ebabe9e5 | | device_owner | compute:nova | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='10.0.0.202', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | | hardware_offload_type | None | | hints | | | id | 9eefea66-897b-4682-883c-b61e22c3c09a | | ip_allocation | None | | mac_address | fa:16:3e:c6:5f:5c | | name | | | network_id | 2a02741e-35f0-4a61-81b0-abd4b5a09f36 | | numa_affinity_policy | None | | port_security_enabled | True | | project_id | bbba2507099c4209a529acd64352939e | | propagate_uplink_status | None | | resource_request | None | | revision_number | 10 | | qos_network_policy_id | None | | qos_policy_id | None | | security_group_ids | 2a206a8d-fb54-4fa7-9db7-e5f197dcc107 | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2024-11-18T08:15:49Z | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+ [root@packstack ~(keystone_admin)]# openstack port list --network public1 +--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+ | 2b215f41-edf8-4c61-8969-383143340444 | | fa:16:3e:30:7e:08 | ip_address='9.12.93.200', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | ACTIVE | | 31d7b194-50a0-4a25-b102-542210e5f3f3 | | fa:16:3e:28:39:a9 | ip_address='9.12.93.203', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | N/A | | 68351942-28a1-4df3-8661-bf157fcd5982 | | fa:16:3e:bf:66:56 | ip_address='9.12.93.201', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | ACTIVE | +--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+ [root@packstack ~(keystone_admin)]# openstack port show 31d7b194-50a0-4a25-b102-542210e5f3f3 +-------------------------+----------------------------------------------------------------------------+ | Field | Value | +-------------------------+----------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | | | binding_profile | | | binding_vif_details | | | binding_vif_type | unbound | | binding_vnic_type | normal | | created_at | 2024-11-15T15:57:42Z | | data_plane_status | None | | description | | | device_id | 3dc9d9c3-28eb-4dfb-a41b-9bbfac9f96da | | device_owner | network:floatingip | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='9.12.93.203', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | | hardware_offload_type | None | | hints | | | id | 31d7b194-50a0-4a25-b102-542210e5f3f3 | | ip_allocation | None | | mac_address | fa:16:3e:28:39:a9 | | name | | | network_id | bcc1c132-074f-45d5-a715-a2d371cdb5be | | numa_affinity_policy | None | | port_security_enabled | False | | project_id | | | propagate_uplink_status | None | | resource_request | None | | revision_number | 2 | | qos_network_policy_id | None | | qos_policy_id | None | | security_group_ids | | | status | N/A | | tags | | | trunk_details | None | | updated_at | 2024-11-15T15:57:43Z | +-------------------------+----------------------------------------------------------------------------+ - I couldn't find something related to the binding port in these logs file. - tail -f /var/log/neutron/openvswitch-agent.log - tail -f /var/log/neutron/server.log - tail -f /var/log/neutron/l3-agent.log [image: r/openstack - [Neutron] - Unable to ping my instance from my Openstack controller node] <https://preview.redd.it/neutron-unable-to-ping-my-instance-from-my-openstac…>[image: r/openstack - [Neutron] - Unable to ping my instance from my Openstack controller node] <https://preview.redd.it/neutron-unable-to-ping-my-instance-from-my-openstac…>[image: r/openstack - [Neutron] - Unable to ping my instance from my Openstack controller node] <https://preview.redd.it/neutron-unable-to-ping-my-instance-from-my-openstac…> Any advise will be welcome

2 2

Live Migration problems
by Gilles Mocellin 22 Nov '24

22 Nov '24

Hi Stackers, I have 2 recurring problems with live migration. If someone already face that and have recommendations... My env : - OpenStack 2023.1, deployed by OpenStack-Ansible 27.5.1 - Network : ML2/LinuxBridges - Some shared Networks on a VLAN provider - Most private networks on VXLAN - OS : Ubuntu 20.04 - libvirt-daemon-system: 8.0.0-1ubuntu7.10~cloud0 - qemu-system-x86: 1:4.2-3ubuntu6.30 Each time we have maintenance to do on compute nodes, we live-migrate the instances, with `nova host-evacuate-live` (still no openstack CLI command to do that). 1. Never ending / failed migrations In general, it works, but each time we have 1 or 2 (on 40 instances) that don't want to complete. It's some memory active instances (Kubernetes controller nodes with etcd...). Even if I had configured live_migration_permit_auto_converge to True. Now, if we try to force complete (openstack server migration force complete), migration fails, instance return to ACTIVE status on source. And I can see libvirt errors : [...] libvirt.libvirtError: operation failed: migration out job: unexpectedly failed Not really helpful. I tried removing live_migration_permit_auto_converge, reduce live_migration_completion_timeout to 60 and set live_migration_timeout_action to force_complete. Same problems. 2. Instance unreachable More problematic, as it's not easy to see : some instances are not reachable after migration. And after a while, DHCP leases expire. The only way to recover is to restart neutron-linuxbridge-agent on the instance compute node. In fact, on all compute nodes, because instances evacuation spread them on all the compute nodes. Then, if DHCP has not expired, the instance is joinable again. We made a script to find those instances, it lists all ports bound to a nova instance. Then connects to each control/network nodes, enter qdhcp network namespaces (found with the MAC adresse in DHCP leases files), and try to ping the instance private IP. The only log I see, but don't know if it has something to do with that is : Device tapxxx-yy has no active binding in host But I can see that message even since, without unjoinable instances... Here it is, if something ring a bell, for an existing bug or configuration error, please let us know ! Thanks. -- Gilles

3 4

[openstack-ansible][horizon]The token used to make the request was domain scoped but the policy requires ['system', 'project'] scope
by Danish Khan 21 Nov '24

21 Nov '24

Dear All, I have upgraded one of my test openstack cluster from victoria to Wallaby. After upgrading, the hypervisor is not showing correct numbers of VMs on compute nodes from browser. But openstack CLI is giving correct number of VMs. I am getting below error in horizon container when I click on "hypervisor" from browser: Nov 20 11:17:29 horizon-container-2b244cbd apache2[43317]: [wsgi:error] [pid 43317:tid 139897181898496] [remote x.x.x.151:37180] /openstack/venvs/horizon-23.3.1.dev4/lib/python3.8/site-packages/oslo_policy/policy.py:1065: UserWarning: Policy os_compute_api:os-migrate-server:migrate failed scope check. The token used to make the request was domain scoped but the policy requires ['system', 'project'] scope. This behavior may change in the future where using the intended scope is required I did try few thing but that did not work. Regards, Danish Khan

3 5