September 2024 - openstack-discuss

Kolla Ansible on Ubuntu 20.04 - cloud-init & other network issues
by Tobias McNulty 10 Oct '24

10 Oct '24

Hi, I'm attempting to use Kolla Ansible 14.6.0 to deploy OpenStack Yoga on a small 3-node Ubuntu 20.04 cluster. The nodes have 128 GB RAM each, dual Xeon processors, and dual 10G Intel NICs. The NICs are connected to access ports on a 10G switch with separate VLANs for the local and external networks. All the playbooks run cleanly, but cloud-init is failing in the Ubuntu 20.04 and 22.04 VMs I attempt to boot. The VM images are unmodified from https://cloud-images.ubuntu.com/, and cloud-init works fine if I mount a second volume with user-data. The error is a timeout attempting to reach 169.254.169.254. This occurs both when booting a VM in an internal routed network and directly in an external network. I tried various neutron plugin agents (ovn, linuxbridge, and openvswitch both with and without firewall_driver = openvswitch <https://docs.openstack.org/kolla-ansible/latest/reference/networking/neutro…>) first with a clean install of the entire OS each time, all with the same result. Running tcpdump looking for 169.254.169.254 shows nothing. As a possible clue, the virtual NICs are unable to pass any traffic (e.g., to reach an external DHCP server) unless I completely disable port security on the interface (even if the associated security group is wide open). But disabling port security does not fix cloud-init (not to mention I don't really want to disable port security). Are there any additional requirements related to deploying OpenStack with Kolla on Ubuntu 20.04? This is a fairly vanilla configuration using the multinode inventory as a starting point. I tried to follow the Quick Start <https://docs.openstack.org/kolla-ansible/yoga/user/quickstart.html> as closely as possible; the only material difference I see is that I'm using the same 3 nodes for control + compute. I am using MAAS so it's easy to get a clean OS install on all three nodes ahead of each attempt. I plan to try again with the standard (non-HWE) kernel just in case, but otherwise I am running out of ideas. In case of any additional clues, here are my globals.yml and inventory file, along with the playbook I'm using to configure the network, images, VMs, etc., after bootstrapping the cluster: https://gist.github.com/tobiasmcnulty/7dbbdbc67abc08cbb013bf5983852ed6 Thank you in advance for any advice! Cheers, Tobias

5 9

Improving stestr... or switching to a (better) compatible test runner?
by Thomas Goirand 09 Oct '24

09 Oct '24

Hi, This is the 2nd time I'm opening a thread about it. Sorry for this, but it really bothers me. Today, running tempest, I got these stats from stestr: - Worker 0 (132 tests) => 0:24:49.209885 - Worker 1 (100 tests) => 0:17:14.791845 - Worker 2 (124 tests) => 0:42:52.690906 - Worker 3 (189 tests) => 0:41:21.307241 - Worker 4 (159 tests) => 0:45:49.503031 - Worker 5 (143 tests) => 0:28:13.282371 - Worker 6 (156 tests) => 3:16:52.364976 - Worker 7 (103 tests) => 0:46:05.366089 So, thread #1 ran 17 minutes, sitting idle for the rest of the 3 hours run of thread #6. While I thought stestr was deviding the number of tests by the number of thread, I don't get why worker #1 only had 100 tests assigned. All together, all tests could have been ran within maybe less than an hour of time (rather than 3h16 above), if not-yet-ran-tests were reassigned to idle treads. I'm currently spending a lot of time on running tempest. Indeed, I'm running tempest on each OpenStack upgrade, from Victoria up to Dalmatian. With the current way stestr run, it may take 2 full days to do that (and that's not even counting when upgrade will fail and will need fixing...), when it could be done in maybe 8 hours (if I count 1h per OpenStack release). So, knowing the above, it might be a good use of my time to dig into stestr and see if I can fix this... or not! Does anyone have a suggestion for another test runner, that's compatible with stestr, at least for the tests selection with a regular expression? As much as I know, pytest cannot take a regexp for test selection, can it? Or is there maybe a plugin for it? If there's no compatible test runner, where should I dig in the stestr code to rewrite things in a smarter way? Cheers, Thomas Goirand (zigo) P.S: On Caracal today, I just had this: ============== Worker Balance ============== - Worker 0 (117 tests) => 0:59:32.291385 - Worker 1 (147 tests) => 1:22:28.228980 - Worker 2 (125 tests) => 0:45:20.969397 - Worker 3 (114 tests) => 1:46:21.667579 - Worker 4 (170 tests) => 0:45:27.577738 - Worker 5 (190 tests) => 2:28:39.744920 - Worker 6 (182 tests) => 2:29:01.402255 - Worker 7 (152 tests) => 2:29:10.183359 this looks better, but that's probably 1/ random 2/ still not perfect, with worker #0, #2 and #4 doing nothing 2/3rd of the time.

2 1

[tc][watcher] No leaders for project team, heading to retirement
by Goutham Pacha Ravi 08 Oct '24

08 Oct '24

Hello Stackers, The "watcher" project [1] did not have any volunteers to lead the project in the 2025.1 election cycle. In the past few years, the project's team has slowly disbanded and there have been no significant changes proposed/merged in the project's repositories [2]. We noticed that there were three deployers that noted their use of Watcher in production in the past user survey [3]. It's possible that the project failed to gather any significant adoption, and hence there's less interest in contributing to, and maintaining the project. I'm sending this notification pursuant to the OpenStack governance recommendations [4] to retire project teams. We can mark a project team "inactive", however, an "inactive" project must have some contributors aiming to build-back a maintainers team around it - and it must pick a project team lead, or adopt a distributed project leadership model by nominating liaisons. Since there are no PTL nominees, we think marking this project "inactive" serves no purpose. Is there any objection to retiring the "openstack/watcher" project (and associated deliverables: python-watcherclient, watcher-dashboard and watcher-tempest-plugin)? Once the project is retired, configuration management and deployment projects can drop their integrations as well. Thanks, On behalf of the OpenStack TC, Goutham Pacha Ravi [1] https://wiki.openstack.org/wiki/Watcher [2] https://review.opendev.org/q/(project:%5Eopenstack/watcher.*+OR+project:ope… [3] https://governance.openstack.org/tc/user_survey/analysis-2023.html [4] https://governance.openstack.org/tc/reference/dropping-projects.html

5 7

Stop bumping major versions when dropping EOL Python versions
by Stephen Finucane 04 Oct '24

04 Oct '24

o/ Python 3.8 is going EOL (upstream, anyway) next month. We've been working to drop support for this from various SDK deliverables, and I'm guessing (though don't yet know for sure) that this will happen across other project deliverables before long (if it hasn't started happening already). One side-effect of this change is that we will be expected to raise the major version of the affected dependencies on our next release. This is annoying for projects like openstacksdk that use the release-with-intermediary release model because it forces us to have a major version release earlier than we would like to. To give a concrete example, we have a RemovedInSDK50Warning deprecation warning that we are using to indicate (as you might guess) things that we plan to remove in openstacksdk 5.0.0. We need to wait 6 months/1 OpenStack release before removing anything flagged with this warning, so this has been mainly applied to legacy deprecation warnings to put a concrete timeline on their removal. However, because we just dropped Python 3.8 support, it is currently expected that our next release be 5.0.0. We last deprecated something new ~2 months ago, so this means we either cannot release a new version of openstacksdk for 3-4 months (to allow 6 months to pass) or we need to either rename RemovedInSDK50Warning to RemovedInSDK60Warning or add a new RemovedInSDK60Warning and move the offending deprecations to this. None of these are nice options and to be honest, I don't think the major version bump really provides all that much value to end-users: they already have a much better signal for the removal of a Python version in the 'python_requires' marker in setup.py/setup.cfg/pyproject.toml/(other build system file). By using this, we ensure that a user simply can't install a version of a package that doesn't support their chosen Python version. I'd be in favour of getting rid of this particular constraint, and instead focusing on API changes when deciding whether to bump the major version. Is this a reasonable suggestion, and if so what would we need to do to tweak our policy to allow it? Cheers, Stephen

4 3

[Kolla-Ansible] [Ironic] Node Deploy Failed with MissingAuthPlugin
by Ali Hussain 01 Oct '24

01 Oct '24

Hi Kolla and Ironic folks, While playing with stable/kolla release to fix the No VIF found error. Looks like adding Readers role to ironic user fixed it. But run into Missing Auth plugin [2] issue while image is being deployed to ironic node. Openstack endpoint list command [1] shows the correct IP of the controller. My question is: 1. Kolla-Ansible needs explicite Ironic keystone configuration in Ironic.conf? Following this link <https://docs.openstack.org/kolla-ansible/yoga/reference/bare-metal/ironic-g…> which does not seems to need any keystone configurations. This issue is observed when 1. --boot-interface redfish-virtual-media 2. --deploy-interface ramdisk 3. --driver-info deploy_iso= 4. -instance-info kernel & ramdisk 2. My passwords.yaml has 1. ironic_database_password: 2. ironic_inspector_database_password: 3. ironic_inspector_keystone_password: 4. ironic_keystone_password: Any suggestions please. Thank you. Ali [1] openstack endpoint list | grep ironic | <ID> | <Region> | ironic-inspector | baremetal-introspection | True | internal | http://<IP>:5050 | | <ID> | <Region> | ironic | baremetal | True | public | http://<IP>:6385 | | <ID> | <Region> | ironic-inspector | baremetal-introspection | True | public | http://<IP>:5050 | | <ID> | <Region> | ironic | baremetal | True | internal | http://<IP>:6385 [2] *Error: *Detected change in error condition on node Node1. Deploy step deploy.deploy failed with MissingAuthPlugin: An auth plugin is required to determine endpoint URL.. An unhandled exception was encountered while aborting. More information may be found in the log file.

2 6

[largescale-sig] Next meeting: September 18, 9utc
by Thierry Carrez 30 Sep '24

30 Sep '24

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 9UTC. You can doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20240918T09 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

3 3

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2024.2/R-0)
by Goutham Pacha Ravi 30 Sep '24

30 Sep '24

Hello Stackers, After 26 weeks of effort across the community, we're finally at the release week for the 2024.2 release [1], codenamed 'Dalmatian'. I'll cue "Who Let the Dogs Out" for 2024-10-02 and toast the successful delivery of the 30th release of OpenStack with all of you. Congratulations, and thank you for your contributions in making this yet another on-time release. Please tune into the OpenInfra Live episode at 1400 UTC on 2024-10-03 [2], where you'll hear from a few Project Team Leads about the new and exciting things included in this release. Over the past week, the OpenStack TC assisted the release team in tagging a few deliverables, and on its behalf, I'd like to commend the release team for pulling out all the stops to ensure yet another on-time release. It takes a village, as Jeremy Stanley (fungi) noted on IRC — the happy, buzzing Stackerton village. We also merged some governance changes in the meantime. Notably, we adopted a community goal to move all existing Ubuntu testing to the latest LTS [3], and we retired several charms repositories after their consolidation into the Sunbeam-charms project [4]. I am pleased to welcome Dmitriy Rabotyagov (noonedeadpunk) as the Vice Chair for the TC, and I thank the outgoing Vice Chair, Dr. Jens Harbott (frickler), for his contributions during the 2024.2 release cycle. Next week, as we step into the 2025.1 ("Epoxy") release cycle formally [5], we'll switch into PTG and planning mode and chart out the remaining release priorities early. === Weekly Meeting === The Technical Committee's weekly meeting was held over IRC on 2024-09-24 [6]. We began the meeting by welcoming new members and thanking outgoing ones. We appointed a TC Chair (me) and a Vice Chair (noonedeadpunk) for the 2025.1 release cycle. We then worked through several 2024.2 release issues. We were presented with a peculiar, but familiar problem involving release deliverables that couldn't proceed without help from project maintainers who were unavailable during the release crunch. The release team prepares release candidate patches a week before the RC1 deadline and encourages project maintainers to vote on them. They also submit changes to the respective repositories to verify the state of the CI system to prevent releasing broken code. However, when a project team doesn't respond, there's a risk to the coordinated release. We brainstormed strategies to deal with the current conundrum. Thanks to noonedeadpunk and the attentiveness of project maintainers in Zun and Barbican, we were able to fix broken CI jobs and tag release candidates for some projects. However, we couldn't determine the state of "kuryr-kubernetes" and "kuryr-tempest-plugin." These projects, flagged earlier as unmaintained, are expected to be retired soon, so there are no releases from these projects in the 2024.2 release stream. If you depend on these projects, please let us know. We are looking for maintainers who can commit to reviving the CI and furthering these projects. The next OpenStack TC meeting is on 2024-10-01 at 1800 UTC. This meeting will be held simultaneously via video and IRC. Please find the agenda and other meeting details in our meeting wiki [7]. I hope you'll be able to join us. Remember, you can always edit the agenda and propose new topics! === Governance Proposals === ==== Merged ==== - Retire all single charm repositories | https://review.opendev.org/c/openstack/governance/+/903490 - Select community-wide goal: migrate CI/CD jobs to Ubuntu Noble | https://review.opendev.org/c/openstack/governance/+/929703 - Add 2025.1 TC chair and vice-chair | https://review.opendev.org/c/openstack/governance/+/930376 - Add chair nomination folder for 2025.1 | https://review.opendev.org/c/openstack/governance/+/929970 - Add GouthamR nomination for 2025.1 TC chair | https://review.opendev.org/c/openstack/governance/+/929971 ==== Open for Review ==== - Appoint Tim Burke as PTL for Swift | https://review.opendev.org/c/openstack/governance/+/928881 - Add Axel Vanzaghi as PTL for Mistral | https://review.opendev.org/c/openstack/governance/+/927962 - Mark kuryr-kubernetes and kuryr-tempest-plugin as Inactive | https://review.opendev.org/c/openstack/governance/+/929698 - Retire kuryr-kubernetes and kuryr-tempest-plugin | https://review.opendev.org/c/openstack/governance/+/922507 === Upcoming Events === - 2024-10-02: OpenInfra Live - Introducing OpenStack Dalmatian 2024.2 [2] - 2024-10-08: OpenInfra Monthly Board Meeting: https://board.openinfra.dev/ - 2024-10-15: OpenInfra Days NA, Indianapolis: https://ittraining.iu.edu/explore-topics/titles/oid-iu/ - 2024-10-21: OpenInfra Project Teams Gathering https://openinfra.dev/ptg/ === How to Contact the TC === You can reach the TC in several ways: 1. Email: send an email with the tag [tc] on this mailing list. 2. Ping us using the 'tc-members' keyword on the #openstack-tc IRC channel. 3. Join us at our weekly meeting: The Technical Committee meets every week on Tuesdays at 1800 UTC [7]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2024.2 "Dalmatian" Release Schedule: https://releases.openstack.org/dalmatian/schedule.html [2] Introducing OpenStack Dalmatian 2024.2: https://openinfra.dev/live/ [3] Goal: migrate CI/CD jobs to Ubuntu Noble: https://review.opendev.org/c/openstack/governance/+/929703 [4] Retire all single charm repositories: https://review.opendev.org/c/openstack/governance/+/903490 [5] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [6] TC Meeting IRC Log, 2024-09-24: https://meetings.opendev.org/meetings/tc/2024/tc.2024-09-24-17.59.log.html [7] TC Meeting Agenda, 2024-10-01: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

1 0

Nova placement
by engineer2024 30 Sep '24

30 Sep '24

Is there a way to get back the correct used resources allocation data, in case of loosing the db ? Does placement update the allocation details after restarting the compute nodes to get the latest resources usage for that node ? Thanks elinux

2 2

[neutron] Proposing Ihar Hrachyshka as neutron stable core reviewer
by Brian Haley 30 Sep '24

30 Sep '24

Hello Neutrinos, I would like to nominate Ihar Hrachyshka for Neutron stable core reviewer (again). Ihar previously led the neutron stable core reviewer team a number of years ago, and I think he'd be a great re-addition to the team. Please vote with a +1/-1. Thanks, -Brian, Neutron PTL

6 6

[kolla] Proposing Matt Crees for Kayobe core
by William Szumski 30 Sep '24

30 Sep '24

Greetings Kayobians, I would like to propose Matt Crees as a core reviewer for Kayobe. With Mark Goddard stepping back from Kayobe maintenance duties, we have a slot to fill in the core team. In my opinion, Matt Crees is the strongest candidate for the position, having provided numerous reviews and contributions over the past few cycles. Please register any objections to this decision before the kolla meeting on 25th of September, otherwise I will proceed with making formal arrangements to get him added. Thanks Will Szumski will(a)stackhpc.com

6 6