- openstack-discuss - lists.openstack.org

[ironic] multi-conductor architecture, standalone and DHCP
by Dmitry Tantsur 01 Dec '23

01 Dec '23

Hi folks, This is detailed context on the discussions we've had recently about multi-conductor in metal3. I'm sending it to potentially interested people and cc'ing the openstack ML. The metal3 side is https://github.com/dtantsur/ironic-operator/issues/3 (although that covers a few more different issues). In metal3 we have Ironic without Neutron (or really any other OpenStack components). So far, we have only supported a single Ironic instance in a cluster, but we want to move towards a multi-conductor setup where every Kubernetes control plane node will host an Ironic (so, 3 Ironics in a normal deployment). The biggest problem stems from DHCP for the provisioning network. With Neutron, Ironic can configure it to direct a node to boot from a specific iPXE server. With Metal3, we have a static DHCP configuration. If we keep it the way it is, it will always direct a node to boot from the iPXE server on the same machine as dnsmasq (regardless of how many dnsmasq replicas we run). This has to be changed. I see several options, none of which are straightforward. (1) iPXE-level fix: boot config API This is what https://bugs.launchpad.net/ironic/+bug/2044561 proposes. If we stop using static iPXE scripts and start going through the Ironic API instead, Ironic can reach out to the correct conductor internally, always serving the right script. (2) dnsmasq DHCP implementation Ironic now has a non-neutron managed DHCP implementation through DHCP host and options files: https://opendev.org/openstack/ironic/src/branch/master/ironic/dhcp/dnsmasq.…. This could be a half of the solution. Since Metal3 does not support new node auto-discovery, we can run 3 copies of dnsmasq with dhcp-ignore=!known and rely on this DHCP provider to configure only 1 dnsmasq for each node. The other 2 will ignore the requests. The problem is in the DHCP range. As far as I understand, the 3 dnsmasqs must be configured with disjoint ranges to prevent leases from clashing. This seems non-trivial with Kubernetes which expects to create identical pods. Adam has suggested using annotations from inside ironic-operator, which could work if we find a way to read them inside the pod after it has started (which seems to pose a chicken-and-egg problem). Other workarounds are possible if we go down this path. E.g. ironic-operator could split the DHCP range and creating a mapping hostIP:subrange,hostIP2:subrange2 and pass it to pods. Then each pod will pick its subrange based on its IP address. Huge downside: every time a Kubernetes control plane node is replaced or changes its IP, the whole Ironic installation will get restarted. (2.1) Pre-created leases Another Metal3 contributor has suggested requiring an IPAM implementation (metal3 actually has one) that will create leases centrally, allowing us to use the same DHCP range. The only problem: how to pass them to dnsmasq in runtime without restarting all pods? We could probably have a small sidecar container next to dnsmasq that watches the Kubernetes API for new leases in IPAM. Or... (2.2) Dynamic DHCP and Kea The same contributor is working on Kea support for their deployment: https://hackmd.io/@7RNIJtmvSIeNpSOMTW9oqA/SkWhyr7Hp. There seem to be limitations in the free version, which caused them to write a tiny downstream plugin: https://gitlab.com/Orange-OpenSource/kanod/kanod-kea/-/blob/controlled-leas…. Changing the DHCP server is a dramatic step, I'm not sure if our small community can afford that. It may also pose problems for downstreams like OpenShift, requiring Metal3 to support both servers in the long run. Any thoughts are welcome. Dmitry

3 3

[neutron] Neutron drivers meeting
by Brian Haley 30 Nov '23

30 Nov '23

Hello Neutrinos, Please remember tomorrow (Friday) we have a Neutron drivers meeting. This is the agenda: https://wiki.openstack.org/wiki/Meetings/NeutronDrivers There is currently one topic: * [RFE] Use call_monitor_timeout of oslo.messaging RPCClient instead of custom backoff mechanism and hardcoded timeouts https://bugs.launchpad.net/neutron/+bug/2045058 Thanks, -Brian

1 0

[OpenInfra Live]: PTG Recap, Dec 7 at 15 UTC
by Kristin Barrientos 30 Nov '23

30 Nov '23

Hi everyone, Thanks for watching this week's episode of OpenInfra Live, featuring Zvonko Kaiser from NVIDIA. In case you missed it, you can find it here: https://www.youtube.com/live/urHjDdtuIa4?feature=shared Tune in next week for another OpenInfra Live, brought to you by the OpenStack and StarlingX community! Episode: Join us for a Project Teams Gathering (PTG) recap! Project leaders from Ironic, Technical Committee, Nova, Sunbeam, Manila, StarlingX and Neutron will provide recaps from discussions held during the PTG. Date and time: Thursday, Dec 7, 15 UTC YouTube: https://www.youtube.com/live/thidlQGX29M?feature=shared LinkedIn: https://www.linkedin.com/events/7132754094531747840/comments/ WeChat: The recording will be posted on OpenInfra Foundation WeChat after the live stream Speakers: Kendall Nelson, Jay Faulkner, Sylvain Bauza, James Page, Carlos Eduardo, Steve Geary and Brian Haley Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

"openstack usage list" doesn't return information about all projects
by Michel Jouvin 30 Nov '23

30 Nov '23

Hi, We are using "openstack usage list" to follow resource usage by projects at our cloud. We recently discovered that ~1/3 of the resource consumed (mostly "big projects") are missing from the JSON/CSV produced where the Admin Overview menu in the Horizon dashboard displays everything. Is there a trivial reason for this? We tried both with a pretty old openstackclient (5.8) and the last one after applying the patch in https://github.com/openstack/python-openstackclient/commit/30462f57d48162d1… to fix the problem with date conversion. The result is the same. Thank you in advance for any help. Cheers, Michel

5 15

openstack Vm shutoff by itself
by AJ_ sunny 30 Nov '23

30 Nov '23

Hi team, I am using openstack kolla ansible on wallaby version and currently I am facing issue with virtual machine, vm is shutoff by itself and and from log it seems libvirt lifecycle stop event triggering again and again Logs:- 2023-10-16 08:48:10.971 7 WARNING nova.compute.manager [req-c7b731db-2b61-400e-917f-8645c9984696 f226d81a45dd46488fb2e19515 848 316d215042914de190f5f9e1c8466bf0 default default] [instance: 4b04d3f1-1fbd-4b63-b693-a0ef316ecff3] Received unexpected - vent network-vif-plugged-f191f6c8-dff5-4c1b-94b3-8d91aa6ff5ac for instance with vm_state active and task_state None. 2023-10-21 22:42:44.589 7 INFO nova.compute.manager [-] [instance: 4b04d3f1-1fbd-4b63-b693-a0ef316ecff3] VM Stopped (Lifecyc Event) 2023-10-21 22:42:44.683 7 INFO nova.compute.manager [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d -] [instance: 4b04d3f1- fbd-4b63-b693-a0ef316ecff3] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from ti e hypervisor (4). Updating power_state in the DB to match the hypervisor. 2023-10-21 22:42:44.811 7 WARNING nova.compute.manager [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d ----] [instance: 4b04d3f 1-1fbd-4b63-b693-a0ef316ecff3] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state : None, original DB power_state: 1, current VM power_state: 4 2023-10-21 22:42:44.977 7 INFO nova.compute.manager [req-1d99b87b-7ff7-462d-ab18-fbdec6bda71d -] [instance: 4b04d3f1-1 fbd-4b63-b693-a0ef316ecff3] Instance is already powered off in the hypervisor when stop is called. Thanks & Regards Arihant Jain +91 8299719369

5 15

How do I query the distribution versions of Nova and Neutron for openstack that have been configured？
by qq597399721＠163.com 30 Nov '23

30 Nov '23

How do I query the distribution versions of Nova and Neutron for openstack that have been configured？ I have tried using the "nova -- version" and "neutron -- version" commands, but the results of the commands show their CLI version, which is not their actual release version. I hope someone can help me find the correct command or path, thank you!

2 1

Failed to allocate the network(s), not rescheduling].
by kjme001＠gmail.com 29 Nov '23

29 Nov '23

hello, when it tries to create instances it gets an error message: Failed to allocate the network(s), not rescheduling]. as it restarts the openstack node it is already possible to create an instance without this error. What could be the reason, does anyone have an idea? Maybe someone will suggest something Greetings

2 2

[openstack][octavia] amphora cannot live migrate
by Nguyễn Hữu Khôi 29 Nov '23

29 Nov '23

Hello everyone. I deploy amphora instances using shared storage but when I do live migrate it to another host, I encounter this error. *Error: *Failed to live migrate instance to host "AUTO_SCHEDULE". Details <https://cloudhn.fpt.net/admin/instances/#message_details> compute12 is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks Woud we have a solution for this? Thank you. Nguyen Huu Khoi

3 11

Seeking Advice on Optimal Linux Distribution for OpenStack Installation
by Sait, Abubakur Abdulgader S 29 Nov '23

29 Nov '23

Hello All, I'm currently pursuing my PhD and my research involves utilizing OpenStack. I've encountered some challenges while attempting to install it on Ubuntu 22.04. I'd greatly appreciate it if you could recommend the most reliable Linux Distribution (along with the version) that has worked well for you. Additionally, any up-to-date tutorial suggestions for installing OpenStack would be incredibly helpful, as many of the resources I've come across seem to be outdated. Thanks for your assistance! Best regards, Bakursait

2 1

Cinder Bug Report 2023-11-29
by Jon Bernard 29 Nov '23

29 Nov '23

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> Undecided - Virtual machine cannot unload data disk <https://bugs.launchpad.net/cinder/+bug/2044313> - Status: New Thanks, -- Jon

1 0

[neutron] Review day(s)
by Brian Haley 29 Nov '23

29 Nov '23

Hi Neutrinos, In our PTG and past weeks' meetings I've brought-up setting aside a day or two for us to prioritize merging some of our in-flight reviews, which currently number ~150 [0][1][2] just in the neutron* repositories. I'm hoping it will both help with our bug count as well as make neutron more stable, since a lot are fixing issues and/or adding new tests. I'm proposing we do this over the next two weeks if possible, I've made a doodle to gather responses at [3], that way we can try and coordinate reviewing and the inevitable re-factor time that comes along with merging code. Think of it as an early Christmas present :) Thanks for participating! -Brian, Neutron PTL [0] https://review.opendev.org/q/project:openstack%252Fneutron+branch:master+st… [1] https://review.opendev.org/q/project:openstack/neutron-lib+branch:master+st… [2] https://review.opendev.org/q/project:openstack/neutron-tempest-plugin+branc… [3] https://doodle.com/meeting/participate/id/dN8opRDa

1 1

[magnum] Weekly Meeting Cancelled - 29 Nov 2023
by Jake Yip 28 Nov '23

28 Nov '23

Hi all, Apologies for cancelling this week's meeting due to a clash. If you have a discussion topic, please add them to the agenda, and we will get to them next week. Regard, Jake

1 0

A questionnaire for unmaintained package users
by smoncaff＠gmail.com 28 Nov '23

28 Nov '23

1 0

Re: [Heat][requirements] Possible feature backport to stable branches
by Martin Magr 27 Nov '23

27 Nov '23

On Mon, Nov 13, 2023 at 4:50 PM Martin Magr <mmagr(a)redhat.com> wrote: > Adding more folks from requirements core to CC. > And also correct mailing list. I'm sorry for the noise. > > On Mon, Nov 13, 2023 at 4:32 PM Martin Magr <mmagr(a)redhat.com> wrote: > >> Greetings, >> >> we have introduced new feature to Aodh [1][2] (which introduced new >> dependency [3]) and Heat [4] recently. >> The changes to both projects are relatively small, it just added >> new respective resources and did not change >> existing logic. Also the new dependency is providing functionality to >> those new resources. >> >> We are building our product on stable/antelope and so ideal state for us >> would be to have those changes >> included in this branch also on upstream side. I'm aware that Heat >> follows stable branch policy and it's written >> in requirements documentation, that new dependency cannot be added in >> stable ranches, but before we are >> going to start building workarounds on this I'd like to ask if there is >> any chance of getting exception >> (given the fact that changes are pretty safe) on backporting the Heat >> patch down to stable/antelope >> and(or at least) including the new dependency in stable/antelope. >> >> Thanks for answer in advance, >> Martin >> >> [1] https://review.opendev.org/c/openstack/aodh/+/890529 >> [2] https://review.opendev.org/c/openstack/python-aodhclient/+/893319 >> [3] https://review.opendev.org/c/openstack/requirements/+/891291 >> [4] https://review.opendev.org/c/openstack/heat/+/893673 >> >> -- >> Martin Mágr >> Principal Software Engineer - Service Telemetry Framework >> Red Hat Czech >> > > -- Martin

5 6

[largescale-sig] Next meeting: Nov. 29, at 16UTC
by Kristin Barrientos 27 Nov '23

27 Nov '23

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 16UTC. Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

[tc] Next Technical Committee Meeting Nov 28 1800 UTC
by Jay Faulkner 27 Nov '23

27 Nov '23

Hi all, The next weekly meeting of the Technical Committee is tomorrow, November 28, at 1800 UTC. It will be held in #openstack-tc in OFTC. Agenda - Roll call - Follow up on tracked action items: https://meetings.opendev.org/meetings/tc/2023/tc.2023-11-07-18.00.html - Rosmaita to propose amendment to unmaintained branch resolution allowing a single review group - Rosmaita to email mailing list about progress implementing unmaintained branch resolution - Gate health check - Leaderless projects (gmann) - Implementation of Unmaintained branch statuses - 2024.1 TC Tracker - https://etherpad.opendev.org/p/tc-2024.1-tracker - Upgrade issue between tooz and cinder - https://review.opendev.org/c/openstack/requirements/+/901131 - https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… - Declare projects inactive for lack of ...? - Working py311 unit tests - Support for sqla2 - Broken CI since 2023.2 release - Do we want/need to look at specific deliverables or only whole projects? - Open Discussion and Reviews Thanks, Jay Faulkner TC Chair

1 0

[all] North Dakota State University Student Opportunity
by Kendall Nelson 27 Nov '23

27 Nov '23

Hello All! I was contacted by a professor at North Dakota State University wondering if there were projects in OpenStack that students could work on over the course of the coming semester (Jan -May). The students will be nearing the end of their Bachelors degree (this course being one of the final requirements to graduate) so should be relatively knowledgeable and adept and picking up the technologies required if they aren't already versed in them. If you have an idea for a project that you would want to work with a group of students on, please let me know ASAP. I will get you connected with the professor. Project descriptions are due to the professor by December 1st so we only have a couple weeks to get them together. Anyone interested? -Kendall Nelson Senior Upstream Developer Advocate OpenInfra Foundation

1 2

Is "nova-manage db online_data_migrations" cumulative across versions?
by Michel Jouvin 27 Nov '23

27 Nov '23

Hi, We realized that after a Stein -> Train upgrade of our cloud, we forgot to run "nova-manage db online_data_migrations" and did the Train -> Ussuri upgrade. We have not seen any bad side effect but where wondering if the online_data_migrations was cumulative across versions and if running it under Ussuri fixed potential changes related to Stein -> Train upgrade that we forgot to fix right after the Train upgrade. If not, are there things to do to cleanup the situation? Cheers, Michel

2 1

[neutron] Bug Deputy Report Nov 20-26
by Oleg Bondarev 27 Nov '23

27 Nov '23

Hello Neutron Team, Easy week, as usual for me :P 1 bug is undecided, more info requested, others are on track or fixed. Critical ----- - https://bugs.launchpad.net/neutron/+bug/2043959 - Unit test jobs with neutron-lib master(3.9.0) failing - Fix released: https://review.opendev.org/c/openstack/neutron/+/901430 - Fixed by yatinkarel High ----- - https://bugs.launchpad.net/neutron/+bug/2044272 - Inconsistent IGMP configuration across drivers - In progress: https://review.opendev.org/c/openstack/neutron/+/901753 - assigned to lucasagomes Medium ----- - https://bugs.launchpad.net/neutron/+bug/2044171 - External shared networks may not be seen by other projects - Invalid, not relevant for master ( https://review.opendev.org/c/openstack/neutron-lib/+/901565 closed) - handled by libosvar - https://bugs.launchpad.net/neutron/+bug/2044484 - Enabling DHCP after setting the subnet's 'enable_dhcp' to false does not recreate the metadata namespace - In progress https://review.opendev.org/c/openstack/neutron/+/901812 - assigned to huanghailun Undecided ----- - https://bugs.launchpad.net/neutron/+bug/2044215 - dns: Keyword is replaced by project_name instead of project_id - Incomplete, more info requested Thanks, Oleg

1 0

[cinder][kolla][OpenstackAnsible] 2023.2 Bobcat cycle-trailing release deadline
by Elõd Illés 27 Nov '23

27 Nov '23

Hello teams with trailing projects, The 2023.2 Bobcat cycle-trailing release deadline is in ~1 months [1], and all projects following the cycle-trailing release model must release their 2023.2 Bobcat deliverables by December 7th, 2023. The following trailing projects haven't been released yet for 2023.2 Bobcat (aside the release candidates versions if exists). Cinder team's deliverables: - cinderlib OSA team's deliverables: - openstack-ansible-roles - openstack-ansible Kolla team's deliverables: - kayobe - kolla - kolla-ansible - ansible-collection-kolla This is just a friendly reminder to allow you to release these projects in time. Do not hesitate to ping us if you have any questions or concerns. [1] https://releases.openstack.org/caracal/schedule.html#c-cycle-trail Thanks, Előd irc: elodilles @ #openstack-release

1 1

[release][puppet] Can we retire old stable branches ?
by Takashi Kajinami 27 Nov '23

27 Nov '23

Hi release team, The puppet-ceph repository in Puppet OpenStack project currently follows the independent release model. This is because the module is more aligned with Ceph releases rather than OpenStack release. In the past, we created a stable branches per Ceph stable release and we currently have the following three stable branches. - stable/jewel - stable/mimic - stable/luminous However we no longer use these branches because the old stable branches (eg train) using these old releases are already EOL. So I'm wondering if someone from the release team can create -eol tag from the latest commits and remove these branches. Ideally we may be able to create a release from the latest commits, but I doublt we can merge any patches cleanly to these ancient branches so I tend to just add -eol tag instead, assuming no one may need releases for these old contents. Thank you, Takashi Kajinami

2 1

[all] [taskflow] asyncore is removed in Python 3.12
by Corey Bryant 27 Nov '23

27 Nov '23

Hi All, As noted in the following bug, taskflow uses asyncore, which is removed in Python 3.12. tasklfow needs to be switched to asyncio as a result. Does anyone happen to be working on this? https://bugs.launchpad.net/ubuntu/+source/python-taskflow/+bug/2024588 Thanks, Corey

5 7

[publiccloud-sig] Reminder - next meeting November 29th - 0800 UTC
by Felix Kronlage-Dammers 27 Nov '23

27 Nov '23

Hi everyone, since we’re switching to even weeks with the publiccloud-sig meeting (as also announced last week[1] in my summary), I’m happy to share the reminder for this weeks meeting :) This Wednesday the next meeting of the public cloud sig is going to happen. We meet on IRC in #openstack-operators at 0800 UTC. See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG felix [1] <url: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack…> -- Felix Kronlage-Dammers Product Owner IaaS & Operations Sovereign Cloud Stack Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[tripleo] stepping down from TripleO core
by Takashi Kajinami 27 Nov '23

27 Nov '23

Hello, As you might know or notice, I recently joined a different company, and my focus has been changed. I'm still involved in the OpenStack community but I may no longer be involved in TripleO. To clarify my status I've removed myself from TripleO core. # Please be aware that I'm removing myself from release liason, # someone from the core team might want to replace me, though # I don't think there may be multiple releases created in the future. I'd like to thank all members in the team (current members and ex-menmbers) for your great collaboration and work in the project. Thank you, Takashi Kajinami

3 2

[puppet] Proposing EOL of stable/wallaby and xena
by Takashi Kajinami 24 Nov '23

24 Nov '23

Hello, Some of you may know this but currently CentOS 8 Stream integration jobs in stable branches are all broken. I've looked into some of these failures but tasks randomly get stuck which eventually caused timeout and I've not yet found any potential cause. While we may be able to spend some time to dig into the problem, I'm wondering if anyone still requires wallaby and xena which support CentOS 8 Stream. Most of the backports submitted to these old branches were for TripleO,but I've seen some patches indicating TripleO wallaby is being retired so I doubt anyone still need keeping these branches open for backports. Please let me know if you have any concern about the proposed EOL and are interested in fixing these jobs. The stable/yoga branch has already passed the EM phase, so is another candidate for EOL but my intention is to keep it for now and re-consider its EOL early next year. We may have to make all CentOS 8 jobs non-voting, though. (We have CentOS 9 jobs in this branch). Thank you, Takashi Kajinami

2 2