November 2023 - openstack-discuss

[swift][OpenStackSDK][keystone] Warnings about (my) swift endpoint URL by keystoneauth1 discovery
by Christian Rohmann 13 Nov '23

13 Nov '23

Hey openstack-discuss, we use Ceph RadosGW (Quincy release) to provide Swift as part of an OpenStack cloud. The endpoints are configured as such: > # openstack endpoint list > +----------------------------------+--------+--------------+----------------+---------+-----------+----------------------------------------------------------------------+ > | ID | Region | Service Name | Service > Type | Enabled | Interface | URL | > +----------------------------------+--------+--------------+----------------+---------+-----------+----------------------------------------------------------------------+ > | 1234567890 | region | swift | object-store | True | > public | > https://object-store.region.cloud.example.com/swift/v1/AUTH_%(tenant_id)s > | > | 2345678901 | region | swift | object-store | True | > internal | > https://object-store.region.cloud.example.com/swift/v1/AUTH_%(tenant_id)s > | > | 3456789012 | region | swift | object-store | True | > admin | > https://object-store.region.cloud.example.com/swift/v1/AUTH_%(tenant_id)s > | > so according to https://docs.ceph.com/en/latest/radosgw/keystone/#cross-project-tenant-acce…, if I am not mistaken. I can also use the openstack client to access containers: > # openstack container list > +---------------+ > | Name | > +---------------+ > | containera | > | containerb | > +---------------+ and there is no warnings or other apparent issues. But when using "project cleanup" via e.g. "openstack project cleanup --dry-run --project $PROJECT" I see multiple warnings like > Failed to contact the endpoint at > https://object-store.region.cloud.example.com/swift/v1/AUTH_f2bc4bd34567ddc… > for discovery. Fallback to using that endpoint as the base url. > but only mentioning the swift endpoint. Following the warning string this originates from the discovery method of keystoneauth1, see https://github.com/openstack/keystoneauth/blob/28048af9593740315df8d9027c3b…. Could anybody help me understand what is issue might be? Why does this warning not appear for every use of object storage / swift via openstackclient? Is there any way for me to "fix" this? Could this even be a bug? Thanks and with kind regards Christian

4 10

report an error
by 许伟祥 13 Nov '23

13 Nov '23

carry out iaas-pre-host.sh report an error Downloading packages: warning: /var/cache/yum/x86_64/7/iaas/packages/python-fixtures-3.0.0-2.el7.noarch.rpm: Header V4 RSA/SHA1 Signature, key ID 764429e6: NOKEY Public key for python-fixtures-3.0.0-2.el7.noarch.rpm is not installed Please Reboot or Reconnect the terminal How to fix it 从 Windows 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>发送

1 0

[nova] what nova filter scheduler are you using?
by Satish Patel 13 Nov '23

13 Nov '23

Folks, Just curious about what schedulers people using in their cloud for better distribution or management of VMs? And why? In entire my life I never touch that config also go with default but just curious what people doing and how?

1 0

[kolla-ansible] Galera DR across multiple datacenters
by Jahns, Jay 13 Nov '23

13 Nov '23

We currently run OpenStack on top of two datacenters, and we need to ensure we have datacenter redundancy for the database. With this, we would have 2-3 nodes in each DC running galera. Its been recommended by multiple people that we use an externally managed database server (or cluster) that has its own redundancies built around the two DC layout, and not have kolla manage that. Is there a way we can use kolla to deploy a galera cluster per DC, and use that to provide redundancy in the event that we lose one of the DCs? We were currently looking at proxysql but folks have recommended against it. Thanks.

4 4

[neutron] Bug Deputy Report November 06 - 12
by Lucas Alvares Gomes 13 Nov '23

13 Nov '23

Hi, This is the Neutron bug report from November 6th to 12th. Quieter week than usual. High: * https://bugs.launchpad.net/neutron/+bug/2042938 - "[OVN] Neutron Service Restart Disrupts Octavia OVN Load Balancer Floating IP" Unassigned * https://bugs.launchpad.net/neutron/+bug/2042941 - "neutron-{ovn,ovs}-tempest-with-sqlalchemy-master jobs not installing sqlalchemy/alembic from source" Unassigned * https://bugs.launchpad.net/neutron/+bug/2043141 - " neutron-lib unit tests need update for sqlalchemy 2.0" Assigned to: Brian Haley Medium: * https://bugs.launchpad.net/neutron/+bug/2042925 - "DNS Integration throws expection logs entries were warnings would be sufficient" Assigned to: Jayce Houtman * https://bugs.launchpad.net/neutron/+bug/2042947 - "[stable branches] nftables job inherits from openvswitch-iptables_hybrid master job" Assigned to: Yatin Cheers, Lucas

1 0

[neutron][ovs][kolla-ansible] Does neutron-openvswitch-agent required for sriov only compute node
by Satish Patel 12 Nov '23

12 Nov '23

Folks, I have an openstack cloud just running SRIOV workload, We don't have a requirement to run standard vms on compute nodes. Currently I have noticed neutron-sriov-agent and openvswitch-agent running on all compute nodes. Technically in my case I don't have any use of openvswitch-agent so I am planning to get rid of those agents to reduce pressure on the neutron-server. Just wanted to check with export if that is ok to not run ovs agent. I hope neutron-server won't freak out.

2 2

[release] Release countdown for week R-20, Nov 13-17
by Elõd Illés 10 Nov '23

10 Nov '23

Development Focus ----------------- The Caracal-1 milestone is next week, on November 16th, 2023! Project team plans for the 2024.1 Caracal cycle should now be solidified. General Information ------------------- Libraries need to be released at least once per milestone period. Next week, the release team will propose releases for any library which had changes but has not been otherwise released since the 2023.2 Bobcat release. PTL's or release liaisons, please watch for these and give a +1 to acknowledge them. If there is some reason to hold off on a release, let us know that as well, by posting a -1. If we do not hear anything at all by the end of the week, we will assume things are OK to proceed. NB: If one of your libraries is still releasing 0.x versions, start thinking about when it will be appropriate to do a 1.0 version. The version number does signal the state, real or perceived, of the library, so we strongly encourage going to a full major version once things are in a good and usable state. Upcoming Deadlines & Dates -------------------------- Caracal-1 milestone: November 16th, 2023 Előd Illés irc: elodilles @ #openstack-release

1 0

[OpenstackAnsible][cinder][newtron][nova]
by Tecstim 10 Nov '23

10 Nov '23

Hi, I am testing to deploy a multi-node environment. I tried openstack-helm because I thought it was cool to have containers and virtual machines in the same environment, but it is not very stable and seems difficult to operate. Now I am testing with openstack ansible. I just deployed AIO in a VM. I want to achieve this: Use a physical interface for networks with vlans, and the gateway and dhcp are managed outside openstack. Use NetApp NFS as storage for persistent data of lxd containers of the openstack-ansible, images for VMs and deplyment and use this https://docs.openstack.org/cinder/2023.2/configuration/block-storage/driver… to create volumes for vms.

1 0

[neutron] CI meeting for next week is cancelled
by Yatin Karel 10 Nov '23

10 Nov '23

Hello Neutron Team !! Next Tuesday we have a Public Holiday so I will not be available to chair the meeting. So, the next Neutron CI Meeting will be on 21st November, 2023. Thanks and Regards Yatin Karel

1 0

[all] Failing sqlalchemy-tips jobs should now be passing again
by Stephen Finucane 10 Nov '23

10 Nov '23

Just FYI, for projects that have sqlalchemy-tips jobs that have been failing recently, they should now be passing again. The root cause was neither sqlalchemy nor alembic but rather oslo.utils [1]. It was fixed in oslo.utils 6.3.0 which was released [2] roughly an hour ago. A recheck should be sufficient to clear errors and as these tips jobs only run on master there should be no issues with stable branches. A further point to add: while I appreciate that broken gates are never a good thing, I would be careful making these jobs non-voting. I've already seen a few instances where SQLAlchemy 2.0 support has regressed and I for one have no desire to re-herd cats as we push to get SQLAlchemy uncapped in the coming weeks. If this happens again then we can re-examine things, naturally. Cheers, Stephen [1] https://review.opendev.org/c/openstack/oslo.utils/+/900343 [2] https://review.opendev.org/c/openstack/releases/+/900382

2 2

[kolla-ansible] group_vars question
by Satish Patel 09 Nov '23

09 Nov '23

Folks, I have multiple kinds of nic in cloud and want to configure SRIOV so instead of defining the following variable for each compute node, I would like to create a group and apply config to the groups. Am I right here to define groups like the following or is there a better way to handle this? What is the default path of group_vars and hosts_vars in kolla-ansible? # vi /etc/kolla/group_vars/sriov_eno49.yml neutron_sriov_physnet_mappings: physnet1: eno49 # vi /etc/kolla/group_vars/sriov_eno50.yml neutron_sriov_physnet_mappings: physnet1: eno50 # vi multinode [sriov_eno49] comp1 comp2 comp3 [sriov_eno50] comp4 comp5 comp6

2 2

[Diversity] Diversity and Inclusion WG November Meeting Change and New Time
by Amy Marrich 09 Nov '23

09 Nov '23

Due to travel next week we are bumping the D&I WG meeting to November 21st at our new winter time of 15:00 UTC. As always we welcome members of all the OpenInfra projects to join us to discuss diversity and WG related topics. Amy (spotz)

1 0

[kolla-ansible] Not to install openvswitch on sriov_agent compute nodes
by Satish Patel 09 Nov '23

09 Nov '23

Folks, I have pure sriov compute nodes where I don't want to deploy openvswitch because SRIOV agents do pass through PCI bugs to VM directly. Currently all my SRIOV compute nodes have two agents running. neutron_openvswitch_agent neutron_sriov_agent I don't think "neutron_openvswitch_agent" has any role on sriov compute nodes. If I want to tell kolla-ansible not to deploy neutron_openvswitch_agent on X compute node then how do I tell. Are there any easy ways to turn it off? ~S

1 0

hi
by Tecstim 09 Nov '23

09 Nov '23

1 0

[kolla-ansible] sriov setup
by Satish Patel 09 Nov '23

09 Nov '23

Folks, I am following this guide to deploy sriov compute with kolla-ansible (I am not using kayobe) - https://www.stackhpc.com/sriov-kayobe.html In the above doc last section neutron sriov agent, I am using the same code but those variables are not reflecting. It just prints everything as it is. # cat /etc/kolla/host_vars/os1-bos-sriov-comp-001.yml neutron_sriov_physnet_mappings: physnet1: eno49 # cat /etc/kayobe/kolla/config/neutron/sriov_agent.ini {% raw %} [sriov_nic] physical_device_mappings = {% for dev, physnet in sriov_physnet_mappings.items() %}{{ (loop.index0 > 0)|ternary(',','') }}{{ physnet }}:{{ dev }}{% endfor %} exclude_devices = {% endraw %} After running deploy it just prints the entire file as its in sriov_agent.ini file instead of proper values of variable. Does kolla-ansible support the above code?

2 2

[ptls] [tc] 2023 OpenStack Deployment Data
by Allison Price 09 Nov '23

09 Nov '23

Hi everyone, Apologies for sending this during the PTG as I intended to send this prior, but below is a link to the anonymized data including responses to many project or TC contributed questions. I have removed a significant number of fields to preserve the anonymity of the organizations, but if there is a particular data point that would help your team, please let me know. Cheers, Allison https://docs.google.com/spreadsheets/d/1ehlabuqOnNK4M7xtN1GXvhLoeHn4JBaISih…

2 1

[ironic] Hold approvals for various gate breakages
by Jay Faulkner 08 Nov '23

08 Nov '23

Hey Ironicers, We have a couple of gate unhappinesses: - bifrost gate is busted until this lands: https://review.opendev.org/c/openstack/bifrost/+/900487 - Ironic gate is busted until the new oslo.utils is released and added to requirements (we vote on a sqlalchemy 2x job) -- https://review.opendev.org/c/openstack/releases/+/900382 appears to be the next change in that chain. Just ensuring folks know what's going on! Please avoid approving changes to these repos until the issues are resolved. Thanks, Jay Faulkner Ironic PTL

1 0

[ironic] 2024.1 vPTG Notes & Work Items
by Jay Faulkner 08 Nov '23

08 Nov '23

Hi all, In typical Ironic tradition, we've documented the outcome of the PTG in a spec: https://specs.openstack.org/openstack/ironic-specs/priorities/2024-1-workit… -- please review it for information on work we plan on at least trying to start this cycle :). For full notes, please see the etherpad: https://etherpad.opendev.org/p/ironic-ptg-october-2023. Thanks, Jay Faulkner Ironic PTL

1 0

[FOSDEM] Call for participation: Virtualization and Cloud infrastructure Room at FOSDEM 2024
by Piotr Kliczewski 08 Nov '23

08 Nov '23

We are excited to announce that the call for proposals is now open for the Virtualization and Cloud infrastructure devroom at the upcoming FOSDEM 2024, to be hosted on February 3rd 2024. This devroom is a collaborative effort, and is organized by dedicated folks from projects such as OpenStack, Xen Project, KubeVirt, QEMU, KVM, and Foreman. We would like to invite all those who are involved in these fields to submit your proposals by December 8th, 2023. About the Devroom The Virtualization & IaaS devroom will feature session topics such as open source hypervisors or virtual machine managers such as Xen Project, KVM, bhyve and VirtualBox as well as Infrastructure-as-a-Service projects such as KubeVirt, Apache CloudStack, OpenStack, QEMU and OpenNebula. This devroom will host presentations that focus on topics of shared interest, such as KVM; libvirt; shared storage; virtualized networking; cloud security; clustering and high availability; interfacing with multiple hypervisors; hyperconverged deployments; and scaling across hundreds or thousands of servers. Presentations in this devroom will be aimed at developers working on these platforms who are looking to collaborate and improve shared infrastructure or solve common problems. We seek topics that encourage dialog between projects and continued work post-FOSDEM. Important Dates Submission deadline: 8th December 2023 Acceptance notifications: 10th December 2023 Final schedule announcement: 15th December 2023 Devroom: 3rd February 2024 Submit Your Proposal All submissions must be made via the Pretalx event planning site[1]. It is a new submission system so you will need to create an account. If you submitted proposals for FOSDEM in previous years, you won’t be able to use your existing account. During submission please make sure to select Virtualization and Cloud infrastructure from the Track list. Please fill out all the required fields, and provide a meaningful abstract and description of your proposed session. Submission Guidelines We expect more proposals than we can possibly accept, so it is vitally important that you submit your proposal on or before the deadline. Late submissions are unlikely to be considered. All presentation slots are 30 minutes, with 20 minutes planned for presentations, and 10 minutes for Q&A. All presentations will be recorded and made available under Creative Commons licenses. In the Submission notes field, please indicate that you agree that your presentation will be licensed under the CC-By-SA-4.0 or CC-By-4.0 license and that you agree to have your presentation recorded. For example: "If my presentation is accepted for FOSDEM, I hereby agree to license all recordings, slides, and other associated materials under the Creative Commons Attribution Share-Alike 4.0 International License. Sincerely, <NAME>." In the Submission notes field, please also confirm that if your talk is accepted, you will be able to attend FOSDEM and deliver your presentation. We will not consider proposals from prospective speakers who are unsure whether they will be able to secure funds for travel and lodging to attend FOSDEM. (Sadly, we are not able to offer travel funding for prospective speakers.) Code of Conduct Following the release of the updated code of conduct for FOSDEM, we'd like to remind all speakers and attendees that all of the presentations and discussions in our devroom are held under the guidelines set in the CoC and we expect attendees, speakers, and volunteers to follow the CoC at all times. If you submit a proposal and it is accepted, you will be required to confirm that you accept the FOSDEM CoC. If you have any questions about the CoC or wish to have one of the devroom organizers review your presentation slides or any other content for CoC compliance, please email us and we will do our best to assist you. Questions? If you have any questions about this devroom, please send your questions to our devroom mailing list. You can also subscribe to the list to receive updates about important dates, session announcements, and to connect with other attendees. See you all at FOSDEM! [1] <https://penta.fosdem.org/submission/FOSDEM17> https://pretalx.fosdem.org/fosdem-2024/cfp [2] virtualization-devroom-manager at fosdem.org

1 0

The difference between kolla-ansible deploy and kolla-ansible upgrade
by bryansoong21＠gmail.com 08 Nov '23

08 Nov '23

Hello team, I have noticed this for the kolla operation documentation: https://docs.openstack.org/kolla-ansible/zed/user/operating-kolla.html This procedure is for upgrading from series to series, not for doing updates within a series. Inside a series, it is usually sufficient to just update the kolla-ansible package, rebuild (if needed) and pull the images, and run kolla-ansible deploy again. Please follow release notes to check if there are any issues to be aware of. We have two different kolla images and the OpenStack version are all Zed, while one container base image si Rocky 9.2 and another is CentOS 9. Based on kolla documentation, we should use kolla-ansible deploy if we want to switch different kolla images. We also tried to run kolla-ansible upgrade and all VMs have been stopped, just wondering what's the difference when running these two commands and why the upgrade shutdown all the VMs? Thanks

1 0

[ironic][nova] A few notes about the Ironic<>Nova driver
by Jay Faulkner 08 Nov '23

08 Nov '23

Hey all, Julia Kreger, John Garbutt, and myself met today in a trio-coding session to review the Ironic virt driver in Nova, as a follow-up to some of the concerns we developed when sharding support was reverted in Bobcat. I wanted to ensure both communities knew about what we discussed. Notes were taken in the Ironic PTG etherpad starting at line 654. *Fix a Bug in list_instances/list_instance_uuids* We found a bug around list_instances and list_instance_uuids in the Ironic driver, and tracked that its main impact was performance. The bug has been documented in https://bugs.launchpad.net/nova/+bug/2043036 and we will work to ensure it's resolved this cycle. *Completing the Partially completed ironicclient -> SDK migration* The Ironic virt driver in Nova began migrating to openstacksdk, but this migration was not completed. This left the driver in a strange, in-between state which made it much easier to misunderstand microversion interactions. Even more unfortunately, we even have places where we're checking the available microversion using ironicclient, then making the actual call with openstacksdk (which may negotiate another microversion altogether). Completing this migration will be a priority; and we will attempt to complete it, barring unforeseen technical issues, before re-proposing the sharding changes. *Sharding re-proposal and testing* For sharding's re-proposal in Caracal, we discussed some details around validation and testing. Primarily, we want to ensure Ironic's sharding API has tempest test coverage, and that we have more complex testing coverage around the integrated Nova/Ironic sharding case, for when it's reproposed. As sharding will be the primary scaling method for integrated Ironic<>Nova users, it's important it's exercised well in our CI. *Pass more metadata to Ironic at provisioning* Ironic implemented several features around RBAC recently, but those have not been available for users of Nova-integrated Ironic installations. We want to change that. John proposed a blueprint, https://blueprints.launchpad.net/nova/+spec/ironic-guest-metadata , which we hope to be approved as specless, which would make the metadata set in Ironic's instance_info more consistent with what is set on a libvirt guest. With that metadata, Ironic developers could make the "automatic_lessee" feature work with Nova and unlock RBAC-goodness for users of fully integrated Ironic<>Nova clouds and Ironic deployers have a new tool they can use to troubleshoot failures using instance_info from Ironic's API. Thanks for reading. This is just a summary of our discussion and links to some artifacts from it; obviously none of this is going to happen until the blueprints are approved, code is written and lands -- but we thought the community might appreciate hearing about what we're thinking about ahead of time. Thanks, Jay Faulkner Ironic PTL

1 1

Cinder Bug Report 2023-11-08
by Jon Bernard 08 Nov '23

08 Nov '23

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> Undecided - NetApp doesn't update the backend LUN name during generic migration <https://bugs.launchpad.net/cinder/+bug/2042662> - Status: New Thanks, -- Jon

1 0

[tc][ptg] 2024.1 PTG Recap: TC Sessions
by Jay Faulkner 08 Nov '23

08 Nov '23

Hi all, The OpenStack Technical Committee met for two three hour sessions during the vPTG, covering a handful of topics. This email serves as an extremely high level recap of those topics. If you want full information on the discussions, please review the full etherpad notes[0]. The TC will be tracking long-term projects, related to these discussion items and otherwise, in the 2024.1 TC Tracker[1]. As always; no official TC policy changes can be made via a meeting. PTGs and other synchronous meetings are useful for rapidly gaining context and consensus, which then must be reflected in a resolution and merged when that consensus results in a policy change. *Leaderless Projects[2]* The TC discussed the multiple projects who did not have a PTL elected, and the various reasons that could cause that. A few interesting items were discussed which may need future action by the TC. First, concerns were raised over the ec2-api PTL candidate not being an Active Contributor in title, despite showing stewardship over a relatively "complete" project. This is an example of how our process of identifying active contributors can be flawed. A workaround in the interim of a PTL in this situation nominating themselves as an extra-AC was identified. Secondly, the recurring issue of smaller projects missing elections, and generally being less active, was a topic of discussion. The TC decided that as a regular course of action, any project that did not successfully participate in electing a PTL would also have the activity of its project reviewed as part of the PTL appointment process -- with a default towards marking such projects inactive[3] unless they have clear signs of activity. *Follow-ups from 2023.2 TC Tracker[4]* There are some ongoing items which the TC has continued to work on cross cycle. Their status was evaluated and discussed; items which needed additional tracking were pulled forward to 2024.1. *Managing Divergence between documented runtime and reality* This was a session originally inspired by a mailing list post from Herve[5] about how the listed versions of some libraries on the releases page for bobcat[6] were not the versions compatible with bobcat. After significant discussion, it became apparent that there was confusion and lack of communication around a change that was not intended to merge yet, and did anyway, breaking project compatibility which had not yet done the SQLAlchemy 2.0 migration. The TC agreed that proposing a new step in the release schedule where we integrate and test libraries with projects before cutting library releases is a potential solution to this problem. *More predictable python minimum supported version* Currently, our documentation indicates we drop support when there is "a solid reason", but recent mailing list discussions[7] make it clear that is not a clear line. After significant discussion, the consensus was that unit testing the minimum supported version of python does very little harm but provides a benefit. While it was desired to have a general pattern we could follow, the realities of testing on LTS platforms make it so that we should continue to look at python version support on a case-by-case basis, comparing support lifecycles in python upstream, our current tested LTS, and the OpenStack release in question. Similarly for new python versions, selecting what new ones to support is a technical concern rooted in library, distro, and ecosystem support moreso than what we'd desire OpenStack to support. *Communication* Communication to/from the TC was raised as a topic due to a misunderstanding around language used around a TC action decided on in a meeting. For OpenStack community members who don't frequent TC meetings, or interact with the TC process often, it can be confusing, but we should always be clear: binding decisions are not made in synchronous meetings. Other items discussed around this communication topic included how more virtual communication methods, such as vPTG, may be limiting our cross-project communication and ability to reach larger groups. *Election Retrospective* This topic also covers many items covered in the communication topic; but essentially, many projects have missed elections -- the TC wants to find a root cause and address it if possible. During the discussions, it became clear there was a significant cultural disconnect. Many pieces of documentation needed for OpenStack leaders -- including election reminder emails, governance docs, and this summary(!) -- are only available in English. Additionally, there is a concern that there are cultural conflicts that extend past language. One set of solutions discussed were alternative PTL models, such as DPL. Some attendees indicated a dislike of the DPL model due to it potentially masking inactivity issues by removing the periodic check-in inherent to the elections process. An idea of electing a PTL for multiple cycles was raised as well, but did not gain wide consensus. Additionally, we discussed how to get new leaders onboarded. A suggestion was made for the TC to send an incoming PTL email, with information and expectations for incoming PTLs. This is a process that traditionally has been handled well, albeit ad-hoc, in larger projects, but is often missed in smaller projects, especially if there is no handoff from one PTL to another. Finally, there was a discussion about the size of our electorate being artificially small. Despite having 580 contributors in bobcat, OpenStack TC only had 309 eligible voters (AC+Foundation Member), and of those 309, only 91 people had opted-in to receive a ballot. Some suggestions were to revise our definition of who is eligible to vote in elections to include OIF non-members, but still require PTLs or TC candidates to be OIF members. *Shadowing Program* There was some discussion as well about the OpenInfra Diversity working group[8] piloting a shadowing program. Many TC members indicated a willingness to be shadowed, but a lack of people willing to take on leadership roles. (Along these lines; if you're reading this and want to shadow someone; ask!) There were suggestions that perhaps the prestige of being a TC member or PTL has gone down, making it difficult to take on those responsibilities or justify them to management. We discussed several ways to try and re-engage people into leadership and contribution with OpenStack. *Footnotes:* 0: https://etherpad.opendev.org/p/tc-ptg-october-2023 1: https://etherpad.opendev.org/p/tc-2024.1-tracker 2: https://etherpad.opendev.org/p/2024.1-leaderless 3: https://governance.openstack.org/tc/reference/emerging-technology-and-inact… 4: https://etherpad.opendev.org/p/tc-2023.2-tracker 5: https://lists.openstack.org/pipermail/openstack-discuss/2023-September/0351… 6: https://releases.openstack.org/bobcat/index.html 7: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… 8: https://etherpad.opendev.org/p/diversity-leadership If you made it this far; thanks! -- Jay Faulkner OpenStack TC Chair

1 0

[Glance] Cancelling Weekly Meeting -9th Nov
by Pranali Deore 08 Nov '23

08 Nov '23

Hello All, Cancelling tomorrow's weekly meeting as most of the team members will not be around for the meeting. Thanks, Pranali Deore

1 0

[publiccloud-sig][ptg] 2024.1 Caracal vPTG summary
by Felix Kronlage-Dammers 07 Nov '23

07 Nov '23

Hi, my attempt at giving an overview of what we discussed in the publiccloud-sig vPTG meeting. We’ve met once on Tuesday for couple hours and used the regular Wednesday slot for a re-cap of the discussion on the day before as well as what we’ve learned in the nova operator hour - but more on that further down below. We've set out to discuss the following topics: - Standard set of properties - way forward - Domain admin support - Alignment of activities of Public Cloud SIG The etherpad for the 2024.1 vPTG is here: https://etherpad.opendev.org/p/oct2023-ptg-publiccloud-sig # Standard Set of properties We've discussed how we (PublicCloud-SIG) think the way forward should look like and aligned ourselves that we need to write the proper specs for nova and glance. As part of the discussion, aside from the SCS (Sovereign Cloud Stack) initiative on standardizing items here, we were made aware of the efforts within gaia-x to standardize properties here[1][2]. With that in mind we (gtema and fkr) joined the nova-operator hour (which was directly after the PublicCloud-SIG meeting)[3]. The outcome of the nova operator hour was then something completely different. It was agreed[3] that we look at the metadefs and possibly also look into a new nova api that gives back flavors based on standard traits and resources (min/max)[4]. # Domain admin support For the domain admin support there is currently quite a bit of activity from the SCS Side of things[5]. From those efforts also items like [6] are coming. # Alignment of activites of Public Cloud SIG From the discussion in the PublicCloud SIG it became apparent to us that with the touch points that the PublicCloud SIG has it is to us to 'represent' the stakeholders of PublicCloud-functionality towards the development efforts of OpenStack projects. Driving this thought further, we (Public Cloud SIG) should become more active and see wether we can fulfil the role of a ProductOwner towards the OpenStack projects to work towards an better alignment between them in order to provide an overall better experience for entities operating public clouds with OpenStack. We will ask OpenInfra wether they can provide us a list of contacts of people at entities that operate public clouds with OpenStack or wether they could mail them for us in order to foster more involvement in the publiccloud SIG meetings. Thanks to everyone who participates as well as the people from the nova operator hour for the insights. felix [1] <url: https://github.com/SovereignCloudStack/gx-self-description-generator/issues…> [2] <url: https://gitlab.com/gaia-x/technical-committee/service-characteristics/-/blo…> [3] <url: https://etherpad.opendev.org/p/oct2023-ptg-operator-hour-nova#L106> [4] <url: https://etherpad.opendev.org/p/oct2023-ptg-operator-hour-nova#L120> [5] <url: https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-03…> [6] <url: https://bugs.launchpad.net/keystone/+bug/2041611>

1 0

[docs] Encourage search engines to show newest version of OpenStack docs first
by Jake Yip 07 Nov '23

07 Nov '23

Hi all, I am not sure if this is encountered by others, but often when I search for something regarding OpenStack, the first few results are of older versions. For example, my last search was "keystone config service_token_roles_required", which returns a Pike[1] URL as first result. For older versions, the documentation does have a header which is something like "This release is no longer supported by the community. The current supported release is 2023.2." It links to new documentation[2], but drops me to the docs root instead of the new version of this page and usually is not useful for me. (I understand it is not possible to drop at the same page as documentation moves around). I often try to change the version in the URL to 'latest' which works sometimes. I wonder if we can help to fix this? Looking at other people with this issue[3], a few strategies seems to have help. (Disclaimer: I am not a SEO expert). 1. Prevent Google from indexing the older versions by using `noindex`[4] 2. Removing old documentation so it is flushed from Google Regards, Jake [1] https://docs.openstack.org/keystone/pike/admin/identity-auth-token-middlewa… [2] https://docs.openstack.org/2023.2/ [3] https://github.com/crossplane/docs/issues/107 [4] https://developers.google.com/search/docs/crawling-indexing/block-indexing -- Jake Yip Technical Lead, Nectar Research Cloud

4 4

[Telemetry][Requirements][TC] When did we stop caring about backwards compatibility
by Erno Kuvaja 07 Nov '23

07 Nov '23

Hi all, I'd like to bring the conversation to a wider audience as the current state is really not sustainable. Since I've got involved with Telemetry project early this year, I honestly think our gates have been blocked more than not and most of that has been due to requirements bumps where the dependencies break backwards compatibility and that is not obvious, like major version bump (pyparsing being the current example [0] [1]), or are just not tested before bumped (like the recent Sphinx/WSME episode [2]). While I do recognize Dr. Harbott's concerns around gnocchi [1] I'd like to point out that a) it's still currently our only supported time series database for metrics and b) providing alternative solution has not been an exactly smooth road either [3]. Unfortunately gnocchi is still very tightly coupled in our Telemetry testing (something I'm working on) and not having an operational gate makes it very difficult to even work on those tests across the Aodh and Telemetry-tempest-plugin repos. We have been capping and blacklisting broken dependencies for years, so I'd like to hear what has changed recently so that we do not care anymore but prefer "the latest" over the "latest working". In my understanding the upper constraints was supposed to be "the latest dependencies you can stand up a functional and working cloud with" not "the latest dependencies existing regardless of what they break". Back to Dr. Harbott's remarks of gnocchi being an acceptable requirement [1], without a working gate we cannot even deprecate it. And yet we would need an appropriate deprecation period before removing it. Should we be forced to rip out our only TS database (gnocchi) to be able to focus on developing support for the alternative (Prometheus) I would really like to see TC resolution for this so we can point the impacted users to it, avoiding all the blame falling to the Telemetry project team. I would also like to see the Telemetry job being in the Requirements gate so we get warnings when something is about to break, rather than a jammed gate because of an upper constraints bump. Thanks, Erno "jokke" Kuvaja [0] https://github.com/pyparsing/pyparsing/issues/504 [1] https://review.opendev.org/c/openstack/requirements/+/898699 [2] https://review.opendev.org/c/openstack/requirements/+/896018 [3] https://review.opendev.org/c/openstack/requirements/+/891291

4 4

Kindly share the current roadmap for neutron
by habib ELalfy 07 Nov '23

07 Nov '23

Dears, Good day, Could you please add me to neutron mailing list and share with me the current roadmap for neutron, thanks. Best Regards, Habib

3 2

[nova] Nov 7 is a spec review day and Nov 14 is implementation review day
by Sylvain Bauza 06 Nov '23

06 Nov '23

Hey folks, I feel a bit sad to had no time yet to provide you a summary for our PTG, but please wait a bit more :-) That said, something you should already know, we will have two spec review days this cycle and two implementation review days as well, so please mark your calendars for the first to come : - Nov 7 will be a spec review day. Please ensure your specs are ready for reviews - Nov 14 will be an implementation review day. Mark your patches up ! As a reminder, the idea behind those review days is to synchronize our efforts between owners and reviewers so people can discuss at the same time. Thanks, -Sylvain

1 1

[tc] Next meeting of TC tomorrow, Tues Nov 7, 1800 UTC
by Jay Faulkner 06 Nov '23

06 Nov '23

Hi all, The next meeting of the TC is tomorrow, Tuesday November 7th at 1800 UTC. As it is our first meeting of the month, it will be held via a Zoom video call, with minutes kept in #openstack-tc on OFTC IRC. Use the following link to connect: https://us06web.zoom.us/j/87108541765?pwd=emlXVXg4QUxrUTlLNDZ2TTllWUM3Zz09 While it is TC practice to hold one video meeting a month to accommodate those who prefer that mode of communication, as always, all official TC decisions are made via commits to governance repositories. *Please note that for most US residents, this will be an hour earlier than it was before our DST switch.* The proposed agenda for the meeting is: - Roll call - Follow up on tracked action items: https://meetings.opendev.org/meetings/tc/2023/tc.2023-10-31-18.00.html - JayF: Before next video meeting, write up a short document on pros/cons of moving TC video meetings to jitsi-meet. - Gate health check - Leaderless projects (gmann) - Implementation of Unmaintained branch statuses - draft email to openstack-discuss while we figure this out: https://etherpad.opendev.org/p/early-caracal-unmaintained-transition - PTG follow-ups - https://etherpad.opendev.org/p/tc-ptg-october-2023 - https://etherpad.opendev.org/p/tc-leaders-interaction-2024-1 - https://etherpad.opendev.org/p/oct2023-ptg-os-dbperf-crossproject - Open Discussion and Reviews Thanks all, Jay Faulkner TC Chair

1 0

[kolla-ansible][Announcement] IMPORTANT! do not install from pypi!
by Sven Kieske 06 Nov '23

06 Nov '23

Hi there! I have seen many issues[0] over the past weeks because people still do "pip install kolla-ansible" or "pip install kolla- ansible==16.1.0". Please don't do that! We currently lack the workforce to do regular releases to pypi, but bugs get fixed in git, so you might miss important fixes! Please use the documented installation procedure[0], e.g. this: "pip install git+https://opendev.org/openstack/kolla-ansible@stable/2023.1" Thanks [0]: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [1]: https://docs.openstack.org/kolla-ansible/2023.1/user/quickstart.html#instal… -- Sven Kieske Senior Cloud Engineer Mail: kieske(a)osism.tech Web: https://osism.tech OSISM GmbH Teckstraße 62 / 70190 Stuttgart / Deutschland Geschäftsführer: Christian Berendt Unternehmenssitz: Stuttgart Amtsgericht: Stuttgart, HRB 756139

5 7

[publiccloud-sig] Reminder - next meeting November 08th - 0800 UTC
by Felix Kronlage-Dammers 06 Nov '23

06 Nov '23

Hi everyone, this Wednesday the next meeting of the public cloud sig is going to happen. We meet on IRC in #openstack-operators at 0800 UTC. A preliminary agenda can be found in the pad: https://etherpad.opendev.org/p/publiccloud-sig-meeting See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG I’ll be attending the SCS Hackathon in Dresden, so I’ll not be around but Tobias (tobberydberg) will be there and facilitate the meeting. Since we did have couple of good rounds at the vPTG (as well as a good discussion in the nova operators hour with some of us joining there) we should still send out a summary - I’ll see to get that out of to this list tomorrow. felix -- Felix Kronlage-Dammers Product Owner IaaS & Operations Sovereign Cloud Stack Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[neutron] bug deputy report of week 2023-10-30
by Bence Romsics 06 Nov '23

06 Nov '23

Hi Team, We only had a handful of new bug reports last week: Unassigned: * https://bugs.launchpad.net/neutron/+bug/2041861 [ovn] instance in a shared network owned by another project is unreachable via floating IP * https://bugs.launchpad.net/neutron/+bug/2042647 doc: Options described in "DHCP High-availability" are outdated Invalid: * https://bugs.launchpad.net/neutron/+bug/2042089 neutron : going to shared network is working, going back not * https://bugs.launchpad.net/neutron/+bug/2042598 neutron_server container suspended in health:starting state Cheers, Bence (rubasov)

1 0

neutron_server suspended in heath:starting
by Bursztynowski Dariusz 03 Nov '23

03 Nov '23

I've installed OpenStack (zed) on a Raspberry Pi cluster using kolla-ansible (version tagged for zed), and all containers are healthy except the neutron_server which is suspended in 'health: starting' state. Network related part of OpenStack seems not to work (snippet form running init-runonce): Configuring neutron. HttpException: 503: Server Error for url: http://192.168.1.60:9696/v2.0/routers, No server is available to handle this request.: 503 Service Unavailable. However, at least some commands not related to networking seem to work as expected (e.g., I can create an image which is reported by openstack image list as 'active'). There are four Raspberry Pi 4B in the cluster (2 x 4GB RAM and 2 x 8GB RAM). They run Debian 11 (bullseaye). Kolla-ansible has been used for the installation strictly following the guidelines form https://docs.openstack.org/kolla-ansible/zed/user/quickstart.html. Notably, I'm using a specific (weird?) configuration of networking on my Pis to mimic two network interfaces on each host as kolla-ansible expects. These are provided as interfaces of veth pairs (more details on that can be found in my report here: https://bugs.launchpad.net/neutron/+bug/2042598) Is it possible that this setup rises problems? I've already contacted OpenStack folks and they see no evident symptoms of neutron bug, suggesting a problem with deployment. Reported a bug to kolla-ansible here: https://bugs.launchpad.net/kolla-ansible/+bug/2042372, but do not know how fas they react there. I could try to install other versions but maybe there's a smarter ways to cope with the problem. Any suggestion will be welcome.

3 2

Provision of 100 HP ProLiant servers
by Leszek Szczepanowski 03 Nov '23

03 Nov '23

Hi all, I have the following use case: I need to install Openshift on 100 HP ProLiant servers, those are HP DL360 Gen10+. They are mounted in the racks, with factory settings. So, BIOS settings are default, as well as RBSU, RAID configs and ILO (ILO5). Is it possible to provision them using Openstack? Set a few parameters into ILO, into RBSU/BIOS, set up RAID controller disks (MegaRAID), and then put some ISO image and boot it? Or, maybe even better: create an Openshift cluster similar to how we do it with cloud providers? I have done this for years manually, via ILO WebUI + HTML5 console, but I'm actually tired of this. Can it be somehow automated? -- Leszek A. Szczepanowski twinsen(a)mspanc.net

4 3

[charms] Nominating Utkarsh Bhatt for charms-ceph-core
by Peter Sabaini 03 Nov '23

03 Nov '23

Hello all, I would like to nominate Utkarsh Bhatt for charms-ceph-core. Utkarsh has worked for the Ceph charms team for 1.5 years now and has provided many contributions to the charms, especially to the Ceph charms[0]. I know him as a diligent and enthusiastic contributor and hope you will support his nomination. Thanks, peter. [0] https://review.opendev.org/q/owner:utkarsh.bhatt%2540canonical.com

4 3

[neutron] Drivers meeting cancelled
by Brian Haley 02 Nov '23

02 Nov '23

Hello Neutrinos: Due to the lack of agenda, Friday's drivers meeting is cancelled. Have a nice weekend! -Brian

1 0

[ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster
by Forrest Fuqua 02 Nov '23

02 Nov '23

Hi there, I'm currently exploring some strategies and seeking advice on the deployment and layout of services on an 8-node cluster. Each node is equipped with dual AMD EYPC 7702s, two 100G Links, and 2TB of RAM. Additionally, I've set up two 13TB NVME drives in a Ceph configuration for each node. At present, the setup involves one management node, while the others are dedicated to running Nova, Neutron, Glance, and Cinder. The workload, while fairly basic, involves handling substantial numbers. It's primarily used for one-off research projects and a few classes, serving around 60 end users. The most demanding scenario occurs during cybersecurity competitions when we need to deploy 3100 VMs at once. When this is spread out over a few days, it's manageable. However, there's a risk of overloading the system, leading to DiskIO issues and Database inconsistencies. Currently, I'm utilizing Kolla for deployment, and I've shared my configurations on this GitHub repository: Link to Configs<https://github.com/RIT-GCI-CyberRange/openstack-configs>. I'd greatly appreciate any insights or advice on optimizing this setup to handle the peak workload more efficiently and prevent potential performance issues. Thanks in advance for your help!

3 7

Call for Writers or OpenInfra Live Participants
by Kristin Barrientos 02 Nov '23

02 Nov '23

Hi everyone, Do you have a topic that you would like to discuss on OpenInfra Live or Superuser? We’d love to hear from you. Please reach out to me via email or IRC if you have any ideas and we can talk next steps. Thanks, Kristin Barrientos IRC: @kbarrientos:matrix.org Marketing Coordinator OpenInfra Foundation

1 0

[nova][ptg] 2024.1 Caracal PTG summary
by Sylvain Bauza 02 Nov '23

02 Nov '23

(sorry folks, was working on other urgencies hence the late summary) Again, as every cycle I say, that's a wrap ! Thanks to all the contributors that were around, we were more than 10 everyday, woohoo ! As a reminder (like every cycle ;-) ), please don't open the main etherpad link [1] if you use a Web browser with an automatic translation modification feature. If you do it, then it would internally translate all the phrases directly in the etherpad as if you were directly modifying the etherpad by yourself. Please also make sure you don't accidentally remove or modify lines. In order to prevent any misusage, this etherpad is a read-only copy of the main one : https://etherpad.opendev.org/p/r.02f889c0423aa279a3fa8e136becad10 Anyway, let's stop discussing about etherpad, and lemme provide you the summary : (oh and I could be wrong for explaining some stuff, no worries, just reply in here if so) (please also grab a coffee or whatever else, you may need it) ### Operator hour ### (read-only etherpad for this here https://etherpad.opendev.org/p/r.06c38397f2cdd81380649312f032b22f) So, we had around 4-5 operators for this operator hour, lovely. This is what we discussed : # flavors explosion for public clouds Basically, there are multiple usecases here : this is hard to discover flavor properties except by looking up the description, but this is also difficult to filter filters based on some properties since they can be different across clouds. What we agreed (as a start) : - Glance metadefs concept [2] could help to provide a catalog of flavor properties definition if operators were using it. - Nova could provide some new API microversion for filtering flavors based on a request of standard traits and/or resources (those queried within minimum and/or maximum bars). We don't know yet who could be the owner of such a specification, contributors welcome. - Another feature could be "Nova, tell me whether my flavor could have some candidates ?" but again, we need hands on deck. # Nova could get a better housekeeping system for orphaned resources # (I'm literally paraphrasing the topic name) OK, so, after discussing with the operator, we eventually found two use-cases : - as an operator, I'd like nova to magically adopt a compute node coming from elsewhere (say a region or another nova deployment). Long story short (lookup the etherpad for more details), the winning strategy that we could implement would be : #1 adopt a whole separate nova cell by using a specific nova-manage command that would create records on the target nova-api DB based from the records from the source api DB. The operator was happy with working on a POC and discussing with the nova community to make it a quickwin. #2 live-migrate my instances between cells. Well, that one is more difficult to implement, so we could punt this until #1 is implemented. - as an operator, I sometimes need to rebuild a compute OS. Well, on that one, we discussed it further more in terms of "project cleanup". Again, long story short, we went into explaining to use 'OSC project cleanup' [3] feature that'd do the job. # I want to be able to detach my root volume Well, that one wasn't really discussed, but the topic was filled in the etherpad. No real owner of such feature request, which absolutely requires a large community design discussion since resolving this isn't trivial (and we probably need to do some trade-offs in terms of guest support). Anyway, if you read those lines and you do care of such usecase, please understand that you need to associate resources and come by the community to bring them. I, as PTL, can help providing some mentoring and onboarding if needed. ### Cross-project meetings ### We had two cross-project sessions, one with Neutron and one with Cinder. # Neutron cross-project session We only discussed one topic, but a very important one : how Neutron would feel if Nova starts to require some optional Neutron API extensions ? :-) Eventually, what we agreed with Neutron is : - Nova will deprecate by Caracal the legacy codepath that doesn't use multiple port binding and provide a LOG.warning - Nova will delete this ^ in the E* release - Sean will provide a nova blueprint explaining which API extensions will be mandatory by Nova but also he will provide a neutron rfe bug report for that. - Eventually, we'll provide release notes and a ML thread for explaining all of this (we being Brian and me) # Cinder cross-project sessions We had two sessions, one in Wednesday, the other in Friday.. Please look at both [4] and [5] for knowing the summary for each of them. ### Bobcat retrospective and Caracal planning ### 17 blueprints were approved, 9 of them landed, one of them got reverted due to late issues discovered with OSC. 34 contributors this cycle, less than previously (mostly less on-off contributors). The list of untriaged bugs exploded, but some of our awesome contributors did a very late scrubbing, so now we're down to 30, huzzah. What we agreed at the PTG : - we gonna abandon the review-priority Gerrit label in favor of using an etherpad for both feature and bugs cycle tracking (we were already doing etherpad tracking for approved blueprints implementation patches) - that cycle tracking etherpad will provide a section for proposing patches, and another section only written by Nova PTL containing a scrubbed list of changes ready to review. Process to be further documented later (when our PTL eventually has time to do it) - PTL will review every approved spec to ensure that the testing requirements are reasonably defined Please also take care of our approved Caracal planning (I'll propose a release patch to add those deadlines into the main Caracal schedule page) : - Spec freeze at the second milestone - Spec review days on Tuesdays of the weeks R-21 and R-17 - Implementation review days on Tuesdays of the weeks R-20 and R-12 ### EOLing our unmaintained branches ### - bauzas will propose a patch for EOLing Ussuri - We'll defer until next cycle to see whether we drop Victoria and Wallaby ### SLURP cadence impact and curating Caracal ### - We are bumping our libvirt minimums early this cycle to Ubuntu Jammy versions - We will also update our next minimum versions later this cycle based on the numbers of the next Ubuntu 24.04 LTS release (once a beta build is available). - we'll forward port our Bobcat deprecation notices in the Caracal release notes - we end up merging all the required bits (mostly ORM changes) for preparing the SQLA 2.0 upgrade in a later release - we'll remove vmwareapi and hyperv support in Caracal - we agreed for the virt drivers removals to not provide an API microversion on the specific API endpoints and rather return a HTTP400 ### python tooling we could use ### - We agreed on using sphinx-lint, making the code changes having doc job to use it. - We agreed on testing codespell tool with the pep8 job and eventually either keep it or remove it depending on whether it finds many false positives - We agreed on reviewing the pyupgrade patches as a try with no clear promises. ### Planning new features ### - the Healthchecks spec is going to be reproposed this cycle, some implementation details about how to do RPC checks have to be identified - we agreed on exposing the value of the pinned AZ in GET /servers/<uuid>/details - we agreed on doing kind of soft-[anti-]affinity for AZs using instance groups (either thru new policies or some new concept of 'destination') - PCI device affinity for libvirt driver seems an effort to be resumed. Testing this is a concern but we'll discuss this in the spec - PCI devices notion of grouping (imagine two PCI devices for the same card, you'll get the idea) seems an interesting usecase, worth spec'ing. We could imagine the notion of a single placement unit being 'pci group' with one PCI device being the main object to track. The neighbouring aspect seems hard to implement so we reserve it for later. - Having console timeouts on token expiry seems a nice security feat, reviews welcome on the spec. - VGPU efforts will be huge this cycle. More to be discussed below in a specific section ### Caracal cleanups, bugs and follow-ups ### - we really want to change our default quota system to use unified limits soon but there are some upgrade concerns with flavors that contain placement resource classes that require ourselves to do further work with oslo.limits or Keystone in order to not break operators that chose to use the unified limits driver. We could also add some checking in the nova-manage migration tool that would emit a warning if resource classes were used. - our config with images_types_backend, use_cow_images and force_raw_images is confusing and errorprone. We agreed on fixing some migration bug as a backportable way, but also deprecating use_cow_images and force_raw_images while providing meaningful config options instead. - we're mixing IP addresses and URLs usage in our configs, we could be more consistent. What we agreed on that is that a new migration_inbound_addr config sounds cool, defaulting to the my_ip config value but being able to provide a FQDN or hostname too ### The VGPU efforts ### - we're actively working on providing upstream testing for mdevs, thanks to the mtty kernel samples framework. mtty live-migration is currently an ongoing patch, but someone proposed to backport this into a custom kernel built on purpose for our usage. Either way, mtty testing will be done as much as possible in nova-next but we could create a new periodic/experimental target for targeting new features like live-migration. - we plan to do mdev live-migration this cycle. There are lots of limitations that we need to document but this sounds doable. We could add some pre-live-migration check about mdev types and this would require some object change. We may also need to persist the mdev/allocation relationship but that will be discussed in the spec - nvidia is going to drop vfio-mdev for their SR-IOV GPUs and use a specifc vfio-pci variant driver. This is a very premature task and our caracal efforts will consist in identifying the necessary nova adaptations. One is already identified (adding a managed flag) and we agreed on a solution. - we're gonna change how nova creates the mediated devices by rather defining a libvirt nodedevice which will also facilitate operator maintenance (they could just mdevctl or define udev rules) - we agreed on deprecating the config default that allows you to only specify mdev types without specifying the addresses. - SRIOV GPUs currently leak some unused VFs into Placement, leading to capacity issues [6]. We could address that problem by either making the device's explicit definition mandatory or adding a new config option that would specify the number of VFs to use per type. - some other bugs were discussed with an operator who was present. We agreed on doing further testing to clearly identify problems, and some patches are already up for reviews (like the evacuate case or the multi-create limitation) but also require some rebase. Anyway, I guess I'm done now. Thanks for having read until that point and I hope your coffee (or tea) was good. HTH and thanks, -Sylvain (on behalf of the whole Nova community) [1] https://etherpad.opendev.org/p/nova-caracal-pt g (please remove the empty char between 'pt' and 'g') [2] https://docs.openstack.org/glance/latest/user/metadefs-concepts.html [3] https://docs.openstack.org/python-openstackclient/latest/cli/command-object… [4] https://wiki.openstack.org/wiki/CinderCaracalPTGSummary#Cross_project_with_… [5] https://wiki.openstack.org/wiki/CinderCaracalPTGSummary#Nova_Cinder_cross_p… [6] https://bugs.launchpad.net/nova/+bug/2041519

1 0

[sunbeam][ptg] vPTG summary
by James Page 02 Nov '23

02 Nov '23

HI All The Sunbeam project participated in the vPTG last week; here is a short summary of the topics that we discussed and plans added for the Caracal cycle. # Full Cycle Retro ## Celebrations - Significant reduction in both the time taken to bootstrap and resize a deployed cluster. - Switch from Antelope to Bobcat in the charms was smooth and relatively quick. - Many new plugins delivered during the cycle, enabling lots more of the OpenStack ecosystem - telemetry, magnum, heat, barbican and designate. - Refactoring of ops-sunbeam to support both k8s and machine charms was a success - Documentation starting to look great and developing lots of depth - Lots of contributions to the open source tools we depend on including snapd, Juju (inc terraform provider), microk8s, traefik and prometheus charms. ## Learnings - Switching to Bobcat for snaps and terraform plans took a bit longer than it should have done - most likely down to this being a largely new process for the team. - Some impact on velocity due to bugs in dependencies outside of OpenStack - Release process needs to be better documented so more of the core team can execute it. - Plugin full functional test coverage still needs some work. ## Actions - Implementation of automated functional testing for plugins - Full documentation and improved automation of the release process # Debugging Problems ## Overview We had quite an extended conversation about how we help users debug both bugs and deployment specific issues in Sunbeam; the general observability feature we're working on will help in this area once a cloud is deployed but we need some extra features to help users before this stage of the deployment is reached. sunbeam inspect is a bit lightweight still; can supplement the information gathered with more extensive information collected from microk8s so we need to document this into the bug reporting guidance. ## Actions - Improve docs topic 'Inspecting the cluster' to include more detail of collecting debug information for reporting bugs - Update the guidance for reporting bugs in Launchpad projects to help users report better more qualified bugs # Monorepo for Sunbeam Charms ## Overview With our current approach of 1:1 mapping between charms and repositories, we're going to have a similar challenge to the OpenStack Charms project when we need to effect a change across the entire charmset (bumping to a new release, updating a dependency or alike). This generates a large number of reviews, and associated costs of testing, which then need to be managed through to landing by one of the core team. Liam proposed that we look at moving to a single mono-repo for all of the charms in the project, reducing the overhead of 'batch' type operations and making it generally easier to manage this part of Sunbeam. We'd also address integrated testing of a change that impacts multiple charms without having to model cross-repo dependencies. The resource required to complete a function test would increase in size, but reduce hugely in number when compared to our current approach. We'll need to figure out tooling to determine which charms are impacted by a specific change so that we only build, test and publish charms that need to be changed, rather than the entire charmset. Some projects in the wider charm ecosystem are using this approach to good effect. ## Actions - Liam to take this idea forward early during the Caracal cycle - will need some coordination and advice from the infra team (see next topic). # Integration testing ## Overview Although the Sunbeam charms do include functional testing, it's not a complete integration test of a full Sunbeam deployment. We'd like to achieve this goal but this will require test instances with more memory than we typically get right now. We discussed that this felt linked to the idea of the mono-repo for charms as it feels unreasonable to ask for increased instance sizes for and increasing set of repositories when we could achieve better resource usage efficiency with a mono-repo. We agreed to fold this into the mono-repo work early this cycle. # Multi-node testing ## Overview All of our multi-node testing is currently not executed as part of the project CI workflows - we really need to test this part of Sunbeam as part of our gating for changes. We discussed a few ideas in this area including use of LXD containers on a big instance type or use of the 3rd party CI provided by Canonical that currently supports the OpenStack Charms project. Both might present viable routes forward and will be considered during the coming cycle. # Managing OCI artefacts references in charms ## Overview The OCI resources used by the Sunbeam K8S charms are managed within a different project; we need to be aware of when new OCI's are published so we can feed these updates into the charms and publish new charm revisions and associated OCI resources. There is some prior art on github modelled as actions to support this type of workflow but we need something that will work across that boundary into the opendev gerrit infrastructure. ## Actions - Reach out to the opendev infrastructure team as to whether there is any prior work on this type of function. # Single Container Multiple Services ## Overview We've spent quite a bit of time this cycle reducing the physical footprint of the OCI resources we use for Sunbeam. We discussed whether this might be taken further by reducing the number of containers we run in each charm - this is made possible by the fact that the OCI entry point is managed using Pebble, which supports multiple service definitions so we can run multiple daemons within the same container. We agreed to prototype this in one of the more complicated charms to see how much benefit might be achieved by taking this approach. ## Actions - POC probably using the nova-k8s charm to assess benefits. We had around 6 active participants during our session with a few others dropping in to listen in on the conversation from time to time - thankyou to all of those who contributed to our discussions and I look forward to the next (v)PTG. Cheers James Sunbeam PTL

1 0

[kolla-ansible][16.1.0] Could not load namespace_haproxy_for_cell
by Satish Patel 02 Nov '23

02 Nov '23

Folks, I am deploying new cloud and planning to use 2023.1 with kolla-ansible tag 16.1.0 but I hit this error and after google I found this reference bug: https://bugs.launchpad.net/kolla-ansible/+bug/2039573 Very curious about the workaround here. How does it pass CI-CD if it's really a bug?

3 3

[nova][compute] HugePages question
by Satish Patel 01 Nov '23

01 Nov '23

Folks, What are the advantages and disadvantages to enable HugePages on compute nodes? I know that HugePages improve performance of VMs because of TLB lookup etc. But if it does improve in most cases then why people why default not turning on HugePage for deployment? Just trying to find out what could go wrong if I turn on by default HugePage for all deployment?

2 2

[kolla-ansible][16.1.0] Error container timeout
by Satish Patel 01 Nov '23

01 Nov '23

Folks, I am deploying the 2023.1 release of openstack using kolla-ansible (tag 16.1.0) . I noticed a very strange error during deployment. This is very close to this bug report: https://bugs.launchpad.net/kolla-ansible/+bug/2038914 RUNNING HANDLER [common : Restart fluentd container] ***************************************************************************************************************************** fatal: [os1-bos-ctrl02]: FAILED! => {"Command": "dumb-init --single-child -- kolla_start", "Created": 1698865177, "HostConfig": {"NetworkMode": "host"}, "Id": "ae17d4d276fd278d15c5df03fdee70eafbff66fecbc78ae0bbae64274daa910b", "Image": "registry.example.com/kolla/fluentd:2023.1-ubuntu-jammy", "ImageID": "sha256:41752d28100fd85074d112942f843ce144df8bb023d7bdf664df0ef7be9a50ea", "Labels": {"build-date": "20231029", "kolla_version": "16.1.1", "maintainer": "Kolla Project (https://launchpad.net/kolla)", "name": "fluentd", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "22.04"}, "Mounts": [{"Destination": "/etc/localtime", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/localtime", "Type": "bind"}, {"Destination": "/etc/timezone", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/timezone", "Type": "bind"}, {"Destination": "/var/lib/fluentd/data", "Driver": "local", "Mode": "rw", "Name": "fluentd_data", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/fluentd_data/_data", "Type": "volume"}, {"Destination": "/var/lib/kolla/config_files", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/kolla/fluentd", "Type": "bind"}, {"Destination": "/var/log/kolla", "Driver": "local", "Mode": "rw", "Name": "kolla_logs", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/kolla_logs/_data", "Type": "volume"}], "Names": ["/fluentd"], "NetworkSettings": {"Networks": {"host": {"Aliases": null, "DriverOpts": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAMConfig": null, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "Links": null, "MacAddress": "", "NetworkID": ""}}}, "Ports": [], "State": "created", "Status": "Created", "changed": true, "msg": "Container timed out"} fatal: [os1-bos-ctrl01]: FAILED! => {"Command": "dumb-init --single-child -- kolla_start", "Created": 1698865177, "HostConfig": {"NetworkMode": "host"}, "Id": "1ca559230483b490830dc2e38791fc8c567702d3280713a6f0c7bc45f2c8a038", "Image": "registry.example.com/kolla/fluentd:2023.1-ubuntu-jammy", "ImageID": "sha256:41752d28100fd85074d112942f843ce144df8bb023d7bdf664df0ef7be9a50ea", "Labels": {"build-date": "20231029", "kolla_version": "16.1.1", "maintainer": "Kolla Project (https://launchpad.net/kolla)", "name": "fluentd", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "22.04"}, "Mounts": [{"Destination": "/etc/timezone", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/timezone", "Type": "bind"}, {"Destination": "/var/lib/fluentd/data", "Driver": "local", "Mode": "rw", "Name": "fluentd_data", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/fluentd_data/_data", "Type": "volume"}, {"Destination": "/var/lib/kolla/config_files", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/kolla/fluentd", "Type": "bind"}, {"Destination": "/var/log/kolla", "Driver": "local", "Mode": "rw", "Name": "kolla_logs", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/kolla_logs/_data", "Type": "volume"}, {"Destination": "/etc/localtime", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/localtime", "Type": "bind"}], "Names": ["/fluentd"], "NetworkSettings": {"Networks": {"host": {"Aliases": null, "DriverOpts": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAMConfig": null, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "Links": null, "MacAddress": "", "NetworkID": ""}}}, "Ports": [], "State": "created", "Status": "Created", "changed": true, "msg": "Container timed out"} fatal: [os1-bos-ctrl03]: FAILED! => {"Command": "dumb-init --single-child -- kolla_start", "Created": 1698865177, "HostConfig": {"NetworkMode": "host"}, "Id": "8f3599681f56695b0e809acb5645d79deb3fbb476dc7bda5d75f838f67ed085d", "Image": "registry.example.com/kolla/fluentd:2023.1-ubuntu-jammy", "ImageID": "sha256:41752d28100fd85074d112942f843ce144df8bb023d7bdf664df0ef7be9a50ea", "Labels": {"build-date": "20231029", "kolla_version": "16.1.1", "maintainer": "Kolla Project (https://launchpad.net/kolla)", "name": "fluentd", "org.opencontainers.image.ref.name": "ubuntu", "org.opencontainers.image.version": "22.04"}, "Mounts": [{"Destination": "/etc/localtime", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/localtime", "Type": "bind"}, {"Destination": "/etc/timezone", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/timezone", "Type": "bind"}, {"Destination": "/var/lib/fluentd/data", "Driver": "local", "Mode": "rw", "Name": "fluentd_data", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/fluentd_data/_data", "Type": "volume"}, {"Destination": "/var/lib/kolla/config_files", "Mode": "ro", "Propagation": "rprivate", "RW": false, "Source": "/etc/kolla/fluentd", "Type": "bind"}, {"Destination": "/var/log/kolla", "Driver": "local", "Mode": "rw", "Name": "kolla_logs", "Propagation": "", "RW": true, "Source": "/var/lib/docker/volumes/kolla_logs/_data", "Type": "volume"}], "Names": ["/fluentd"], "NetworkSettings": {"Networks": {"host": {"Aliases": null, "DriverOpts": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAMConfig": null, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "Links": null, "MacAddress": "", "NetworkID": ""}}}, "Ports": [], "State": "created", "Status": "Created", "changed": true, "msg": "Container timed out"}

1 0

[ironic] Announcement: Future of Metalsmith
by Jay Faulkner 01 Nov '23

01 Nov '23

Hi all, Just announcing that at the vPTG, the Ironic core team came to a consensus about the future of the metalsmith project. It was originally created to improve the user experience around provisioning Ironic nodes when using Ironic's API directly. At this, it was mostly successful and has completed this task. It's our intention to work on the Ironic API and python-ironicclient to provide a similarly simple deployment user experience, even for users without nova, integrating lessons learned from the metalsmith project. To this end; we are letting the overall community know we no longer expect to add additional functionality to metalsmith. We will ensure it remains maintained and well-tested until it's functionality has been duplicated, but all new work in this area will be directed at the primary Ironic API and Ironic client projects. To this end, and as discussed at the PTG, I've proposed a change to the Metalsmith readme, notifying users of this state. https://review.opendev.org/c/openstack/metalsmith/+/899761 This change, or this mailing list post, is a great place to comment if you have any comments on this decision. Thanks, Jay Faulkner Ironic PTL

3 3

Cinder Bug Report 2023-11-01
by Jon Bernard 01 Nov '23

01 Nov '23

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> Undecided - hitachi and OEM : output REST API token <https://bugs.launchpad.net/cinder/+bug/2040966> - Status: New - hitachi AIX as host OS type does not work <https://bugs.launchpad.net/cinder/+bug/2041784> - Status: In Progress - The 'signature_verified' parameter should not be added to image when upload volume to image <https://bugs.launchpad.net/cinder/+bug/2041793> - Status: In Progress - Missing encryption param handling in Quobyte driver <https://bugs.launchpad.net/cinder/+bug/2042102> - Status: New Thanks, -- Jon

1 0

[manila] 2024.1 Caracal PTG summary
by Carlos Silva 01 Nov '23

01 Nov '23

Hello. Here is a summary of all topics we have discussed during last week's PTG and the AIs that were a result of the discussion. Thank you to all contributors that participated. If you missed one or more sessions, we got you covered. The Manila PTG, as well as the operator hour were recorded and the videos are available in the OpenStack Manila YouTube Channel [0]. Deferred deletion ------------------------- We discussed a bug that exists and the need for a deferred deletion approach. We agreed that the deferred deletion would be ideal in case the driver needs to do multiple things in the deletion and we don't want the operators to wait. #Agreed: The service should release user quotas as soon as the driver says it is processing the deletion; and handle errors internally AI: kpdev and NetApp/SAP will look into the deferred deletion implementation and provide a spec All things CephFS -------------------------- We discussed deprecating support for "standalone" NFS-Ganesha with the CephFS/NFS driver. We've supported using a clustered Ceph NFS service since the Zed release. We've also added upgrade helpers to migrate from a standalone NFS-Ganesha service to the Ceph NFS service; this code was backported to stable/2023.1 (Antelope) Testing and stabilization: We'll continue testing cephfs/nfs with standalone NFS-Ganesha until we drop support for ganesha's DBUS mechanism from the CephFS/NFS driver. IBM is currently testing the scale, performance and stability of clustered-nfs-ganesha and both, nfs-ganesha and cephadm are being updated accordingly Mulitnode devstack ceph in CI: - devstack testing of cephfs/nfs has been stable, we're trying to run this job on stable/2023.1 - AI: ashrodri will enable ingress service with cephadm Ceph NFS upgrades The code patches for upgrading to a cephadm deployed NFS cluster were merged and backported to stable/2023.1 (Antelope). When upgrading, access rules are reapplied and export locations will indicate the preferred export path for mounting shares. Deprecating use of DBUS API with NFS-Ganesha As the clustered nfs solution is hardened, and an upgrade mechanism is invented, we want to stop supporting the use of "standalone-nfs/ganesha" server with CephFS. AI: add a deprecation warning in the Caracal cycle and communicate it in the mailing list Multiple CephFS filesystems We discussed pros/cons of using a single driver representing multiple file systems as Manila storage pools - there are not a lot of benefits to this, especially considering the ease in deployment/configuration of the manila share services currently For multi-fs environments, the user experience to mount shares with Native CephFS is poor since the export paths no longer expose the filesystem name Operators suggested that we encode the filesystem name in export location metadata so automation tools (e.g.: manila CSI plugin, nova/virtiofs) can consume it programmatically AI: Elias Wimmer will report a bug against the CephFS driver AI: Ceph driver documentation will be updated regarding multi-fs and the behavior of the driver Ceph Mon address is never re-validated [3] Today, the only way to re-export a share is by restarting the manila-share manager service. We discussed an API based trigger to the ensure shares logic that will recalculate the export locations of the shares. We agreed that this implementation should add a micro-state to the share and that the API should deny other management operations on the shares when "ensuring". AI: carloss will propose a specification to implement this change Human readable export locations ----------------------------------------------- Export locations are constructed for machines, and they do not use human friendly names. NetApp proposed user customizable export locations a few releases ago [4] We discussed the proposal, and find it is still relevant - we brainstormed using a share type extra specification for customizing a prefix to the export location names We agreed that updating the export location of a share is bound to be disruptive and will not be part of the implementation. We may consider this enhancement if there is a very strong use case AI: NetApp folks will propose updates to the lite-spec and work on the implementation Share backups --------------------- A generic NFS implementation of share backups was introduced in the previous (Bobcat) cycle We discussed a share-manager driven advantaged share backup driver implementations, as well as the data service backup driver interface Cern found that the backup driver interface called back into the data service, and the driver interface needs changes AI: Luis from Cern will update the Backup driver to decouple the data manager interface on-demand in preparation for their implementation of the restic backup driver Tech debt -------------- During the tech debt session, gouthamr walked us through some multi-release tech debt items, such as the progress on scenario tests and the implementation of mechanisms to prevent race conditions. It is a good opportunity for someone in the community to step up and do some of the work, which can be a good technical challenge. AI: Discuss the specification in weekly IRC meetings and find new owner/s Barbican integration ----------------------------- Backend data encryption has been supported for several releases in Manila, but users don't have the ability to configure their encryption keys SAP is pursuing an integration with Barbican for Manila and has worked on a specification. #Agreed: Manila should have a Barbican/Castellan API layer to interact with the keystore, and share backends shouldn't have code to interact directly with Barbican. AI: kpdev will update the proposed spec [5] also documenting the interactions with Castellan Manila/Cinder cross-project session --------------------------------------------------- We organized a cross-project session with the Cinder team to discuss a couple of things: - An enhancement to the manila volume extend API calls in the generic driver: - as the Cinder APIs evolved and Manila would benefit from it. Rajat can work on the enhancement in case the Manila contributors don't have the bandwidth. - The way DHSS=True works in Manila - we brainstormed hypothetical support in Cinder for service instances/storage virtual machines and tenant driven volume networking - We also talked about the progress both the projects achieved with respect to Secure RBAC, OpenStackClient and OpenStackSDK. Expectations while submitting features -------------------------------------------------------- We are working on documenting the process one should go through while submitting features [6]. This also covers what are the deadlines and what reviewers expect in terms of feature submission. AI: carloss will update the new documentation with a checklist that will illustrate the definition of done for a feature Howcasts -------------- This is something we have been ideating on for a while. The idea is to produce short videos that will help newcomers to the project. We have an idea of what to work on [7] and help from the community would be appreciated. AI: we'll start the work on the howcasts and try to split the load within the community members SQLAlchemy 2.0 ----------------------- stephenfin has been working on the SQLAlchemy 2.0 changes for Manila and we have made good progress so far. One of the problems to test the changes is that we need to wait a whole entire CI run that can take up to 3 hours Agreed: - We'll work with stephenfin to reduce the amount of wait time to get CI results by reducing the tests using the dummy drivers - We'll review the changes and help figure out the issues with migration of share replicas and share instances queries Tech Debt Roundup - 2; multi-release efforts that have attention --------------------------------------------------------------------------- ---------------- Metadata spec We currently have metadata implemented for three resources (shares, share snapshots and share network subnets). We have open patches for access rules and export locations. AI: Update the export locations and access rules metadata changes and work on them during the Caracal cycle. OSC Support: We have parity between OSC and the manila shell client and we added a deprecation warning in the previous cycle and the removal will happen prior to the next SLURP release (E). This means that Manila's shell client will not be shipped as part of the D release. - More negative functional tests would be welcome, as we have implemented only the happy-paths during a hackathon. - AI: consider a OSC test hackathon during the Caracal release hackathon - AI: all documentation will be updated to utilize OSC in examples SDK We have completed a lot of this with the help of university students/outreachy interns. Most of the work that was pressing and features that were needed to integrate with other services were covered. AI: Continue working towards full parity between openstacksdk and manilaclient SDK Agreed: There are no plans to deprecate manilaclient SDK Share Transfers: We have share transfers between projects for DHSS=False, but we are still lacking support for DHSS=True. This is a cool technical challenge to work in Manila, and having someone willing to continue this work would be appreciated. Follow up on Resource locks ------------------------------------------ The two featureful changes for resource locks were merged in the Bobcat cycle (share resource locks and locks for access rules deletion and visibility restrictions) There are two changes yet to be merged, being the OpenStack SDK patch and the manila-tempest-plugin tests for the access rules locks. We brainstormed implementations of other resource locks, such as a visibility lock on sensitive fields of a security service, or deletion locks for snapshots, backups and replicas AI: We'll implement new resource locks opportunistically during the Caracal release Operator Hour --------------------- We have reflected over the results of the latest OpenStack user survey [9] and talked about the progress of the most requested features and also requested feedback from operators. We discussed the use cases for the disabling manila services feature, and chatted about one possible enhancement to this feature which would be to add backends as disabled and enable them later. AI: we should enhance our documentation to talk about start and stopping services, as well as the differences between stopping and disabling them. [0] https://www.youtube.com/@openstackmanila [1] https://github.com/openstack-k8s-operators/manila-operator/ - a replacement for TripleO [2] https://prow.ci.openshift.org/job-history/gs/origin-ci-test/pr-logs/directo… (Native CephFS) [3] https://bugs.launchpad.net/manila/+bug/1996793 [4] https://opendev.org/openstack/manila-specs/src/commit/baf6209da8e5b47228024… [5] https://review.opendev.org/c/openstack/manila-specs/+/898999 [6] https://review.opendev.org/c/openstack/manila/+/898855 [7] https://etherpad.opendev.org/p/manila-howcasts [8] https://etherpad.opendev.org/p/caracal-ptg-manila-operator-hour [9] https://drive.google.com/file/d/1ZM722TOX3ouP_7Cet2JdURFttTuI5ZLe/view?usp=… [10] https://etherpad.opendev.org/p/caracal-ptg-manila If you have questions or would like to follow-up on any of the topics, don't hesitate to reach out. Thank you! carloss

1 0

[manila] Nov 2nd upstream meeting is cancelled
by Carlos Silva 01 Nov '23

01 Nov '23

Hello! As some of our regular contributors will be on PTO tomorrow and we are following up on all-things-ptg, we are cancelling this week's weekly meeting. The next meeting will take place next week (Nov 9th). Thanks and regards, carloss

1 0

Volumes not deleting
by Russell Stather 01 Nov '23

01 Nov '23

Hi We have a juju openstack installation and we are finding that when we delete servers the associated volume doesnt get deleted. The volume stays as showing attached to a non existent server instance. We are having to marked them as not attached in the database and then the volume delete works. Any ideas what could be wrong here? Thanks Russell

2 1

Openstack volume deletion
by Vijay Thilak S 01 Nov '23

01 Nov '23

Hi, OpenStack Kolla Ansible has been installed on Ubuntu 20 and integrated with Ceph storage. The volume associated with the instance is not being deleted by Cinder when I terminate the instance with this option (Delete Volume on Instance Delete), which is causing me problems. I need to log in to the server and run the cinder commands to manually delete the volume that is not associated with instances. I would appreciate it if you could provide a solution. Thanks, Vijay

4 10

[kolla] Cancelling weekly meeting
by Michał Nasiadka 01 Nov '23

01 Nov '23

Hello Koalas, Weekly meeting today is cancelled due to public holiday in some of the countries including Poland. Best regards, Michal

1 0

[Octavia] weekly meeting 11/1 canceled
by Gregory Thiemonge 01 Nov '23

01 Nov '23

Hi Folks Many contributors are off today (public holiday), so we are canceling this week's meeting, Thanks

1 0

RE: openstack-discuss Digest, Vol 61, Issue 1 Openstack- Instance creation
by Asma Naz Shariq 31 Oct '23

31 Oct '23

Hi Openstack Team, When I tried to create instance in Openstack dashboard, I encountered below error: Error: Failed to perform requested operation on instance "Test-Instance", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance f8cb7bb1-64db-4242-a26a-65f3568ff4e9.]. Can anyone guide me what is the root cause for this error? Regards, -----Original Message----- From: openstack-discuss-request(a)lists.openstack.org <openstack-discuss-request(a)lists.openstack.org> Sent: Wednesday, November 1, 2023 7:26 AM To: openstack-discuss(a)lists.openstack.org Subject: openstack-discuss Digest, Vol 61, Issue 1 Send openstack-discuss mailing list submissions to openstack-discuss(a)lists.openstack.org To subscribe or unsubscribe via email, send a message with subject or body 'help' to openstack-discuss-request(a)lists.openstack.org You can reach the person managing the list at openstack-discuss-owner(a)lists.openstack.org When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..." Today's Topics: 1. Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster (Forrest Fuqua) ---------------------------------------------------------------------- Message: 1 Date: Wed, 1 Nov 2023 02:26:02 +0000 From: Forrest Fuqua <fffics(a)rit.edu> Subject: Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster To: Karl Kloppenborg <kkloppenborg(a)resetdata.com.au>, "openstack-discuss(a)lists.openstack.org" <openstack-discuss(a)lists.openstack.org> Message-ID: <CO6PR16MB420923931283EAF219118489D0A7A(a)CO6PR16MB4209.namp rd16.prod.outlook.com> Content-Type: multipart/alternative; boundary="_000_CO6PR16MB420923 931283EAF219118489D0A7ACO6PR16MB4209namp_" I am currently doing COW images in Ceph right now, that seems to be the same thing as volume backed images, I have about 40 images I use ( I even published the set here:http://mirrors.rit.edu/oszoo/ ) Ill take a look at Openstack Helm, I've never deployed Kerb before. ________________________________ From: Karl Kloppenborg <kkloppenborg(a)resetdata.com.au> Sent: Tuesday, October 31, 2023 8:03 PM To: Forrest Fuqua <fffics(a)rit.edu>; openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster Hi Forrest, One of the first recommendations I would suggest is to utilise volume-backed-images. https://docs.openstack.org/cinder/latest/admin/volume-backed-image.html If you have say a set of linux and windows images, you can use this to instead of RBD imaging, ceph will snapshot from the base image the RBD and this can provision in seconds. I would also suggest (and this is completely biased because I’m on the core team) looking at OpenstackHelm – we run this at scale with many thousands of VM’s. Thanks, Karl. From: Forrest Fuqua <fffics(a)rit.edu> Date: Wednesday, 1 November 2023 at 2:15 am To: Karl Kloppenborg <kkloppenborg(a)resetdata.com.au>, openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster In response to Karl's request, Here is last year's major load I ran: http://mirrors.rit.edu/cptc/2023/virtual-machines/1-README.txt<http://mirrors.rit.edu/cptc/2023/virtual-machines/1-README.txt> The TL;DR its a mix of Windows and Linux VMs at about 4GB of Ram, and 4vCPU at about 50 machines per team, and having up to 50 teams at a time, with all the extras support VMs at various workloads, we do about 2500 VMs just for the yearly compitations All of these VMs will be under high load since they will be under a consistent cyberattack by students. The biggest issues I've had is Ceph starts to slow down on RBD creation of a COW image and Nova will timeout the creation, If it gets too back, entries like volumes, and VMs can get stuck in weird error states of error_deleting or some other weird state (I've had Volumes stuck attached to VMs that don't exist!) At peak, Netdata was showing about 500kIOP/s across the whole cluster before it really started to go north and went into a death spiral, A single windows VM was a 4 hour boot time. Ceph started showing slow writes with up to 40s commit latency When I was running a multi-node management, the MariaDB and RabbitMQ would get super mad and start dropping messages and database issues, having a single dedicated compute node has worked well so far. but I am thinking of moving away from Kolla and going to Ansible during my summer rebuild. ________________________________ From: Karl Kloppenborg <kkloppenborg(a)resetdata.com.au> Sent: Monday, October 30, 2023 6:44 PM To: Forrest Fuqua <fffics(a)rit.edu>; openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster Hi Forrest, Based on the configuration specified, I will assume that 1.8TB is assignable to VM resources, less the management node (though honestly why not just share over the cluster, doesn’t seem like it’s a production workload?) However, (1.8TB*7)*1000 = 12600GB usable ram, with no oversubscription. (roughly 180-200GB each system for host and ceph services) Therefore 12600/3100 = ~4GB ram per VM at peak load, no oversubscription. Based on the CPU arch: 128*2 threads = 256 vCPU cores (which in this model I would count as vCPU cores, there’s a LOT of contention about what defines a vCPU, there’s many camps and religious arguments on this)… However, we want some space for the host so as a general thought, I will put 250 vCPU available in this calculation. 250*7 = 1750 vCPU cores avail. There are 3100-1750 = 1,350 deficit vCPU cores. As such to make up a 1vCPU core to VM at peak workload an oversubscription would be 1.8 1750*1.8 = 3150 vCPU available. Honestly, depending on these workloads and your requirements you can probably bust this subscription ratio up more and again, I would instead share the management workloads over the cluster more and induct the first node as another compute resource. I would like to know more about the workloads, what is the flavour of each VM, Is the workload disk intensive, cpu or memory, or io? In terms of running the workload, I don’t see an obvious issue with your nova config, the allocation ratio is high but that’s not going to probably be an issue here based on the numbers I mentioned. Ensuring host reservation memory is good, which I can see you have around 128GB in reservation and ram_allocation_ratio as 1.0 will ensure no OOM scenarios. Overall, I think your settings are sane and my only comment would be to add that management node into the compute mix if you can but otherwise without knowing more on the workload, seems fine. Have you done any scale testing? Spun up a workload of 3100 VMs? ( easiest way to do this would likely be to heat template…) then run an image with a default start workload of something you want to test with. --Karl. From: Forrest Fuqua <fffics(a)rit.edu> Date: Tuesday, 31 October 2023 at 5:40 am To: openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: [ops] Seeking Advice on Service Deployment and Layout for an 8-Node Cluster Hi there, I'm currently exploring some strategies and seeking advice on the deployment and layout of services on an 8-node cluster. Each node is equipped with dual AMD EYPC 7702s, two 100G Links, and 2TB of RAM. Additionally, I've set up two 13TB NVME drives in a Ceph configuration for each node. At present, the setup involves one management node, while the others are dedicated to running Nova, Neutron, Glance, and Cinder. The workload, while fairly basic, involves handling substantial numbers. It's primarily used for one-off research projects and a few classes, serving around 60 end users. The most demanding scenario occurs during cybersecurity competitions when we need to deploy 3100 VMs at once. When this is spread out over a few days, it's manageable. However, there's a risk of overloading the system, leading to DiskIO issues and Database inconsistencies. Currently, I'm utilizing Kolla for deployment, and I've shared my configurations on this GitHub repository: Link to Configs<https://github.com/RIT-GCI-CyberRange/openstack-configs>. I'd greatly appreciate any insights or advice on optimizing this setup to handle the peak workload more efficiently and prevent potential performance issues. Thanks in advance for your help!

1 0