October 2022 - openstack-discuss - lists.openstack.org

[kolla] [train] Endpoints fail when one controller is down
by Albert Braden 25 Oct '22

25 Oct '22

Some of our clusters are heavily used, and in those clusters we get complaints when we reboot a controller (or sometimes when we deploy and containers restart). Is that normal, or does it mean that we have something configured wrong? The symptoms are intermittent 504 from endpoints, and VM creation/deletion failing or partially completing, for example the VM is created but without DNS records. We are not following the "Removing existing controllers" procedure [1] before rebooting the controller; is that necessary to avoid these issues? 1. https://docs.openstack.org/kolla-ansible/latest/user/adding-and-removing-ho…

1 0

[kolla-ansible][xena][Masakari] masakarimonitors-instancemonitor won't start
by wodel youchi 25 Oct '22

25 Oct '22

Hi, I have this in my computes nodes; masakarimonitors-instancemonitor won't start 2022-10-25 14:37:27.354 7 INFO masakarimonitors.service [-] Starting masakarimonitors-instancemonitor 2022-10-25 14:37:27.357 7 WARNING masakarimonitors.instancemonitor.instance [-] Error from libvirt : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found) *2022-10-25 14:37:27.357 7 ERROR oslo_service.service [-] Error starting thread.: libvirt.libvirtError: authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found) 2022-10-25 14:37:27.357 7 ERROR oslo_service.service Traceback (most recent call last): 2022-10-25 14:37:27.357 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_service/service.py"* , line 806, in run_service 2022-10-25 14:37:27.357 7 ERROR oslo_service.service service.start() 2022-10-25 14:37:27.357 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/masakarimonitors/service .py", line 62, in start 2022-10-25 14:37:27.357 7 ERROR oslo_service.service self.manager.main() 2022-10-25 14:37:27.357 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/masakarimonitors/instanc emonitor/instance.py", line 166, in main 2022-10-25 14:37:27.357 7 ERROR oslo_service.service self._virt_event(uri) 2022-10-25 14:37:27.357 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/masakarimonitors/instanc emonitor/instance.py", line 128, in _virt_event 2022-10-25 14:37:27.357 7 ERROR oslo_service.service vc = libvirt.openReadOnly(uri) 2022-10-25 14:37:27.357 7 ERROR oslo_service.service File "/usr/lib64/python3.6/site-packages/libvirt.py", line 350, in openReadOnl y 2022-10-25 14:37:27.357 7 ERROR oslo_service.service raise libvirtError('virConnectOpenReadOnly() failed') 2022-10-25 14:37:27.357 7 ERROR oslo_service.service libvirt.libvirtError: authentication failed: Failed to start SASL negotiation: - 4 (SASL(-4): no mechanism available: No worthy mechs found) 2022-10-25 14:37:27.357 7 ERROR oslo_service.service 2022-10-25 14:37:27.359 7 INFO masakarimonitors.service [-] Stopping masakarimonitors-instancemonitor I have verified the auth.conf file on both containers the password is the same. In the other hand, in /var/log/kolla/libvirt/libvirtd.log I have this : 2022-10-25 13:29:16.911+0000: 1391031: error : virNetSocketReadWire:1792 : End of file while reading data: Input/output error 2022-10-25 13:30:18.226+0000: 1391031: error : virNetSocketReadWire:1792 : End of file while reading data: Input/output error 2022-10-25 13:31:19.516+0000: 1391031: error : virNetSocketReadWire:1792 : End of file while reading data: Input/output error 2022-10-25 13:32:20.817+0000: 1391031: error : virNetSocketReadWire:1792 : End of file while reading data: Input/output error 2022-10-25 13:33:22.066+0000: 1391031: error : virNetSocketReadWire:1792 : End of file while reading data: Input/output error 2022-10-25 13:34:23.359+0000: 1391031: error : virNetSocketReadWire:1792 : End of file while reading data: Input/output error 2022-10-25 13:35:24.740+0000: 1391031: error : virNetSocketReadWire:1792 : End of file while reading data: Input/output error Any ideas? Regards.

1 0

[kolla-ansible][Yoga] How to configure rabbitmq heartbeat values
by wodel youchi 25 Oct '22

25 Oct '22

Hi, I need to tweak these variables in my Rabbitmq : heartbeat_timeout_threshold heartbeat_rate heartbeat_interval When I tried to add heartbeat_timeout_threshold = 600 to rabbitmq.conf and reconfigure, the service failed to start with : 13:39:41.756 [error] You've tried to set heartbeat_timeout_threshold, but there is no setting with that name. 13:39:41.758 [error] Did you mean one of these? 13:39:41.838 [error] socket_writer.gc_threshold 13:39:41.838 [error] handshake_timeout 13:39:41.838 [error] auth_http.http_method 13:39:41.838 [error] You've tried to set heartbeat_rate, but there is no setting with that name. 13:39:41.838 [error] Did you mean one of these? 13:39:41.888 [error] heartbeat 13:39:41.889 [error] cluster_name 13:39:41.889 [error] channel_max 13:39:41.889 [error] Error preparing configuration in phase transform_datatypes: 13:39:41.889 [error] - Conf file attempted to set unknown variable: heartbeat_rate 13:39:41.889 [error] - Conf file attempted to set unknown variable: heartbeat_timeout_threshold How can I modify those variables? Regards.

2 2

[TripleO] 2023.1 Antelope PTG summary
by Rabi Mishra 25 Oct '22

25 Oct '22

Hi All, Hope y'all had a great PTG last week and a great weekend. I would like to thank everyone who participated/contributed to Tripleo PTG sessions and helped all of us "share and learn". Though personally I don't like long emails, this summary is going to be a bit longer, please bear with me. We had a total of ten sessions spread over two days with very good participation (most sessions with 35+ [max 44] in attendance) and we shared/ discussed a multitude of topics. # Welcome and Zed Retrospective The highlights of the Zed cycle was collaboration across teams to add some new features and fine-tune features implemented in previous cycles that includes; Standalone roles, nftables switch, OVN RAFT support, ephemeral heat for upgrade use cases, CI jobs for Deployed Ceph and Ceph Container Promotion in CI etc. There was a discussion on the suggestion to stop backporting feature removals from stable branches as we did for wallaby in the last few cycles. Though TripleO does not follow stable policy, backporting feature removals with dependent patches across repos more often than not leads to broken CI and user support cases. We would be well placed if we can identify and do those before zed release and avoid the firefighting later. [Action Items] - Update zed review priorities etherpad[1] with deprecation/removal patches [All] # Standalone Ansible Roles Status Update James provided an update on the current status of "Standalone Ansible Roles' POC and we discussed the possible future scope of it beyond the current "scale-out compute" use case. He also demoed molecule based automation of deploying/testing a compute node with the new meta-role. Chandan shared the current work on CI jobs/scenarios to deploy computes with standalone roles. [Action Items] - Prioritize review/merge of proposed patches[2] on this topic [tripleo-core] - Finish the CI jobs for testing standalone roles and make them voting [Chandan/tripleo-ci-core] # Standalone Ansible Roles and External Ceph John presented his work to integrate computes deployed with standalone roles with external ceph. We don't plan to extend this effort to include tripleo deploying ceph using standalone roles, unless there is a requirement for that in the future. [Action Items] - Prioritize review of ceph related standalone role patches.[tripleo-core] # Multi-rhel support for Compute Brendan gave an update on the ongoing effort for multi-rhel support, different options evaluated and current POC status with composable roles approach. There is still some work left on automating role split during upgrade, which is expected to be completed this cycle. [Action Items] - Finish composable roles POC and test upgrade procedure manually [upgrades team] - Add CI jobs to test the proposed procedure upstream as feasible [upgrades team] - Implement role-split automation during upgrade [Takashi/Rabi] # OpenStack SDK, Ansible Collections OpenStack and TripleO We discussed the issues around openstacksdk backward incompatibility and the current pinning for it in RDO/TripleO. We agreed on an action plan to pre-release ansible-openstack-collection (AOC) for RDO/TripleO (though it is still not fully compatible with latest openstacksdk) and remove the pin so that Zed RDO can be released with openstacksdk 0.101.0. [Action Items] - Bump openstacksdk to 0.101.0 for RDO master and zed [Jakob/Alfredo] - Update RDO ansible-collections-openstack to the 2.0.0 pre-release [Jakob/Alfredo] # TripleO and Distributed Project Leadership (DPL) In this session, we discussed the possibility of opting for DPL in TripleO and sharing the responsibilities across multiple leaders in the team. There were a number of volunteers for different liaison roles and we in general agreed that it would be a good thing to do. We would propose the required change to governance sometime this cycle, unless there is any concern. [Action Items] - Finalize all required liaison roles and propose patch to openstack governance repo [Rabi] # Migration from puppetlabs/apache As part of the broader effort to reduce the puppet footprint, Cedric presented the ongoing effort to replace puppetlabs/apache with the new ansible role "tripleo_httpd_vhost"[3]. A few services have already been migrated and remaining ones are expected to be done this cycle with broader participation from sub teams. [Action Items] - Review/merge the already proposed patches for migration [tripleo-core] - Plan and work on migrating remaining services [Cedric] # State of CI CI team presented on a vast range of topics that included, - Added coverage of upgrade jobs across upstream check/gate, periodic component pipeline and periodic integration pipeline in zed (Marios) - Upstream jobs for multi-rhel testing (Marios) - Current state of OVB Jobs (Chandan) - TripleO CI Jobs on IBM Cloud and the new feature to hold nodes for troubleshooting (Chandan) - Tempest Allow List (Arx/Pooja/Soniya) - Tempest Dashboard (Arx/Lukas) # Container Capabilities We revisited the topic of privileged containers for services and how we can possibly avoid those by allowing the required limited container capabilities. As all these services are mostly for the compute role, we agreed that it would probably be better to align this effort with "Standalone Roles" for better testing. # OS Migrate Status Update Jirka provided an update on the tenant migration tool 'os-migrate', an ansible collection that is not a TripleO umbrella project, but used by the upgrades team. It's possible that I missed some important points from these discussions. Please feel free to add them by replying to this thread. All session etherpad links are in the main schedule etherpad[4] for reference. I'll share the session recordings once they're made available to us. Thanks Again. [1] https://etherpad.opendev.org/p/tripleo-zed-review-priorities [2] https://review.opendev.org/q/topic:standalone-roles [3] https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ans… [4] https://etherpad.opendev.org/p/oct2022-ptg-tripleo -- Regards, Rabi Mishra

1 0

[cloudkitty] Antelope PTG summary
by Rafael Weingärtner 25 Oct '22

25 Oct '22

Dear all, The CloudKitty project had a great PTG meeting. Thanks to the OpenInfra Foundation for holding it. Also, thanks to all who have dedicated precious time to attend. We started the meeting by revisiting the latest release and discussing the review format we have adopted in the past two years. Looking at the result of our process, we decided to maintain it, as it seems to bring results and concentrate our efforts on getting CloudKitty better and evolving throughout time. While reviewing the past release, we discussed the situation with patch [1] (Allow rating rules that have 12 digits in the integer part of the number). This patch is a bug fix, and we discussed the idea of backporting it. However, as it is a patch that affects the database schema, it can be troublesome to backport it. Therefore, before doing any work, we will consult the mailing list to gather others opinions on this matter. Users could upgrade CloudKitty to address this issue, if they need to, instead of applying a patch. Pierre volunteered to execute this "inquiry" with the CloudKitty user base in the mailing list. While discussing the ElasticSearch (ES) improvements in CloudKitty to support ES 7, we raised the concern that Kolla-ansible is moving from ElasticSearch to OpenSearch. Therefore, we need to evaluate if this can have an impact in CloudKitty. Most certainly it will impact people using ES as the backend. The goal is to revisit this discussion in our next bi-weekly meeting, and then communicate for users and community (Topics for meeting of 31/10/2022 [2]). Furthermore, as the current CloudKitty team is a bit small, we have decided to focus on the following topics for the Antelope cycle. - Multiple rating types for the same metric -- One metric being used to generate different rates - Different time frame group by options -- provide group by options to group by day of the year, week of the year, month, and year. - Improve some timeouts handling when using Gnocchi backends. This one needs some checking with Zigo, who was the one that reported this situation. Zigo, I added you to this e-mail, can you reply back with more details regarding this situation? - A regression API that takes the resource ID Or project ID or scope ID, and executes a regression (prediction) based on the collected data so far for the given resources/elements/scope. - Add CloudKitty API reference docs ( https://docs.openstack.org/cloudkitty/latest/api-reference/index.html) to https://docs.openstack.org/api-ref/ We also briefly discussed the situation with Monasca, which has already been deprecated from Kolla-ansible, and might be losing some traction and support. Therefore, we (as a community), might need to start thinking about deprecating it, and removing some non-tested code from our code base. We do understand that deprecating and removing support is complicated for users, but nobody is maintaining and testing that integration. Therefore, it tends to get broken and unstable over time. However, if somebody wants to volunteer to maintain and improve it, you are welcome :) In summary that is the outcome of our small, but efficient PTG meeting. Let's keep improving and making CloudKitty better. If I missed something, please do not hesitate to reply back. Also, if somebody needs any help, you can always ping me :) See you guys. Have a nice week! [1] https://review.opendev.org/c/openstack/cloudkitty/+/837200 [2] https://etherpad.opendev.org/p/cloudkitty-meeting-topics -- Rafael Weingärtner

1 0

[neutron] Antelope PTG summary
by Rodolfo Alonso Hernandez 25 Oct '22

25 Oct '22

Hello Neutrinos: This is a quick summary of what we have discussed during the PTG week. Zed summary and retrospective: - Here is the output of this session: https://etherpad.opendev.org/p/neutron-antelope-ptg#L83 Spec handling: - To have a core reviewer assigned per spec. This reviewer will engage the community to review the spec and will be a kind of "godfather". - To have a weekly status report of the active specs in the community, to visibilize them, boost their impact and "coerce" the community to review them. Neutron CLI deprecation: - To remove the CLI code. - Python bindings: - Investigate and report the effort needed to make this migration in neutron client consumers (Nova, Horizon, Heat, etc). - Request other projects using the python bindings to move to OpenStack SDK. - Stop merging new features. Quota classes: - Information: https://docs.openstack.org/project-team-guide/technical-guides/unified-limi… - Neutron RFE: https://bugs.launchpad.net/neutron/+bug/1993288 - Still under development in the community, this is still not an accepted community goal, thus we won't start the development until some cons are discussed with the TC members (see the Neutron RFE description). Pyroute2 session: - IPmock, a new class to mock some pyroute2 objects: https://lab.pyroute2.org/ - Pluggable netlink engines. Still not compatible with NetNS (namespaces) - Transactional engine: - A daemon is always reading the netlink and inotify, and builds a DB with the system information. Data is read faster. This could be not compatible with the current sync calls using privsep. - Using NDB class. - Customize message parser, to reduce the amount of data retrieved from the kernel. - Dump/list operators will be generators (to check the compatibility with privsep). Remove the failed devstack migration and get back to what is called "neutron-legacy" (that was discussed before in a PTG but we didn't start the reversion). DNS subdomain support in ML2/OVN: - https://review.opendev.org/c/openstack/neutron-specs/+/832658 - On hold until there is a use case. DNS resolver in ML2/OVN: - https://bugzilla.redhat.com/show_bug.cgi?id=2104568 - The main devoplment part falls in the OVN core team. Once finished, the Neutron part should be almost trivial. ML2/OVN flow metering. - Same as we currently have with ML2/OVS + iptables firewall (currently not supported with native firewall because we can't count OF flows). - Same as the previous feature, the main development is in the OVN core team. Neutron QA: - Implement a grenade job based on the SLURP cadence, testing from Y to A (done). - Have the "tempest-integrated-networking" job in our gate/check queues (patch under review). PCI Device Tracking In Placement (Nova-Neutron session): - The different approaches are still under development. - Spec to be proposed during this release (see etherpad to check the possible implementation options). Port binding switchdev capabilities (Nova-Neutron session): - We agreed that the current implementation is incorrect: Neutron should not modify the port binding dictionary. Nova is not reading it but overwriting it. - We won't change the current code but no new features will rely on it. Nova mutable MTU support (Nova-Neutron session): - For now, Neutron will document that a network MTU change requires a VM reboot (ot port detach/attach operations) to be applied on this port. S-RBAC: - Neutron has three implemented personas: system wide admin, project user and project reader. - Tempest will integrate all RBAC testing projects during this release. Once done, the new RBAC flag will be enabled by default. - Neutron will create the corresponding RBAC job/jobs. - Neutron needs to identify what is called "service role" calls (done from other projects). This is a new concept that will be included in next releases. nftables migration: - The current iptables legacy interface could be deprecated. During this release, Neutron will start an epic to move the current iptables API to the new nftables API; if possible, incrementally, mixing both APIs in the same agent. Please, if there is any missing topic, let me know. Regards and thank you for attending and participating.

1 0

[Neutron] Bug Deputy Report Oct 17 - 24
by Oleg Bondarev 25 Oct '22

25 Oct '22

Hello Neutron Team, It was probably the most quiet week in the history of Bug Reports: Low ----- - https://bugs.launchpad.net/neutron/+bug/1993181 - [OVN] OVN metadata "MetadataProxyHandler" not working if workers=0 - Fix Released: https://review.opendev.org/c/openstack/neutron/+/861649 - Fixed by Rodolfo Alonso - https://bugs.launchpad.net/neutron/+bug/1993502 - failing unit tests when not running them all - Fix Released: https://review.opendev.org/c/openstack/neutron/+/861869 - Fixed Rodolfo Alonso - https://bugs.launchpad.net/neutron/+bug/1993498 - [CI] Create oslo master branch jobs (apart from py39)8 - In progress: https://review.opendev.org/c/openstack/neutron/+/861859 - assigned to Rodolfo Alonso RFEs: ----- - https://bugs.launchpad.net/neutron/+bug/1993288 - RFE: Adopt Keystone unified limits as quota driver for Neutron - New (PTG decision - wait till it become more mature) - Unassigned Thanks, Oleg

1 0

[Cinder][Nova]Zombie process prevention for Cinder and Nova APIs
by yu-kishimoto＠kddi.com 25 Oct '22

25 Oct '22

Hi all, I'm trying to fix issues of spawning zombie processes for Cinder and Nova APIs during instantiation in an IaC (Ansible which is not openstack-ansible). Does someone can kindly help me to solve the issue or give me some clues like parameter changes that can be expected to be effective? - Issues The cinder api service process(port: 8776) existed in a zombie state. The neutron api service process(port: 9696) required for nova existed in a zombie state. - What the IaC does Use an existing tenant to create a network and subnet, then use 8volumes to instantiate 8VMs. - Workarounds and what I've done so far Identify and kill zombie processes. - Environment OS: CentOS Stream 8 Kernel: 4.18.0-408.el8.x86_64 OpenStack: Yoga(Deployed by PackStack: https://www.rdoproject.org/install/packstack/) Nova: 25.0.1 Neutron: 20.2.0 Cinder: 20.0.1 KeyStone: 21.0.0 -- Yukihiro Kishimoto Technologist KDDI Co., Ltd. Tokyo Japan

1 1

[magnum] Antelope PTG summary
by Jake Yip 24 Oct '22

24 Oct '22

Dear all, Magnum had a great meeting during the latest PTG. Thanks to OpenInfra Foundation for holding it. Also, thanks to all who has found precious time to attend. As the current Magnum team is quite small, we have decided on the following: - Deprecate the Fedora Atomic driver - This is EOL. - Deprecate the CoreOS driver - This is similarly EOL. - Deprecate the Docker Swarm driver. Kubernetes has became the container orchestration engine of choice now. With our limited resources, it will be more efficient to support the most popular choice well, instead of multiple choices. - Mark branches before Victoria as unmaintained. These branches only support Kubernetes versions that has been EOL. - Most excitingly, move towards adding Cluster API support for Kubernetes. This allows the Magnum project to leverage off upstream effort in creating and managing a Kubernetes cluster. We understand that removing support is difficult for users, but we believe this is for the better. Our main aim is to keep the project moving fast enough to keep up with upstream Kubernetes. If anyone is willing to help maintain any of the items we plan on deprecating, please drop me a message. Regards, Magnum Team. -- Jake Yip DevOps Engineer, ARDC Nectar Research Cloud

1 0

[nova][placement] 2023.1 Antelope PTG summary
by Sylvain Bauza 24 Oct '22

24 Oct '22

We're on a wetty Monday here and I guess this is time for me to take my pen again and write a PTG recap email for the current release, which is 2023.1 Antelope. Yet again, I *beg* you to *not* use any translation tool when reading any etherpad, including Google Chrome embedded translation or it would fully translate the whole etherpad content for *all* readers. In order to prevent any accidental misusage, here is a readonly copy of the Nova etherpad we had for the week https://etherpad.opendev.org/p/r.8ae4e0ef997aebfe626b2b272ff23f1b Again, I'm human and while I can write bugs (actually I write a lot of them), I can also write wrong things in this email and I can misinterpretate something precise. Apologies if so, and please don't refrain them yourself to correct me by replying to this email. Last but not the least, I'm more than open to any questions or remarks that would come from the reading of this long email. Good luck with this very long thread, you should maybe grab a coffee before starting to read it (and I promise to keep it as short as I can). ### Operator hours ### After the success of the productive Nova meet-and-greet session we held in Berlin (packed room w/ lots of feedback), we were eager to again have a discussion with operators, this time virtually which would have hopefully lowered the entry barrier by not requiring an in-person event in order to attend the session. Consequently, we allocated three timeslots of one hour, two back-to-back on Tuesday and one on Wednesday (on different times to arrange time differences between operators). Unfortunately, I have to write here that we only had a very small attendance on Tuesday with only three operators joining (but providing great feedback, thanks btw. !) and *none* on the 1-hour Wednesday slot. As a result, I won't provide statistics about release and project usage here as numbers are too low to be representative, but here is what we discussed : (again, a readonly etherpad is available here https://etherpad.opendev.org/p/r.aa8b12b385297b455138d35172698d48 for further context) # Upcoming features and general discussions - the possibility to mount Manila shares in Nova seems promising. - As nova doesn't support multiple physnets per neutron network today, this prevents the use of routed networks in some cases. - modifying our mod_wsgi usage to be allowed to pass some arguments is requested - operators seem interested in having PCI devices tracked in Placement # Pain points - getting inventories or used/availables resources in Placement becomes costly as it requires N calls, with N be the number of Resource Providers (instead of a single HTTP call). Problem has been acknowledged by the team and some design discussion occurred. - We should paginate on the flavors API and we need to fix the private > public flavor bug. - mediated devices disappearing at reboot is a problem. This has been discussed during the contributor session later, see below. - routing metrics are hard to manage when you have a long list of multiple ports attached. We eventually agreed on the fact the proposal fix could be an interesting feature for Neutron team to develop. That's it for the operators discussions. Let's now discuss about what we discussed in the contributors PTG agenda : ### Cross-project discussions Two projects this time were discussing with the Nova community : # Ironic-Nova The whole session was about the inconsistencies that were happening if a nova-compute was failing down with Ironic rebalancing the nodes. - we agreed on the fact this wasn't an easy problem to solve - a consensus was around the fact Ironic should support a sharding key so the Nova compute could use it instead of using hash rings. - JayF and johnthetubayguy agreed on codifying this idea into a spec - we'd hear feedback from operators about what they feel of the above migration (sharding their node cloud into sharding pieces and providing sharding details to nova) - in parallel of the above, documentation has to be amended in order to recommend for *existing deployments* to setup active/passive failover mode for Nova computes (instead of relying on hashring rebalances) # Neutron-Nova (I'll briefly cover the details here, as it will also be covered by the Neutron PTG recap) - while Nova is finishing up tracking of PCI devices in Placement this cycle, we agreed on defering the modeling of Neutron-based PCI requests until next cycle. This won't have any impact in terms of existing features that will continue to operate seamlessly. - the way we define switchdev capabilities as of now is incorrect. We agreed on modfying Nova to allow it to report such capabilities to Neutron. This may be a specless blueprint or just a bugfix (and then potential backportability), to be determined later. - MTUs are unfortunately immutable. If you change the MTU value by Neutron, you need to restart the instance( (or reattach the port). We consequently agreed on documenting this caveat, which is due to libvirt/qemu not able to change the MTU while the guest is running. - We eventually agreed on letting --hostname (parameter of an instance creation request) to represent a FQDN thru a nova microversion. This value won't be sanitized by nova and directly passed to Neutron as the hostname value. We also agreed on the fact Nova WON'T ever touch the port domain value in metadata, as this is not in the project scope to manage name domains. ### Procedural discussions (skip it if you're not interacting with the Nova contributors) ### # Zed retrospective - We had a productive Zed release and we made a good job on reducing bugs and bug reports. Kudos to the team again. - The microversions etherpad we had was nice but planning microversions usage is hard. WE agreed on rather providing a document to contributors explaining them how to write an API change that adds a new microversion and how to easily rebase this change if a merge conflict occurs (due to the micoversion being taken by another patch that merged) - We agreed on keeping an etherpad for tracking all API changes during milestone-3 - We agreed on filing blueprints for TC goals that impact Nova - bauzas will promote again the use of review-priority label in Gerrit during weekly meetings in order for cores to show their interest in reviewing a particular patch. # Promoting backlog features to new contributors and enhance mentoring - Starting this cycle, we'll put our list of small and easily actionable blueprints into Launchpad bug reports that have a "Wishlist" status and both a 'low-hanging-fruit' and a 'rfe' tag. New contributors or people wanting to mentor a new upstream member are more than welcome to consume that list of 'rfe' bugs and identify the ones they're willing to work on. A detailed process will be documented for helping newcomers to join. - We'll also draft our pile of work we defer into 'backlog specs' if they require further design (and are less actionable from a newcomer perspective) # Other procedural and release discussions - we'll stop using Storyboard for tracking Placement features and bugs and we'll pivot back to Launchpad for the 'placement' sub-project. - we agreed on a *feature* review day possibly 4 weeks before the feature freeze in order to catch up any late design issue we could have missed when reviewing the spec. - we will EOL stein and older nova branches with elodilles proposing the deletion patch. We agred on discussing EOL'ing train at next PTG. - gibi will work on providing a static requirements file for nova that would use minimum versions of our dependencies (non transitively) and modify unit and functional test jobs to rather use this capped requirements file for testing. - we discussed the User Survey and we agreed on discussing any question we may want to add in the next survey during the next weekly meetings. - "2023.1 Antelope" naming is a bit confusing, but we agreed on we should continue to either use "2023.1" or "2023.1 Antelope" for naming our docs. We also wait for guidelines in order to consistently name our next stable branch (2023.1 possibly). (That's the end of procedural and release discussions, please resume here if you skipped the above) ### Technical bits ### # VMware driver status - As currently no job runs happened since April, we agreed on communicating in the nova-compute logs at periodic times that the code isn't tested so operators running it would know its upstream status. - we'll update the supported matrix documentation to refiect this 'UNTESTED' state # TC goals this cycle - for the FIPS goal, we agreed on the fact the current FIPS job (running on centos 9 stream) shouldn't be running on gate and check pipelines. As the current job is busted (all periodic runs go into TIMEOUT state), we also want to modify the job timeout to allow 30 mins more time for running (due to a reboot in the job definition) - for the oslo.privsep goal, no effort is required from a nova perspective (all deliverables are already using privsep). sean-k-mooney can propose a 'rfe' bug (see the note above on new contributors) for modifying our privsep usage in nova (using different privsep context by callers) - for the Ubunutu 2022.4 goal, gmann already provided changes. We need to review them. - for the RBAC goal, let's discuss it in a proper section just below # Next steps on RBAC - we need to test new policy defaults in a full integrated tempest testsuite (meaning with other projects), ideally before the first milestone. - once we check everything works fine as expected, we can flip the default (enabling new policies) before milestone-2 - we'd like to start drafting the new service role usage by two new specs (one for nova, the other one for placement) # Power management in Nova A pile of work I'm proud we gonna start this cycle. This is about disabling/enabling cores on purpose, so power savings occur. - we agreed on supporting it directly in Nova (and to not design and support an external tool which would suppose to draft some heave interface for an easy quickwin). This would just be a config flag to turn on that would enable CPU cores on demand. - a potential follow-up *may* be to use a different CPU governor depending on flavors or images but this won't be scoped for this release. # Power monitoring in Nova I don't really like this name, as monitoring isn't part of the Nova mission statement so I'll clarify. Here, this is about providing an internal readonly interface for power monitoring tools running on guests that would be able to capture host consumption metrics. One example of such monitoring tools is Scaphandre, demonstrated during the last OpenInfraSummit at a keynote. - we agreed on reusing virtiofs support we're gonna introduce in this release for the Manila share attachment usecase - this share would be readonly and would be unique per instance (ie. we wouldn't be supporting multiple guest agents reading different directories) - this share would be enabled thru a configuration flag per compute, and would only be mounted per instance by specific flavor or image extraspec/metadata. # Database soft-deleted records and the archive/purge commands - We don't want to deprecate DB soft-deleted records as some APIs continue to rely on those records. - We rather prefer to add a new parameter to the purge command that will directly delete the 'soft-deleted' records from the main DBs and not only the shadow tables (skipping the need to archive, if the operator wants to use this parameter) # Nova-compute failing at reboot due to vGPU missing devices Talking of bug #1900800, the problem is that mediated devices aren't persistent so nova-compute isn't able to respawn the instances after a reboot if those are vGPU-flavored. - we agreed on the fact that, like any other device, Nova shouldn't try to create them and we should rather ask the operator to pre-create the mediated devices, exaclty like we do for SR-IOV VFs. - we'll accordingly deprecate the mdev creation in the libvirt driver (but we'll continue to support it) and we'll log a warning if Nova has to create one mdev. - we'll change the nova-compute init script to raise a better exception explaining which mdev is missing - we'll document a procedure for explaining how to get existing mdev information and persist them by udev rules (for upgrade cases) # Robustify our compute hostname changes To be clear, at first step, we will continue to *NOT* support nova-compute hostname changes but we'll better detect the hostname change to prevent later issues. - first step to persist the compute UUID on disk seems a good candidate so a spec is targeted this cycle. - next steps could be to robustify our relationships between instance, compute node and service object records but this design will be deferred for later in a backlog spec. # Move to OpenStackClient and SDK OSC is already fully supported in Zed but it continues to rely on novaclient python bindings for calling the Nova API. - we agreed on modifying OSC to rather use openstacksdk (instead of novaclient) for communicating to the Nova APIs. Contributors welcome on this move. - we agreed on stopping to use project client libraries in nova services (eg. cinderclient used by nova-compute) and rather use openstacksdk directly. A 'rfe' bug per project client will be issued, anyone willing to work on it is welcome. - we also agreed on continuing to support novaclient for a couple of releases, as operators or other consumers of this package could require substantial efforts to move to the sdk. - we agreed on changing the release model for novaclient to be independent so we can release anytime we need. # Evacuate to target state We understand the usecase (evacuate an instance shouldn't always turn the evacuated instance on) - that said, we don't want to amend the API for passing a parameter as this would carry a tech debt indefinitely) - we prefer to propose a new microversion that would stop the instance eventually instead of starting it - operators wanting to keep the original behaviour would need to negociate an older microversion to the Nova API as we don't intend to make spawning optionally chosen by API on evacuate. (aaaaaaaaaaaaaaaand that's it for the PTG recap) Kudos, you were brave, you reached that point. Hope your coffee was good and now you feel rejuvenated. Anyway, time for me now to rest my fingers and to enjoy a deserved time off. As said, I'm all up for any questions or remarks that would come from the reading of this enormous thread. -Sylvain

2 1