October 2022 - openstack-discuss - lists.openstack.org

Fwd: DPDK+OVS with OpenStack
by Mark Wittling 23 Apr '25

23 Apr '25

Looking for someone who knows OpenStack with OpenVSwitch, and in addition to that, DPDK with OpenStack and OVS.I am using OpenStack Queens, with OpenVSwitch. The architecture I am using is documented here: https://docs.openstack.org/neutron/queens/admin/deploy-ovs-provider.htmlThe OVS I am using on the Compute Node, is compiled with DPDK, and I have enabled the datapath to netdev (DPDK) on br-prv (provider network bridge), and br-tun (tunneling bridge). But these two bridges, br-tun and br-prv, are patched into another OpenStack bridge, called br-int. I wasn’t actually sure about whether to tinker with this bridge, and wondered what datapath it was using.Then, I realized there is a parameter in the openvswitch_agent.ini file, which I will list here: # OVS datapath to use. 'system' is the default value and corresponds to the # kernel datapath. To enable the userspace datapath set this value to 'netdev'. # (string value) # Possible values: # system - <No description provided> # netdev - <No description provided> #datapath_type = system datapath_type = netdev So in tinkering with this, what I realized, is that when you set this datapath_type to system or netdev, it will adjust the br-int bridge to that datapath type.So here is my question. How can I launch a non-DPDK VM, if all of the bridges are using the netdev datapath type?Here is another question. What if one of the flavors don’t have the largepages property set on them? I assumed OpenStack would revert to a system datapath and not use DPDK for those VM interfaces. Well, I found out in testing, that is not the case. If you set all your bridges up for netdev, and you don’t set the property on the Flavor of the VM (largepages), the VM will launch, but it simply won’t work.Is there no way, on a specific Compute Host, to support both DPDK (netdev datapaths) and non-DPDK (system datapaths)?Either on a VM interface level (VM has one interface that is netdev DPDK and another that is system datapath non-DPDK)?Or on a VM by VM basis (VM 1 has 1 or more netdev datapath interfaces and VM 2 has 1 or more system datapath interfaces)?Am I right here? Once you set up a Compute Host for DPDK, it’s DPDK or nothing on that Compute Host? (edited)

3 2

How to config DPDK compute in OVN
by dangerzone ar 22 Apr '25

22 Apr '25

Hi....as far I understand ovs does support dpdk compute to be deployed. I'm referring to the guideline ' *https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html <https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html>* ' ... It's too brief.. May I know how to config dpdk compute in OVN networking openstack? Hope someone could help what is the config to be done.... Currently compute is run without dpdk over openstack train ovn based. Please need some info on how I can proceed. Thank you

2 1

ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat.| Openstack tripleo wallaby version
by Swogat Pradhan 24 Dec '24

24 Dec '24

Hi, I am currently trying to deploy openstack wallaby using tripleo arch. I created the network jinja templates, ran the following commands also: #openstack overcloud network provision --stack overcloud --output networks-deployed-environment.yaml custom_network_data.yaml # openstack overcloud network vip provision --stack overcloud --output vip-deployed-environment.yaml custom_vip_data.yaml # openstack overcloud node provision --stack overcloud --overcloud-ssh-key /home/stack/sshkey/id_rsa overcloud-baremetal-deploy.yaml and used the environment files in the openstack overcloud deploy command: (undercloud) [stack@hkg2director ~]$ cat deploy.sh #!/bin/bash THT=/usr/share/openstack-tripleo-heat-templates/ CNF=/home/stack/ openstack overcloud deploy --templates $THT \ -r $CNF/templates/roles_data.yaml \ -n $CNF/workplace/custom_network_data.yaml \ -e ~/containers-prepare-parameter.yaml \ -e $CNF/templates/node-info.yaml \ -e $CNF/templates/scheduler-hints.yaml \ -e $CNF/workplace/networks-deployed-environment.yaml \ -e $CNF/workplace/vip-deployed-environment.yaml \ -e $CNF/workplace/overcloud-baremetal-deployed.yaml \ -e $CNF/workplace/custom-net-bond-with-vlans.yaml Now when i run the ./deploy.sh script i encounter an error stating: ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat. The generated environments from 'openstack overcloud baremetal provision' and 'openstack overcloud network provision' must be included with the deployment command.: tripleoclient.exceptions.InvalidConfiguration: Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat. The generated environments from 'openstack overcloud baremetal provision' and 'openstack overcloud network provision' must be included with the deployment command. 2022-04-19 13:47:16.582 735924 INFO osc_lib.shell [-] END return value: 1 Can someone tell me where the mistake is? With regards, Swogat Pradhan

4 4

Live migration fails
by Szabo, Istvan (Agoda) 07 Aug '24

07 Aug '24

Hi, I have couple of compute nodes where the live migration fails with existing vms. When I quickly spawn a vm and try live migration it works so I assume shouldn't be a big problem with the compute node. However I have many existing vms where it fails with a servername not found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 ERROR nova.conductor.tasks.migrate [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Unable to find record for source node servername on servername: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Failed to compute_task_migrate_server: Compute host servername could not be found.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Setting instance to ACTIVE state.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.672 227612 ERROR oslo_messaging.rpc.server [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Exception during message handling: ComputeHostNotFound: Compute host am-osfecn-4025 Tried with this command: nova live-migration --block-migrate id. Any idea? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 16

[puppet] Proposing core team clean up
by Takashi Kajinami 23 Aug '23

23 Aug '23

Hello, I went through the current core reviewers and found out that the current list includes some people no longer working in the OpenStack community. I'd like to propose removing the following persons from the core reviewer team as we haven't seen these people making any activity in the community recently. * Pradeep Kilambi - puppet-aodh, puppet-ceilometer, puppet-gnocchi, puppet-panko * Richard Megginson - puppet-keystone * Denis Egorenko - puppet-murano * Serg Melikyan - puppet-murano * Sergey Kolekonov - puppet-neutron * David Moreau Simard - puppet-openstack-integration * Yu xingchao - all modules Also, I confirmed that Wes Hayutin is no longer working for OpenStack and he agreed to retire from the core reviewer position. Thus I'll remove him as well. I'll be open for any feedback for one week to hear any thoughts/feedback about this removal. I'd like to take this moment to thank all people for their past work. We always welcome them back if they resume their work in this project. Thank you, Takashi KAJINAMI

3 4

Re: [all] [oslo.messaging] Interest in collaboration on a NATS driver
by Tobias Urdin 22 Jul '23

22 Jul '23

Hello Kai, Great to hear that you’re interested! I’m currently just testing this as a POC directly in devstack so there is real-world testing (not should it yet I think, needs more development first). Best regards Tobias > On 30 Aug 2022, at 18:32, Kai Bojens <kb(a)kbojens.de> wrote: > > Am 29.08.22 um 15:46 schrieb Tobias Urdin: > >> If anybody, or any company, out there would be interested in collaborating in a project to bring this support and maintain it feel free to >> reach out. I’m hoping somebody will bite but atleast I’ve put it out there for all of you. > > Hi, > I am very much interested to take a closer look at your work and maybe contribute to it. Although I'm working with OpenStack for my employer at the moment, I'd do this in my spare time as I'm not sure that I can convince him to add another staging system with a full OpenStack installation just for developing a RabbitMQ replacement. That's not on our agenda as we are mostly users and not developers of OpenStack. > > As I'm pretty new to the messaging topic and had just heard of NATS and your idea, I'll first would have to dive into NATS and then I would take a closer look at your code and maybe try to create some documentation or tests. So, I can't promise anything but as I said I'm very interested in your approach as I also see the massive load from RabbitMQ. > > This brings me to the most important question: Where do you run and test your code? > > Greetings, > Kai

8 30

[ironic][stable] Proposing EOL of ironic project branches older than Wallaby
by Jay Faulkner 02 Jun '23

02 Jun '23

Hi all, Ironic has a large amount of stable branches still in EM. We need to take action to ensure those branches are either retired or have CI repaired to the point of being usable. Specifically, I'm looking at these branches across all Ironic projects: - stable/queens - stable/rocky - stable/stein - stable/train - stable/ussuri - stable/victoria In lieu of any volunteers to maintain the CI, my recommendation for all the branches listed above is that they be marked EOL. If someone wants to volunteer to maintain CI for those branches, they can propose one of the below paths be taken instead: 1 - Someone volunteers to maintain these branches, and also report the status of CI of these older branches periodically on the Ironic whiteboard and in Ironic meetings. If you feel strongly that one of these branches needs to continue to be in service; volunteering in this way is how to save them. 2 - We seriously reduce CI. Basically removing all tempest tests to ensure that CI remains reliable and able to merge emergency or security fixes when needed. In some cases; this still requires CI fixes as some older inspector branches are failing *installing packages* in unit tests. I would still like, in this case, that someone volunteers to ensure the minimalist CI remains happy. My intention is to let this message serve as notice and a waiting period; and if I've not heard any response here or in Monday's Ironic meeting (in 6 days), I will begin taking action on retiring these branches. This is simply a start; other branches (including bugfix branches) are also in bad shape in CI, but getting these retired will significantly reduce the surface area of projects and branches to evaluate. I know it's painful to drop support for these branches; but we've provided good EM support for these branches for a long time and by pruning them away, we'll be able to save time to dedicate to other items. Thanks, Jay Faulkner

3 5

live resize
by Parsa Aminian 15 Mar '23

15 Mar '23

hello on openstack with ceph backend is it possible to live resize instances ? I want to change flavor without any down time .

5 6

[kolla-ansible] [yoga] [magnum] [k8s] cannot attach persistent volume to pod
by Stuart Whitman 24 Feb '23

24 Feb '23

Hello, When I try to attach a persistent cinder volume to a pod, I get FailedMount and FailedAttachVolume timeout events. I also get these errors in the log of the csi-cinder-controllerplugin-0 pod: E1020 13:38:41.747511 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.VolumeAttachment: the server could not find the requested resource E1020 13:38:41.748187 1 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.CSINode: the server could not find the requested resource I fixed a CrashLoopBackoff error with the csi-snapshotter container in the csi-cinder-controllerplugin-0 pod by providing the label "csi_snapshotter_tag=v4.0.0" when I created the cluster template. I found that suggestion in an issue on the GitHub cloud-provider-openstack project. I'm not finding any help with this error on Google. Thanks, -Stu _____________________________________ The information contained in this e-mail and any attachments from Group W may contain confidential and/or proprietary information and is intended only for the named recipient to whom it was originally addressed. If you are not the intended recipient, be aware that any disclosure, distribution, or copying of this e-mail or its attachments is strictly prohibited. If you have received this e-mail in error, please notify the sender immediately of that fact by return e-mail and permanently delete the e-mail and any attachments to it.

3 5

[charms] Team Delegation proposal
by Chris MacNaughton 08 Feb '23

08 Feb '23

Hello All, <tl;dr> I would like to propose some new ACLs in Gerrit for the openstack-charms project: - openstack-core-charms - ceph-charms - network-charms - stable-maintenance </tl;dr> With an increasing focus split among the openstack-charmers team, I'm observing that people are focused on more specific subsets of the charms, and would like to propose that new ACLs are created to allow us to recognize that officially. I've chosen the breakdown above as it aligns neatly with where the focus lines are at this point, letting developers work on their specific focus areas. This proposal would not reduce permissions for anybody who is currently a core on the openstack-charms project and, in fact, future subteam core members could aspire to full openstack-charmers core as well. Ideally, this approach will let us escalate developers to "core" developers for the subteam(s) where they have demonstrated the expertise we expect in a core-charmer. It also allows a more gradual escalation to being a core in the openstack-charms project, making it a progression rather than a single destination. As a related addition, I'm appending to this proposal the creation of a stable-maintenance ACL which would allow members to manage backports without a full core-charmer grant. --- Chris MacNaughton

5 7

[cinder][drivers] third-party CI for os-brick changes
by Brian Rosmaita 04 Jan '23

04 Jan '23

To all third-party CI maintainers, As you are aware, cinder third-party CI systems are required to run on all cinder changes. However, the os-brick library used in cinder CI testing is the latest appropriate *released* version of os-brick. Thus, it is possible for changes to be happening in os-brick development that might impact the functionality of your driver. If you aren't testing os-brick changes, you won't find out about these until *after* the next os-brick release, which is bad news all around. Therefore, at last week's cinder midcycle [0], the cinder project team agreed to require that cinder third-party CI systems run on all os-brick changes in addition to all cinder changes. This is a nice-to-have for the current (Zed) development cycle, but will be required in order for a driver to be considered 'supported' in the 2023.1 (Antelope) release [1]. If you have comments or concerns about this policy, please reply on the list to this email or put an item on the agenda [2] for the cinder weekly meeting. [0] https://etherpad.opendev.org/p/cinder-zed-midcycles [1] https://wiki.openstack.org/wiki/Cinder/tested-3rdParty-drivers#What_changes… [2] https://etherpad.opendev.org/p/cinder-zed-meetings

2 1

Glance api deployed only on a single controller on multi-controller deployment [kolla]
by A Monster 30 Dec '22

30 Dec '22

I've deployed openstack xena using kolla ansible on a centos 8 stream cluster, using two controller nodes, however I found out after the deployment that glance api is not available in one node, I tried redeploying but I got the same behavior, although the deployment finished without displaying any error. Thank you. Regards

4 4

[all][TC] Bare rechecks stats week of 25.07
by Slawek Kaplonski 17 Dec '22

17 Dec '22

Hi, Here are fresh data about bare rechecks. +--------------------+---------------+--------------+-------------------+ | Team | Bare rechecks | All Rechecks | Bare rechecks [%] | +--------------------+---------------+--------------+-------------------+ | requirements | 2 | 2 | 100.0 | | keystone | 1 | 1 | 100.0 | | OpenStackSDK | 3 | 3 | 100.0 | | tacker | 3 | 3 | 100.0 | | freezer | 1 | 1 | 100.0 | | rally | 1 | 1 | 100.0 | | OpenStack-Helm | 14 | 15 | 93.33 | | cinder | 6 | 7 | 85.71 | | kolla | 20 | 25 | 80.0 | | neutron | 4 | 5 | 80.0 | | Puppet OpenStack | 10 | 13 | 76.92 | | OpenStack Charms | 3 | 6 | 50.0 | | nova | 11 | 22 | 50.0 | | tripleo | 8 | 19 | 42.11 | | ironic | 1 | 3 | 33.33 | | Quality Assurance | 1 | 3 | 33.33 | | octavia | 0 | 1 | 0.0 | | OpenStackAnsible | 0 | 1 | 0.0 | | designate | 0 | 1 | 0.0 | | Release Management | 0 | 1 | 0.0 | | horizon | 0 | 1 | 0.0 | +--------------------+---------------+--------------+-------------------+ Those data should be more accurate as my script now counts only comments from last 7 days, not all comments from patches updated in last 7 days. Reminder: "bare rechecks" are recheck comments without any reason given. If You need to do recheck for patch due to failed job(s), please first check such failed job and try to identify what was the issue there. Maybe there is already opened bug for that or maybe You can open new one and add it as explanation in the recheck comment. Or maybe it was some infra issue, in such case short explanation in the comment would also be enough. -- Slawek Kaplonski Principal Software Engineer Red Hat

6 10

[neutron] Switching the ML2 driver in-place from linuxbridge to OVN for an existing Cloud
by Christian Rohmann 26 Nov '22

26 Nov '22

Hello openstack-discuss and neutron experts, I'd like to ask for your input and discussion on the idea of changing the ML2 driver for an existing cloud, read: changing a tire while still riding the bike. I actually like to find out if it's feasible to switch from the trusted Linuxbridge driver to the more modern SDN stack of OVN - in place. With all existing user networks, subnets, security groups and (running) instances in the database already. And while I know there a push of migrating from OVS to OVN and a clear migration path documented (https://docs.openstack.org/neutron/latest/ovn/migration.html) but they are much more similar in their data plane. And to get this out of the way: I am not asking to do be able to do this without downtime, interruptions, migrations or any active orchestration of the process. I just want to know all the possible options apart from setting up a new cloud and asking folks to migrate all of their things over... 1) Are the data models of the user managed resources abstract (enough) from the ML2 used? So would the composition of a router, a network, some subnets, a few security group and a few instances in a project just result in a different instantiation of packet handling components, but be otherwise transparent to the user? 2) What could be possible migration strategies? While it might be a little early to think about the actual migration steps. But just to consider more than a full cloud shutdown following a cold start with modified neutron config then using the OVN ML2. I know there is more than just the network and getting a virtual layer 2 network. There are DHCP, DNS and the metadata service and last but not least the gateway / router and the security groups. But if using VXLAN for OVN (as Linuxbridge uses currently) could the shift from one implementation to the other potentially be done node by node? Or project by project by changing the network agents over to nodes already running OVN? If not, is there any other hot cut-over approach that would avoid having to shutdown all the instances, but only cause them some network downtime (until the next DHCP renew or similar?) Has anybody ever done something similar or heard about this being done anywhere? Thanks for your time and input, Christian

3 5

[all][stable][ptl] Propose to EOL Queens series
by Elõd Illés 26 Nov '22

26 Nov '22

Hi, As more and more teams decide about moving their Queens branches to End of Life, it looks like the time has come to transition the complete Queens stable release for every project. The reasons behind this are the following things: - gates are mostly broken - minimal number of bugfix backports are pushed to these branches - gate job definitions are still using the old, legacy zuul syntax - gate jobs are based on Ubuntu Xenial, which is also beyond its public maintenance window date and hard to maintain - lack of reviews / reviewers on this branch Based on the above, if no objection comes from teams, then I'll start the process of EOL'ing Queens stable series. Please let the community know what you think, or indicate if any of the projects' stable/queens branch should be kept open in Extended Maintenance. Thanks, Előd Illés irc: elodilles @ #openstack-stable / #openstack-release

1 1

[designate] How to avoid NXDOMAIN or stale data during cold start of a (new) machine
by Christian Rohmann 23 Nov '22

23 Nov '22

Hello openstack-discuss, I have a designate setup using bind9 as the user-serving DNS server. When starting a machine with either very old or no zones at all, NXDOMAIN or other actually stale data is sent out to clients as designate is not done doing an initial full sync / reconciliation. * What is the "proper" way to tackle this cold-start issue and to keep the bind from serving wrong data? ** Did I miss on any options to handle this startup case? * What is the usual runtime for an initial sync that you observe in case the backend DNS server has no zones at all anymore? Regards Christian

2 7

[all][openstack-dev][ptls] Migrating devstack jobs to Jammy (Ubuntu LTS 22.04)
by Dmitriy Rabotyagov 21 Nov '22

21 Nov '22

Hi everyone, According to a 2023.1 community-wide goal [1], base-jobs including but not limited to devstack-minimal, devstack, devstack-ipv6, devstack-multinode, tox, will be switched from Ubuntu Focal (20.04) to Jammy (22.04). That will also bring python 3.10 as a default python interpreter for these jobs. Migration will affect most of the projects, so to make it smooth, we will split the process into the following steps: 1. Patches for switching base jobs to Jammy will be proposed and marked as WIP (Work In Progress). It will allow projects to check if their code is compatible without breaking gates. These patches are ready and you can check them out here [2] 2. All projects should ensure that the code is compatible and merge changes if required. To properly track and test these changes you should: * Create a DNM patch that will contain in its commit message: "Depends-On: https://review.opendev.org/c/openstack/tempest/+/861110" to consume new nodesets. You can check the sample patch for Nova below [3] * In case of any job failure due to Ubuntu 22.04 or Python 3.10, land requires changes to support this new distribution * If you have overridden nodeset for your project jobs, ensure that you also have jobs against Ubuntu 22.04 and these jobs set to voting. * Set topic to all related changes to "migrate-to-jammy" * Please use etherpad [4] for tracking patches, bug reports, current status, or any other activity related to this topic 3. On R-18, which is the first 2023.1 milestone that will happen on the 18th of November 2022, base-jobs patches mentioned in step 1 will be merged. Please ensure you have verified compatibility for your projects and landed the required changes if any were needed before this date otherwise, they might fail. Please, do not hesitate to raise any questions or concerns. [1] https://governance.openstack.org/tc/goals/selected/migrate-ci-jobs-to-ubunt… [2] https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/861116 https://review.opendev.org/c/openstack/tempest/+/861110 https://review.opendev.org/c/openstack/devstack/+/860795 [3] https://review.opendev.org/c/openstack/nova/+/861111 [4] https://etherpad.opendev.org/p/migrate-to-jammy

2 2

[kolla-ansible][Yoga] Deployment stuck
by wodel youchi 16 Nov '22

16 Nov '22

Hi, I am trying to deploy a new platform using kolla-ansible Yoga and I am trying to upgrade another platform from Xena to yoga. On both platforms the prechecks went well, but when I start the process of deployment for the first and upgrade for the second, the process gets stuck. I tried to tail -f /var/log/kolla/*/*.log but I can't get hold of the cause. In the first platform, some services get deployed, and at some point the script gets stuck, several times in the modprobe phase. In the second platform, the upgrade gets stuck on : Escalation succeeded [204/1859] <20.3.0.28> (0, b'\n{"path": "/etc/kolla/cron", "changed": false, "diff": {"before": {"path": "/etc/kolla/cro n"}, "after": {"path": "/etc/kolla/cron"}}, "uid": 0, "gid": 0, "owner": "root", "group": "root", "mode": "07 70", "state": "directory", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 70, "invocation": {"module_ args": {"path": "/etc/kolla/cron", "owner": "root", "group": "root", "mode": "0770", "recurse": false, "force ": false, "follow": true, "modification_time_format": "%Y%m%d%H%M.%S", "access_time_format": "%Y%m%d%H%M.%S", "unsafe_writes": false, "state": "directory", "_original_basename": null, "_diff_peek": null, "src": null, " modification_time": null, "access_time": null, "seuser": null, "serole": null, "selevel": null, "setype": nul l, "attributes": null}}}\n', b'') ok: [20.3.0.28] => (item={'key': 'cron', 'value': {'container_name': 'cron', 'group': 'cron', 'enabled': True , 'image': '20.3.0.34:4000/openstack.kolla/centos-source-cron:yoga', 'environment': {'DUMMY_ENVIRONMENT': 'ko lla_useless_env', 'KOLLA_LOGROTATE_SCHEDULE': 'daily'}, 'volumes': ['/etc/kolla/cron/:/var/lib/kolla/config_f iles/:ro', '/etc/localtime:/etc/localtime:ro', '', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) => { "ansible_loop_var": "item", "changed": false, "diff": { "after": { "path": "/etc/kolla/cron" }, "before": { "path": "/etc/kolla/cron" } }, "gid": 0, "group": "root", How to start debugging the situation. Regards.

1 5

Zun connector for persistent shared files system Manila
by Vaibhav 13 Nov '22

13 Nov '22

Hi, I am using zun for running containers and managing them. I deployed cinder also persistent storage. and it is working fine. I want to mount my Manila shares to be mounted on containers managed by Zun. I can see a Fuxi project and driver for this but it is discontinued now. With Cinder only one container can use the storage volume at a time. If I want to have a shared file system to be mounted on multiple containers simultaneously, it is not possible with cinder. Is there any alternative to Fuxi. is there any other mechanism to use docker Volume support for NFS as shown in the link below? https://docs.docker.com/storage/volumes/ Please advise and give a suggestion. Regards, Vaibhav

2 4

[neutron][release] Proposing to EOL Queens, Rocky and Stein (all Neutron related projects)
by Rodolfo Alonso Hernandez 11 Nov '22

11 Nov '22

Hello: In the last PTG, the Neutron team has decided [1] to move the stable branches Queens, Rocky and Stein to EOL (end-of-life) status. According to the steps to achieve this [2], we need first to announce it. That will affect all Neutron related projects. The patch to mark these branches as EOL will be pushed in one week. If you have any inconvenience, please let me know in this mail chain or in IRC (ralonsoh, #openstack-neutron channel). You can also contact any Neutron core reviewer in the IRC channel. Regards. [1]https://etherpad.opendev.org/p/neutron-antelope-ptg#L131 [2] https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-l…

1 1

[keystone][cache] How to tune role cache
by Eugen Block 08 Nov '22

08 Nov '22

Hi *, one of our customers has two almost identical clouds (Victoria), the only difference is that one of them has three control nodes (HA via pacemaker) and the other one only one control node. They use terraform to deploy lots of different k8s clusters and other stuff. In the HA cloud they noticed keystone errors when they purged a project (cleanly) and started the redeployment immediately after that. We did some tests to find out which exact keystone cache it is and it seems to be the role cache (default 600 seconds) which leads to an error in terraform, it reports that the project was not found and refers to the previous ID of the project. The same deployment seems to work in the single-control environment without these errors, it just works although the cache is enabled as well. I already tried to reduce the cache_time to 30 seconds but that doesn't help (although it takes more than 30 seconds until terraform is ready after the prechecks). But the downside of disabling the role cache entirely leads to significantly longer response times when using the dashboard or querying the APIs. Is there any way to tune the role cache in a way so we could have both a reasonable performance as well as being able to redeploy projects without a "sleep 600"? Any comments or recommendations are appreciated! Regards, Eugen

1 1

(openstack-ansible) Container installation in openstack
by Adivya Singh 08 Nov '22

08 Nov '22

Hi Team, Any input on this, to install container service in openstack using ansible. standard global parametre Regards Adivya Singh

2 5

[Kolla-ansible][Xena] How to update an Openstack deployment with new containers?
by wodel youchi 01 Nov '22

01 Nov '22

Hi, The documentation of kolla-ansible Xena does not talk about updating an existing Xena deployment with new containers. Could you please help with this? I found some lines about that in Yoga version, saying that, to update an existing deployment you have to : 1 - Update kolla-ansible it self : $ source xenavenv (xenavenv) $ pip install --upgrade git+ https://opendev.org/openstack/kolla-ansible@stable/xena 2 - Update the container Images with Docker pull 3 - Update my local registry if I am using one, and in my case I am, so I deleted the registry images then I recreated them. 4 - Then finally deploy again (xenavenv) $ kolla-ansible -i multinode deploy Is this the right procedure? because I followed the same procedure and I think it didn't change anything. For example I am taking nova-libvirt container as and example, in my local registry I have this : [root@rcdndeployer2 ~]# docker images | grep nova-libvirt 192.168.2.34:4000/openstack.kolla/centos-source-nova-libvirt xena 5be83d680102 31 hours ago 2. 34GB quay.io/openstack.kolla/centos-source-nova-libvirt xena *5be83d680102* *31 hours ago * 2. 34GB [root@rcdndeployer2 ~]# docker inspect -f '{{ .Created }}' *5be83d680102 * *2022-10-25*T02:33:13.172550584Z But in my compute nodes I have this : root@computehci24 ~]# docker ps | grep nova-lib b56a12bfd482 192.168.2.34:4000/openstack.kolla/centos-source-nova-libvirt:xena "dumb-init --single-…" *5 months ag**o Up Up 5 months (healthy) nova_libvirt* Regards.

1 1

Delivery reports about your e-mail
by mark＠stackhpc.com 01 Nov '22

01 Nov '22

The original message was received at Tue, 1 Nov 2022 07:26:06 +0900 from stackhpc.com [42.37.153.10] ----- The following addresses had permanent fatal errors ----- <openstack-discuss(a)lists.openstack.org>

1 0

[OSSN-0091] BMC emulators developed in OpenStack community do not preserve passwords on VMs
by Jay Faulkner 01 Nov '22

01 Nov '22

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 ## Summary ## When deploying VirtualBMC or Sushy-Tools in an unsupported, production-like configuration, it can remove secret data, including VNC passwords, from a libvirt domain permanently. Operators impacted by this vulnerability must reconfigure any secret data, including VNC passwords, for the libvirt domain. These virtual machine emulators are tools to help emulate a physical machine's Baseboard Management Controller (BMC) to aid in development and testing of software that would otherwise require physical machines to perform integration testing activities. They are not intended or supported for production or long-term use of any kind. ## Affected Services / Software ## * Sushy-Tools, <=0.21.0 * VirtualBMC, <=2.2.2 There is no impact to any OpenStack software or services intended for production use. ## Patches ## * VirtualBMC: https://review.opendev.org/c/openstack/virtualbmc/+/862620 * Sushy-Tools: https://review.opendev.org/c/openstack/sushy-tools/+/862625 ## Discussion ## To perform some advanced operations on Libvirt virtual machines, the underlying XML document describing the virtual machine's domain must be extracted, modified, and then updated. These specific actions are for aspects such as "setting a boot device" (VirtualBMC, Sushy-Tools), Setting a boot mode (Sushy-Tools), and setting a virtual media device (Sushy-Tools). This issue is triggered when a VM has any kind of "secure" information defined in the XML domain definition. If an operator deploys VirtualBMC or Sushy-Tools to manage one of these libvirt VMs, the first time any action is performed that requires rewriting of the XML domain definition, all secure information -- including a VNC console password, if set -- is lost and removed from the domain definition, leaving the libvirt VM's exposed to a malicious console user. ## Recommended Actions ## Operators who may have been impacted by this vulnerability should immediately remove use of VirtualBMC and/or Sushy-Tools from their production environment. Then, validate and if necessary, reconfigure passwords for VNC access or any other impacted secrets. ## Notes ## The OpenStack team will ensure documentation is updated to clearly state these software packages are intended for development/CI use only, and are not safe to run in production. ## Credits ## Julia Kreger from Red Hat ## References ## Author: Jay Faulkner, G-Research Open Source Software This OSSN: https://wiki.openstack.org/wiki/OSSN/OSSN-0090 Original Storyboard bug: https://storyboard.openstack.org/#!/story/2010382 Mailing List : [Security] tag on openstack-discuss(a)lists.openstack.org OpenStack Security Project : https://launchpad.net/~openstack-ossg CVE: CVE-2022-44020 -----BEGIN PGP SIGNATURE----- Version: FlowCrypt Email Encryption 8.3.8 Comment: Seamlessly send and receive encrypted email wsFzBAEBCgAGBQJjYDhhACEJEGt12Tm0JMbUFiEEvF1YmsGLSYuWqE+ta3XZ ObQkxtSCUw/9FEeakvlf06BWrk5Lc3TGwUKV0WiLaE4M0xjljkzg/3/580/E nhOTl/raPszlzgkGdrQTaH3Sj4AwUTdPHqqxjyK/Xb1DIm+CfS3bdbP0aLHG Y3Su4Z74unMaKbnbyDYhM1qMIzPyBruLpqiyYJGhSuzU/fu1O/LCWfSicvKK YDmAHJ9TjXuTMdWrLrkMknvJaLe0aJrNW5iqDnIh6YrUC2Pioi5h+OFKwDpn Ea+YnlAxKR7OQGRGcY3AwP1Jq87pdHZagcVThc/wnCATKT/FtaIogDkUnoMn qI+6MNjV3R4kyQCbyo35KeIDWm+541XsK0GoR5hcvR1AkwciSPBAkt3VHxpa p0g9hVcNTv+tWwN8LrdLRPMDuqKA51eNUvQCV8W+H42wS0uoaMPXglbZIuwv AmEoK8UC8Gii8cPoIkiZGSSOo4i+tlE/q+L/Mgs1opyt1Klcxs/Lm1PNylET XqLw70qKrfqWabZpKUxMS3F9JwyCkgnD5+t2x/qsqg5Hq+kUZqP8be3Oc7K7 He/gIneWDMpH1+J9Tm5ofyxtJCA+V96+cXoXYk8SncVf/O5djgd48UkQo1iJ NZlKxJsaKH5+JyPuXkR6hyqDrIkmbJRh4aU9nJBQyFho0fXuQVlC2iUOGa0F BgUFs5J6oQtglAAuyUoNuhuBJBwdW09NxQ4= =6yXE -----END PGP SIGNATURE-----

1 1

[oslo][tooz][openstack-ansible] Discussion about coordination (tooz), too many backend options, their state and deployment implications
by Christian Rohmann 31 Oct '22

31 Oct '22

Hallo openstack-discuss, apologies for this being quite a long message - I tried my best to collect my thoughts on the matter. 1) The role of deployment tooling in fulfilling the requirement for a coordination backend I honestly write this, triggered by openstack-ansible plans to add coordination via the Zookeeper backend (https://lists.openstack.org/pipermail/openstack-discuss/2022-October/031013…) On 27/10/2022 13:10, Dmitriy Rabotyagov wrote: > * Add Zookepeer cluster deployment as coordination service. > Coordination is required if you want to have active/active > cinder-volume setup and also actively used by other projects, like > Octavia or Designate. Zookeeper will be deployed in a separate set of > containers for LXC path First of all I believe it's essential for any OpenStack deployment tooling to handle the deployment of a coordination backend as many OS projects just rely in their design and code to have it in place. But I am convinced though there too many options, that some stronger guidance should be given to people designing and then deploying OS for their platform. This guidance certainly can be in the form of a comparison table - but when it comes to using deployment tooling like openstack-ansible, the provided "default" component or options for something might just be worth more than written text explaining all of the possible approaches. This hold especially true to me as you can get quite far with no coordination configured which then results in frustration and invalid bugs being raised. And it's not just openstack-ansible thinking about coordination deployment / configurations. To just point to a few: * Kolla-Ansible: https://lists.openstack.org/pipermail/openstack-discuss/2020-November/01883… * Charms: https://bugs.launchpad.net/charm-designate/+bug/1759597 * Puppet: https://review.opendev.org/c/openstack/puppet-oslo/+/791628/ * ... 2) Choosing the "right" backend driver I've recently been looking into the question what would be the "best" tooz driver to cover all coordination use cases the various OS projects require. Yes, the dependencies and use of coordination within the OS projects (cinder, designate, gnocchi, ...) are very different. I don't want to sound unfair, but most don't communicate which of the Tooz services they actually require. In any case, I might just like something to cover all possible requirements to "set it (up) and forget it, no matter what OS projects run on the platform. Apart from basic compatibility, there are qualities I would expect (in no particular order) from a coordination backend: * no "best-effort" coordination, but allowing for actual reliance on it (CP if talking CAP) * HA - this needs to be working just as reliably as my database as otherwise the cloud cannot function * efficient in getting the job done (e.g. support for events / watches to reduce latency) * lightweight (deployment), no additional components, readily packaged * very little maintenance operations * easy monitoring I started by reading into the tooz drivers (https://docs.openstack.org/tooz/latest/user/drivers.html) of which there are more than enough to require some research. Here are my rough thoughts: a) I ruled out the IPC, file or RDBMs (mysql, postgresql) backend options as they all have strong side-notes (issues when doing replication or no HA at all). Additionally they usually are not partition tolerant or support watches. b) Redis seems quite capable, but there are many side notes about HA and this also requires setting up and maintaining sentinel. c) Memcached supports all three services (locks, groups, leader-election) tooz provides and is usually already part of an OpenStack infrastructure. So looked appealing. But it's non-replicating architecture and lack of any strong consistency guarantees make it less of a good "standard". I was even wondering how tooz would try it's best to work with multiple memcached nodes (https://bugs.launchpad.net/python-tooz/+bug/1970659) d) Then there only is Zookeeper left, which also ticks all the (feature-)boxes (https://docs.openstack.org/tooz/latest/user/compatibility.html) and is quite a proven tool for coordination also outside of the OpenStack ecosystem. On the downside it's not really that well known and common (anymore) outside the "data processing" context (see https://cwiki.apache.org/confluence/display/ZOOKEEPER/PoweredBy) Being a Java application it requires a JVM and its dependencies and is quite memory heavy to store just a few megabytes of config data. Looking at more and more people putting their OS control plane into something like Kubernetes it also seems even less suitable to be "moved around" a lot. Another issue might be the lack of a recent and non-EoL version packaged in Ubuntu - see https://bugs.launchpad.net/ubuntu/+source/zookeeper/+bug/1854331. Maybe (!) this could be an indication of how commonly it is used outside of e.g. Support from TLS was only added in 3.5.5 (https://zookeeper.apache.org/doc/r3.5.5/zookeeperAdmin.html#Quorum+TLS) e) Consul - While also well known and loved, it has, like Zookeeper, quite a big footprint and is way more than just a CP-focused database. It's more of an application with man use cases. f) A last "strong" candidate is etcd. It did not surprise me to see it on the list of possible drivers and certainly is a tool known to many from running e.g. Kubernetes. It's actually already part of openstack-ansible deployment code as a role (https://github.com/openstack/openstack-ansible/commit/2f240dd485b123763442a…) as it is required when using Calico as SDN. While etcd is also something one must know how to monitor and operate, I allow me to say it might just be more common to find this operational knowledge. Also etcd has a smaller footprint than Zookeeper and it beeing "just a Golang binary" comes with (no) less dependencies. But I noticed that it does not even support "grouping", according to the feature matrix. But apparently this is just a documentation delay, seehttps://bugs.launchpad.net/python-tooz/+bug/1995125. What's left to implement would be leader-election, but there seems to be no technical reason why this cannot be done. this by no means is a comparison with a clear winner. I just want to stress how confusing having lots of options with no real guidance are. The requirement to chose and deploy coordination might not be a focus when looking into designing an OS cloud. 3) Stronger guidance / "common default", setup via OS deployment tooling and also used for DevStack and tested via CI To summarize, there are just too many options and implications in the compatibility list to quickly chose the "right" one for one's own deployment. While large-scale deployments might likely not mind for coordination to have a bigger footprint and requiring more attention in general. But for smaller and even mid-size deployments, it's just convenient to offload the configuration of coordination and the selection the backend driver to the deployment tooling. Making it way too easy for such installations to not use coordination and running into issues or every other installation using a different backend creates a very fragmented landscape. Add different operating system distributions and versions, different deployment tooling, different set and versions of OS projects used, there will be so many combinations. This will likely just cause OS projects to receive more and non-reproducible bugs. Also not having (a somewhat common) coordination (backend) used within CI and DevStack does not expose the relevant code paths to enough testing. I'd like to make the analogy to having "just" MySQL as the default database engine, while still allowing other engines to be used (https://governance.openstack.org/tc/resolutions/20170613-postgresql-status.…) Or labeling certain options as "experimental" as Neutron just did with "linuxbridge" (https://docs.openstack.org/neutron/latest//admin/config-experimental-framew…) or cinder with naming drivers unsupported (https://docs.openstack.org/cinder/ussuri/drivers-all-about.html#unsupported…) My point is that just having all those backends and no active guidance might make Tooz a very open and flexible component. I myself would wish for some less confusion around this topic and having a little less to think about this myself. Maybe the "selection" of Zookeeper by openstack-ansible is just that? I would love to hear your thoughts on coordination and why and how you ended up with using what. And certainly what your opinion on the matter of a stronger communicated "default" is. Thanks for your time and thoughts! Christian

3 3

[neutron] Bug Deputy Report October 24-30
by Nurmatov Mamatisa 31 Oct '22

31 Oct '22

Hi, Below is the week summary of bug deputy report for last week. One RFE was proposed and already discussed on drivers meeting, more information should be provided. Details: Critical -------- - https://bugs.launchpad.net/neutron/+bug/1995091 - [CI] "neutron-functional-with-oslo-master" failing with timeout - Confirmed - Unassigned - https://bugs.launchpad.net/neutron/+bug/1994491 - Functional tests job is failing on Ubuntu 22.04 - Incomplete - Assigned to Rodolfo Alonso Medium ------ - https://bugs.launchpad.net/neutron/+bug/1995031 - [CI][periodic] neutron-functional-with-uwsgi-fips job failing - Confirmed - Unassigned - https://bugs.launchpad.net/neutron/+bug/1994635 - [CI][tempest] Error in "test_multiple_create_with_reservation_return" - Confirmed - Assigned to Rodolfo Alonso Undecided --------- - https://bugs.launchpad.net/neutron/+bug/1995078 - OVN: HA chassis group priority is different than gateway chassis priority - New - Unassigned RFEs ---- - https://bugs.launchpad.net/neutron/+bug/1994137 - [RFE] Specify the precedence of port routes if multiple ports attached to a VM - Incomplete - Unassigned Best regards, Mamatisa Nurmatov Advanced Software Technology Lab / Cloud Technologies Research

1 0

[all][tc] What's happening in Technical Committee: summary 2022 Oct 28: Reading: 5 min
by Ghanshyam Mann 29 Oct '22

29 Oct '22

Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * We had this week's meeting on Oct 27. Most of the meeting discussions are summarized in this email. Meeting logs are available @ https://meetings.opendev.org/meetings/tc/2022/tc.2022-10-27-15.00.log.html * Next TC weekly meeting will be on Nov 3 Thursday at 15:00 UTC, feel free to add the topic to the agenda[1] by Nov 2. 2. What we completed this week: ========================= * 2021 User Survey TC Question Analysis[2] * Appointed hongbin as Zun PTL[3] 3. Activities In progress: ================== TC Tracker for 2023.1 cycle --------------------------------- * I have prepared the 2023.1 tracker etherpad which includes the TC working items for the 2023.1 cycle[4]. Open Reviews ----------------- * Four open reviews for ongoing activities[5]. 2023.1 cycle TC PTG Summary ------------------------------------- I sent the TC sessions and TC+Leader discussion summary on ML[7][8]. User Survey: --------------- TC worked on the modification of TC questions for the 2023 survey[9]. TC chair nomination & election process ----------------------------------------------- We are formalizing the process of TC chair nomination process. Two options are up for the review[10][11]. Fixing Zuul config error ---------------------------- We request projects having zuul config error to fix them, Keep supported stable branches as a priority and Extended maintenance stable branch as low priority[12]. Project updates ------------------- * None. 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[13]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15:00 UTC [14] 3. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. See you all next week in PTG! [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://review.opendev.org/c/openstack/governance/+/836888 [3] https://review.opendev.org/c/openstack/governance/+/860759 [4] https://etherpad.opendev.org/p/tc-zed-tracker [5] https://review.opendev.org/q/projects:openstack/governance+status:open [7] https://lists.openstack.org/pipermail/openstack-discuss/2022-October/030954… [8] https://lists.openstack.org/pipermail/openstack-discuss/2022-October/030953… [9] https://etherpad.opendev.org/p/tc-2023-user-survey-questions [10] https://review.opendev.org/c/openstack/governance/+/862772 [11] https://review.opendev.org/c/openstack/governance/+/862774 [12] https://etherpad.opendev.org/p/zuul-config-error-openstack [13] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [14] http://eavesdrop.openstack.org/#Technical_Committee_Meeting -gmann

1 0

[tacker][ptg] Antelope PTG Summary
by Yasufumi Ogawa 28 Oct '22

28 Oct '22

Hi all, We had great PTG sessions in Tacker team, and hopefully all. This is a summary of what we discussed during our sessions in three days. All of items and the results of discussion is here [1]. Day 1 (18 Oct 2022) 1. Reducing resources for FTs - We shared suggestions for fixing critical issues in functional tests in which we've had some unreasonable failures, especially at the end of releases. - Removing/deprecating unecessary legacy features and its tests. - Quick analysis for the failures [1]. - Wishlist for the item [2]. - Agree to focus on jobs has the large num of failures to revise tests. 2. Support tacker-db-manage for Multi DB backend - Support for PostgreSQL doesn't work, for insatnce, some issues in tacker-db-manage. - We agreed to not only fix the issues, but also take care other backends. - Revise tacker-db-manage for supporting multi DB backend. Start to support Postgres in antelope first, and other ones in the next releases then. - Proposed design is not so mature for now, so continue to discuss to revise. 3. Migrate OpenStack testing from Ubuntu Focal (20.04) to Ubuntu Jammy (22.04) - We agreed to migrate Focal to Jammy Ubuntu release as one of the community goal in this release [4]. - Candidate patches for the update have already proposed by manpreetk not only for tackder but also other releated projs. - https://review.opendev.org/c/openstack/tacker/+/861137 - https://review.opendev.org/c/openstack/tacker-horizon/+/861571 - https://review.opendev.org/c/openstack/python-tackerclient/+/861572 - https://review.opendev.org/c/openstack/tosca-parser/+/861136 - https://review.opendev.org/c/openstack/heat-translator/+/861158 https://etherpad.opendev.org/p/migrate-to-jammy 4. Bug Triage in Tacker - Add item of bug triage in IRC etherpad to discuss about bug triage. Day 2 (19 Oct 2022) 1. Operator Hour - Guys from KDDI and NTT Docomo joined and proposed their proposals as telcom operators [5]. 1.1. Support of redanduncy - For availability, essential to make Tacker redundant for commercial use, but less than we expected for the current impl. - What is the tacker specific consideration point is store VNF packages. 1.2. Error handling and recovering operation - More tests for changing error status, disaster recovery are required. 1.3. Detailed logs - Not enough log messages especially for management drivers. - Error message in tacker-server.log and tacker-conductor.log is usefull and enough for vnfm-admin, but like openstack vnflcm op show comannd and api's information is not enough for vnfm-user. 1.4. Roadmap - Having explicit roadmap is helpful for oprators to be interested. - Comply with ETSI NFV Rel4 is one of the good example for the direction. 2. Add sample coordinate VNF script using coordination API - We agreed to implement interfaces ETSI NFV SOL002 v3.5.1 defines the Coordination VNF interface for coordination with external management components. 3. Revise automatic operation state transition for N-Act configuration - v2API has a function to automatically transition opocc in an intermediate state to the final state when the tacker-conductor starts up, to recover inoperable opocc in the PROCESSING state by conductor down. - It causes a problem in N-Act setting. The opocc which are being handled by other working conductors are also automatically transitioned to the final state. We propose solutions for limiting opocc to be transitioned to those handled by the downed conductor. 4. Set different number of instances for same delta and aspectId - Fix overwriting the value for same aspect and delta id. - Multi VDUs Scale of V1 API isn’t supported by the issue. 5. Continue to update the remaining patch from Zed version - Move forward reviews on gerrit. Day 3 (20 Oct 2022) 1. Enhancement of Tacker API Resource Access Policy - Current Tacker policy control access to API resources by default role such as admin or any only. - We propose Fine-grained API resource access management based on user and VNF information according to operator usecases. 2. Secure RBAC: Implement support for project-reader persona in Tacker - As per the TC and community wide goal, next cycle 2023.1 is the must for all projects to implement the phase-1 (project personas). Plan to do it in Tacker in 2023.1 cycle will be helpful in community wide goal [6][7]. - Conclusions: Clarify conflicts can be caused while introducing new roles of S-RBAC to implement the fine grained APIs 3. Support Tacker auto-scale and auto-heal without NFVO (k_fukaya) - Zed release supported, Fault Management/ Performance Management(FM/PM) interfaces, and AutoHeal and AutoScale with external monitoring. However, Heal or Scale execution must be triggered by NFVO. This feature proposes implementing support receiving alerts from external monitoring tools, which can be VNFM driven AutoHeal and AutoScale without NFVO. 4. Discuss Tacker support versions and updates for K8s/Helm/Prometheus - The support versions in Antelope should be determined, as some versions may be inconsistent and support may expire before release. Considering the support period and development risk, it seems good to decide on the following. - k8s : 1.25 (current latest version) - helm : 3.10 (current latest version) - prometheus : 2.42 (next LTS version) - Update test patch for k8s 1.25.2 and helm 3.10.1 - https://review.opendev.org/c/openstack/tacker/+/860633 (Zuul +1) - Add version information under user guide. 5. AWS vim support - Add support for EC2 as VIM for the first step for AWS. - Whole discussion is here [8]. 6. Marking Deprecated and obsoleting of Legacy API - Should start to discussion about deprecation and obsoleting Legacy Tacker API (excluding VIM feature). - Deprecation process should follow as "Deprecation Guidelines" [9]. - The key point is - APIs should be marked deprecated before obsoleting and should be marked for at least 12 months. - An email thread will be started on openstack-discuss to determine how many people are using the deprecated API. - We will continue our discussion and move towards removing the Legacy API (excluding VIM feature). [1] https://etherpad.opendev.org/p/tacker-antelope-ptg [2] https://etherpad.opendev.org/p/tacker-antelope-failures-analysis [3] https://bugs.launchpad.net/tacker/+bug/1993187 [4] https://governance.openstack.org/tc/goals/selected/migrate-ci-jobs-to-ubunt… [5] https://etherpad.opendev.org/p/oct2022-ptg-operator-hour-tacker [6] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… [7] https://lists.openstack.org/pipermail/openstack-discuss/2022-October/030863… [8] https://etherpad.opendev.org/p/tacker-antelope-aws-vim-support [9] https://docs.openstack.org/project-team-guide/deprecation.html#guidelines

1 0

[horizon] missing reviews for cinder backup filtering
by Mohammed Naser 28 Oct '22

28 Oct '22

Hi folks, We're nearing a very long time that this patch has been open, it's undergone several revisions from our team and sitting idle almost 2 months. https://review.opendev.org/c/openstack/horizon/+/791532/ Can we please help get this landed, we're trying our best to stick to our upstream only :) Thanks Mohammed -- Mohammed Naser VEXXHOST, Inc.

1 0

[Ceilometer] RADOS GW metrics : cannot get radosgw.objects.size metric
by Taltavull Jean-François 28 Oct '22

28 Oct '22

Hello, I'm trying to get the 'radosgw.objects.size' metric, that is the total bucket objects size per tenant. I expected to get one sample per tenant but I get one sample per bucket instead, as with the 'rados.containers.objects.size' metric. Here is my pollster definition: ''' - name: "radosgw.objects.size" sample_type: "gauge" unit: "B" value_attribute: ". | value['usage'] | value.get('rgw.main',{'size':0}) | value['size']" url_path: "FQDN/admin/bucket?stats=True" module: "awsauth" authentication_object: "S3Auth" authentication_parameters: my_access_key,my_secret_key,FQDN user_id_attribute: "owner | value.split('$') | value[0]" project_id_attribute: "tenant" resource_id_attribute: "id" ''' I tried with "resource_id_attribute: "tenant" but it does not work better. Any idea ? Is there something wrong in the pollster definition ? Regards, Jean-Francois

2 5

Outreachy Mentorship Programme
by Victor Ekwueme 28 Oct '22

28 Oct '22

Hello, I am a programmer who is visually impaired. I set up devstack on a Digital Ocean droplet on Ubuntu 20.04 Server. It installed successfully. I am trying to make a contribution on 1599140. So my questions are: 1. How do I set up a development environment for this or any other contribution? 2. Do I work directly on the droplet and push from to the repo from there? Any assistance will be of tremendous help. Regards, Victor O. Ekwueme

2 1

[neutron] CI meeting Nov 1st cancelled
by Slawek Kaplonski 28 Oct '22

28 Oct '22

Hi, As we discussed on IRC, Nov 1st is public holiday for me and many other folks who usually attend Neutron CI meeting so meeting next week is cancelled. See You on the CI meeting on Nov 8th. -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

Help us plan the next PTG!
by Kendall Nelson 28 Oct '22

28 Oct '22

Hello Everyone! Congratulations on a great virtual PTG last week :) The OpenInfra Foundation is hosting the OpenInfra Summit in Vancouver [1], June 13 -15, 2023. We are trying to determine the level of interest among contributors to OpenInfra projects on attending a co-located PTG in Vancouver. *The exact format and dates are still being determined.* At this time, we are evaluating the level of interest from an attendee and employer perspective. Please complete the following poll so we can measure the level of interest and plan accordingly. Future updates will be distributed to the project mailing lists as well as previous PTG attendees. Poll: https://openinfrafoundation.formstack.com/forms/openinfra_ptg_2023 As a reminder, we are also gathering feedback for the virtual PTG here: https://etherpad.opendev.org/p/Oct2022_PTGFeedback -Kendall Nelson (diablo_rojo) [1] https://openinfra.dev/summit/

6 8

network monitor
by Parsa Aminian 28 Oct '22

28 Oct '22

hello Im using openstack kolla-ansible . Could you please tell me how I can monitor instances' network usage ? 1-download and upload speed 2-traffic usage for example send and receive per month for each instance

3 3

[neutron] Neutron drivers meeting Oct 28th
by Rodolfo Alonso Hernandez 27 Oct '22

27 Oct '22

Hello Neutrinos: The drivers meeting will take place tomorrow at 14UTC. The agenda has one single topic: * https://bugs.launchpad.net/neutron/+bug/1994137: [RFE] Specify the precedence of port routes if multiple ports attached to a VM Regards.

1 0

[openstack-ansible][ptg] Antelope PTG results
by Dmitriy Rabotyagov 27 Oct '22

27 Oct '22

Hi everyone! As you might know, the OpenStack-Ansible team had Project Team Gathering on October 18, 2022 and we had quite good discussions. This email is a relatively short sum-up of decisions that were taken during it: * Add Zookepeer cluster deployment as coordination service. Coordination is required if you want to have active/active cinder-volume setup and also actively used by other projects, like Octavia or Designate. Zookeeper will be deployed in a separate set of containers for LXC path * Move out Ubuntu Jammy from experimental state except Ceph integration. This is because community repo does not have ceph packages built for Ubuntu 22.04, so we will switch default to use Ubuntu's native repository to get ceph from. With that you won't be able to pick what ceph version to use, but it will work for most usecases. * Continue supporting Ubuntu Focal (20.04) until Antelope release as upgrade path from Yoga to Antelope should exist and on TC PTG it was decided that it should be tested by projects on Ubuntu 20.04. * Update docs regarding level of support for integrated ceph-ansible deployment with openstack-ansible to address expectations on upgrade path. * Address concerns regarding Glance configuration with COW backends, with regards to OSSN notice: https://wiki.openstack.org/wiki/OSSN/OSSN-0090 * Switch distro path to experimental due to lack of maintainers. We should document that and mention that help is needed to keep it alive. If not - the feature can be removed in the future. * Cover OVN with TLS encryption. Leverage PKI role for that. * We also agreed to switch the default network driver used by OpenStack-Ansible to OVN as default. However, at the moment there is not much hands-on experience on OVN from the core team, so we can not switch default right now as we won't be efficiently helping with OVN deployments. So following switching plan was defined: ** Document reference OVN architecture that we are going to provide ** Document implementation of extra networks that are required for octavia, trove, etc. ** Switch AIO to OVN as default ** Add LXB jobs to neutron role ** Switch default for non-AIO ** Ensure that users who were relying on previous default (and using LXB) have proper override during upgrade. * Improve documentation for the Ironic role with adding a couple of scenarios. * Convert our dynamic inventory to an inventory plugin which will be installed during ansible bootstrap. * Create a repository for the skyline role. We are allowed to take https://github.com/jrosser/openstack-ansible-os_skyline as base and move it under openstack-ansible umbrella * Simplify documentation regarding provider_networks in openstack_user_config and suggest using neutron_provider_networks instead. * Continue working on improving our CI coverage, including returning to molecule testing but rely on the ansible and constraints version from the integrated repo. At the same time finish cleanup of old functional tests, including run_tests.sh, tox, etc. * Deprecate rsyslog_server/client roles as they are hardly used as of today. You can also check etherpad [1] where notes were taken during PTG and that we aim to update with progress on these points implementation: [1] https://etherpad.opendev.org/p/osa-antelope-ptg

1 0

[all][tc] Technical Committee next weekly meeting on 2022 Oct 27 at 1500 UTC
by Ghanshyam Mann 27 Oct '22

27 Oct '22

Hello Everyone, The technical Committee's next weekly meeting is scheduled for 2022 Oct 27, at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Oct 26 at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[cinder] Problem with restoring purestorage volume backup
by Ришат Азизов 26 Oct '22

26 Oct '22

Hello! We have a problem with cinder-backup with cephbackup driver and volume on purestorage. When backing up purestorage volume with cinder-backup with cephbackup driver it creates in ceph pool and everything is ok. But when we try to restore the backup, it is not restored with an error "rbd.ImageNotFound" in the screenshot attached to this email. This happens because the original image is not in rbd, it is in purestorage. It is not clear why the cinder is trying to look for a disk in the ceph. Could you please help with this? Thanks. Regards.

2 1

[cinder] Bug report from 10-12-2022 to 10-26-2022.
by Sofia Enriquez 26 Oct '22

26 Oct '22

This is a bug report from 10-12-2022 to 10-26-2022. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Undecided - https://bugs.launchpad.net/cinder/+bug/1994018 "The volume multiattach and in-use after retyping another backend, then can not detach it." - https://bugs.launchpad.net/cinder/+bug/1994021 "Cinder cannot work when 1 node of 3 rabbit node cluster down." Low - https://bugs.launchpad.net/cinder/+bug/1993282 "Fail to get pools by volume_type filter key." Assigned to zhaoleilc. No fix proposed to master yet. - https://bugs.launchpad.net/os-brick/+bug/1994083 "FileNotFoundError: [Errno 2] No such file or directory: is raised when dmidecode is not installed." Fix proposed to os-brick master. Wishlist - https://bugs.launchpad.net/cinder/+bug/1994150 "Image signature verification does not verify certificates." Unassigned. - https://bugs.launchpad.net/cinder/+bug/1992493 "Cinder fails to backup/snapshot/clone/extend volumes when the pool is full." Unassigned. - https://bugs.launchpad.net/cinder/+bug/1992685 "Automate generation of snapshot transfer api-ref samples for MV 3.55." Assigned. No fix proposed to master yet. Invalid - https://bugs.launchpad.net/cinder/+bug/1993577 " [JovianDSS] Unable to provide target prefix through iscsi_target_prefix ." Cheers, -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[neutron] Neutron team meeting cancelled Nov 1st
by Rodolfo Alonso Hernandez 26 Oct '22

26 Oct '22

Hello Neutrinos: The next Neutron team meeting on November 1st is cancelled (All Saints' Day [1]). The next one will take place at the same time on November 8th. Regards. [1]https://en.wikipedia.org/wiki/All_Saints%27_Day

1 0

[manila] Antelope PTG Summary
by Carlos Silva 25 Oct '22

25 Oct '22

Hello, Zorillas and interested stackers. Thank you for the productive PTG we had last week. Here goes the summary for the past week's discussions. In case you would like to see the expanded versions, please refer to [0] or check out the recordings in the Manila YouTube Channel [11]. Different approaches for the images we use in our jobs - Recent changes in the Manila CI (related to the service image) made the jobs take longer to run and become more resource intensive, causing jobs to fail more often due to lack of resources in dsvm - We have been looking for approaches to tackle this issue - In the short term, if a job keeps failing due to such issues, we will split it into two jobs - One to run scenario tests (these tests spawn VMs) - One to run API tests - We agreed that this can get worse in the future with dsvm images that could be even more resource demanding, and that we will look for containerized approaches to try and solve this issue Share backup - (kpdev) The proposed approach for share backup has changed - An existing specification [1] was restored - The idea is to have new APIs for share backup and a new generic Backup driver in a similar way to what Cinder does. - This would allow backends to have their specific implementations for share backups - Reviews for this specification are ongoing Secure RBAC changes - Goutham has shared with the community all the good progress we had in Manila, and also mentioned what we did in Zed, after all the operator's feedback. - Now, we are entering a phase where we want to test a lot! - We have a CI jobs and some test cases covered, but we still want to have more coverage - During Antelope, we will promote a hackathon in order to accelerate the functional tests coverage Outreachy Internships - On this topic, ashrodri, fmount, gouthamr and carloss shared the two proposed Outreachy project that are Manila related - ashrodri and fmount are proposing a project to have an intern creating a multi-node test environment with Devstack and Ceph [2], as these tests are becoming more resource demanding - carloss and gouthamr are proposing a project for an intern to work in Manila UI [3]. The idea is to close down the gap of features between Manila UI and the Manila API. OpenStackSDK Manila Status - This has been an effort for some time, and during the Zed cycle some code was merged for integrating share groups and share group snapshots. - This is a good candidate project for an internship, and the idea is that we will be able to get some interns working on this project (potentially a project for a college capstone project) OSC Updates - Zed was the release we targeted to reach feature parity with the native client, and we made it! - We had an idea to add a deprecation warning to the native manila client, but that will need to wait - There is still a missing bit for adding the deprecation warning: having version autonegotiation working, which is something we are targeting for Antelope - OSC is now the primary focus of the project when doing implementations for CLIs. Bandit Testing / VMT - The Manila team was introduced to the VMT (vulnerability management team) and Bandit, and we had contributors sharing their ideas around having manila under the VMT and running bandit tests. - VMT - The audience agreed with the conditions to be under the overseen repos. - Goutham has volunteered to be the security liaison for Manila - Bandit - There were some errors pointed out in a preliminar test run [4]. - We will add a bandit non-voting job and file bugs against third party drivers that have issues pointed out by the new job - New bugs will be filed and will be distributed across community members All Things CephFS - Ingress aware NFS Ganesha - Ceph Orchestrator is now capable of creating an ingress service to front-end a cluster of active-active NFS Ganesha instances via a single load-balanced IP address - Previously, we would only use one single instance of NFS Ganesha. This introduces a SPOF and is not suitable for production environments without HA being handled externally (like TripleO deployments do). With this change, Manila CephFS NFS users would be able to deploy multiple NFS Ganesha servers and leverage inbuilt HA capabilities - Currently though manila client restrictions (access rules) will not work since NFS-Ganesha sees the ingress service's IP instead of the client's IP address. - ffilz proposed his design to support the PROXY protocol with NFS-Ganesha. Ceph Ingress can pass the Client's IP address to NFS-Ganesha [5] - An alternative would be to have cephadm assign stable IP addresses to ganesha instances - No driver changes are anticipated in either approach - AI: Investigating deploying ceph-ingress with devstack - Migration tool for Manila with CephFS NFS - After the new helper was introduced to the driver, we are able to interact with cephadm-deployed CephNFS service (which comes with a lot of benefits) - Now, we need to figure out how to upgrade deployments with CephFS NFS using the current helper (which interacts with NFS Ganesha through DBUS) - There are two main issues: - Representing a change in exports when migrating across NFS Servers (current use case) or when decommissioning NFS Servers - Representing exports from multiple servers - a special case enabling migration where an old server will eventually be decommissioned: - These issues were talked through and possible solutions were proposed and will be worked on in the following cycles. More details on [6] - DHSS=True for CephFS driver - Currently, CephFS driver, whether you'd like to use it for Native CephFS, NFS via DBUS, NFS via ceph-mgr only supports DHSS=False. The request of this feature was raised during the OpenInfra Summit in Berlin from several Manila users. - We discussed some alternatives for making it happen: - 1) Operator determines Ceph cluster limits, and creates isolated file systems and declares multiple manila backends with each filesystem - 2) Subvolume groups - pinning a subvolume group to mds can isolate/dedicate mds load - CephFS CI issues - The lack of resources in the dsvms is affecting the CephFS Native and NFS jobs, causing them to be unstable. The jobs often run into timeouts - The situation could get worse as Jammy Jellyfish packages are not available yet - We will ask the ceph-users if jammy release bits can be made available Oversubscription enhancements - Storage pools supporting thin provisioning are open to oversubscription. This caused some problems mentioned in [7]. - We have an open specification which we intend to merge in the Antelope cycle, as well as the changes to address such issue FIPS - We have shared our testing status with regards to FIPS. We have jobs merged on stable branches up to wallaby for both manila and python-manilaclient repositories. - The next steps would be a more in-depth code audit to identify non-compliant libraries and making our jobs voting - We agreed to make our jobs voting when the Ubuntu images supporting FIPS are out - For drivers using non FIPS compliant libraries, we will notify the maintainers Metadata API update - Metadata APIs for share snapshots were added during the Zed cycle - The goal for this cycle is to have functional testing and the CLI merged. Both patches are under review and expected to be merged not late in the release. Manila CSI - The CSI plugin's been pretty stable in the last six months [8] - There was a good talk presented at KubeCon by Robert Vasek [9] - The next steps include getting a fix for an issue involving long snapshot names in the CephFS backend and supporting volume expansion in the OpenShift Manila CSI driver operator Manila Configuration of VLAN Network information - A issue was found where Contrail Neutron Plugin did not return the VLAN ID during port allocation [10] - To tackle this issue, we have agreed to add metadata to the share network APIs so the administrators would be able to add VLAN as metadata and drivers would be able to consume it. *Better use of bug statuses* - Our bugs were stuck in "New" instead of "Confirmed" or "Triaged" and this could be misleading. - We agreed to tag the bugs as confirmed or triaged depending on the outcome of our triaging and not leave bugs as "new" - We will ask the bug assignees to update their open bugs in case one of them has the new status, so we can have a better visibility [0] https://etherpad.opendev.org/p/antelope-ptg-manila [1] https://review.opendev.org/c/openstack/manila-specs/+/330306 [2] https://www.outreachy.org/outreachy-december-2022-internship-round/communit… [3] https://www.outreachy.org/outreachy-december-2022-internship-round/communit… [4] https://gist.github.com/gouthampacha/c0d96966670956761b2b620be730efe2 [5] https://docs.google.com/document/d/1orjNjtEeeyRrgvbQuFU5BdsCnZG9gF3MymDCZl-… [6] https://etherpad.opendev.org/p/antelope-ptg-manila-cephfs [7] https://etherpad.opendev.org/p/manila_oversubscription_enhancements [8] https://github.com/kubernetes/cloud-provider-openstack/commits/master/pkg/c… [9] https://www.youtube.com/watch?v=XfpP9pBTXfY&t=1145s [10] https://bugs.launchpad.net/charm-manila/+bug/1987315 [11] https://www.youtube.com/playlist?list=PLnpzT0InFrqBzbSP6lcYDStKpbEl3GNHK Thanks, carloss

1 0

[Glance] Antelope PTG Summary
by Pranali Deore 25 Oct '22

25 Oct '22

Hello Everyone, We had our sixth virtual PTG between 18th OCt to 21st OCT 2022. Thanks to everyone who joined the virtual PTG sessions. Using bluejeans app we had all the discussion around different topics for glance, glance + cinder, glance + ceph, fips and secure RBAC. I have created etherpad[1] with notes from the session and also included the recordings of each session. Here is the short summary of the discussions. Tuesday, OCT 18th 2022 # Zed Retrospective On the positive note, we have merged a number of useful features this cycle. We managed to implement glance-download internal plugin to download the image from remote glance, Implemented support for immediate caching of an image, Extended the functionality of stores-detail API to expose store details of other stores and we have removed the scope scheck from scope_types for all project resources and done with phase 1 as per the revised community goal. In addition to that we have successfully organized the review party before each milestone to perform group review to cover the review load till the final milestone. On the other side we decided to organize a midcycle general cross project meetup/drivers meetup towards the end of the 2nd milestone to increase our presence in cross-projects. Recording: https://bluejeans.com/s/II_@CAqrZdd - Chapter 1 # Distributed responsibilities among cores/team This cycle also we have decided to follow the distributed leadership internally, we are going to distribute below responsibilities among the team, Release management:pranali Bug management: cyril Meetings: pranali Stable branch management: cyril and Erno Cross project communication: abhishekk Mailing lists: share responsibility, pranali PTG/summit preparation:pranali Vulnerability management: glance-coresec group Infra management: periodic-jobs - abhishekk migration of test jobs -abhishekk Recording: https://bluejeans.com/s/II_@CAqrZdd - Chapter 2 # Default Glance to configure multiple stores Glance has deprecated single store configuration since Stein Cycle and this cycle we are going to start putting our efforts to deploy glance using multistore by default and then remove the single store support from glance. Recording: https://bluejeans.com/s/II_@CAqrZdd - Chapter 3 # Add missing CLI support for some Glance API in Openstack Client The CLI support for all glance APIS is already there in GlanceClient and the similar CLI support we need to have in OpenstackClient, in this Cycle we are going to put efforts to have OSC support for all the missing glance APIs. Recording: https://bluejeans.com/s/II_@CAqrZdd - Chapter 4 # Glance Cache improvements, restrict duplicate downloads This is abou how we can avoid multiple downloading of the same image in cache on first download Spec : https://review.opendev.org/c/openstack/glance-specs/+/734683 We had this spec in the last cycle & decided to break the image into chunks and when the first request gets to the backend store it will start caching that image and if any other request comes in between and if caching is still in process it will read from the chunks created by the first request. But currently we have made one caching state to check if the image is still in caching but that was relying on md checksum to check if the image iterator has read the image completely or not but in new images we might not have checksum, multihash or size of the image, bcz if that is not present with the image we won't be able to change the state of image and thus we will never be able to resolve the issue of checking the image is in caching or not. Decided to dig more on the size verification & need to revisit this topic during mid-cycle meeting and update the spec with the solution for handling the multiple request and multiple chunks Recording: https://bluejeans.com/s/II_@CAqrZdd - Chapter 5 Wednesday OCT 19th , 2022 # Image uploads to the filesystem driver are not fully atomic No efficient way to reproduce this issue, so it's decided to mark it as 'Won't fix' Recording: https://bluejeans.com/s/j_UgZFw_jEV - Chapter 1 # DB migration constant change handling Till now we have all the migration scripts by the name of the cycle and since currently release has been change to 2023.1 which is going to break the migration test because when our DB sync tool runs it will check the initial version liberty and it finds the migration script from the liberty and traverse through all the directories till the current release and executes all the scripts available in that path. Decided to fix this by updating the data migration current release to '2023.1' and check with actual migration script to check whether there is any regression or not and check if it executes the scripts in serial manner. Recording: https://bluejeans.com/s/j_UgZFw_jEV - Chapter 2 # Configurable Soft Delete Stephen initiated this topic for nova and oslo.db but also sent out a mail for glance, cinder and for other projects, if we would be interested in the idea https://lists.openstack.org/pipermail/openstack-discuss/2022-October/030729… But Glance replies a lot on soft delete and doesn't allow hard delete as glance allows the user to specify the UUID of the image , so it's part of a security promise of an immutable images that you are not able to delete an image and then recreate image with same UUID right after, hence soft delete model can't be removed from Glance. Recording: https://bluejeans.com/s/j_UgZFw_jEV - Chapter 3 # Secure RBAC As per the revised community goal, till zed cycle we must have the project prosona implementation & drop the system scope which we have already done in glance. We have implemented project scope for image apis in wallaby & in Xena we have managed to move all policy checks to API layer and implemented project scope of metadef APIs. During zed cycle, we had discussed that like which apis should be exposed to system scope but after the operators feedback & as per the revised community goal it was decided to drop the system scope, we just had to remove the scope scheck from scope_types for all project resources, so we are done with Phase 1. In Antelope Cycle glance is going to switch the new defaults flag ON, once it is verified by tempest for all the services. Recording: https://bluejeans.com/s/j_UgZFw_jEV - Chapter 4 # Image Export with Metadata This is about exporting images with associated metadata for importing into another Glance deployment. This cycle we need some volunteers to work on this, the glance team will help in terms of reviews/finalizing the design etc otherwise we will revisit this in the next cycle. This session was not recorded as it was a small discussion # Fips Overview Path forward: - Investigate on the time outing of the existing glance fips job - Try to do the centos9 jobs stable till wallaby, so could possibly move to voting - Have ubuntu jobs working & running and make them stable Recording: https://bluejeans.com/s/j_UgZFw_jEV - Chapter 5 Thursday, OCT 20th, 2022 # Option needed to create image in erasure coded ceph pool To use the erasure coded ceph pool feature where the data will be held needs to be specified during image creation, a config option is needed. Decided to write a spec first describing this in detail and modify the existing devstack job for ceph. Recording: https://bluejeans.com/s/cNEJ@Yq_hv5 - Chapter 1 # Parallelization of RADOS image writes when creating an image This is for parallelization of image writes into ceph, by increasing the amount of data written at once. Decided to not have this without testing to measure if there's an actual gain. Recording: https://bluejeans.com/s/cNEJ@Yq_hv5 - Chapter 2 # Add chunk download support for rbd backend Rbd supports random reading, and the rbd driver is designed to support partial download. But the current version disables this feature, and doesn't implement the chunk support. There is potential future use for it, but there's nothing that users would gain from this. If not usable through the API then we will drop this. Recording: https://bluejeans.com/s/cNEJ@Yq_hv5 - chapter 3 # Operators hour No operator joined the session # Speedup upload images for Swift backend Upload images using Swift backend is slow due to the synchronous way of uploading fragments and which causes loading that can take several hours, especially in cases with very large images. Decided to update the spec with two implementations one as traditional & one for multithreaded one, by having a new configuration option ``swift_store_thread_pool_size`` to the Swift store backend. Recording: https://bluejeans.com/s/cNEJ@Yq_hv5 - Chapter 4 Friday, OCT 21st, 2022 # Cross project meet with Cinder # Glance Image Direct URL access Issue Glance has OSSN-0090[2] describing the security risk when you are operating glance with 'show_multiple_locations' or if the end user facing glance-api has 'show_image_direct_url' options set to true. When glance shares a common storage backend with nova & cinder, it is possible to open some known attack vendors by which malicious data modification can occur. Decided to fix this by going with the solution proposed by Rajat during last cycle, to remove the show_multiple_locations config option and to have below 2 new location APIs[3] which will replace the image-update mechanism for consumers like cinder and nova in glance, 1. Location ADD API: Design this api in way that the location will be added only once during image create when the image is in QUEUED state and no-one should be allowed to add location after the image is active This wouldn't require the 'service role' and a basic check on the glance side to check image status should suffice. 2. Location GET API: This will show all the locations associated with an existing image. Returns an empty list if an image contains no locations. This would still require the 'service role' since we don't want to expose locations to end users. Glance has dependency on keystone for 'service role' which is going to be implemented in this cycle as per phase 2 target mentioned in SRBAC community goals[4]. Recording : https://bluejeans.com/s/B4Rlifuwx_l - Chapter 1 You will find the detailed information about the same in the PTG etherpad[1] along with the recordings of the sessions and milestone wise priorities at the bottom of the etherpad. Kindly let me know if you have any questions about the same. [1]: https://etherpad.opendev.org/p/antelope-glance-ptg [2]: https://wiki.openstack.org/wiki/OSSN/OSSN-0090 [3]: https://specs.openstack.org/openstack/glance-specs/specs/zed/approved/glanc… [4]: https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… Thanks, Pranali

1 0

[cinder] Cancelling this weeks meeting 26 Oct
by Rajat Dhasmana 25 Oct '22

25 Oct '22

Hello Argonauts, Since we just had PTG last week, it seems reasonable to skip this week's meeting. Also I will be on leave tomorrow so cancelling this week's cinder meeting (26 October, 2022). - Rajat Dhasmana

1 0

[kolla] [train] Endpoints fail when one controller is down
by Albert Braden 25 Oct '22

25 Oct '22

Some of our clusters are heavily used, and in those clusters we get complaints when we reboot a controller (or sometimes when we deploy and containers restart). Is that normal, or does it mean that we have something configured wrong? The symptoms are intermittent 504 from endpoints, and VM creation/deletion failing or partially completing, for example the VM is created but without DNS records. We are not following the "Removing existing controllers" procedure [1] before rebooting the controller; is that necessary to avoid these issues? 1. https://docs.openstack.org/kolla-ansible/latest/user/adding-and-removing-ho…

1 0

[kolla-ansible][xena][Masakari] masakarimonitors-instancemonitor won't start
by wodel youchi 25 Oct '22

25 Oct '22

Hi, I have this in my computes nodes; masakarimonitors-instancemonitor won't start 2022-10-25 14:37:27.354 7 INFO masakarimonitors.service [-] Starting masakarimonitors-instancemonitor 2022-10-25 14:37:27.357 7 WARNING masakarimonitors.instancemonitor.instance [-] Error from libvirt : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found) *2022-10-25 14:37:27.357 7 ERROR oslo_service.service [-] Error starting thread.: libvirt.libvirtError: authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found) 2022-10-25 14:37:27.357 7 ERROR oslo_service.service Traceback (most recent call last): 2022-10-25 14:37:27.357 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_service/service.py"* , line 806, in run_service 2022-10-25 14:37:27.357 7 ERROR oslo_service.service service.start() 2022-10-25 14:37:27.357 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/masakarimonitors/service .py", line 62, in start 2022-10-25 14:37:27.357 7 ERROR oslo_service.service self.manager.main() 2022-10-25 14:37:27.357 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/masakarimonitors/instanc emonitor/instance.py", line 166, in main 2022-10-25 14:37:27.357 7 ERROR oslo_service.service self._virt_event(uri) 2022-10-25 14:37:27.357 7 ERROR oslo_service.service File "/var/lib/kolla/venv/lib/python3.6/site-packages/masakarimonitors/instanc emonitor/instance.py", line 128, in _virt_event 2022-10-25 14:37:27.357 7 ERROR oslo_service.service vc = libvirt.openReadOnly(uri) 2022-10-25 14:37:27.357 7 ERROR oslo_service.service File "/usr/lib64/python3.6/site-packages/libvirt.py", line 350, in openReadOnl y 2022-10-25 14:37:27.357 7 ERROR oslo_service.service raise libvirtError('virConnectOpenReadOnly() failed') 2022-10-25 14:37:27.357 7 ERROR oslo_service.service libvirt.libvirtError: authentication failed: Failed to start SASL negotiation: - 4 (SASL(-4): no mechanism available: No worthy mechs found) 2022-10-25 14:37:27.357 7 ERROR oslo_service.service 2022-10-25 14:37:27.359 7 INFO masakarimonitors.service [-] Stopping masakarimonitors-instancemonitor I have verified the auth.conf file on both containers the password is the same. In the other hand, in /var/log/kolla/libvirt/libvirtd.log I have this : 2022-10-25 13:29:16.911+0000: 1391031: error : virNetSocketReadWire:1792 : End of file while reading data: Input/output error 2022-10-25 13:30:18.226+0000: 1391031: error : virNetSocketReadWire:1792 : End of file while reading data: Input/output error 2022-10-25 13:31:19.516+0000: 1391031: error : virNetSocketReadWire:1792 : End of file while reading data: Input/output error 2022-10-25 13:32:20.817+0000: 1391031: error : virNetSocketReadWire:1792 : End of file while reading data: Input/output error 2022-10-25 13:33:22.066+0000: 1391031: error : virNetSocketReadWire:1792 : End of file while reading data: Input/output error 2022-10-25 13:34:23.359+0000: 1391031: error : virNetSocketReadWire:1792 : End of file while reading data: Input/output error 2022-10-25 13:35:24.740+0000: 1391031: error : virNetSocketReadWire:1792 : End of file while reading data: Input/output error Any ideas? Regards.

1 0

[kolla-ansible][Yoga] How to configure rabbitmq heartbeat values
by wodel youchi 25 Oct '22

25 Oct '22

Hi, I need to tweak these variables in my Rabbitmq : heartbeat_timeout_threshold heartbeat_rate heartbeat_interval When I tried to add heartbeat_timeout_threshold = 600 to rabbitmq.conf and reconfigure, the service failed to start with : 13:39:41.756 [error] You've tried to set heartbeat_timeout_threshold, but there is no setting with that name. 13:39:41.758 [error] Did you mean one of these? 13:39:41.838 [error] socket_writer.gc_threshold 13:39:41.838 [error] handshake_timeout 13:39:41.838 [error] auth_http.http_method 13:39:41.838 [error] You've tried to set heartbeat_rate, but there is no setting with that name. 13:39:41.838 [error] Did you mean one of these? 13:39:41.888 [error] heartbeat 13:39:41.889 [error] cluster_name 13:39:41.889 [error] channel_max 13:39:41.889 [error] Error preparing configuration in phase transform_datatypes: 13:39:41.889 [error] - Conf file attempted to set unknown variable: heartbeat_rate 13:39:41.889 [error] - Conf file attempted to set unknown variable: heartbeat_timeout_threshold How can I modify those variables? Regards.

2 2

[TripleO] 2023.1 Antelope PTG summary
by Rabi Mishra 25 Oct '22

25 Oct '22

Hi All, Hope y'all had a great PTG last week and a great weekend. I would like to thank everyone who participated/contributed to Tripleo PTG sessions and helped all of us "share and learn". Though personally I don't like long emails, this summary is going to be a bit longer, please bear with me. We had a total of ten sessions spread over two days with very good participation (most sessions with 35+ [max 44] in attendance) and we shared/ discussed a multitude of topics. # Welcome and Zed Retrospective The highlights of the Zed cycle was collaboration across teams to add some new features and fine-tune features implemented in previous cycles that includes; Standalone roles, nftables switch, OVN RAFT support, ephemeral heat for upgrade use cases, CI jobs for Deployed Ceph and Ceph Container Promotion in CI etc. There was a discussion on the suggestion to stop backporting feature removals from stable branches as we did for wallaby in the last few cycles. Though TripleO does not follow stable policy, backporting feature removals with dependent patches across repos more often than not leads to broken CI and user support cases. We would be well placed if we can identify and do those before zed release and avoid the firefighting later. [Action Items] - Update zed review priorities etherpad[1] with deprecation/removal patches [All] # Standalone Ansible Roles Status Update James provided an update on the current status of "Standalone Ansible Roles' POC and we discussed the possible future scope of it beyond the current "scale-out compute" use case. He also demoed molecule based automation of deploying/testing a compute node with the new meta-role. Chandan shared the current work on CI jobs/scenarios to deploy computes with standalone roles. [Action Items] - Prioritize review/merge of proposed patches[2] on this topic [tripleo-core] - Finish the CI jobs for testing standalone roles and make them voting [Chandan/tripleo-ci-core] # Standalone Ansible Roles and External Ceph John presented his work to integrate computes deployed with standalone roles with external ceph. We don't plan to extend this effort to include tripleo deploying ceph using standalone roles, unless there is a requirement for that in the future. [Action Items] - Prioritize review of ceph related standalone role patches.[tripleo-core] # Multi-rhel support for Compute Brendan gave an update on the ongoing effort for multi-rhel support, different options evaluated and current POC status with composable roles approach. There is still some work left on automating role split during upgrade, which is expected to be completed this cycle. [Action Items] - Finish composable roles POC and test upgrade procedure manually [upgrades team] - Add CI jobs to test the proposed procedure upstream as feasible [upgrades team] - Implement role-split automation during upgrade [Takashi/Rabi] # OpenStack SDK, Ansible Collections OpenStack and TripleO We discussed the issues around openstacksdk backward incompatibility and the current pinning for it in RDO/TripleO. We agreed on an action plan to pre-release ansible-openstack-collection (AOC) for RDO/TripleO (though it is still not fully compatible with latest openstacksdk) and remove the pin so that Zed RDO can be released with openstacksdk 0.101.0. [Action Items] - Bump openstacksdk to 0.101.0 for RDO master and zed [Jakob/Alfredo] - Update RDO ansible-collections-openstack to the 2.0.0 pre-release [Jakob/Alfredo] # TripleO and Distributed Project Leadership (DPL) In this session, we discussed the possibility of opting for DPL in TripleO and sharing the responsibilities across multiple leaders in the team. There were a number of volunteers for different liaison roles and we in general agreed that it would be a good thing to do. We would propose the required change to governance sometime this cycle, unless there is any concern. [Action Items] - Finalize all required liaison roles and propose patch to openstack governance repo [Rabi] # Migration from puppetlabs/apache As part of the broader effort to reduce the puppet footprint, Cedric presented the ongoing effort to replace puppetlabs/apache with the new ansible role "tripleo_httpd_vhost"[3]. A few services have already been migrated and remaining ones are expected to be done this cycle with broader participation from sub teams. [Action Items] - Review/merge the already proposed patches for migration [tripleo-core] - Plan and work on migrating remaining services [Cedric] # State of CI CI team presented on a vast range of topics that included, - Added coverage of upgrade jobs across upstream check/gate, periodic component pipeline and periodic integration pipeline in zed (Marios) - Upstream jobs for multi-rhel testing (Marios) - Current state of OVB Jobs (Chandan) - TripleO CI Jobs on IBM Cloud and the new feature to hold nodes for troubleshooting (Chandan) - Tempest Allow List (Arx/Pooja/Soniya) - Tempest Dashboard (Arx/Lukas) # Container Capabilities We revisited the topic of privileged containers for services and how we can possibly avoid those by allowing the required limited container capabilities. As all these services are mostly for the compute role, we agreed that it would probably be better to align this effort with "Standalone Roles" for better testing. # OS Migrate Status Update Jirka provided an update on the tenant migration tool 'os-migrate', an ansible collection that is not a TripleO umbrella project, but used by the upgrades team. It's possible that I missed some important points from these discussions. Please feel free to add them by replying to this thread. All session etherpad links are in the main schedule etherpad[4] for reference. I'll share the session recordings once they're made available to us. Thanks Again. [1] https://etherpad.opendev.org/p/tripleo-zed-review-priorities [2] https://review.opendev.org/q/topic:standalone-roles [3] https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ans… [4] https://etherpad.opendev.org/p/oct2022-ptg-tripleo -- Regards, Rabi Mishra

1 0

[cloudkitty] Antelope PTG summary
by Rafael Weingärtner 25 Oct '22

25 Oct '22

Dear all, The CloudKitty project had a great PTG meeting. Thanks to the OpenInfra Foundation for holding it. Also, thanks to all who have dedicated precious time to attend. We started the meeting by revisiting the latest release and discussing the review format we have adopted in the past two years. Looking at the result of our process, we decided to maintain it, as it seems to bring results and concentrate our efforts on getting CloudKitty better and evolving throughout time. While reviewing the past release, we discussed the situation with patch [1] (Allow rating rules that have 12 digits in the integer part of the number). This patch is a bug fix, and we discussed the idea of backporting it. However, as it is a patch that affects the database schema, it can be troublesome to backport it. Therefore, before doing any work, we will consult the mailing list to gather others opinions on this matter. Users could upgrade CloudKitty to address this issue, if they need to, instead of applying a patch. Pierre volunteered to execute this "inquiry" with the CloudKitty user base in the mailing list. While discussing the ElasticSearch (ES) improvements in CloudKitty to support ES 7, we raised the concern that Kolla-ansible is moving from ElasticSearch to OpenSearch. Therefore, we need to evaluate if this can have an impact in CloudKitty. Most certainly it will impact people using ES as the backend. The goal is to revisit this discussion in our next bi-weekly meeting, and then communicate for users and community (Topics for meeting of 31/10/2022 [2]). Furthermore, as the current CloudKitty team is a bit small, we have decided to focus on the following topics for the Antelope cycle. - Multiple rating types for the same metric -- One metric being used to generate different rates - Different time frame group by options -- provide group by options to group by day of the year, week of the year, month, and year. - Improve some timeouts handling when using Gnocchi backends. This one needs some checking with Zigo, who was the one that reported this situation. Zigo, I added you to this e-mail, can you reply back with more details regarding this situation? - A regression API that takes the resource ID Or project ID or scope ID, and executes a regression (prediction) based on the collected data so far for the given resources/elements/scope. - Add CloudKitty API reference docs ( https://docs.openstack.org/cloudkitty/latest/api-reference/index.html) to https://docs.openstack.org/api-ref/ We also briefly discussed the situation with Monasca, which has already been deprecated from Kolla-ansible, and might be losing some traction and support. Therefore, we (as a community), might need to start thinking about deprecating it, and removing some non-tested code from our code base. We do understand that deprecating and removing support is complicated for users, but nobody is maintaining and testing that integration. Therefore, it tends to get broken and unstable over time. However, if somebody wants to volunteer to maintain and improve it, you are welcome :) In summary that is the outcome of our small, but efficient PTG meeting. Let's keep improving and making CloudKitty better. If I missed something, please do not hesitate to reply back. Also, if somebody needs any help, you can always ping me :) See you guys. Have a nice week! [1] https://review.opendev.org/c/openstack/cloudkitty/+/837200 [2] https://etherpad.opendev.org/p/cloudkitty-meeting-topics -- Rafael Weingärtner

1 0

[neutron] Antelope PTG summary
by Rodolfo Alonso Hernandez 25 Oct '22

25 Oct '22

Hello Neutrinos: This is a quick summary of what we have discussed during the PTG week. Zed summary and retrospective: - Here is the output of this session: https://etherpad.opendev.org/p/neutron-antelope-ptg#L83 Spec handling: - To have a core reviewer assigned per spec. This reviewer will engage the community to review the spec and will be a kind of "godfather". - To have a weekly status report of the active specs in the community, to visibilize them, boost their impact and "coerce" the community to review them. Neutron CLI deprecation: - To remove the CLI code. - Python bindings: - Investigate and report the effort needed to make this migration in neutron client consumers (Nova, Horizon, Heat, etc). - Request other projects using the python bindings to move to OpenStack SDK. - Stop merging new features. Quota classes: - Information: https://docs.openstack.org/project-team-guide/technical-guides/unified-limi… - Neutron RFE: https://bugs.launchpad.net/neutron/+bug/1993288 - Still under development in the community, this is still not an accepted community goal, thus we won't start the development until some cons are discussed with the TC members (see the Neutron RFE description). Pyroute2 session: - IPmock, a new class to mock some pyroute2 objects: https://lab.pyroute2.org/ - Pluggable netlink engines. Still not compatible with NetNS (namespaces) - Transactional engine: - A daemon is always reading the netlink and inotify, and builds a DB with the system information. Data is read faster. This could be not compatible with the current sync calls using privsep. - Using NDB class. - Customize message parser, to reduce the amount of data retrieved from the kernel. - Dump/list operators will be generators (to check the compatibility with privsep). Remove the failed devstack migration and get back to what is called "neutron-legacy" (that was discussed before in a PTG but we didn't start the reversion). DNS subdomain support in ML2/OVN: - https://review.opendev.org/c/openstack/neutron-specs/+/832658 - On hold until there is a use case. DNS resolver in ML2/OVN: - https://bugzilla.redhat.com/show_bug.cgi?id=2104568 - The main devoplment part falls in the OVN core team. Once finished, the Neutron part should be almost trivial. ML2/OVN flow metering. - Same as we currently have with ML2/OVS + iptables firewall (currently not supported with native firewall because we can't count OF flows). - Same as the previous feature, the main development is in the OVN core team. Neutron QA: - Implement a grenade job based on the SLURP cadence, testing from Y to A (done). - Have the "tempest-integrated-networking" job in our gate/check queues (patch under review). PCI Device Tracking In Placement (Nova-Neutron session): - The different approaches are still under development. - Spec to be proposed during this release (see etherpad to check the possible implementation options). Port binding switchdev capabilities (Nova-Neutron session): - We agreed that the current implementation is incorrect: Neutron should not modify the port binding dictionary. Nova is not reading it but overwriting it. - We won't change the current code but no new features will rely on it. Nova mutable MTU support (Nova-Neutron session): - For now, Neutron will document that a network MTU change requires a VM reboot (ot port detach/attach operations) to be applied on this port. S-RBAC: - Neutron has three implemented personas: system wide admin, project user and project reader. - Tempest will integrate all RBAC testing projects during this release. Once done, the new RBAC flag will be enabled by default. - Neutron will create the corresponding RBAC job/jobs. - Neutron needs to identify what is called "service role" calls (done from other projects). This is a new concept that will be included in next releases. nftables migration: - The current iptables legacy interface could be deprecated. During this release, Neutron will start an epic to move the current iptables API to the new nftables API; if possible, incrementally, mixing both APIs in the same agent. Please, if there is any missing topic, let me know. Regards and thank you for attending and participating.

1 0

[Neutron] Bug Deputy Report Oct 17 - 24
by Oleg Bondarev 25 Oct '22

25 Oct '22

Hello Neutron Team, It was probably the most quiet week in the history of Bug Reports: Low ----- - https://bugs.launchpad.net/neutron/+bug/1993181 - [OVN] OVN metadata "MetadataProxyHandler" not working if workers=0 - Fix Released: https://review.opendev.org/c/openstack/neutron/+/861649 - Fixed by Rodolfo Alonso - https://bugs.launchpad.net/neutron/+bug/1993502 - failing unit tests when not running them all - Fix Released: https://review.opendev.org/c/openstack/neutron/+/861869 - Fixed Rodolfo Alonso - https://bugs.launchpad.net/neutron/+bug/1993498 - [CI] Create oslo master branch jobs (apart from py39)8 - In progress: https://review.opendev.org/c/openstack/neutron/+/861859 - assigned to Rodolfo Alonso RFEs: ----- - https://bugs.launchpad.net/neutron/+bug/1993288 - RFE: Adopt Keystone unified limits as quota driver for Neutron - New (PTG decision - wait till it become more mature) - Unassigned Thanks, Oleg

1 0

[Cinder][Nova]Zombie process prevention for Cinder and Nova APIs
by yu-kishimoto＠kddi.com 25 Oct '22

25 Oct '22

Hi all, I'm trying to fix issues of spawning zombie processes for Cinder and Nova APIs during instantiation in an IaC (Ansible which is not openstack-ansible). Does someone can kindly help me to solve the issue or give me some clues like parameter changes that can be expected to be effective? - Issues The cinder api service process(port: 8776) existed in a zombie state. The neutron api service process(port: 9696) required for nova existed in a zombie state. - What the IaC does Use an existing tenant to create a network and subnet, then use 8volumes to instantiate 8VMs. - Workarounds and what I've done so far Identify and kill zombie processes. - Environment OS: CentOS Stream 8 Kernel: 4.18.0-408.el8.x86_64 OpenStack: Yoga(Deployed by PackStack: https://www.rdoproject.org/install/packstack/) Nova: 25.0.1 Neutron: 20.2.0 Cinder: 20.0.1 KeyStone: 21.0.0 -- Yukihiro Kishimoto Technologist KDDI Co., Ltd. Tokyo Japan

1 1

[magnum] Antelope PTG summary
by Jake Yip 25 Oct '22

25 Oct '22

Dear all, Magnum had a great meeting during the latest PTG. Thanks to OpenInfra Foundation for holding it. Also, thanks to all who has found precious time to attend. As the current Magnum team is quite small, we have decided on the following: - Deprecate the Fedora Atomic driver - This is EOL. - Deprecate the CoreOS driver - This is similarly EOL. - Deprecate the Docker Swarm driver. Kubernetes has became the container orchestration engine of choice now. With our limited resources, it will be more efficient to support the most popular choice well, instead of multiple choices. - Mark branches before Victoria as unmaintained. These branches only support Kubernetes versions that has been EOL. - Most excitingly, move towards adding Cluster API support for Kubernetes. This allows the Magnum project to leverage off upstream effort in creating and managing a Kubernetes cluster. We understand that removing support is difficult for users, but we believe this is for the better. Our main aim is to keep the project moving fast enough to keep up with upstream Kubernetes. If anyone is willing to help maintain any of the items we plan on deprecating, please drop me a message. Regards, Magnum Team. -- Jake Yip DevOps Engineer, ARDC Nectar Research Cloud

1 0

[nova][placement] 2023.1 Antelope PTG summary
by Sylvain Bauza 24 Oct '22

24 Oct '22

We're on a wetty Monday here and I guess this is time for me to take my pen again and write a PTG recap email for the current release, which is 2023.1 Antelope. Yet again, I *beg* you to *not* use any translation tool when reading any etherpad, including Google Chrome embedded translation or it would fully translate the whole etherpad content for *all* readers. In order to prevent any accidental misusage, here is a readonly copy of the Nova etherpad we had for the week https://etherpad.opendev.org/p/r.8ae4e0ef997aebfe626b2b272ff23f1b Again, I'm human and while I can write bugs (actually I write a lot of them), I can also write wrong things in this email and I can misinterpretate something precise. Apologies if so, and please don't refrain them yourself to correct me by replying to this email. Last but not the least, I'm more than open to any questions or remarks that would come from the reading of this long email. Good luck with this very long thread, you should maybe grab a coffee before starting to read it (and I promise to keep it as short as I can). ### Operator hours ### After the success of the productive Nova meet-and-greet session we held in Berlin (packed room w/ lots of feedback), we were eager to again have a discussion with operators, this time virtually which would have hopefully lowered the entry barrier by not requiring an in-person event in order to attend the session. Consequently, we allocated three timeslots of one hour, two back-to-back on Tuesday and one on Wednesday (on different times to arrange time differences between operators). Unfortunately, I have to write here that we only had a very small attendance on Tuesday with only three operators joining (but providing great feedback, thanks btw. !) and *none* on the 1-hour Wednesday slot. As a result, I won't provide statistics about release and project usage here as numbers are too low to be representative, but here is what we discussed : (again, a readonly etherpad is available here https://etherpad.opendev.org/p/r.aa8b12b385297b455138d35172698d48 for further context) # Upcoming features and general discussions - the possibility to mount Manila shares in Nova seems promising. - As nova doesn't support multiple physnets per neutron network today, this prevents the use of routed networks in some cases. - modifying our mod_wsgi usage to be allowed to pass some arguments is requested - operators seem interested in having PCI devices tracked in Placement # Pain points - getting inventories or used/availables resources in Placement becomes costly as it requires N calls, with N be the number of Resource Providers (instead of a single HTTP call). Problem has been acknowledged by the team and some design discussion occurred. - We should paginate on the flavors API and we need to fix the private > public flavor bug. - mediated devices disappearing at reboot is a problem. This has been discussed during the contributor session later, see below. - routing metrics are hard to manage when you have a long list of multiple ports attached. We eventually agreed on the fact the proposal fix could be an interesting feature for Neutron team to develop. That's it for the operators discussions. Let's now discuss about what we discussed in the contributors PTG agenda : ### Cross-project discussions Two projects this time were discussing with the Nova community : # Ironic-Nova The whole session was about the inconsistencies that were happening if a nova-compute was failing down with Ironic rebalancing the nodes. - we agreed on the fact this wasn't an easy problem to solve - a consensus was around the fact Ironic should support a sharding key so the Nova compute could use it instead of using hash rings. - JayF and johnthetubayguy agreed on codifying this idea into a spec - we'd hear feedback from operators about what they feel of the above migration (sharding their node cloud into sharding pieces and providing sharding details to nova) - in parallel of the above, documentation has to be amended in order to recommend for *existing deployments* to setup active/passive failover mode for Nova computes (instead of relying on hashring rebalances) # Neutron-Nova (I'll briefly cover the details here, as it will also be covered by the Neutron PTG recap) - while Nova is finishing up tracking of PCI devices in Placement this cycle, we agreed on defering the modeling of Neutron-based PCI requests until next cycle. This won't have any impact in terms of existing features that will continue to operate seamlessly. - the way we define switchdev capabilities as of now is incorrect. We agreed on modfying Nova to allow it to report such capabilities to Neutron. This may be a specless blueprint or just a bugfix (and then potential backportability), to be determined later. - MTUs are unfortunately immutable. If you change the MTU value by Neutron, you need to restart the instance( (or reattach the port). We consequently agreed on documenting this caveat, which is due to libvirt/qemu not able to change the MTU while the guest is running. - We eventually agreed on letting --hostname (parameter of an instance creation request) to represent a FQDN thru a nova microversion. This value won't be sanitized by nova and directly passed to Neutron as the hostname value. We also agreed on the fact Nova WON'T ever touch the port domain value in metadata, as this is not in the project scope to manage name domains. ### Procedural discussions (skip it if you're not interacting with the Nova contributors) ### # Zed retrospective - We had a productive Zed release and we made a good job on reducing bugs and bug reports. Kudos to the team again. - The microversions etherpad we had was nice but planning microversions usage is hard. WE agreed on rather providing a document to contributors explaining them how to write an API change that adds a new microversion and how to easily rebase this change if a merge conflict occurs (due to the micoversion being taken by another patch that merged) - We agreed on keeping an etherpad for tracking all API changes during milestone-3 - We agreed on filing blueprints for TC goals that impact Nova - bauzas will promote again the use of review-priority label in Gerrit during weekly meetings in order for cores to show their interest in reviewing a particular patch. # Promoting backlog features to new contributors and enhance mentoring - Starting this cycle, we'll put our list of small and easily actionable blueprints into Launchpad bug reports that have a "Wishlist" status and both a 'low-hanging-fruit' and a 'rfe' tag. New contributors or people wanting to mentor a new upstream member are more than welcome to consume that list of 'rfe' bugs and identify the ones they're willing to work on. A detailed process will be documented for helping newcomers to join. - We'll also draft our pile of work we defer into 'backlog specs' if they require further design (and are less actionable from a newcomer perspective) # Other procedural and release discussions - we'll stop using Storyboard for tracking Placement features and bugs and we'll pivot back to Launchpad for the 'placement' sub-project. - we agreed on a *feature* review day possibly 4 weeks before the feature freeze in order to catch up any late design issue we could have missed when reviewing the spec. - we will EOL stein and older nova branches with elodilles proposing the deletion patch. We agred on discussing EOL'ing train at next PTG. - gibi will work on providing a static requirements file for nova that would use minimum versions of our dependencies (non transitively) and modify unit and functional test jobs to rather use this capped requirements file for testing. - we discussed the User Survey and we agreed on discussing any question we may want to add in the next survey during the next weekly meetings. - "2023.1 Antelope" naming is a bit confusing, but we agreed on we should continue to either use "2023.1" or "2023.1 Antelope" for naming our docs. We also wait for guidelines in order to consistently name our next stable branch (2023.1 possibly). (That's the end of procedural and release discussions, please resume here if you skipped the above) ### Technical bits ### # VMware driver status - As currently no job runs happened since April, we agreed on communicating in the nova-compute logs at periodic times that the code isn't tested so operators running it would know its upstream status. - we'll update the supported matrix documentation to refiect this 'UNTESTED' state # TC goals this cycle - for the FIPS goal, we agreed on the fact the current FIPS job (running on centos 9 stream) shouldn't be running on gate and check pipelines. As the current job is busted (all periodic runs go into TIMEOUT state), we also want to modify the job timeout to allow 30 mins more time for running (due to a reboot in the job definition) - for the oslo.privsep goal, no effort is required from a nova perspective (all deliverables are already using privsep). sean-k-mooney can propose a 'rfe' bug (see the note above on new contributors) for modifying our privsep usage in nova (using different privsep context by callers) - for the Ubunutu 2022.4 goal, gmann already provided changes. We need to review them. - for the RBAC goal, let's discuss it in a proper section just below # Next steps on RBAC - we need to test new policy defaults in a full integrated tempest testsuite (meaning with other projects), ideally before the first milestone. - once we check everything works fine as expected, we can flip the default (enabling new policies) before milestone-2 - we'd like to start drafting the new service role usage by two new specs (one for nova, the other one for placement) # Power management in Nova A pile of work I'm proud we gonna start this cycle. This is about disabling/enabling cores on purpose, so power savings occur. - we agreed on supporting it directly in Nova (and to not design and support an external tool which would suppose to draft some heave interface for an easy quickwin). This would just be a config flag to turn on that would enable CPU cores on demand. - a potential follow-up *may* be to use a different CPU governor depending on flavors or images but this won't be scoped for this release. # Power monitoring in Nova I don't really like this name, as monitoring isn't part of the Nova mission statement so I'll clarify. Here, this is about providing an internal readonly interface for power monitoring tools running on guests that would be able to capture host consumption metrics. One example of such monitoring tools is Scaphandre, demonstrated during the last OpenInfraSummit at a keynote. - we agreed on reusing virtiofs support we're gonna introduce in this release for the Manila share attachment usecase - this share would be readonly and would be unique per instance (ie. we wouldn't be supporting multiple guest agents reading different directories) - this share would be enabled thru a configuration flag per compute, and would only be mounted per instance by specific flavor or image extraspec/metadata. # Database soft-deleted records and the archive/purge commands - We don't want to deprecate DB soft-deleted records as some APIs continue to rely on those records. - We rather prefer to add a new parameter to the purge command that will directly delete the 'soft-deleted' records from the main DBs and not only the shadow tables (skipping the need to archive, if the operator wants to use this parameter) # Nova-compute failing at reboot due to vGPU missing devices Talking of bug #1900800, the problem is that mediated devices aren't persistent so nova-compute isn't able to respawn the instances after a reboot if those are vGPU-flavored. - we agreed on the fact that, like any other device, Nova shouldn't try to create them and we should rather ask the operator to pre-create the mediated devices, exaclty like we do for SR-IOV VFs. - we'll accordingly deprecate the mdev creation in the libvirt driver (but we'll continue to support it) and we'll log a warning if Nova has to create one mdev. - we'll change the nova-compute init script to raise a better exception explaining which mdev is missing - we'll document a procedure for explaining how to get existing mdev information and persist them by udev rules (for upgrade cases) # Robustify our compute hostname changes To be clear, at first step, we will continue to *NOT* support nova-compute hostname changes but we'll better detect the hostname change to prevent later issues. - first step to persist the compute UUID on disk seems a good candidate so a spec is targeted this cycle. - next steps could be to robustify our relationships between instance, compute node and service object records but this design will be deferred for later in a backlog spec. # Move to OpenStackClient and SDK OSC is already fully supported in Zed but it continues to rely on novaclient python bindings for calling the Nova API. - we agreed on modifying OSC to rather use openstacksdk (instead of novaclient) for communicating to the Nova APIs. Contributors welcome on this move. - we agreed on stopping to use project client libraries in nova services (eg. cinderclient used by nova-compute) and rather use openstacksdk directly. A 'rfe' bug per project client will be issued, anyone willing to work on it is welcome. - we also agreed on continuing to support novaclient for a couple of releases, as operators or other consumers of this package could require substantial efforts to move to the sdk. - we agreed on changing the release model for novaclient to be independent so we can release anytime we need. # Evacuate to target state We understand the usecase (evacuate an instance shouldn't always turn the evacuated instance on) - that said, we don't want to amend the API for passing a parameter as this would carry a tech debt indefinitely) - we prefer to propose a new microversion that would stop the instance eventually instead of starting it - operators wanting to keep the original behaviour would need to negociate an older microversion to the Nova API as we don't intend to make spawning optionally chosen by API on evacuate. (aaaaaaaaaaaaaaaand that's it for the PTG recap) Kudos, you were brave, you reached that point. Hope your coffee was good and now you feel rejuvenated. Anyway, time for me now to rest my fingers and to enjoy a deserved time off. As said, I'm all up for any questions or remarks that would come from the reading of this enormous thread. -Sylvain

2 1

[nova] Proposing EOL of nova project's old stable branches (Stein, Rocky, Queens)
by Elõd Illés 24 Oct '22

24 Oct '22

Hi, At the 2023.1 Antelope PTG, Nova team discussed that currently open Queens, Rocky and Stein branches for Nova teams' repositories will be moved to End of Life. There are multiple reasons behind this decision: - gate of these branches are broken for a couple of months now - zuul job definitions for these branches use the pre-"zuul-v3" [1] style, which makes the maintenance harder of these jobs - CI on Rocky and Queens use Ubuntu Xenial, which is also beyond its public maintenance window date And last but not least bugfix backports to these branches don't really get enough reviews anymore. If anyone has any interest to keep any of the above branches open then this is the last opportunity to step up, otherwise, as Nova team decided on the PTG, these branches will transition to End of Life. Please let us know before November 7th, we will proceed with the transition in case no responses will come till that day. [1] https://docs.openstack.org/project-team-guide/zuulv3.html Thanks in advance, Előd Illés irc:elodilles @ #openstack-nova

1 0

[telemetry][cloudkitty][ceilometer] Billing windows instances
by Thomas Goirand 24 Oct '22

24 Oct '22

Hi there! We're using telemetry+cloudkitty for our rating. We're happy of it, though we're having the issue that we would like to bill instances running Windows. The example given in the Cloudkitty doc shows how to bill more when an instance runs Windows. That works in theory, though in practice, Microsoft has a license model based on how many vCPU the instannce runs. So the model that the Cloudkitty documentation shows simply doesn't work with the SPLA thingy. We've looked at options. One way would of course writing a new custom pollster, but we don't like the idea: this would mean polling for all of the thousands of instances that are running in our deployments. So we would very much prefer having ceilometer running on compute node (with polling_namespaces=compute) to do the work, as this scales a way better. However, Ceilometer polls libvirt, which only has the information about the image ID, not the metadata associated with the image (like the property os_type=windows, for example). So, is there a better way than a dynamic pollster? Can this be done with ceilometer on the compute nodes? Cheers, Thomas Goirand (zigo)

2 3

Openstack cluster cannot create instances when 1 of 3 rabbitmq cluster node down
by Nguyễn Hữu Khôi 24 Oct '22

24 Oct '22

Description =========== I set up 3 controllers and 3 compute nodes. My system cannot work well when 1 rabbit node in cluster rabbitmq is down, cannot launch instances. It stucked at scheduling. Steps to reproduce =========== Openstack nodes point rabbit://node1:5672,node2:5672,node3:5672// * Reboot 1 of 3 rabbitmq node. * Create instances then it stucked at scheduling. Workaround =========== Point to rabbitmq VIP address. But We cannot share the load with this solution. Please give me some suggestions. Thank you very much. I did google and enabled system log's debug but I still cannot understand why. Nguyen Huu Khoi

5 9

Openstack Cinder and Nova Services cannot work when rabbitmq cluter node down
by Nguyễn Hữu Khôi 24 Oct '22

24 Oct '22

Title: Openstack cluster cannot create when 1 of 3 rabbitmq cluster node down Bug description: Description =========== I set up 3 controllers and 3 compute nodes. My system cannot work when 1 rabbit node in cluster rabbitmq is down, cannot create volume or launch instance. It stucked at creating and scheduling respectively. Steps to reproduce =========== Openstack nodes point rabbit://node1:5672,node2:5672,node3:5672// * Reboot 1 of 3 rabbitmq node. * Create volume or launch instance then it stucked at creating and scheduling respectively. Workaround =========== I need reboot cinder and nova services to create volume and launch instance . More Info: I see in cinder_scheduler, it looks like cinder cannot change to another rabbitmq node. I hope we have ideas for that.. 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task Traceback (most recent call last): 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return self._queues[msg_id].get(block=True, timeout=timeout) 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/queue.py", line 322, in get 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return waiter.wait() 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/queue.py", line 141, in wait 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return get_hub().switch() 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return self.greenlet.switch() 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task _queue.Empty 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task During handling of the above exception, another exception occurred: 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task Traceback (most recent call last): 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task task(self, context) 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 9716, in _sync_power_states 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task db_instances = objects.InstanceList.get_by_host(context, self.host, 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_versionedobjects/base.py", line 175, in wrapper 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task result = cls.indirection_api.object_class_action_versions( 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return cctxt.call(context, 'object_class_action_versions', 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/client.py", line 189, in call 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task result = self.transport._send( 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/transport.py", line 123, in _send 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return self._driver.send(target, ctxt, message, 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task return self._send(target, ctxt, message, wait_for_reply, timeout, 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task result = self._waiter.wait(msg_id, timeout, 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task message = self.waiters.get(msg_id, timeout=timeout) 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task raise oslo_messaging.MessagingTimeout( 2022-10-24 14:23:01.945 7 ERROR oslo_service.periodic_task oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID c8a676a9709242908dcff97046d7976d Nguyen Huu Khoi

1 0

[large-scale][oslo.messaging] RPC workers and db connection
by Arnaud Morin 24 Oct '22

24 Oct '22

Hey all, TLDR: How can I fine tune the number of DB connection on OpenStack services? Long story, with some inline questions: I am trying to figure out the maximum number of db connection we should allow on our db cluster. For this short speech, I will use neutron RPC as example service, but I think nova is acting similar. So, to do so, I identified few parameters that I can tweak: rpc_workers [1] max_pool_size [2] max_overflow [3] executor_thread_pool_size [4] rpc_worker default is half CPU threads available (result of nproc) max_pool_size default is 5 max_overflow default is 50 executor_thread_pool_size is 64 Now imagine I have a server with 40 cores, So rpc_worker will be 20. Each worker will have a DB pool with 5+50 connections available. Each worker will use up to 64 "green" thread. The theorical max connection that I should set on my database is then: rpc_workers*(max_pool_size+max_overflow) = 20*(5+50) = 1100 Q1: am I right here? I have the feeling that this is huge. Now, let's assume each thread is consuming 1 connection from the DB pool. Under heavy load, I am affraid that the 64 threads could exceed the number of max_pool_size+max_overflow. Also, I noticed that some green threads were consuming more than 1 connection from the pool, so I can reach the max even sooner! Another thing, I notice that I have 21 RPC workers, not 20. Is it normal? [1] https://docs.openstack.org/neutron/latest/configuration/neutron.html#DEFAUL… [2] https://docs.openstack.org/neutron/latest/configuration/neutron.html#databa… [3] https://docs.openstack.org/neutron/latest/configuration/neutron.html#databa… [4] https://docs.openstack.org/neutron/latest/configuration/neutron.html#DEFAUL… Cheers, Arnaud.

3 4

RE: RE: [yoga][cinder] Cinder NFS backend: Compute service cannot access volume file (UID/GID problem)
by Felix Hüttner 24 Oct '22

24 Oct '22

Sorry, no idea about that, for us the group also has write permissions -- Felix Huettner From: 박경원 <park0kyung0won(a)dgist.ac.kr> Sent: Monday, October 24, 2022 9:24 AM To: Felix Hüttner <felix.huettner(a)mail.schwarz>; openstack-discuss(a)lists.openstack.org Subject: RE: RE: [yoga][cinder] Cinder NFS backend: Compute service cannot access volume file (UID/GID problem) Hello Felix Thank you very much for kind reply Do I also need to change permission setting of volume file in /var/lib/nova/mnt/... ? By default its: drwxr-x--- 2 64061 64061 11 Oct 24 04:19 99c4f7e8b15983b65e20cb7d37db899f group has only read and execute permission, no write permission ---------- 원본 메일 ---------- 보낸사람: "Felix Hüttner" <felix.huettner(a)mail.schwarz<mailto:felix.huettner@mail.schwarz>> 받는사람: "park0kyung0won(a)dgist.ac.kr<mailto:park0kyung0won@dgist.ac.kr>" <park0kyung0won(a)dgist.ac.kr<mailto:park0kyung0won@dgist.ac.kr>>, "openstack-discuss(a)lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>" <openstack-discuss(a)lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> 날짜: 2022-10-24 (월) 16:12:48 제목: RE: [yoga][cinder] Cinder NFS backend: Compute service cannot access volume file (UID/GID problem) Hi, we are solving this issue for us by creating a “cinder” group on all hypervisors with the same gid (64061 in your case). Then we add the nova user to the cinder group and we are fine afterwards. You might need set “dynamic_ownership = 0" In your libvirt qemu.conf -- Felix Huettner From: 박경원 <park0kyung0won(a)dgist.ac.kr<mailto:park0kyung0won@dgist.ac.kr>> Sent: Monday, October 24, 2022 7:21 AM To: openstack-discuss(a)lists.openstack.org<mailto:openstack-discuss@lists.openstack.org> Subject: [yoga][cinder] Cinder NFS backend: Compute service cannot access volume file (UID/GID problem) Hi I'm trying to setup cinder-volume service with NFS backend When I create a new VM instance with a volume from web UI, cinder-volume service on storage node creates volume file just fine But I get the following error on compute node and instance fails to spawn. 2022-10-24 02:14:25.347 402789 ERROR nova.compute.manager [req-47ec9fb1-9daa-4c24-8673-538797a217cc 8769cfaf608349bd9fbb36f92b188fe3 e1e8e8397cde49899b00d09dec76b29e - default default] [instance: 5acb1dc3-0685-4980-977b-b6dfff6dfb45] Instance failed to spawn: libvirt.libvirtError: internal error: process exited while connecting to monitor: 2022-10-24T02:14:24.819644Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/var/lib/nova/mnt/99c4f7e8b15983b65e20cb7d37db899f/volume-8f478992-dde3-4c20-9005-61cd34eacf30","aio":"native","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: Could not open '/var/lib/nova/mnt/99c4f7e8b15983b65e20cb7d37db899f/volume-8f478992-dde3-4c20-9005-61cd34eacf30': Permission denied I've added appropriate configs to apparmor profile. (Using Ubuntu 22.04) Apparmor isn't blocking this access. While the instance is spawning, I've checked ownership of the volume file on compute node: root@compute-node:/var/lib/nova/mnt$ ls -al total 17 drwxr-xr-x 3 nova nova 4096 Oct 24 04:19 . drwxr-xr-x 12 nova nova 4096 Oct 24 02:14 .. drwxr-x--- 2 64061 64061 11 Oct 24 04:19 99c4f7e8b15983b65e20cb7d37db899f It seems like cinder user on storage node creates volume file with UID/GID of 64061 (cinder user's UID/GID) But nova user on compute node has UID/GID of 64060, therefore cannot open volume file(/var/lib/nova/mnt/99c4f7e8b15983b65e20cb7d37db899f/volume-8f478992-dde3-4c20-9005-61cd34eacf30) Should I manually set the UID/GID of nova user on compute node to 64061, so both nova user on compute node and cinder user on storage node would have the same UID/GID? Feels like this duct taping isn't a proper solution. Did I miss something? Thank you Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hier<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.daten…>. Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarz>.

1 0

RE: RE: [yoga][cinder] Cinder NFS backend: Compute service cannot access volume file (UID/GID problem)
by 박경원 24 Oct '22

24 Oct '22

1 0

[yoga][cinder] Cinder NFS backend: Compute service cannot access volume file (UID/GID problem)
by 박경원 24 Oct '22

24 Oct '22

2 1

[largescale-sig] No meeting this week
by Thierry Carrez 23 Oct '22

23 Oct '22

Hi everyone, With the PTG just over, the Large Scale SIG will not be meeting this week. Our next regular IRC meeting will be November 9, at 1500utc on #openstack-operators on OFTC. Feel free to add topics to the agenda: https://etherpad.openstack.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 0

single Network interface
by Parsa Aminian 23 Oct '22

23 Oct '22

Hello I use kolla ansible wallaby version . my compute node has only one port . is it possible to use this server ? as I know openstack compute need 2 port one for management and other for external user network . Im using provider_networks and it seems neutron_external_interface could not be the same as network_interface because openvswitch need to create br-ex bridge on separate port is there any solution that i can config my compute with 1 port ?

7 9

[tc][all][policy][rbac] RBAC 2023.1 Antelope cycle Virtual PTG discussions summary
by Ghanshyam Mann 23 Oct '22

23 Oct '22

Hello Everyone, Last week, I tried to attend all the RBAC-related sessions on various projects. We mainly discussed the next steps on RBAC and implementing the project personas if not yet done. I captured the summary and outcome of the discussions of those projects in the below etherpad: Look for the section: "RBAC/Policy 2023.1 Antelope PTG summary". If I have missed any project discussion, please add it to the etherpad. - https://etherpad.opendev.org/p/rbac-goal-tracking One of the big outcomes, we agreed on all the projects (implemented phase-1) to enable the scope checks and new defaults by default. But before doing that, we need to test the new defaults and scope in Tempest (tempest plugin) integrated gate. I have started working on the job enabling the scope and new defaults[1]. [1] https://review.opendev.org/c/openstack/tempest/+/614484 -gmann

1 0

[tc][all][ Technical Committee 2023.1 Antelope cycle Virtual PTG discussions summary
by Ghanshyam Mann 23 Oct '22

23 Oct '22

Hello Everyone, I am summarizing the Technical Committee discussion that happened in the 2023.1 Antelope cycle PTG last week. Also, in case anyone wanted to search for all the projects etherpad later after ptgbot cleans them, I have added those to this wiki[1] Attendance: ========= * ~30 on Monday, ~25 on Thursday, ~20 on Friday Improvement in project governance: =========================== We continued the discussion from Monday and discussed a few high-level criteria which can be the potential criteria to monitor inactive projects. Along with contributors, event, and meeting occurrence is one of them but not all. JayF will be working on this to list the exact criteria/things TC can monitor to find and work on help inactive projects. User Survey: ========== * 2021 user Survey result analysis: We started with the pending review of the 2021 survey analysis[2]. There wasn't 2020 analysis so, the comparison was done using 2019. Overall, the trends in the survey were positive. We discussed adding the question about UI participants especially if users need new features to be implemented in UI then what prevents them to help and contribute? * 2022 User Survey result: knikolla and I will be working on the analysis of this survey result. jungleboyj will help on documenting the process and tips to analyze the survey xls which is not an easy task. Thanks to jungleboyj for doing the survey analysis and helping with the 2022 survey too. * 2023 User Survey question review: While doing the 2021 survey analysis, we discussed the below changes in 2023 survey. ** Remove "Other ways users participate:" ** Add: 1. how users are consuming OS. From pypi packages? From RHOSP? etc. 2. How are users interacting with OpenStack? Through Horizon? CLI? Skyline? An internally developed tool? etc. 3. Adding a sub-question about UI development help in "To which projects does your organization contribute maintenance resources such as patches for bugs and reviews on master or stable branches?" 4. Asking about what all OSS/software/services are used in your OpenStack provisioned cloud/resource. * Action: As the next step, we will work on the exact wording for the above questions and send it to Allison before the end of Oct month. Next step for OSC work ================== We started the discussion on sdk/osc should continue supporting the old interfaces even if we are not able to test them properly. Anyone running a very old cloud should be able to use the sdk/osc latest. We agree and Artem mentioned being more careful about removing the things. Next, we discussed where we stand on OSC work and direction from the project side and good progress from nova, neutron, and manila (might be a few more) but at the same time, many other projects need work on fixing the feature gaps. Glance is also in agreement to work towards OSC. There was a question on when and how we should work towards deprecating/removing the project's python client. The best way is to start implementing the new features in OSC and try their best to fill the old feature gap. Based on the current state, we agree to start this work as one of the community-wide and Artem volunteers to help here. * Action: ** Propose this as a community-wide goal "focus on fixing the feature gaps in osc not to remove the pythonclient which can be another goal" ** Look at devstack's use of OSC to see if we can enable token caching and perhaps use a smaller venv for performance Consistent and Secure Default RBAC ============================ As a good amount of the services are ready with the project personas, We will be testing them in the integrated gate (tempest and tempest plugins jobs), and based on the results we will enable the scope to enforce and new defaults enforce by default in this cycle. Also, services can start implementing phase 2 of the goal which is the service-to-service role. On Horizon question of moving to new defaults. because of the explicit policy files (with defaults) on the horizon side, the operator will not be able to just switch the flag until they change the policy file. For a better migration path and getting one more cycle to test the new defaults by default on the service side, it is better for Horizon to move to new defaults in the next cycle. FIPs goal ======= Ade lee provided the current status[3] of the goal which is good progress. There is work going on having the ubuntu based FIPs enabled job[4] and that will be to achieve the goal of having of voting job in the check/gate pipeline. Migrate CI/CD jobs to Ubuntu 22.04 (Jammy Jellyfish) ========================================= Projects are testing the jobs on Jammy but not all[5], we encourage the projects to start testing your project jobs before the deadline which is m-1 (Nov 18), and fix the issues if there are any. One known issue is Ceph job failing on Jammy as there aren't jammy upstream packages. There are packages in UCA and Ubuntu proper that could be used instead. The community-wide goal process in a world with few contributors =================================================== This topic is on how we can improve the community-wide goal progress and productivity but at the same time reduce the overhead on projects. Also, to highlight that the Champion of a goal is not required to do all the work and it is up to the champion. knikolla will work on the goal documentation to try goal overhead minimum. We will be more careful with goal selection to make sure if any work can be done without a community-wide goal then we should proceed that way. * Action: knikolla: see opportunities for rewording the community goal document. mention: striving to keep goals to a minimum to reduce overhead on teams and try to build consensus first. Pop-Up Team checks ================ We have two active popup teams, 1. Image Encryption 2. Policy(RBAC). Both have pending work to finish so will continue in this cycle also. Support policy for 32-bit platforms support ================================= Zigo is testing the OpenStack on 32-bit and filed bugs[6]. As we do not test it in the upstream CI/CD, we will not be committing to its support. But it is completely ok to fix the bug or add a skip in tests. Thanks to zigo for testing and reporting the bugs. Fixing docs build issues with recent Sphinx ================================ Sphinx is capped at 4.5.0 in upper constraints for a long time and whether we should move our doc to 6.0 or not is the question. Moving to 6.0 needs more work in openstackdocstheme. Until we have someone fixing things, it is ok to continue on Sphinx 4.5.0. Election retrospective: ================= We had a good amount of discussion on elections and many ideas to improve them in the future. Also by discussing the k8s election process with k8s steering committee members. We also discussed TC Chair election process to record the nomination in a better way. This is still not concluded and we continue the discussion in TC. Overall it was a productive discussion and we came up with below action items: *Action: 1. Extend the nomination and voting period to two weeks and also communicate the election well in advance. 2. Along with existing projects/SIG repos, add a few more related repos (governance, etc) in election tooling to count the electorate. 3. Add in the election process: call for AC on openstack-discuss or any other ML before the deadline. 4. Appoint a TC liaison as a point of contact to track the election dates more carefully and ICAL for election tasks will be a good idea to send/add. 5. Update the TC charter to mention the election period, and deadline explicitly. Cross-community sessions with k8s steering committee team: =============================================== We invited Kubernetes Steering Committee members to TC sessions. Tim and Christoph joined the sessions. This is a great way to collaborate between two communities. Following the introduction from both sides, we discussed various topics and share the process, challenges, and feedback from both sides. The election process, contributor recruiting, and how to engage them as long-term contributors. Also having part-time or non-corporate contributors is still a challenge for both communities. We also discussed the operator engagement challenges OpenStack is facing. Kubernetes operator thing is a bit complicated and they have app dev, cluster, app, and infra operator. Similar to OpenInfra foundation to the OpenStack community, CNCF foundation plays an important role for them to connect the operator with the community as much as possible. Discuss and clarify the supported upgrade-path testing in PTI: =============================================== To provide a better upgrade path, we decided to test the old distro version also whenever we will bump the distro version in our CI/CD. Below are details of the upgrade testing we will be following: Agree: * Supporting two distro versions when we bump the new distro version in any release: (this is only for the release bump of the distro and after that, we can go back to a single distro version testing) ** Run single tempest job in project gate on old distro version, not all jobs ** Add the previous python version unit test job in the new release testing template. * For non-SLURP releases, we will try not to change the testing runtime unless it is very much required due to the EOL of versions, we are using in testing. * In case the project has to add some feature that needs a new version of deps which is not supported in the old distro version then they need to be explicit about it and communicate in a better way. * Action: I will document the above in PTI. Guidelines for using the OpenStack release version/name and project version in 1. releasenotes 2. Documentation ===================================================================================== In Zed cycle, TC passed a resolution[7] and also prepared the guidelines[8] on using the release number as a primary identifier. During nova PTG, there was a question on what is the recommendation for using OpenStack version and package version (say Nova 27.0.0) in project documentation, releasenotes etc. After discussing the multiple options listed in etherpad we agree to go for the below: * OpenStack <release version> (<project> <project vesion>) Example: OpenStack 2023.1 (Nova 27.0.0) I will add this to the release identifier page[8] so that all projects can use it consistently. Discussion on projects (like neutron, ceilometer ) in Upper Constraints: ===================================================== Having projects in u-c makes it difficult for users to have a consistent deployment of those services. But we do not recommend using the u-c in production and it can be explicitly mentioned in the requirement and project-team-guide document. * Action: tonyb to document the upper constraints usage expectation (especially for production usage) in the requirement document as well as in project-team-guide[9]. Zed Retrospective: ============== In the end, we discussed the Zed cycle retrospective. * What went well? ** Good amount of work in the zed cycle[10] ** New TC members ** Good participation in meetings especially video call ** TC & Community Engagement (leaders interaction) improving *** i18 SIG team having there helped to proceed with i18 SIG work One thing to improve next time is to explicitly call out the team/members with a courtesy ping for future PTG if any related discussion. We do send the agenda on ML in advance but no harm in ping also. Meeting time check: =============== TC weekly Video calls are more productive compared to text meetings and we will do two video calls in a month. We will also start a poll to select the meeting time. 2023.1 cycle TC Tracker ================== I prepared the TC tracker for 2023.1 and listed all the actionable working items that came up during the PTG discussion. This is helpful for tracking the working items. - https://etherpad.opendev.org/p/tc-2023.1-tracker Thank you for reading the summary or I will say detailed summary :), have a nice weekend everyone. [1] https://wiki.openstack.org/wiki/PTG/2023.1/Etherpads [2] https://review.opendev.org/c/openstack/governance/+/836888 [3] https://etherpad.opendev.org/p/fips_goal_status [4] https://review.opendev.org/c/openstack/project-config/+/861457/ [5] https://etherpad.opendev.org/p/migrate-to-jammy [6] https://bugs.launchpad.net/glance-store/+bug/1991406 [7] https://governance.openstack.org/tc/resolutions/20220524-release-identifica… [8] https://governance.openstack.org/tc/reference/release-naming.html [9] https://docs.openstack.org/project-team-guide/dependency-management.html [10] https://etherpad.opendev.org/p/tc-zed-tracker -gmann

1 0

[tc][all][ptg] "Technical Committee + Leaders interaction" 2023.1 cycle vPTG sessions summary
by Ghanshyam Mann 23 Oct '22

23 Oct '22

Hello Everyone, I am writing the Technical Committee + Leader interaction session discussion summary that happened in the 2023.1 Antelope cycle PTG this week. We continue the interaction session in this PTG also. The main idea here is to interact with community leaders and ask for their feedback on TC. I am happy to see ~30 attendance and had a good amount of discussions. Below are the topics we discussed in this feedback session. Updates from TC: ============= * Conveying current selected Community-wide goals. * Avoid Bare recheck * TC PTG slots: 15-19 UTC on Thursday and Friday * Status on previous PTG interaction sessions' Action items/feedback: ** PTLs/Leaders to start spreading the word and monitor ignorant recheck in their weekly meeting Status: Good progress in this, many projects started monitoring it and slaweq continues contacting the PTL for this. But we still encourage PTLs/Leaders to publish the blog/video of their stories/project updates etc at least once in a cycle. You can check the places where to publish such blogs/videos [1]. ** Recognize the contribution: Status: Not done, we are continuing this AI and will work on it this cycle. ** Bare Recheck: Status: Done. TC is monitoring the bare recheck in the weekly meetings and slaweq reaching out to projects to do the same in their weekly meeting. Feedback on TC activities or anything Community Leaders would like TC to spend more time on: ========================================================================= * Continue the TC weekly summary email which is helpful * TC to build a review process that would happen periodically to check the project status. Basically to ensure the project status as per "Emerging Technology and Inactive projects"[2]. JayF volunteered to work on this to write some documentation or resolution. ** Action Item: 1. JayF to write a resolution on criteria/flag to identify/action on the inactive projects. Renovate translation SIG i18: ====================== Ian and Seongsoo joined the discussion to provide the current state and help needed in i18 SIG. We collected some information on translation usage, language, and content(GUI, doc, log etc). Motoki updated information from the Japan group but still, needs more information from other groups/translation users and agrees to add some queries in the 2023 user survey. Also, Brian will investigate the possibility of migration to weblate tool. * Action Item: 1. rosmaita: draft questions for the survey, get feedback from ian and seongsoo 2. rosmaita: follow up with weblate service to determine whether OpenStack qualifies for gratis service 3. Ian to reach out to contributors/users to know the translation requirements for applicable language. Possibilities to avoid RC time issues with Oslo release: ======================================== After discussing the issue that occurred for Sqlalchemy 2.0 related merge in the zed cycle, we discuss one option to propose the non-client feature freezes to m-2 and get more feedback from Oslo or other non-client lib maintainers. Also, we encourage adding the lib forward testing which tests the integration gate with the master version of lib which can help to find the issues while the code merges itself. As discussed Elod proposed the review to get more feedback[3] and also email it to ML[4]. After getting feedback from Oslo maintainer it seems it is not a good idea to preponed the lib feature freeze. Discussion is going on in ML[4]. Clean up Zuul config errors: ===================== At the end of the session, we discussed the current zuul config error in OpenStack projects. TC is actively working on this and knikolla and other members trying to fix them as much as possible. Please check your projects and fix them, priority is to fix the master and supported branches, not the EM state branches. [1] https://docs.openstack.org/project-team-guide/spread-the-word.html [2] https://governance.openstack.org/tc/reference/emerging-technology-and-inact… [3] https://review.opendev.org/c/openstack/releases/+/861900 [4] https://lists.openstack.org/pipermail/openstack-discuss/2022-October/030914… -gmann

1 0

Follow up on https://storyboard.openstack.org/#!/story/2010377
by rajiv mucheli 22 Oct '22

22 Oct '22

Hi, Unlike Openstack Yoga, where the setuptools version was set in upper-contstraints.txt. This is not set in Openstack Zed, which uses jammy with python 3.10. I get the below while performing few tests : - [[ -f /failure ]] - echo Wheel failed to build - cat /failure Wheel failed to build thrift===0.16.0 python-nss===1.0.1 We will need to stay with setuptools <58 to fix the 2to3 compiler issue, else anyjson packages adds up to the above list, but thrift is removed when setuptools >63. Regards, Rajiv

2 5

[PTL][TC] library *feature* freeze at Milestone-2
by Elõd Illés 22 Oct '22

22 Oct '22

Hi, During 'TC + Community leaders interaction' [1] a case was discussed, where a late library release caused last minute fire fighting in Zed cycle, and people discussed the possibility to introduce a (non-client) library *feature* freeze at Milestone-2 to avoid similar issues in the future. I've started to propose the possible schedule change [2] (note: it's not ready yet as it does not emphasize that at Milestone-2 we mean *feature* freeze for libraries, not "final library release"). The patch already got some reviews from library maintainers so I'm calling the attention to this change here on the ML. Thanks everyone for the responses in advance, Előd [1] https://lists.openstack.org/pipermail/openstack-discuss/2022-October/030718… [2] https://review.opendev.org/c/openstack/releases/+/861900

4 5

[neutron]: DVR with OVN
by Farhad Sunavala 22 Oct '22

22 Oct '22

Hi, Just want to get a feel with how users are using DVR with OVN especially for a medium sized installation (500 nodes, 3000-4000 VMs)Are you just using it in the default configuration with all the complexities and just take the hit when troubleshooting why a particular path doesn't work as expected? Are you taking precautions such as providing VMs with a separate direct path to the public network in case the main interface connected to DVR fails?See OpenInfra 17th minute of Live Ep. 26 Large Scale Neutron Best Practices @ (340) OpenInfra Live Ep. 26: Large Scale OpenStack: Neutron Scaling Best Practices - YouTube | | | | | | | | | | | OpenInfra Live Ep. 26: Large Scale OpenStack: Neutron Scaling Best Pract... | | | Is DVR with OVN reasonably solid for production environments that makes you comfortable enough to use it in installations with around 5000 VMs? thanks,Fred.

1 0

[tripleo] gate blocker centos 9 content provider - trending fixed
by Marios Andreou 21 Oct '22

21 Oct '22

Hello FYI (few folks asking in irc apologies should have sent sooner) gate blocker on the content provider https://bugs.launchpad.net/tripleo/+bug/1984237/comments/5 thanks chkumar we are trying to get a fix through gate at https://review.opendev.org/c/openstack/tripleo-quickstart/+/856582 please hold your recheck until that merges or use depends-on thank you!

2 1

Track Chair Nominations Close in One Week!
by Helena Spease 21 Oct '22

21 Oct '22

Hey everyone! Track Chair nominations <https://openinfrafoundation.formstack.com/forms/trackchairnominations2023> for the 2023 OpenInfra Summit in Vancouver (June 13-15, 2023) are closing soon! Please submit your nominations before they close on October 28, 2022 Track Chairs for each Track will help build the Summit schedule, and are made up of individuals working in open infrastructure. Responsibilities include: - Help the Summit team put together the best possible content based on your subject matter expertise - Promote the individual Tracks within your networks - Review the submissions and Community voting results in your particular Track - Determine if there are any major content gaps in your Track, and if so, potentially solicit additional speakers directly to submit - Ensure diversity of speakers and companies represented in your Track - Avoid vendor sales pitches, focusing more on real-world user stories and technical, in-the-trenches experiences 2023 Summit Tracks: - 5G/NFV & Edge - AI/Machine Learning/HPC - CI/CD - Container Infrastructure - Getting Started - Hardware Enablement - Open Development - Private & Hybrid Cloud - Public Cloud - Security - Hands On Workshops Full track descriptions are available here <https://openinfra.dev/summit/vancouver-2023/summit-tracks/>. If you’re interested in nominating yourself or someone else to be a member of the Summit Track Chairs for a specific Track, please fill out the nomination form <https://openinfrafoundation.formstack.com/forms/trackchairnominations2023>. Nominations will close on October 28th, 2022. Track Chairs selections will occur before we close the Call for Presentations (CFP) so that the Chairs can host office hours to consult on submissions, and help promote the event. CFP will be opening in November, registration <https://openinfrasummit2023.eventbrite.com/> and sponsorship <http://openinfra.dev/summit/vancouver-2023/summit-sponsor/> information are already available. Please email speakersupport(a)openinfra.dev with any questions or feedback. Cheers, Helena Spease

1 0

[ptl][release][stable][EM] Extended Maintenance - Wallaby
by Elõd Illés 21 Oct '22

21 Oct '22

Hi, In less than a two weeks Wallaby is planned to transition to Extended Maintenance phase [1] (planned date: November 2nd, 2022). The release patches for the transition have been generated [2], these patches mark the latest release in stable/wallaby with wallaby-em. PTLs and release liaisons are encouraged to approve them as soon as possible, by latest before the transition date. After the transition stable/wallaby will be still open for bug fixes, but there won't be any official releases. Note: if a team wants to do a final release before the transition, then it can be done, but be careful to not release fixes that could break things to avoid broken releases as *final* stable Wallaby release. Thanks, Előd irc: elodilles @ #openstack-releases [1] https://releases.openstack.org/ [2] https://review.opendev.org/q/topic:wallaby-em

1 0

Seeking help for Keystone Error while assign_role_to_user for a Project in Openstack SDK
by Ayush Jaiswal 21 Oct '22

21 Oct '22

Hi Team, I am using the Openstack SDK in my django rest api project where I have to create a new project and assign various users with different roles to that project in openstack. However, while adding a user to a project I am getting the following error. [image: image.png] Complete details of the steps I followed that resulted in this error are in the text file attached with this email. Kindly help me resolve this issue. Your support is highly appreciated. -- Thanks and Regards, Ayush Jaiswal

1 0

[victoria] Loose network on all instance.
by Albert Shih 21 Oct '22

21 Oct '22

Hi, I've a small openstack installation running Victoria on Ubuntu 20 LTS. After a update (not upgrade) of Ubuntu, I loose all network on all my instance. On a instance console I got : [[0;32m OK [0m] Started [0;1;39mifup for ens3[0m. Starting [0;1;39mRaise network interfaces[0m... [[0;1;31m*[0m[0;31m* [0m] A start job is running for Raise ne��ork interfaces (1min 4s / 6min 2s) M[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Raise ne��ork interfaces (1min 4s / 6min 2s) M[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Raise ne��ork interfaces (1min 5s / 6min 2s) M[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Raise ne��ork interfaces (1min 5s / 6min 2s) M[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Raise ne��ork interfaces (1min 6s / 6min 2s) M[K[ [0;31m*[0;1;31m*[0m] A start job is running for Raise ne��ork interfaces (1min 6s / 6min 2s) M[K[ [0;31m*[0m] A start job is running for Raise ne��ork interfaces (1min 7s / 6min 2s) M[K[ [0;31m*[0;1;31m*[0m] A start job is running for Raise ne��ork interfaces (1min 7s / 6min 2s) M[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Raise ne��ork interfaces (1min 8s / 6min 2s) M[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Raise ne��ork interfaces (1min 8s / 6min 2s) M[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Raise ne��ork interfaces (1min 9s / 6min 2s) M[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Raise ne��ork interfaces (1min 9s / 6min 2s) M[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Raise ne��rk interfaces (1min 10s / 6min 2s) M[K[[0m[0;31m* [0m] A start job is running for Raise ne��rk interfaces (1min 10s / 6min 2s) M[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Raise ne��rk interfaces (1min 11s / 6min 2s) M[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Raise ne��rk interfaces (1min 11s / 6min 2s) M[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Raise ne��rk interfaces (1min 12s / 6min 2s) M[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Raise ne��rk interfaces (1min 12s / 6min 2s) M[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Raise ne��rk interfaces (1min 13s / 6min 2s) M[K[ [0;31m*[0;1;31m*[0m] A start job is running for Raise ne��rk interfaces (1min 13s / 6min 2s) M[K[ [0;31m*[0m] A start job is running for Raise ne��rk interfaces (1min 14s / 6min 2s) ...... ...... M[K[ [0;31m*[0;1;31m*[0m] A start job is running for Raise ne��rk interfaces (2min 59s / 6min 2s) M[K[ [0;31m*[0;1;31m*[0m[0;31m*[0m] A start job is running for Raise network interfaces (3min / 6min 2s) M[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Raise network interfaces (3min / 6min 2s) M[K[ [0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Raise ne��ork interfaces (3min 1s / 6min 2s) M[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Raise ne��ork interfaces (3min 1s / 6min 2s) M[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Raise ne��ork interfaces (3min 2s / 6min 2s) M[K[[0m[0;31m* [0m] A start job is running for Raise ne��ork interfaces (3min 2s / 6min 2s) M[K[[0;1;31m*[0m[0;31m* [0m] A start job is running for Raise ne��ork interfaces (3min 3s / 6min 2s) M[K[[0;31m*[0;1;31m*[0m[0;31m* [0m] A start job is running for Raise ne��ork interfaces (3min 3s / 6min 2s) M[K[[0;32m OK [0m] Finished [0;1;39mRaise network interfaces[0m. [K[[0;32m OK [0m] Reached target [0;1;39mNetwork[0m. Starting [0;1;39mInitial cloud-ini�� (metadata service crawler)[0m... [ 186.709234] cloud-init[514]: Cloud-init v. 20.4.1 running 'init' at Thu, 20 Oct 2022 20:12:41 +0000. Up 186.69 seconds. [ 186.727021] cloud-init[514]: ci-info: ++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++ [ 186.728928] cloud-init[514]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+ [ 186.730709] cloud-init[514]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address | [ 186.732694] cloud-init[514]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+ [ 186.734765] cloud-init[514]: ci-info: | ens3 | True | fe80::f816:3eff:fec3:2e6c/64 | . | link | fa:16:3e:c3:2e:6c | [ 186.736681] cloud-init[514]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . | [ 186.738473] cloud-init[514]: ci-info: | lo | True | ::1/128 | . | host | . | [ 186.740246] cloud-init[514]: ci-info: +--------+------+------------------------------+-----------+-------+-------------------+ [ 186.741723] cloud-init[514]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++ [ 186.742306] cloud-init[514]: ci-info: +-------+-------------+---------+-----------+-------+ [ 186.742881] cloud-init[514]: ci-info: | Route | Destination | Gateway | Interface | Flags | [ 186.743460] cloud-init[514]: ci-info: +-------+-------------+---------+-----------+-------+ [ 186.744075] cloud-init[514]: ci-info: | 1 | fe80::/64 | :: | ens3 | U | [ 186.744681] cloud-init[514]: ci-info: | 3 | local | :: | ens3 | U | [ 186.745255] cloud-init[514]: ci-info: | 4 | multicast | :: | ens3 | U | [ 186.745833] cloud-init[514]: ci-info: +-------+-------------+---------+-----------+-------+ in the log of neutron-linuxbridge-agent.log it seem everything should work : 2022-10-20 20:17:54.967 59611 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-9e6b23d3-1e0e-4c48-8f0e-30d761233188 - - - - -] Port tap4f68bcf1-4d updated. Details: {'device': 'tap4f68bcf1-4d', 'network_id': '9223f9ff-7ab0-4268-9b7f-3b5966625c65', 'port_id': '4f68bcf1-4d32-4ad3-a0df-0415f7170c5d', 'mac_address': 'fa:16:3e:c3:2e:6c', 'admin_state_up': True, 'network_type': 'flat', 'segmentation_id': None, 'physical_network': 'provider', 'mtu': 1500, 'fixed_ips': [{'subnet_id': '12cdcc06-bbf0-48f2-b303-bdfab460c696', 'ip_address': '145.238.137.52'}], 'device_owner': 'compute:nova', 'allowed_address_pairs': [], 'port_security_enabled': True, 'qos_policy_id': None, 'network_qos_policy_id': None, 'profile': {}, 'propagate_uplink_status': False} But I'm unable to ping any instance. Any idea ? Regards -- Albert SHIH 🦫 Observatoire de Paris France Heure locale/Local time: jeu. 20 oct. 2022 22:47:15 CEST

2 1

[designate] [release] Proposing to EOL Designate stable branches Queens, Rocky, Stein, and Train
by Michael Johnson 21 Oct '22

21 Oct '22

Today at the Designate PTG session we discussed the number of stable branches (10) we are carrying for Designate and the lack of interest in maintaining them. For example, there has not been a patch merged on the Queens branch for over two years. Based on the discussion, and lack of patches merged to the older branches, I am proposing we move the Queens, Rocky, Stein, and Train stable branches from extended maintenance to end-of-life status. The definition of this process is part of the project team guide[1]. I plan to propose patches for the required changes next week. If you would like to maintain one of these branches, please respond to this email and post a patch to the branch to resolve any existing issues on the older branches. Michael [1] https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-l…

1 0

[i18n] questions for the user survey
by Brian Rosmaita 21 Oct '22

21 Oct '22

Hello people interested in internationalization, At the PTG this week, the i18n SIG drafted some questions for the 2023 OpenStack User Survey. Please look them over and leave any comments on the etherpad: https://etherpad.opendev.org/p/oct2022-ptg-openstack-i18n-user-survey-quest… We need to get these to the Foundation quickly, so please leave your comments before 1200 UTC on Monday 24 October.

1 0

[glance] Slow image download when using glanceclient
by Lucio Seki 20 Oct '22

20 Oct '22

Hi glance experts, I'm using the following code to download a glance image: ``` from glanceapi import client ... glance = client.Client(GLANCE_API_VERSION, session=sess) ... with open(path, 'wb') as image_file: data = glance.images.data(image_id) for chunk in tqdm(data, unit='B', unit_scale=True, unit_divisor=1024): image_file.write(chunk) ``` And I get a speed around 3kB/s. It would take months to download an image. I'm using python3-glanceclient==3.6.0. I even tried: ``` for chunk in tqdm(data, unit='B', unit_scale=True, unit_divisor=1024): pass ``` to see if the bottleneck was the disk I/O, but didn't get any faster. In the same environment, when I use the glance CLI instead: ``` glance image-download --file $path $image_id ``` I get hundreds of MB/s download speed, and it finishes in a few minutes. Is there anything I can do to improve the glanceclient performance? I'm considering using subprocess.Popen(['glance', 'image-download', ...]) if nothing helps... Regards, Lucio

3 6

[all][tc] 2023.1 Antelope TC-PTG Planning
by Ghanshyam Mann 20 Oct '22

20 Oct '22

Hello Everyone, As you already know that the 2023.1 cycle virtual PTG will be held between Oct 17th - 21[1]. I have started the preparation for the Technical Committee PTG sessions. Please do the following: 1. Fill the below poll as per your availability. - https://framadate.org/yi8LNQaph5wrirks 2. Add the topics you would like to discuss to the below etherpad. - https://etherpad.opendev.org/p/tc-2023-1-ptg NOTE: this is not limited to TC members only; I would like all community members to fill the doodle poll and, add the topics you would like or want TC members to discuss in PTG. [1] https://lists.openstack.org/pipermail/openstack-discuss/2022-August/030041.… -gmann

2 4

[PTG][nova][neutron] Nova-Neutron cross-team meetings
by Rodolfo Alonso Hernandez 20 Oct '22

20 Oct '22

Hello all: The Nova-Neutron cross-team sessions today, starting at 13UTC, will be on Nova channel "bexar". In any case, I'll send an update in our IRC channel and I'll keep "mitaka" session open to redirect people to "bexar". Regards.

1 0

[dev][infra][tact-sig] Updating Zuul's default-ansible-version to 6
by Jeremy Stanley 19 Oct '22

19 Oct '22

Just a heads up for folks not following the OpenDev Collaboratory's service-announce mailing list... now that Zed is officially released, we'll be increasing the default Ansible version for Zuul jobs from 5 to 6 in preparation for Zuul to drop Ansible 5 support in coming weeks. See the full announcement here: https://lists.opendev.org/pipermail/service-announce/2022-October/000046.ht… -- Jeremy Stanley

2 1

[horizon] Cancelling Two Weekly Meeting
by vishal manchanda 19 Oct '22

19 Oct '22

Hello Team, As discussed in Yesterday's PTG meeting, there will be no horizon weekly meeting for today and next week. The next horizon weekly meeting will be on 2nd November. Thanks & Regards, Vishal Manchanda

1 0

DC DR Setup Queries
by KK CHN 19 Oct '22

19 Oct '22

List, We are having a client DC running on HP( HP simplivity) HCI servers, With VMware ( Vsphere 7.0) only few VMs running on it. (11 VMs maximum all Linux VMs). The DR site also having the same HCI setup another location. ( The VMs are replicated to DR site with HP simplivity). We are planning to use Openstack for both DC and DR solutions with Wallaby or Xena version with KVM as hypervisor to replace the proprietary S/W and H/W vendor locking. The requirement is to Setup a Stable DC- DR solution. Totally confused How to setup a best Dc- DR solution for this purpose. The DR setup can be possible / advisable with Zero down time ?( or manual DR site uping with downtime of hours ) ? What are the available/suggested DC-DR replication mechanisms for high degree of application data protection and service availability? Kindly advise.. Thanks in advance, Krish

2 2

[cloudkitty] Another core team cleanup
by Pierre Riteau 18 Oct '22

18 Oct '22

Hello, Almost exactly two years since the last core team cleanup [1], it's probably time to have another one. I don't think we have heard from these contributors in the last couple of years: Justin Ferrieu jferrieu(a)objectif-libre.com Luis Ramirez luis.ramirez(a)opencloud.es Luka Peschke mail(a)lukapeschke.com Maxime Cottret maxime.cottret(a)gmail.com Stéphane Albert sheeprine(a)nullplace.com Jeremy Liu liuj285(a)chinaunicom.cn Is everyone okay with removing cloudkitty-core membership for these users? Cheers, Pierre Riteau (priteau) [1] https://lists.openstack.org/pipermail/openstack-discuss/2020-October/017751…

2 3

[ovn][neutron] RE: OVN BGP Agent query
by Ihtisham ul Haq 18 Oct '22

18 Oct '22

Hi Luis and Daniel, Please see inline response. > From: Daniel Alvarez Sanchez <dalvarez(a)redhat.com> > Sent: 29 September 2022 11:37 > Subject: Re: OVN BGP Agent query > > Hi Ihtisham and Luis, > > On Thu, Sep 29, 2022 at 7:42 AM Luis Tomas Bolivar <ltomasbo(a)redhat.com> wrote: > > Some comments and questions inline > > > > On Tue, Sep 27, 2022 at 1:39 PM Ihtisham ul haq <ihtisham.uh(a)hotmail.com> wrote: > > > Hi Luis, > > > > > > Thanks for your work on the OVN BGP Agent. We are planning > > > to use it in our OVN deployment, but have a question regarding it. > > > > Great to hear! Can you share a bit more info about this environment? like > > openstack version, target workload, etc. We plan to use this with Yoga version. Our workload consist of enterprise users with VMs running on Openstack and connected to their enterprise network via transfer network(to which the customer neutron router is attached to). And we also have public workload but with the ovn-bgp we only want to we want to advertise the former. > > > > > The way our current setup with ML2/OVS works is that our customer VM IP routes > > > are announced via the router IP(of the that customer) to the leaf switch instead of > > > the IP of the host where the neutron BGP agent runs. And then even if the > > > router fails over, the IP of the router stays the same and thus the BGP route > > > doesn't need to be updated. > > > > Is this with Neutron Dynamic Routing? When you say Router IP, do you mean the virtual neutron router and its IP associated with the provider network? What type of IPs are you announcing with BGP? IPs on provider network or on tenant networks (or both)? Yes, that's with Neutron DR agent, and I meant virtual neutron router with IP from the provider network. We announce IPs of our tenant network via the virtual routers external address. > > If the router fails over, the route needs to be updated, doesn't it? Same IP, but exposed in the new location of the router? Correct. > The route to the tenant network doesn't change, ie. > 192.168.0.0 via 172.24.4.100 (this route remains the same regardless of where 172.24.4.100 is). > If there's L2 in the 172.24.4.0 network, the new location of 172.24.4.100 will be learnt via GARP announcement. In our case, this won't happen as we don't have L2 so we expose directly connected routes to overcome this "limitation". Right, in our case we have a stretched L2 transfer network(mentioned above) to which our gateway nodes and customer routers are connected to, so we can advertise the IPs from the tenant network via the virtual router external IP and thus the location of the router isn't relevant in case of failover as its address will be relearned. > In the case of Neutron Dynamic Routing, there's no assumption that everything is L3 so GARPs are needed to learn the new location. > > > > We see that the routes are announced by the ovn-bgp-agent via the host IP(GTW) in our > > > switch peers. If that's the case then how do you make sure that during failover > > > of a router, the BGP routes gets updated with the new host IP(where the router > > > failed over to)? > > > > The local FRR running at each node is in charge of exposing the IPs. For the IPs on the provider network, the traffic is directly exposed where the VMs are, without having to go through the virtual router, so a router failover won't change the route. > > In the case of VMs on tenant networks, the traffic is exposed on the node where the virtual router gateway port is associated (I suppose this is what you refer to with router IP). In the case of a failover the agent is in charge of making FRR to withdraw the exposed routes on the old node, and re-advertise them on the new router IP location > > > > > Can we accomplish the same route advertisement as our ML2/OVS setup, using the ovn-bgp-agent? > > I think this is technically possible, and perhaps you want to contribute that functionality or even help integrating the agent as a driver of Neutron Dynamic Routing? Sounds good, our plan currently is to add this to the ovn-bgp-agent, so we can announce our tenant routes via virtual routers external address on a stretched L2 network, to make it work with our use case. -- Ihtisham ul Haq Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für die Verwertung durch den vorgesehenen Empfänger bestimmt. Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den Absender bitte unverzüglich in Kenntnis und löschen diese E Mail. Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarz>.

3 5

[kolla] Monasca removal, Kafka and Zookeeper deprecation/removal
by Michał Nasiadka 18 Oct '22

18 Oct '22

Hello, As Kolla/Kolla-Ansible projects are implementing switch to OpenSearch - it became obvious to remove Elasticsearch (which is EOL in Kolla-provided version) and orchestrate a direct upgrade into OpenSearch. Part of this work is removal of elasticsearch and log stash Kolla container images and elasticsearch role in Kolla-Ansible. Since Monasca is relying on Elasticsearch and Logstash - and there are no contributors willing to make it work with OpenSearch inside Kolla/Kolla-Ansible - we decided to drop Monasca support in Zed release (especially that the CI jobs have been long failing). One of the Monasca dependencies is Kafka (and ZooKeeper) - is there anybody using it outside of Monasca deployment in Kolla/Kolla-Ansible? If in 4 weeks there will be no answers to this question - we will also remove Kafka and ZooKeeper container images in Kolla (and their respective roles in Kolla-Ansible). Thanks Michal

1 0

[cinder] cancelling this week's meeting (19 Oct)
by Rajat Dhasmana 18 Oct '22

18 Oct '22

Hello Argonauts, Since we've PTG this week, we will not be conducting the weekly meeting on 19th Oct 2022 but everyone is recommended to attend the PTG instead[1]. See you all at the PTG! [1] https://etherpad.opendev.org/p/antelope-ptg-cinder Thanks and regards Rajat Dhasmana

1 0

[kolla] Weekly meeting 20th Oct 2022
by Michał Nasiadka 18 Oct '22

18 Oct '22

Hello, Since this is PTG week - this week meeting is cancelled. Best regards, Michal

1 0

[tacker] Cancelling IRC meeting on Oct 18
by Yasufumi Ogawa 18 Oct '22

18 Oct '22

Hi team, Since we're going to have the vPTG session, I'd like to cancel IRC meeting today. Thanks, Yasufumi

1 0

[keystone] Cancelling weekly meeting for 10/18/22
by Dave Wilde 18 Oct '22

18 Oct '22

Cancelling due to the PTG

1 0

[ptls][tc] 2022 OpenStack User Survey Project Question Responses
by Allison Price 17 Oct '22

17 Oct '22

Hi everyone, Please find attached the responses to the project questions from the 2022 OpenStack User Survey. Based on feedback last year, I included additional, non-identifiable information that will hopefully help provident deployment context for the responses to your questions. If you need a reminder of your project question, you can review the OpenStack User Survey [1]. During the PTG, I would encourage you and your teams to review the responses and decide if you would like to make any changes to your question for the 2023 OpenStack User Survey. It is live now, but we can make changes ahead of significant promotion. Please reach out to me directly with any changes. If you have any questions on how to read the results, please let me know. Have a great week at the PTG! Cheers, Allison [1] https://www.openstack.org/usersurvey

2 4

[kolla][ptg] Meetings on Tuesday 18th Oct
by Michał Nasiadka 17 Oct '22

17 Oct '22

Hello Koalas, Due to exhaustion of topics - we’re meeting tomorrow only on the Kolla Operator Hour at 16 UTC. Kolla Operator Hour Etherpad: https://etherpad.opendev.org/p/oct2022-ptg-operator-hour-kolla Best regards, Michal Nasiadka

1 0

[manila] Cancelling two weekly meetings
by Carlos Silva 17 Oct '22

17 Oct '22

Hello, Zorillas! As agreed in the previous week, we will be cancelling the next two weekly meetings: from this week (Oct 20th) and next week (Oct 27th). The next weekly meeting for Manila will be on Nov 3rd. Thank you, carloss

1 0

[release] Release countdown for week R-22, Oct 17 - 21
by Előd Illés 17 Oct '22

17 Oct '22

Hi, Welcome back to the release countdown emails! These will be sent at major points in the 2023.1 Antelope development cycle, which should conclude with a final release on March 22nd, 2023. Development Focus ----------------- At this stage in the release cycle, focus should be on planning the 2023.1 Antelope development cycle, assessing 2023.1 Antelope community goals and approving 2023.1 Antelope specs. General Information ------------------- 2023.1 Antelope is a 24 weeks long development cycle. In case you haven't seen it yet, please take a look over the schedule for this release: https://releases.openstack.org/antelope/schedule.html By default, the team PTL is responsible for handling the release cycle and approving release requests. This task can (and probably should) be delegated to release liaisons. Now is a good time to review release liaison information for your team and make sure it is up to date: https://opendev.org/openstack/releases/src/branch/master/data/release_liais… By default, all your team deliverables from the Zed release are continued in 2023.1 Antelope with a similar release model. Upcoming Deadlines & Dates -------------------------- Virtual PTG: October 17-21, 2022 Antelope-1 milestone: November 17th, 2022 Előd Illés irc: elodilles @ #openstack-release

1 0

[openstack-helm] No Weekly IRC Meeting - PTG
by Gage Hugo 17 Oct '22

17 Oct '22

Hey team, This week is the PTG so the weekly IRC meeting is cancelled. We will be meeting for an hour at the same time slot for our PTG session: https://ptg.opendev.org/ptg.html

1 0

[0SSN-0090] Best practices when configuring Glance with COW backends
by Brian Rosmaita 17 Oct '22

17 Oct '22

Best practices when configuring Glance with COW backends --- ### Summary ### When deploying Glance in a popular configuration where Glance shares a common storage backend with Nova and/or Cinder, it is possible to open some known attack vectors by which malicious data modification can occur. This note reviews the known issues and suggests a Glance deployment configuration that can mitigate such attacks. ### Affected Services / Software ### Glance, all supported releases (Queens through Zed) ### Discussion ### This note applies to you if you are operating an end-user-facing glance-api service with the 'show_multiple_locations' option set to True (the default value is False) or if your end-user-facing glance-api has the 'show_image_direct_url' option set to True (default value is False). Our recommendation is that the image "locations" and "direct_url" fields [0] *never* be displayed to end users in a cloud. This can be accomplished by running two glance-api services: - A "user-facing" glance-api that is accessible to end users and which appears in users' service catalogs. - An "internal-only-facing" glance-api that is accessible only to those services that require access to the 'direct_url' or image location fields, and which is protected by firewalls from access by end users. (Nova, Cinder, and Ironic all have configuration options to specify the Glance API endpoint each service uses [1].) This dual glance-api deployment was suggested in "Known Issues" in Glance release notes in the Rocky [2] through Ussuri releases, but it seems that the idea has not received sufficient attention. Hence this security note. The attack vector that becomes available when image locations are exposed to end users was originally outlined in OSSN-0065 [3], though that note was not clear about the attack surface or mitigation, and contained some forward-looking statements that were not fulfilled. The attack vector is: [A] malicious user could create an image in Glance, set an additional location on that image pointing to an altered image, then delete the original location, so that consumers of the original image would unwittingly be using the malicious image. Note, however, that this attack vector cannot change the original image's checksum, and it is limited to images that are owned by the attacker. OSSN-0065 suggests that this is only an issue when users do not checksum their image data. It neglects the fact that in some popular deployment configurations in which Nova creates a root disk snapshot, data is never uploaded to Glance, but instead a snapshot is created directly in the backend and Nova creates a Glance image record with "size" 0 and an empty "os_hash_value" [4], making it impossible to compare the hash of downloaded image data to the value maintained by Glance. Further, when Nova is configured to use the same storage for ephemeral disks that is used as a Glance image store, Nova efficiently creates a server root disk directly in the backend without checksumming the image data. This is an intentional design choice to optimize storage space and host resources, but an implication is that even if the image record has a recorded hash, it is not being checked at the point of image consumption. Similarly, when using a shared backend, or a cinder glance_store, Cinder will efficiently clone a volume created from an image directly in the backend without checksumming the image data. Again, this is done intentionally in order to optimize resources, but it is important to be aware of the security tradeoff being made by this configuration. In other words, if the image data is not going to be checked at the point of image consumption, then extra care needs to be taken to ensure the integrity of the data path. OSSN-0065 suggested that the attack vector of substituting image data by modifying the image locations could be addressed by using policies, but that has turned out not to be the case. The only way currently to mitigate this vector is to deploy glance-api in a dual configuration as described above, namely with an internal-only-facing glance-api used by Nova and Cinder (that has show_multiple_locations enabled), and an end-user-facing glance-api (that has show_multiple_locations disabled). So far the focus has been on 'show_multiple_locations'. When that setting is disabled in Glance, it is not possible to manipulate the locations via the OpenStack Images API. Keep in mind, however, that in any Glance/Nova/Cinder configuration where Nova and/or Cinder do copy-on-write directly in the image store, image data transfer takes place outside Glance's image data download path, and hence the os_hash_value is *not* checked. Thus, if the backend store is itself compromised and image data is replaced directly in the backend, the substitution will *not* be detected. This brings us to the 'show_image_direct_url' option, which includes a "direct_url" field in the image-show response that can be used by various OpenStack services to consume images directly from the storage backend. Exposing the 'direct_url' to end users leaks information about the storage backend. What exactly that information consists of depends upon the backend in use and how it is configured, but in general, it is not a good idea to provide hints that could be useful to malicious actors in their attempts to compromise the backend storage by some type of independent exploit. The 'direct_url', being read-only, may appear innocuous, but its use by services is usually to perform some kind of optimized image data access that most likely does not include computing a hash of the image data. We therefore recommend that OpenStack services that require exposure of the 'direct_url' image property be similarly configured to use an internal-only-facing glance-api. It is worth nothing that end users who wish to download an image do not require access to the 'direct_url' image property because they can simply use the image-data-download API call [5]. ### Recommended Actions ### A glance-api service with 'show_multiple_locations' enabled should *never* be exposed directly to end users. This setting should only be enabled on an internal-only-facing glance-api that is used by OpenStack services that require access to image locations. This could be done, for example, by running two glance-api services with different configuration files and using the appropriate configuration options for each service to specify the Image API endpoint to access, and making sure the special internal endpoint is firewalled in such a way that only the appropriate OpenStack services can contact it. Similarly, enabling 'show_image_direct_url' exposes information about the storage backend that could be of use to malicious actors in as yet unknown exploits, so it should likewise only be enabled on an internal-only-facing glance-api. ### Notes / References ### [0] https://docs.openstack.org/api-ref/image/v2/index.html#show-image-schema [1] Nova and Ironic use 'endpoint_override' in the '[glance]' section of the configuration file; Cinder uses 'glance_api_servers' in the '[DEFAULT]' section. [2] OSSN-0065: https://wiki.openstack.org/wiki/OSSN/OSSN-0065 [3] The Glance "multihash" metadata pair of 'os_hash_algo' and 'os_hash_value' were introduced in Rocky to replace the legacy md5 'checksum' field. The python-glanceclient has used multihash checksumming for download verification since version 2.13.0. [4] https://docs.openstack.org/releasenotes/glance/rocky.html#known-issues [5] https://docs.openstack.org/api-ref/image/v2/index.html?#download-binary-ima… ### Contacts / References ### Author: Brian Rosmaita, Red Hat This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0090 Original LaunchPad Bug : https://bugs.launchpad.net/ossn/+bug/1990157 Mailing List : [Security] tag on openstack-discuss(a)lists.openstack.org OpenStack Security Project : https://launchpad.net/~openstack-ossg CVE: none

1 0

[swift] upgrade newton to train in containers
by Marcel 17 Oct '22

17 Oct '22

I'm planning an upgrade for our newton based swift cluster (OOO vm based) to a (kolla image based containers) train cluster So far I have tested the upgrade for the proxies, account_servers and container servers and it looks promising. I have in a test environment: - switched to the new proxies with the old ring files and it looks like everything works normally - Added the new (train) account and container servers to the rings and it looks like all is fine - Removed the old account and container servers, still fine - tested fall back, also fine My question actually is: Did the account and container database format change between newton and train in such a way that I might run into troubles trying the upgrade as tested above in ways that I did not yet foresee? Thanks Marcel

1 0

[Neutron] Bug Deputy Report Oct 10 - 17
by Jakub Libosvar 17 Oct '22

17 Oct '22

Hi all, I was the bug deputy last week, here is the report. Nothing that requires immediate attention. Medium ------ - [ovn-octavia-provider] Detach OVN-LB LS from the LR breaks OVN-LB connectivity https://bugs.launchpad.net/neutron/+bug/1992363 assigned to Luis Fix proposed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/860781 - [scale] Setting a gateway on router is killing database https://bugs.launchpad.net/neutron/+bug/1992950 Assigned to Arnaud Morin Fix proposed: https://review.opendev.org/c/openstack/neutron/+/861322 - Disable in-band flow management in the ovs bridges https://bugs.launchpad.net/neutron/+bug/1992953 Assigned to Slawek Fix proposed: https://review.opendev.org/c/openstack/neutron/+/861351 Wishlist ------- - fip will loss when it migrate between dvr-sant agent and dvr_no_external in Rocky https://bugs.launchpad.net/neutron/+bug/1992542 The reported bug uses unsupported configuration dvr_snat agent mode on the compute nodes. Cheers Kuba

1 0

[kolla][ptg] Using Meetpad instead of Zoom
by Michał Nasiadka 17 Oct '22

17 Oct '22

Hello, If you’re planning on joining the Kolla PTG - please use Meetpad - https://meetpad.opendev.org/kolla-antelope-ptg Sorry for last minute changes Best regards, Michal

1 0

[ptg][nova][placement] Nova/Placement at the PTG, howto
by Sylvain Bauza 17 Oct '22

17 Oct '22

Hey stackers, Got some questions about how/when joining the Nova community at the PTG during this week, so I'll provide you a howto ;) *What is the main document for the Nova PTG ?* This : https://etherpad.opendev.org/p/nova-antelope-ptg Please, please, don't try to translate this etherpad directly as you'll see. *I'm an operator and I want to tell you how terrible you folks are* There : https://etherpad.opendev.org/p/oct2022-ptg-operator-hour-nova Again, don't try to translate this document or you'll create problems for people who don't know your language. *When do you folks discuss ?* Monday : 14:00-15:00 UTC Ironic/Nova cross-project session Tuesday : 13:00 - 15:00 UTC Operator hours. If you are an operator, please join us ! :-) 15:00 - 17:00 UTC Nova sessions for contributors 15:10 UTC (possibly) : Nova team photo Wednesday : 13:00 - 16:00 UTC Nova sessions for contributors 16:00 - 17:00 Operator hour. If you are an operator, please join us ! :-) Thursday : 13:00 - 15:00 UTC Neutron/Nova cross-project session 15:00 - 17:00 UTC Nova sessions for contributors Friday : 13:00 - 17:00 UTC Nova sessions for contributors *How to join ?* Simple. You can look at https://ptg.opendev.org/ptg.html if you want so you'll see what topic we are currently discussing. If you want to arrive, just click either on "operator-hour" nova tag or "nova-placement" tag and it will automatically open the Google Meet client for joining the Nova meeting. If you also want, you can directly join us by using this link https://www.openstack.org/ptg/rooms/bexar (we'll use the same virtual room for all our sessions) *What if you can't join the whole days ?* No worries ! Just make sure you use an IRC client and just go to the #openstack-nova channel. Once you're there, add your IRC in the etherpad for every topic you'd like to be around and then I'll ping you when we start discussing for those topics you added your nick. *What if I want to add some topic I'd like to discuss ?* Easy peasy. Two possibilities : - you are a contributor and you want to add a new feature in nova. Just add your topic in the main etherpad (link above). Make sure your name is marked in the topic title (like every other topic) and I'll ping you when we start discussing your topic. - you are an operator and you want to discuss wishlists or bugs or whatever else. Good news, we have specific timeslots for you ! See the agenda above, we have *operator hours* that are intended for operators to join and engage on specific deployment and operational topics. Just then look at https://etherpad.opendev.org/p/oct2022-ptg-operator-hour-nova and add your points, we'll discuss them during the operator hours (given above). Hope you'll find your way and you'll appreciate the PTG as much productive as we do. In any case, my IRC nick is bauzas. Don't hesitate to bug me over IRC (#openstack-dev, #openstack-nova or #openinfra-events) or reach me by email (sbauza(a)redhat.com) I'll try to answer you as much as I can. HTH and thanks, -Sylvain

1 0