June 2023 - openstack-discuss - lists.openstack.org

[ceilometer][cinder] How to properly set up Block Storage meters / cinder-volume-usage-audit
by Christian Rohmann 21 Jun '23

21 Jun '23

Hello OpenStack-Discuss, when setting up telemetry (ceilometer, gnocci, aodh) I followed the documentation at https://docs.openstack.org/ceilometer/latest/install/cinder/install-cinder-… to set up sending metrics for cinder to ceilomter. 1) The instruction to run cinder-volume-usage-audit as a timer / cron every 5 minutes > */5 * * * * /path/to/cinder-volume-usage-audit --send_actions has me confused. This would cause the usage data for the last full month, to be sent to the notifications queue every 5 minutes. There is no progress tracker or pointer limiting this to only sent data that has not yet been sent. Looking at how openstack-ansible configures this (https://github.com/openstack/openstack-ansible-os_cinder/blob/1af3003e163e0…) it seems they were aware that running this without bounds and every 5 minutes is wrong. I also opened a bug (https://bugs.launchpad.net/ceilometer/+bug/2024475) about this. 2) Extending on 1) I believe setting the > volume_usage_audit_period in the cinder configuration, to e.g. month or hour, will already set the interval for which the data is emitted. Then the important thing is to only run cinder-volume-usage-audit in the same frequency. 3) Looking at the "--send_actions" options, I am wondering if the "create/delete" actions for volumes, snapshosts and backups are not sent by cinder anyways and why they would need to be sent again in batch and by cinder-volume-usage-audit? I am referring to > [oslo_messaging_notifications] > driver = messagingv2 > being enabled. This might also be the reasoning behind this bug, https://bugs.launchpad.net/openstack-ansible/+bug/1968734, asking for this parameter to be behind a config switch to avoid duplicate actions to be sent. 4) Thinking about the whole setup more questions arise * What happens when I run this more than once in the same interval? What will ceilometer do with the (redundant) data? * I am also wondering how well this single piece of software scales and how this could be split up more? * Is there no way to implement a pointer to allow running this command more often and only send "new data"? Just like the timestamp up to which data was sent. * What about any locking / coordination to enable multiple instances? Regards Christian

1 0

[ceilometer][gnocchi][telemetry] (User) frontend or tool to create reports on telemetry data (GPU, RAM, disk, network, ...)
by Christian Rohmann 21 Jun '23

21 Jun '23

Hello OpenStack-Discuss, I was wondering if there are any simple frontends or similar solutions to provide ceilometer's telemetry data, such as CPU, RAM, volume throughput and IOPs, network throughput, ... to users? Something like a basic horizon dashboard? Or does anybody have written something using the API to create some nice graphs? I know there is Cloudkitty, but that looks more targeted towards doing rating and billing with this kind of data? Thanks and with regards Christian

3 2

HOT template help
by Antony P 21 Jun '23

21 Jun '23

Hi team, This is user data jenkins_password=$(sudo cat /var/lib/jenkins/secrets/initialAdminPassword) i want to pass this as output like Output: value: password How can i resolve this? *--Warm Regards* *Antony P* *Jr. DevOps Engineer* antony(a)edgebricks.com Mob No: +919498079898 www.edgebricks.com

1 0

Cinder Bug Report 2023-06-21
by Sofia Enriquez 21 Jun '23

21 Jun '23

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> *Wishlist* - Restore RBD Incremental Backup Fails When Using the --container Argument. <https://bugs.launchpad.net/cinder/+bug/2024484> - *Status*: Fix proposed to master. <https://review.opendev.org/c/openstack/cinder/+/788319> *Low* - Hitachi: wrong exception when deleted volume is busy. <https://bugs.launchpad.net/cinder/+bug/2024418> - *Status*: Fix proposed to master <https://review.opendev.org/c/openstack/cinder/+/885986>. Cheers, -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

versioning hidden container accounting
by Geert Geurts 21 Jun '23

21 Jun '23

Hello I've implemented the newer type of versioning (x-versions-enabled: True) for a container. The documentation (https://docs.openstack.org/swift/latest/middleware.html#object-versioning) states that actual data is written to a hidden container. Now my question is would it be possible to set the account for the data of the current ("is_latest": true) objects to a different account? Would it be possible to set headers on the event that 'current' changes? Can I see/change the properties of this "hidden" container? Thanks allot for any help/pointers! Best regards, Geert

1 0

[ironic] [networking-generic-switch] NGS performance and moving to an agent-based design
by Baptiste Jonglez 20 Jun '23

20 Jun '23

Hello, We have been hitting performance issues with networking-generic-switch (NGS), the Neutron ML2 plugin that performs dynamic reconfiguration of physical switches, most notably used by Ironic. In our quest to improve NGS performance, we have designed and implemented an agent-based system for NGS, and we would now like to upstream it. Since this is a fairly big change, and some parts of the design such as HA are not completely clear yet, feel free to join the discussion on the RFE: https://bugs.launchpad.net/networking-generic-switch/+bug/2024385 Regards, Baptiste -- Baptiste Jonglez Research Engineer, Inria <https://www.inria.fr/> STACK team <https://stack-research-group.gitlabpages.inria.fr/web/>

2 1

[kolla] nova-compute image build failed with invalid signature
by Satish Patel 20 Jun '23

20 Jun '23

Folks, Just wanted to check with folks if I did anything wrong here. Building 2023.1 image of nova-compute and encounter errors. root@docker-reg:/opt/kolla/etc/kolla# kolla-build --registry docker-reg:4000 --config-file kolla-build.conf --debug --cache --threads 1 --skip-existing --push --format none nova-compute INFO:kolla.common.utils:Using engine: docker INFO:kolla.common.utils:Found the container image folder at /usr/local/share/kolla/docker INFO:kolla.common.utils:Added image nova-compute-ironic to queue INFO:kolla.common.utils:Added image nova-compute to queue INFO:kolla.common.utils:Attempt number: 1 to run task: BuildTask(nova-compute-ironic) DEBUG:kolla.common.utils.nova-compute-ironic:Processing INFO:kolla.common.utils.nova-compute-ironic:Building started at 2023-06-20 17:36:33.771585 DEBUG:kolla.common.utils.nova-compute-ironic:Turned 0 plugins into plugins archive DEBUG:kolla.common.utils.nova-compute-ironic:Turned 0 additions into additions archive INFO:kolla.common.utils.nova-compute-ironic:Step 1/4 : FROM docker-reg:4000/kolla/nova-base:2023.1 INFO:kolla.common.utils.nova-compute-ironic: ---> 0ebe71c922b4 INFO:kolla.common.utils.nova-compute-ironic:Step 2/4 : LABEL maintainer="Kolla Project (https://launchpad.net/kolla)" name="nova-compute-ironic" build-date="20230620" INFO:kolla.common.utils.nova-compute-ironic: ---> Using cache INFO:kolla.common.utils.nova-compute-ironic: ---> 5209e0cc6403 INFO:kolla.common.utils.nova-compute-ironic:Step 3/4 : RUN apt-get --error-on=any update && apt-get -y install --no-install-recommends genisoimage nvme-cli && apt-get clean && rm -rf /var/lib/apt/lists/* INFO:kolla.common.utils.nova-compute-ironic: ---> Running in 196bc3715e45 INFO:kolla.common.utils.nova-compute-ironic:Get:1 http://ubuntu-cloud.archive.canonical.com/ubuntu jammy-updates/antelope InRelease [5,463 B] INFO:kolla.common.utils.nova-compute-ironic:Get:2 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [108 kB] INFO:kolla.common.utils.nova-compute-ironic:Err:1 http://ubuntu-cloud.archive.canonical.com/ubuntu jammy-updates/antelope InRelease INFO:kolla.common.utils.nova-compute-ironic: At least one invalid signature was encountered. INFO:kolla.common.utils.nova-compute-ironic:Get:3 http://mirrors.ubuntu.com/mirrors.txt Mirrorlist [3,463 B] INFO:kolla.common.utils.nova-compute-ironic:Get:6 http://archive.ubuntu.com/ubuntu jammy-security InRelease [110 kB] INFO:kolla.common.utils.nova-compute-ironic:Err:2 http://archive.ubuntu.com/ubuntu jammy-backports InRelease INFO:kolla.common.utils.nova-compute-ironic: At least one invalid signature was encountered. INFO:kolla.common.utils.nova-compute-ironic:Get:5 https://mirror.dal.nexril.net/ubuntu jammy-updates InRelease [119 kB] INFO:kolla.common.utils.nova-compute-ironic:Err:6 http://archive.ubuntu.com/ubuntu jammy-security InRelease INFO:kolla.common.utils.nova-compute-ironic: At least one invalid signature was encountered. INFO:kolla.common.utils.nova-compute-ironic:Get:4 http://ftp.usf.edu/pub/ubuntu jammy InRelease [270 kB] INFO:kolla.common.utils.nova-compute-ironic:Err:5 https://mirror.dal.nexril.net/ubuntu jammy-updates InRelease INFO:kolla.common.utils.nova-compute-ironic: At least one invalid signature was encountered. INFO:kolla.common.utils.nova-compute-ironic:Err:4 http://ftp.usf.edu/pub/ubuntu jammy InRelease INFO:kolla.common.utils.nova-compute-ironic: At least one invalid signature was encountered. INFO:kolla.common.utils.nova-compute-ironic:Reading package lists... INFO:kolla.common.utils.nova-compute-ironic:W: GPG error: http://ubuntu-cloud.archive.canonical.com/ubuntu jammy-updates/antelope InRelease: At least one invalid signature was encountered. INFO:kolla.common.utils.nova-compute-ironic:E: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu jammy-updates/antelope InRelease' is not signed. INFO:kolla.common.utils.nova-compute-ironic:W: GPG error: http://archive.ubuntu.com/ubuntu jammy-backports InRelease: At least one invalid signature was encountered. INFO:kolla.common.utils.nova-compute-ironic:E: The repository 'http://archive.ubuntu.com/ubuntu jammy-backports InRelease' is not signed. INFO:kolla.common.utils.nova-compute-ironic:W: GPG error: http://archive.ubuntu.com/ubuntu jammy-security InRelease: At least one invalid signature was encountered. INFO:kolla.common.utils.nova-compute-ironic:E: The repository 'mirror://mirrors.ubuntu.com/mirrors.txt jammy-security InRelease' is not signed. INFO:kolla.common.utils.nova-compute-ironic:W: GPG error: https://mirror.dal.nexril.net/ubuntu jammy-updates InRelease: At least one invalid signature was encountered. INFO:kolla.common.utils.nova-compute-ironic:E: The repository 'mirror://mirrors.ubuntu.com/mirrors.txt jammy-updates InRelease' is not signed. INFO:kolla.common.utils.nova-compute-ironic:W: GPG error: http://ftp.usf.edu/pub/ubuntu jammy InRelease: At least one invalid signature was encountered. INFO:kolla.common.utils.nova-compute-ironic:E: The repository 'mirror://mirrors.ubuntu.com/mirrors.txt jammy InRelease' is not signed. INFO:kolla.common.utils.nova-compute-ironic: INFO:kolla.common.utils.nova-compute-ironic:Removing intermediate container 196bc3715e45 ERROR:kolla.common.utils.nova-compute-ironic:Error'd with the following message ERROR:kolla.common.utils.nova-compute-ironic:The command '/bin/sh -c apt-get --error-on=any update && apt-get -y install --no-install-recommends genisoimage nvme-cli && apt-get clean && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100 INFO:kolla.common.utils:Attempt number: 2 to run task: BuildTask(nova-compute-ironic) DEBUG:kolla.common.utils.nova-compute-ironic:Processing Full trace - https://paste.opendev.org/show/b3IYgcUXsvg0kxVpEaLM/

2 2

[neutron][ptg] PTG summary
by Rodolfo Alonso Hernandez 20 Jun '23

20 Jun '23

Hello all: Thank you for joining us during the Forum session and the PTG slots. It was very productive and we were able to check the pulse of OpenStack and Neutron in particular. This time both the Forum session and the PTG were attended mainly by operators and users; the number of developers was reduced and this is something we need to improve. A healthy community is nourished by the active participation of their members, not only pushing new code but reviewing and helping fixing the new bugs. === Forum session === That was a good surprise for me to find so many people in the room. I wasn't expecting that after checking the etherpad link provided before the meeting. During this short session, focused on issues/pain points of Neutron, we could find out that: * The ML2/OVN backend is increasing its adoption rate. * Most users have deployments between Wallaby and Antelope. * Hardware offload is increasing its demand due to the network BW increasing requirements. * BGP/eVPN is deployed in most of the environments. >From this session I would highlight some lessons learned for next events: * If the Forum session time slot is reduced (30 mins), the attendant interventions should be limited to 2-3 minutes, enough to describe the issue. Any further discussion should be taken in the PTG sessions. * Request attendance "checkin" in the etherpad, to be prepared for the event. * Request people to add their questions in the etherpad in advance. === PTG === We didn't have technical sessions but many questions about issues and possible bugs. These are the most relevant ones: *** OVN L3 scheduler issue *** This issue has been reproduced in an environment with more than 5 chassis with gateway ports. The router GW ports are assigned to the GW chassis using a manual scheduler implemented in Neutron (the default one is ``OVNGatewayLeastLoadedScheduler``). If one of the chassis is stopped, the GW ports should be re-assigned to the other GW chassis. This is happening but all ports fall under the same one; this re-scheduling should share the ports among the other active chassis. * Action item: I'll open a LP bug and investigate this issue. *** Size of the OVN SB "HA_Chassis_Group" table *** The OVN SB "HA_Chassis_Group" increases its size indefinitely with each operation creating a router and assigning a new external gateway (external network). This table never decreases, * Action item: I'll open a LP bug, investigate this issue and if this is a core OVN issue, report it. *** Live migration with ML2/OVN *** This is a common topic and not only for ML2/OVN. The migration time has many factors (memory size, applications running, network BW, etc) that could slow down the migration time and trigger a communication gap during this process. * Action item: to create better documentation, both in Nova and Neutron, about the migration process, what has been done to improve it (for example, the OVN multiple port binding) and what factors will affect the migration. *** ML2/OVN IPv6 DVR *** This spec was approved during the last cycle [1]. The implementation [2] is under review. * Action item: to review the patch (for Neutron reviewers) * Action item: to implement the necessary tempest tests (for the feature developers) *** BGP with ML2/OVS, exposing address blocks *** This user has successfully deployed Neutron with ML2/OVS and n-d-r. This user is currently making public a certain set of FIPs. However, for other VMs without FIPs, the goal is to make the router GW port IP address public, using the address blocks functionality; this is not working according to the user. * Action item: (for this user) to create a LP bug describing the architecture of the deployment, the configuration used and the API commands used to reproduce this issue. *** Metadata service (any backend) *** Neutron is in charge of deploying the Metadata service on the compute nodes. Each time the metadata HTTP server is called, it requests from the Neutron API the instance and tenant ID [3]. This method implies a RPC call. In "busy" compute nodes, where the VMs are created and destroyed very fast, this RPC communication is a bottleneck. * Action item: open a LP bug to implement the same ``CacheBackedPluginApi`` used in the OVS agent. This RPC cached class creates a set of subscriptions to the needed resources ("ports" in this case). The Neutron API will send the port updated info and cached locally; that makes unnecessary the RPC request if the resources are stored locally. *** ML2/OVN + Ironic nodes *** This user has deployed ML2/OVN with Ironic nodes, and is using ovn-bgp-agent with the eVPN driver to make public the private ports (IP and MACs) to the Ironic node ports. More information in [4]. *** BGP acceleration in ML2/OVN *** Many questions related to this topic, both with DPDK and HW offload. I would refer (once the link is available) to the talk "Enabling multi-cluster connectivity using dynamic routing via BGP in Openstack" given by Christophe Fontaine during this PTG. You'll find it very interesting how this new implementation moves all the packet processing to the OVS datapath (removing any Linux Bridge / iptables processing). The example provided in the talk refers to the use of DPDK. I hope this PTG was interesting for you! Don't hesitate to use the usual channels that are the mailing list and IRC. Remember we have the weekly Neutron meeting every Tuesday at 1400UTC. Regards. [1] https://specs.openstack.org/openstack/neutron-specs/specs/2023.1/ovn-ipv6-d… [2]https://review.opendev.org/c/openstack/neutron/+/867513 [3] https://github.com/openstack/neutron/blob/cbb89fdb1414a1b3a8e8b3a9a4154ef62… [4]https://ltomasbo.wordpress.com/2021/06/25/openstack-networking-with-evpn/

3 6

Help Needed: Configuring OpenStack Glance with VMware
by Anupam Panta 20 Jun '23

20 Jun '23

Dear OpenStack Community, I hope this email finds you well. I am reaching out to seek assistance and guidance regarding the configuration of OpenStack integration with VMware. I have encountered some difficulties while trying to configure Glance, and I would greatly appreciate any help or insights you can provide. Here is a brief overview of the issue I am facing: * Configuration Challenge: I am currently working on integrating OpenStack with VMware as the backend infrastructure for image storage and management. * Specific Problem: While configuring Glance, I am stuck with an error message stating "Store for identifier vmware not found". * Configuration Attempts: I have reviewed the relevant documentation and followed the recommended steps, tried changing parameters to configure the VMware store backend in Glance. However, despite my efforts, the error persists. I kindly request your guidance and expertise to help me troubleshoot and resolve this issue. Any advice, best practices, or configuration examples related to Glance integration with VMware would be highly appreciated. To provide you with more context, here are some relevant details of my environment: I am using kolla installation for openstack deployment. My global.yml file consists of: ``` #### GLANCE CONFIG glance_backend_vmware: "yes" glance_backend_insecure: "yes" #### VMWARE CONFIG vmware_vcenter_host_ip: <my-vcenter-ip-address> vmware_vcenter_host_username: <my-vcenter-admin-username> vmware_vcenter_host_password: <my-vcenter-admin-password> vmware_datastore_name: glance-datastore vmware_vcenter_name: <my-vcenter-domain-name> vmware_vcenter_cluster_name: "openstack" vmware_vcenter_datacenter_name: <my-vcenter-datacenter-name> vmware_vcenter_insecure: "True" ``` After running reconfigure command, my glance-api container stuck in restart loop, with error message "Store for identifier vmware not found". I also tried changing parameters in glance-api.conf file, specific parameters `default_backend= vsphere`, `stores= file, http, swift, vmware`. Your assistance will not only help me but also contribute to the collective knowledge of the community. Thank you in advance for your time and support. I look forward to your valuable insights and suggestions. Regards, Anupam Panta | LinkedIn<http://www.linkedin.com/in/anupam-panta> System Engineer anupam.panta(a)f1soft.com 9802079915 Movers & Shakes Tower, Pulchowk, Nepal [cid:78d40d57-4926-472e-bbe6-ea05244f879f] <https://f1soft.com/> Disclaimer This e-mail and any attachments may contain confidential and privileged information. If you are not the intended recipient, please notify the sender immediately by return e-mail, followed by deleting this e-mail and destroying any copies thereof Any dissemination or use of this information by a person other than the intended recipient is unauthorized and may be illegal unless consented by the sender. F1soft International Private Limited or its employees are not responsible for any auto-generated spurious messages that you may receive from F1soft email addresses.

2 1

[cinder] in-use volume retype error "Expecting to find domain in project"
by Eugen Block 20 Jun '23

20 Jun '23

Hi, while trying to reproduce the issue mentioned in [1] I stumbled upon [2], I commented there but because of the "wishlist" importance I don't expect too much attention. I have a production cloud with Victoria where I see the same issue as described in [2], trying to retype an in-use volume results in a stack trace in cinder: 2023-06-20 12:45:52.900 1085571 ERROR oslo_messaging.rpc.server keystoneauth1.exceptions.http.BadRequest: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-6b412e96-a735-4b94-a5e6-8252dd949297) I have also a test cloud installed the same way as the production cloud with V and upgraded to Wallaby where this stack trace does not occur, retyping an in-use volume just works there. I tried the described workaround in [1] and added a [nova] section to the cinder.conf, and that worked. There is no such section in the W cloud. I checked the Wallaby release notes and the only thing related to retype with ceph as backend is [3]. Could that make the difference here? One difference between V and W I noticed was that in V I see two volumes with the same name (but a different UUID, of course) during the retype process while in W I see only one volume at all times. So I assume that W handles the retype differently and it doesn't require a nova section. I'd appreciate any comment. Thanks, Eugen [1] https://lists.openstack.org/pipermail/openstack-discuss/2023-June/034160.ht… [2] https://bugs.launchpad.net/cinder/+bug/1941062 [3] https://bugs.launchpad.net/cinder/+bug/1886543

1 0