October 2023 - openstack-discuss - lists.openstack.org

Regarding Nova's inability to delete the Cinder volume for creating virtual machines (version Y)
by 无数的星球 30 Oct '23

30 Oct '23

When creating a virtual machine in the dashboard, create a volume and choose to delete the virtual machine while also deleting the volume. When deleting the virtual machine, there is no normal uninstallation of the volume and the volume is not deleted. The relevant error logs are shown in the image, but the openstack CLI can delete its volume. The specific commands are as follows. CLI： source /etc/keystone/admin-openrc.sh (Verify password file) openstack volume set --detached　191e555c-3947-4928-be46-9f09e2190877（volumeID） openstack volume delete 191e555c-3947-4928-be46-9f09e2190877（volumeID） It seems that Nova is unable to interact with the Cinder API to delete(or detached) commands, but I am not very professional. I don't know if it's a bug? Note that I have seen similar bug patch files, but they are invalid. URL:   BUG page   https://bugs.launchpad.net/nova/+bug/2016173   Page for patching files   https://review.opendev.org/c/openstack/nova/+/880399 Is not working!!!

2 1

[openstack-ansible][ptg] OpenStack-Ansible 2024.1 (Caracal) vPTG Summary
by Dmitriy Rabotyagov 30 Oct '23

30 Oct '23

Hey everyone, We had quite a productive discussion on the PTG and I will provide a summary of them below. It was divided into multiple sections, like things we need to finish for 2023.2 (Bobcat) - since OpenStack-Ansible has trailing release model we have some time to wrap things up; plans for 2024.1 (Caracal) and some things that moved out of scope until better times. You can also find original PTG with discussions here [1] Finalizing 2023.1 (Bobcat): * Documentation on using pretty endpoint names (ie identity.cloud.org instead of cloud.org:5000). For 2024.1 we will try to simplify the way of using such naming to make it more convenient for end-users. * Finalize support of RabbitMQ Quorum queues for all services [2]. With that they will be disabled by default at least for 2023.2 release and we will revise the defaults for 2024.1 * Implement openstack_resources common rolem that will handle different resources creation, like OpenStack users, projects, networks, images, aggregates, etc. It will be also used in roles like Tempest, Octavia, Trove and Magnum to create such resources, instead of each role implementing its own flow of doing that. We agreed to add a separate playbook that will allow deployers to leverage the role and run it after setup-openstack. * Debian 12 support. Most patches have landed, but we also need to reflect it's support in documentation * Review and release Skyline role. Tough discussion was about defaults and how Skyline should co-exist with Horizon. As a result we decided to deploy both options, when both are defined in the inventory (as usual), but the playbook that is run last will basically configure HAProxy backends to itself, fully neglecting other available services. So if you want to have both Skyline and Horizon running and functional - you will need to apply some overrides for that. There are also no guarantees that we will be able to merge and release the role in time, so we may end up with making it as part of OSA only for 2024.1. Scoped improvements for 2024.1 (Caracal): * Adoption of PKI role to be used with PIPE modules, that will allow the role to be pointed to any file (like encrypted with ansible-vault) or external storage that supports lookup, instead of plain text files on a deploy host. With that we will also take a look at using Hishcorp Vault as a PKI source and adding it as another driver in addition to the "standalone" one. * Natively supporting multi-AZ deployments for Neutron. At the moment for OVS/LXB you need to define some overrides to make that working, while for OVN you would need to manually execute a bunch of commands. POC patch has been already proposed [3], but it still needs some love. * Finalize support of OVN driver for Octavia. Initial patch has been proposed, but it also needs some love, despite it works in general [4] * Resume work on ProxySQL implementation as a replacement for HAProxy balancing for MariaDB deployment. That should improve load distribution between MariaDB backends. * Allow to select OS for LXC containers. At the moment we're forcing to use the host OS, while in some cases it might be preferred to use a different one, like during OS upgrades or switching the OpenStack deployment from one distro to another. * With that we are excited about incus (LXD fork) having the first release and it being packaged with traditional packages (only .deb at the moment). This opens the door for implementation of incus as a manager for LXC containers, that may simplify our flows and day2 operations significantly. * Invest some time into decreasing the runtime, both for CI/CD and for end-user deployments. * In case improvements for Quorum Queues will land to the oslo.messaging [5] - add a set of variables to roles for nicely controlling the behaviour. [1] https://etherpad.opendev.org/p/osa-caracal-ptg [2] https://review.opendev.org/q/topic:osa%252Fquorum_queues [3] https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/858271 [4] https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/868462 [5] https://review.opendev.org/q/topic:bug-2031497

1 0

[Glance] Caracal PTG Summary
by Pranali Deore 30 Oct '23

30 Oct '23

Hello Everyone, We had our eighth virtual PTG between 23rd OCt to 26th OCT 2023. Thanks to everyone who joined the virtual PTG sessions. Using Opendev Meetpad we had all the discussion around different topics for glance. I have created etherpad[1] with notes from the session and also included the recordings of each session. Here is the short summary of the discussions. *Tuesday , OCT 24th 2023* # Bobcat Retrospective On a positive note we have completed few important things like adding support for weighting mechanism to store, support for RBD driver to move images to the trash if they cannot be deleted immediately due to having snapshots, putting efforts in bridging the gap between glanceclient and openstackclient and started new location API work which will replace the image-update mechanism for consumers like cinder and nova for OSSN-0090 and OSSN-0065. On the other side we decided to speedup the work we've had in flight and we decided to have recent patches reviews meeting once in a month and backlogs review parties before each milestone to speedup the reviews. Recording: https://drive.google.com/drive/u/0/folders/1oYrgVLV5L89keDZLx7r1JWcG8pk65918 - Bobcat Retrospective # Common Tempest Failures for Glance Glance has faced so many random CI failure and timeouts since bobcat mid cycle, though progress has been made on general issues and few of the project-specific issue also been addressed like post failures causing by image conversion failure, tried reducing concurrency but still we see some random timeouts, one of the common tempest failure has been observed that some glance related jobs like glance-multistore-cinder-import, nova-ceph-multistore are failing with race condition while saving image data, there is a short term solution proposed by sending image_id while creating the image to ensure that each test can utilize own image and race condition could be avoided. https://docs.google.com/document/d/1S41X3FSjqGF87ehTvqLEC9QsDIkULjS1NiqCagZ… Decided to reproduce 1st failure from the gdoc and debug more on failures Recording: https://drive.google.com/drive/u/0/folders/1oYrgVLV5L89keDZLx7r1JWcG8pk65918 - Common tempest failures # Add missing CLI support for some Glance API in Openstack Client The migration work for adding missing CLI was initiated in the antelope cycle. We have added the SDK support for cache API , images API and metadef API(in progress). On the client side support for cache API and images API has been added and for metadef is in progress. This cycle we have decided to cover the remaining client and sdk support for APIs by distributing the work among the team. Recordings: https://drive.google.com/drive/u/0/folders/1oYrgVLV5L89keDZLx7r1JWcG8pk65918 - Adding missing OSC support # Default Glance to configure multiple stores Glance has deprecated single stores configuration since Stein cycle and will remove the support of same in upcoming cycle by dividing the wrok in small parts like remove/migrate single store unit/functional tests. Goal for this cycle: list down the work we have to do with regards to tests. and migrate unit/functional test from "single-store only" to "multistore-only" and send email to the mailing list to urge users to migrate to multi-store. Recording: https://drive.google.com/drive/u/0/folders/1oYrgVLV5L89keDZLx7r1JWcG8pk65918 - Default glane to multistore support.webm *Wednesday, OCT 25th 2023* # Remove Location Strategy In Bobcat we added support for the weighing mechanism to store. So we can safely remove the location strategy and by default use the latest weighing mechanism which will be very easy and efficient. Decided to deprecate location strategy in C and remove it in D, notify operators and users on ML and have a spec for the same in Caracal Cycle. Recording: https://drive.google.com/drive/u/0/folders/1oYrgVLV5L89keDZLx7r1JWcG8pk65918 - Remove Location Strategy # Improve process of restoring image to active state which is set as 'pending_delete' At the moment glance supports delaying of image-data from the store by setting delayed_delete to True and when image gets deleted it actually keeps image data in the backend and then Scrubber daemon deletes it permanently later. To restore the image, the deployer needs to set the UUID of that image in glance-api.conf to the 'restore' option under the Default section. If there are multiple images that need to be restored then this process will be kind of time consuming as each time a new Id will be needed to set in the configuration file and rerun the scrubber if it is not running as Daemon. Decided to introduce admin only API to restore the image and restrict Scrubber daemon to delete the images which are in pending_delete state. The restore part from the scrubber for a couple of cycles until the new API is stable.The restore option from the scrubber will be deprecated in this cycle and then will be removed in the following cycle. Recording: https://drive.google.com/drive/u/0/folders/1oYrgVLV5L89keDZLx7r1JWcG8pk65918 - Restore Image # Deprecate cachemanage middleware Cache operations (CR(u)D) is now part of glance API. Even though we need to depend on cache/cachemanage middleware to enable it. There are two middlewares related to caching, one is `cache` and other is `cachemanage` and we can now get rid of `cachemanage` middleware and glance-cache-manage command line utility. Agreed on deprecating cachmanage middleware in C & remove in D cycle. Decided to have a spec lite for the same after verifying how the cache APIs will respond if cache middleware is not enabled. Recording: https://drive.google.com/drive/u/0/folders/1oYrgVLV5L89keDZLx7r1JWcG8pk65918 - Deprecate Cachemanage Middleware # Use mysql database for caching instead of sqlite There are two different database(s) used for glance, MySql for regular operations and sqlite for caching related operations. We have decided to use single database i.e. MySql for all glance related operations so that it will reduce the efforts to maintain both databases. The Spec is up for review now: https://review.opendev.org/c/openstack/glance-specs/+/899367/4 Recording: https://drive.google.com/drive/u/0/folders/1oYrgVLV5L89keDZLx7r1JWcG8pk65918 - Centralize DB for Caching *Thursday, OCT 26th 2023* # Unified limits Topic was discussed and decided to postpone it to the next cycle since no one volunteered to work on it. # New Location APIs We have decided to have support of new location api which will replace the image-update mechanism for consumers like cinder and nova for OSSN-0090 and OSSN-0065. The implementation patches are up for review along with the unit/functional tests, documentation, api version bump, client side support and tempest coverage[2] The Poc patches of nova and cinder are also there to test this new location api functionality. In this cycle we are going to work on some missing test scenarios & verify PoC patches of consumers like nova and cinder are working with new location api functionality. Recording: https://drive.google.com/drive/u/0/folders/1oYrgVLV5L89keDZLx7r1JWcG8pk65918 - New Location API Apart from above topics you can find other miscellaneous topics discussed in PTG etherpad [1] along with the recordings of the sessions and milestone wise priorities at the end. If you have any questions/suggestions then please join us in our weekly meeting(each Thursday on #openstack-meeting irc channel at 1400 UTC) [1]: https://etherpad.opendev.org/p/oct2023-ptg-glance#L10 [2]: https://review.opendev.org/q/topic:New-Location-Apis Thanks & Regards, Pranali Deore

1 0

[qa][ptg] QA Caracal vPTG Summary
by Martin Kopec 30 Oct '23

30 Oct '23

Hi everyone, here is a summary of the discussions. The full agenda can be found here [1]. == Bobcat Retrospective == Considering the still shrinking number of active contributors, we still manage making releases and doing code reviews. However, less contributors make it hard to react to sudden failures or provide as stable jobs as possible - that's e.g. the case of the Centos 9 topic. == Stop supporting Centos 9 Stream? == No, Centos 9 stays. As concluded during the session, Centos 9 is required for FIPS as other distros (their latest versions) are far behind with supporting FIPS. Apart from that, CS9 is required for Manila, Cinder integration with Ceph as Ceph is tested on Centos 9 too. The biggest pain point is that Centos9 Stream is not versioned and thus it's hard to build a reliable gate. One of the goals for the cycle is improving the connection between Centos and OpenStack community - it's beneficial for both sides. One of the ways might be a 3rd party CI or simply a solution that would provide a quick feedback to the Centos group, e.g. by running Devstack with the default configuration, but at the same time wouldn't affect all the jobs on the OpenStack side that run the distro. This way we could at least avoid simple issues found right away (e.g. some deps issues). The QA team is happy to assist. == Triage previous and write new priority items == The list of the priority items planned for this cycle as well as the ones which were carried over from the previous cycles, can be found here [2]. [1] https://etherpad.opendev.org/p/oct2023-ptg-qa [2] https://etherpad.opendev.org/p/qa-caracal-priority Thank you all for attending the PTG and contributing to the discussion, -- Martin Kopec Principal Software Quality Engineer Red Hat EMEA IM: kopecmartin

1 0

[interop][ptg] Interop Caracal vPTG Summary
by Martin Kopec 30 Oct '23

30 Oct '23

Hi everyone, here is a summary of the discussions. The full agenda can be found here [1]. We had only one topic on the agenda - a new view on interoperability testing - don't confuse it with interop which is a certification program. The program just uses the interop tests for the certification purposes. The tests that are part of interop were long ago considered as enough for testing interoperability. However, as it's been pointed out as well as discussed many times already [2][3], the tests don't guarantee interoperability of tested clouds - they test only basic API thus they can be used for the certification program (related to OpenStack brand) at most. Interoperability is subjective, every user may be looking for a different level of interoperability. Read the ending of the article [2]. If there is someone who is interested in interoperability testing in OpenStack or even someone who requires that for various reasons, we could do something like this [4] - we could gather various tests from various testing platforms that test specific functionalities. It would provide anyone with metrics and thus allow them to compare clouds. It would be like when we're shopping, e.g. for a new computer - we have various metrics that help us decide which one suits us better like dimensions, weight, RAM, CPU, storage space etc. From the last PTG (and several one before) it, however, doesn't look like the community is interested in running actual interoperability testing. Let us know if you disagree, we don't want to develop something that is not needed by anyone. [1] https://etherpad.opendev.org/p/oct2023-ptg-interop [2] https://superuser.openinfra.dev/articles/new-view-on-interoperability-in-op… [3] https://etherpad.opendev.org/p/bobcat-ptg-interop [4] https://etherpad.opendev.org/p/openstack-interop-2.0 Thank you all for attending the PTG and contributing to the discussion, -- Martin Kopec Principal Software Quality Engineer Red Hat EMEA IM: kopecmartin

1 0

[neutron] Bugs deputy report October 23rd - 29th
by Miguel Lavalle 29 Oct '23

29 Oct '23

Hi Neutrinos, Here's this week's bugs deputy report: Critical ====== High ==== https://bugs.launchpad.net/neutron/+bug/2040166 [OVN] The neutron_sync_mode conf is ignored for SB DB sync. Assignee: Lucas Gomes. https://bugs.launchpad.net/neutron/+bug/2040172 [OVN] OvnSbSynchronizer - clean/delete segmenthostmappings for unrelated hosts. Assignee: Harald Jensas. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/899077 https://bugs.launchpad.net/neutron/+bug/2041609 FIP update removes QoS policy. Assignee Lajos Katona. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/899469 Medium ====== https://bugs.launchpad.net/neutron/+bug/2040517 host not removed from table ml2_vxlan_endpoints with the agent delete. Needs and owner https://bugs.launchpad.net/neutron/+bug/2041612 [n-d-r] DRAgent removed from bgp speaker automatically. Needs verification and owner Low === RFEs ==== https://bugs.launchpad.net/neutron/+bug/2040001 [N-D-R] Change neighbor connect mode when the BGP peer does not support passive mode. Assignee: Tiago Pires. This RFE was initially discussed during this week's vPTG. Proposed spec: https://review.opendev.org/c/openstack/neutron-specs/+/899210 Undecided ======== Invalid ======== https://bugs.launchpad.net/neutron/+bug/2040242 [ipam] Why prevent first ip < (subnet.first + 1) if version of subnet is ipv6. Assignee Liu Xie. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/899374 https://bugs.launchpad.net/neutron/+bug/2040457 [allocation_pools] Need re-generation new pools from cidr and gateway-ip after update one subnet with empty 'allocation_pools'. Assignee: Liu Xie. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/899373

1 0

Regarding Nova's inability to delete the Cinder volume for creating virtual machines (version Y)
by 无数的星球 27 Oct '23

27 Oct '23

When creating a virtual machine in the dashboard, create a volume and choose to delete the virtual machine while also deleting the volume. When deleting the virtual machine, there is no normal uninstallation of the volume and the volume is not deleted. The relevant error logs are shown in the image, but the openstack CLI can delete its volume. The specific commands are as follows. CLI： source /etc/keystone/admin-openrc.sh (Verify password file) openstack volume set --detached　191e555c-3947-4928-be46-9f09e2190877（volumeID） openstack volume delete 191e555c-3947-4928-be46-9f09e2190877（volumeID） It seems that Nova is unable to interact with the Cinder API to delete(or detached) commands, but I am not very professional. I don't know if it's a bug?

1 0

horizon - dashboard behavior - shows elements from previous project
by Miguel Casique Meneses 27 Oct '23

27 Oct '23

There is a horizon chart behaving quite strangely version: horizon-0.3.2 To recreate the issue: login to openstack >> change project (using dropdown) >> show containers (odd behaviour, such as change to old project, or lists containers from previous project) >> show instances (odd behaviour, changes projects, or at least shows the wrong project in dropdown) Basically, it sometimes shows the elements from the previous browsed project and not the current one that has been selected. If we reload the page, it goes back to project that has been selected, but sometimes it does not The logs are not showing nothing relevant, but we can see when project has been swapped

3 2

[Glance] Caracal PTG Updates
by Pranali Deore 27 Oct '23

27 Oct '23

Hello Everyone, We have concluded Glance Caracal vPTG yesterday. I will share the summary early next week. You can find tentative milestone wise priorities for glance in PTG etherpad[1]. Join us in the weekly meeting on Thursday if you have any doubts/suggestions. Thanks & Regards, Pranali [1]: https://etherpad.opendev.org/p/oct2023-ptg-glance#L277

1 0

[keystone] Weekly Meeting 11/1 Canceled
by Dave Wilde 27 Oct '23

27 Oct '23

Hello all, I'm on PTO the week of 11/1 so I'm canceling the Weekly meeting as well as the reviewathon on 11/3. Thanks! /Dave

1 0