July 2024 - openstack-discuss - lists.openstack.org

Magnum issues http requests to other services when internal/admin endpoint is https
by Michel Jouvin 04 Jul '24

04 Jul '24

Hi, We have reconfigured our cloud with a Nginx sever handling TLS in front of every API service, as described in https://docs.openstack.org/security-guide/secure-communication/secure-refer…. Previously we used to have Magnum/Heat/Barbican endpoints using http rather than https (not good!). Since this change, we have seen problems in Magnum (starting in Antelope when we did the change but still present in Caracal) where some "internal" requests to Barbican and Heat are done in http rather than https. We worked around this defining a redirect to https on 497 status code, as documented in Nginx documentation but it is not really satisfying as it means all clients, not only Magnum service, could use http on the https port and get serviced. I remember seing an issue on Launchpad about this issue but forgot the issue number. We were wondering wether it would not be better, at least until the problem is fixed, to define the internal+admin endpoints with http rather than https. According to our tests, it fixes the problem, as expected. Thanks in advance for any thought or experience on this. Best regards, Michel

2 2

[nova] volume information was duplicated when Live Migration failed
by Junya Noguchi (Fujitsu) 04 Jul '24

04 Jul '24

To whom it may concern Hi. I encountered an issue with OpenStack where volume information was duplicated when Live Migration failed. Could you please provide a solution to this issue? [Issue] After a Live Migration failure, when I checked the volume where the error occurred using the 'openstack volume list' command, the following was displayed: +--------------------------------------+-----------------------------------+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+-----------------------------------+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------+ | 9ba9734f-575e-41de-8bb4-60839388e0ad | | in-use | 10 | Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda | I expect that the 'Attached to' display should only appear once as there is one volume connected, but it is duplicated for each failed attempt. The versions of OpenStack and nova are as follows: OpenStack Zed $ openstack --version openstack 6.0.0 $ nova --version nova CLI is deprecated and will be a removed in a future release 18.1.0 [Details] The volume display is duplicated under the following conditions: - When Live Migration of an instance fails If the communication between the ports required for Live Migration is closed, Live Migration will fail when executed. $ sudo netstat -tuln | grep 16509 $ The output of nova-compute.log at the time of Live Migration failure is as follows: --- 2024-02-28 17:04:32.119 2785 ERROR nova.virt.libvirt.driver [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] Live Migration failure: unable to connect to server at 'XXXXX.com:16509': Connection refused: libvirt.libvirtError: unable to connect to server at 'XXXXX.com:16509': Connection refused 2024-02-28 17:04:32.119 2785 DEBUG nova.virt.libvirt.driver [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] Migration operation thread notification thread_finished /usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py:10516 2024-02-28 17:04:32.607 2785 DEBUG nova.virt.libvirt.migration [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] VM running on src, migration failed _log /usr/lib/python3.9/site-packages/nova/virt/libvirt/migration.py:432 2024-02-28 17:04:32.608 2785 DEBUG nova.virt.libvirt.driver [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] Fixed incorrect job type to be 4 _live_migration_monitor /usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py:10330 2024-02-28 17:04:32.608 2785 ERROR nova.virt.libvirt.driver [-] [instance: 521eb55f-535d-4fa7-a27e-447b0bbae9b4] Migration operation has aborted --- After Live Migration, when I check the volume where the error occurred using the 'openstack volume list' command, the following is displayed: +--------------------------------------+-----------------------------------+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+-----------------------------------+-----------+------+---------------------------------------------------------------------------------------------------------------------------------------------+ | 9ba9734f-575e-41de-8bb4-60839388e0ad | | in-use | 10 | Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to | | | | | | 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda Attached to 73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda | '73c4c6e8-fe34-4f87-9f26-e70b4cc593ba on /dev/vda' is displayed as many times as Live Migration failed. The duplicate display did not disappear even after Live Migration was successful. Thanks,

2 2

[kolla-ansible][letsencrypt] containers are running but not getting certificates.
by Jonathan Proulx 04 Jul '24

04 Jul '24

Hi All, I'm trying to do a test multinode deploy using 2023.2 I have letsencrypt_webserver and letsencrypt_lego contsainers running and I'm seeing random traffic in the /var/log/kolla/letsencrypt/letsencrypt-webserver-access.log so fairly confident they're plumbed through to the public internet properly, but I don't seem to be getting certificates. how can I trigger a renewal attempt so I can maybe see what I've screwed up? Thanks, -Jon -- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL

3 6

[OSSA-2024-001] OpenStack Cinder, Glance, Nova: Arbitrary file access through custom QCOW2 external data (CVE-2024-32498)
by Jeremy Stanley 03 Jul '24

03 Jul '24

======================================================================= OSSA-2024-001: Arbitrary file access through custom QCOW2 external data ======================================================================= :Date: July 02, 2024 :CVE: CVE-2024-32498 Affects ~~~~~~~ - Cinder: <22.1.3, >=23.0.0 <23.1.1, ==24.0.0 - Glance: <26.0.1, ==27.0.0, >=28.0.0 <28.0.2 - Nova: <27.3.1, >=28.0.0 <28.1.1, >=29.0.0 <29.0.3 Description ~~~~~~~~~~~ Martin Kaesberger reported a vulnerability in QCOW2 image processing for Cinder, Glance and Nova. By supplying a specially created QCOW2 image which references a specific data file path, an authenticated user may convince systems to return a copy of that file's contents from the server resulting in unauthorized access to potentially sensitive data. All Cinder deployments are affected; only Glance deployments with image conversion enabled are affected; all Nova deployments are affected. Patches ~~~~~~~ - https://review.opendev.org/923247 (2023.1/antelope(cinder)) - https://review.opendev.org/923277 (2023.1/antelope(glance)) - https://review.opendev.org/923278 (2023.1/antelope(glance)) - https://review.opendev.org/923279 (2023.1/antelope(glance)) - https://review.opendev.org/923280 (2023.1/antelope(glance)) - https://review.opendev.org/923281 (2023.1/antelope(glance)) - https://review.opendev.org/923282 (2023.1/antelope(glance)) - https://review.opendev.org/923283 (2023.1/antelope(glance)) - https://review.opendev.org/923288 (2023.1/antelope(nova)) - https://review.opendev.org/923289 (2023.1/antelope(nova)) - https://review.opendev.org/923290 (2023.1/antelope(nova)) - https://review.opendev.org/923281 (2023.1/antelope(nova)) - https://review.opendev.org/923246 (2023.2/bobcat(cinder)) - https://review.opendev.org/923266 (2023.2/bobcat(glance)) - https://review.opendev.org/923267 (2023.2/bobcat(glance)) - https://review.opendev.org/923268 (2023.2/bobcat(glance)) - https://review.opendev.org/923269 (2023.2/bobcat(glance)) - https://review.opendev.org/923270 (2023.2/bobcat(glance)) - https://review.opendev.org/923271 (2023.2/bobcat(glance)) - https://review.opendev.org/923272 (2023.2/bobcat(glance)) - https://review.opendev.org/923284 (2023.2/bobcat(nova)) - https://review.opendev.org/923285 (2023.2/bobcat(nova)) - https://review.opendev.org/923286 (2023.2/bobcat(nova)) - https://review.opendev.org/923287 (2023.2/bobcat(nova)) - https://review.opendev.org/923245 (2024.1/caracal(cinder)) - https://review.opendev.org/923259 (2024.1/caracal(glance)) - https://review.opendev.org/923260 (2024.1/caracal(glance)) - https://review.opendev.org/923261 (2024.1/caracal(glance)) - https://review.opendev.org/923262 (2024.1/caracal(glance)) - https://review.opendev.org/923263 (2024.1/caracal(glance)) - https://review.opendev.org/923264 (2024.1/caracal(glance)) - https://review.opendev.org/923265 (2024.1/caracal(glance)) - https://review.opendev.org/923273 (2024.1/caracal(nova)) - https://review.opendev.org/923274 (2024.1/caracal(nova)) - https://review.opendev.org/923275 (2024.1/caracal(nova)) - https://review.opendev.org/923276 (2024.1/caracal(nova)) - https://review.opendev.org/923244 (2024.2/dalmatian(cinder)) - https://review.opendev.org/923248 (2024.2/dalmatian(glance)) - https://review.opendev.org/923249 (2024.2/dalmatian(glance)) - https://review.opendev.org/923250 (2024.2/dalmatian(glance)) - https://review.opendev.org/923251 (2024.2/dalmatian(glance)) - https://review.opendev.org/923252 (2024.2/dalmatian(glance)) - https://review.opendev.org/923253 (2024.2/dalmatian(glance)) - https://review.opendev.org/923254 (2024.2/dalmatian(glance)) - https://review.opendev.org/923255 (2024.2/dalmatian(nova)) - https://review.opendev.org/923256 (2024.2/dalmatian(nova)) - https://review.opendev.org/923257 (2024.2/dalmatian(nova)) - https://review.opendev.org/923258 (2024.2/dalmatian(nova)) Credits ~~~~~~~ - Martin Kaesberger (CVE-2024-32498) References ~~~~~~~~~~ - https://launchpad.net/bugs/2059809 - http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-32498 Notes ~~~~~ - Due to the scope of the problem and complexity of the resulting fixes, regressions and additional bypasses were reported in the original bug by downstream stakeholders during the coordinated disclosure period. As a result, our initially chosen publication date was rescheduled, which put the advisory four days past our promised ninety day maximum embargo length. Additional revised patches and regression fixes were supplied to stakeholders as soon as possible, but we understand the unfortunate timing of these last-minute changes resulted in a lot of additional work for everyone involved. -- Jeremy Stanley OpenStack Vulnerability Management Team

2 1

[Cinder] Extending Specs Freeze Deadline to Jun 7
by Jon Bernard 03 Jul '24

03 Jul '24

Hello, As discussed in the last cinder meeting [1] we are extending the cinder specs freeze deadline to June 7. The previous week contains holidays for some on our team, so the weekend days are added to make sure everything that needs to merge has sufficient time to do so. Please make sure your specs are updated before the new deadline. [1]: https://meetings.opendev.org/meetings/cinder/2024/cinder.2024-06-26-14.00.l… Thanks, -- Jon

1 0

any idiomatic testtools / stestr patterns for asyncio?
by Mike Bayer 03 Jul '24

03 Jul '24

hey gang - I'm adding asyncio support to oslo.db. Fairly major block hit, which I can work around, is that testtools / stestr do not seem to have any provision for asyncio present. An initial attempt working with it gracefully looked like this (using oslotests base, which is the testtoolsTestBase): import unittest from oslotest import base class MyTestSuite(unittest.IsolatedAsyncioTestCase, base.BaseTestCase): # ... but that didn't work because the degree to which testtools modifies the run() method of unittest.TestCase prevents IsolatedAsyncioTestCase from running awaitable test cases. Next attempt is this, remove testtools / oslotest from the class hierarchy (which is bad, because we lose all of our testtools patterns): class MyTestSuite(unittest.IsolatedAsyncioTestCase): # ... this runs, however stestr sends into the test suite a Result object that is not a subclass of unittest.result.TestResult, it's a "subunit.test_results.AutoTimingTestResultDecorator" that does not TestResult in its MRO; unittest shows a warning that there is no addDuration() method on the result object. final workaround, patch one on, here's my base: class BaseAsyncioCase(unittest.IsolatedAsyncioTestCase): def run(self, result=None): # work around stestr sending a result object that's not a # unittest.TestResult result.addDuration = lambda test, elapsed: None return super().run(result=result) now I can run asyncio tests. But only by ditching testtools entirely. What would be the correct approach to introducing asyncio support in our test suites given our dependency on testtools?

4 6

Regarding Cinder Volume as a backend not mounting on Compute Node
by Adivya Singh 02 Jul '24

02 Jul '24

Hi Team, I am facing a issue, where my Cinder Volume is not getting mounted on Compute node, it creates Successfully, Here is my export file input on storage server /cinder/volume 172.29.244.0/22(rw,no_subtree_check,no_root_squash) and here is my NFS mount Option on Cinder as a backend nfs_mount_options: noatime,rw,bg,intr,async,timeo=60 it mounts successfully on Controller, but compute it get struck Regards Adivya Singh

1 0

[tacker] Cancel weekly meeting
by Yasufumi Ogawa 02 Jul '24

02 Jul '24

Hi team, I'm canceling today's meeting since I'm not available no new topic on the etherpad. Please feel free to host someone else if you have any topic. Thanks, Yasufumi

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2024.2/R-13)
by Goutham Pacha Ravi 02 Jul '24

02 Jul '24

Hello Stackers, We're officially at Milestone 2 of the 2024.2 ("Dalmatian") release cycle [1]. Alongside spec reviews and other project-specific deadlines, this is the week to look out for intermediary releases of several OpenStack projects and close out bug fixes, or re-target open ones for a future milestone. We're soon going to see some patches to add the code name that was picked for the 2025.1 release (31st release!) of OpenStack: "Epoxy" [2]. I probably jinxed the animal theme, but Kendall Nelson (diablo_rojo) told me that Epoxy will be "the glue that holds our clouds together." The Technical Committee was reviewing project activity in Monasca [3], Freezer, and Skyline [4]. As you've probably gathered from the discussion on this list and elsewhere, Monasca seems ready to leave the "inactive" project list and Skyline seems set to graduate from the "emerging" project list [5]. The respective project teams will need to make these proposals, and I'll keep you posted on how that evolves. The Freezer project team would like to continue to keep the "inactive" project tag through this release as they continue to stabilize the codebase and the CI jobs. This means that there will be no releases from the Freezer team during the 2024.2 ("Dalmatian") release cycle. Hervé Beraud (hberaud) and Mike Bayer (zzzeek) will be joining us at the Technical Committee monthly video meeting tomorrow (2024-07-02) to preview the work ahead of us to remove eventlet from OpenStack [6][7]. If you are an OpenStack project maintainer, packager, or operator, please consider joining so we can have a productive discussion. === Weekly Meeting === The Technical Committee met last week on OFTC's #openstack-tc IRC channel [8]. We discussed several in-progress changes, including a proposal to address affiliation diversity violations that may occur during TC elections. There was an emphasis on gate health. Thanks to a few patches to devstack, tempest, and devstack plugins, several integrated gate jobs were more stable over the past couple of weeks. As noted last week, CentOS Stream 8 is now EOL, and hence any test jobs that were relying on this OS are now perma-failing. This is a call-out to project maintainers to clean up their test jobs across all code branches. Currently, a search of trunk branches reveals that centos-stream-8 nodes are required for test jobs in Tempest, Rally, Tenks, Ansible tests, and tempest plugins of Octavia, Telemetry, and Cinder. Please do look at cruft left behind in older stable and "unmaintained" branches as well. TC members also brainstormed on projects that are seemingly active, but the project's core team is unresponsive to new contributors. This is against the Open Community principles that we adhere to [9]. If you're aware of instances of this happening, please do let the TC know. Our next weekly meeting is held simultaneously on Video and IRC. Details, including the meeting's agenda, are in [7]. I hope you'll be able to participate. As always, our IRC meetings are logged to eavesdrop [10], and our Video meeting recordings are uploaded to YouTube [11]. === Governance Proposals === ==== Merged ==== Show TC and SIG repos under reference | https://review.opendev.org/c/openstack/governance/+/918488 ==== Open for review/discussion ==== Retire kuryr-kubernetes and kuryr-tempest-plugin | https://review.opendev.org/c/openstack/governance/+/922507 Update to include docs and miscellaneous repos for AC status | https://review.opendev.org/c/openstack/governance/+/915021 Adding a rule to manage the affiliation diversity requirement in the TC | https://review.opendev.org/c/openstack/governance/+/922512 Update criteria for the inactive projects to become active again | https://review.opendev.org/c/openstack/governance/+/921500 Remove Eventlet From OpenStack | https://review.opendev.org/c/openstack/governance/+/902585 === Upcoming Events === 2024-07-02: Monthly Technical Committee Video+IRC meeting 2024-07-02: OpenInfra Board Meeting -: https://board.openinfra.dev/meetings/ 2024-07-04: Milestone-2 of the 2024.2 release cycle Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] Release Schedule: https://releases.openstack.org/dalmatian/schedule.html [2] OpenStack "Epoxy": https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [3] Reviving Monasca: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [4] Graduating Skyline: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [5] Emerging Technology and Inactive Projects: https://governance.openstack.org/tc/reference/emerging-technology-and-inact… [6] Goal proposal to "Remove Eventlet from OpenStack": https://review.opendev.org/c/openstack/governance/+/902585 [7] Agenda for the OpenStack TC Meeting on 1st July 2024: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [8] TC Meeting log (2024-06-25): https://meetings.opendev.org/meetings/tc/2024/tc.2024-06-25-18.00.log.html [9] "Open Community": https://opendev.org/openinfra/four-opens/src/branch/master/doc/source/openc… [10] Past TC meetings: https://meetings.opendev.org/meetings/tc/ [11] YouTube channel recordings of past TC meetings: https://www.youtube.com/watch?v=34w1Aoai1EA&list=PLhwOhbQKWT7Ui1KPL7MLIq4ye…

1 0

problem with deleting a volume from OpenStack
by kjme001＠gmail.com 02 Jul '24

02 Jul '24

I have a problem with deleting a volume from OpenStack from a machine that no longer exists because it has crashed. If it executes the command: openstack volume delete volume_name receives the deleting status and the volume is not deleted, it is still in the deleting state. How can I delete such a volume? Can I remove it from the postgres database? Regards

2 4