- openstack-discuss - lists.openstack.org

Openstack Instance Snapshot and Rescue Problem
by meteccinar＠gmail.com 23 Aug '24

23 Aug '24

Hi everyone, We use Ceph in Cinder as Block Storage. Firsly, we create a VM with volume (volume create automatically) and we write a data in the VM and take an snapshot after wrote data. Then we write data in main VM again and take second snapshot. But, when we try to rescue instance related main VM with first snapshot, there was an error occured. Nova Log: 2024-08-22 08:23:25.594 19 WARNING nova.compute.api [None req-eb3b4856-c8be-4926-be51-99aaec4de0e0 41040b4a21e34904b6906c12b228d8e9 67ddd9f771be452a808ba3458af544f8 - - default default] Unable to rescue an instance using a volume snapshot image with img_block_device_mapping image properties set Horizon Output: Requested rescue image 'ddc2df28-c4f5-434b-9f9a-030e56425fc5' is not supported (HTTP 400) (Request-ID: req-bd8037a1-3cd4-43fe-a945-e809b1f8ce16) Ceph Block: https://imgtr.ee/images/2024/08/22/2108d8aacee12eaf800377593a229659.png Instance Snapshot Image: https://imgtr.ee/images/2024/08/22/ca1fa81683acc0518ac161e7f97fda0e.jpeg Could it be any configuration wrong ? We will wait for your answers. Thanks.

5 8

[nova][OpenStackSDK][vitrage][zaqar] broken gates in libraries
by Elõd Illés 22 Aug '24

22 Aug '24

Hi teams in $subject, Since we are close to Feature Freeze period and Dalmatian-3 milestone, the relmgt team checked the status of the gates of libraries [1] and the following repositories seem to have broken gates on their master branches: python-novaclient osc-lib python-vitrageclient python-zaqarclient Could the teams check the broken jobs please and fix them as soon as possible? [1] https://review.opendev.org/q/topic:release-health-check-cwi-dalmatian+is:op… Thanks in advance, Előd Illés irc: elodilles @ #openstack-release

1 0

[neutron] dsvm-fullstack test suite: issues with ovs datapaths when running locally?
by Ihar Hrachyshka 22 Aug '24

22 Aug '24

Hi all, This must be simple to do, but when I try to follow the guide in a vagrant box to run the fullstack suite locally, I am hitting a wall. I suspect I miss something obvious but I can't figure it out. I'm following this doc: https://docs.openstack.org/neutron/latest/contributor/testing/fullstack.htm… I start with an empty ubuntu 22.04 vagrant box with enough memory, disk, cpu. Then I check out the neutron repo and devstack into my home dir. I also `mkdir -p /opt/stack/ && chmod a+rwx /opt/stack` and clone `requirements` repo in /opt/stack/requirements. (Otherwise configure_for_func_testing script fails.) Then I execute: export VENV=dsvm-fullstack tools/configure_for_func_testing.sh ~/devstack -i (I am not sure setting VENV envvar is needed, but that's what is in the doc) It successfully completes (I see "Phew, we're done!" message from the script.) I then: $ tox -e dsvm-fullstack neutron.tests.fullstack.test_logging (or any other test case that triggers Open vSwitch) and the test just fails with timeout. And when I look into logs, I see interface additions failing and in vswitchd log: Aug 21 20:24:26 ubuntu ovs-vswitchd[42747]: ovs|00032|bridge|ERR|failed to create bridge br-eth68909740e: No such file or directory Aug 21 20:24:26 ubuntu ovs-vswitchd[42747]: ovs|00031|ofproto|ERR|failed to open datapath br-eth68909740e: No such file or directory Aug 21 20:24:26 ubuntu ovs-vswitchd[42747]: ovs|00030|ofproto_dpif|ERR|failed to open datapath of type system: No such file or directory (there are more errors of the same...) I disabled apparmor as the doc suggests. I also confirmed that the `openvswitch` kernel module is loaded (it is not loaded after ovs and ovn are built from source as part of configure_for_func_testing.sh execution). I also tried to run the suite as root. (I don't think this is needed, but I figured I'll check.) Anyone else running into this? How do you guys execute the suite locally? Ihar

2 2

[all][elections][ptl][tc] Combined PTL/TC 2025.1 cycle Election Nominations Kickoff
by Slawomir Kaplonski 22 Aug '24

22 Aug '24

Nominations for OpenStack PTLs (Project Team Leads) and TC (Technical Committee) positions (4 positions) are now open and will remain open until Aug 28, 2024 23:45 UTC. All nominations must be submitted as a text file to the openstack/election repository as explained at https://governance.openstack.org/election/#how-to-submit-a-candidacy Please make sure to follow the candidacy file naming convention: candidates/2025.1/<Project_Name or "TC">/<email_address> (for example, "candidates/2025.1/TC/stacker(a)example.org"). The name of the file should match an email address for your current OpenStack Foundation Individual Membership. Take this opportunity to ensure that your OSF member profile contains current information: https://www.openstack.org/profile/ Any OpenStack Foundation Individual Member can propose their candidacy for an available, directly-elected seat on the Technical Committee. In order to be an eligible candidate for PTL you must be an OpenStack Foundation Individual Member. PTL candidates must also have contributed to the corresponding team during the 2024.1 to 2024.2 timeframe, Sep 15, 2023 00:00 UTC - Aug 28, 2024 00:00 UTC. Your Gerrit account must also have a verified email address matching the one used in your candidacy filename. Both PTL and TC elections will be held from Sep 04, 2024 23:45 UTC through to Sep 18, 2024 23:45 UTC. The electorate for the TC election are the OpenStack Foundation Individual Members who have a code contribution to one of the official teams over the 2024.1 to 2024.2 timeframe, Sep 15, 2023 00:00 UTC - Aug 28, 2024 00:00 UTC, as well as any Extra ACs who are acknowledged by the TC. The electorate for a PTL election are the OpenStack Foundation Individual Members who have a code contribution over the 2024.1 to 2024.2 timeframe, Sep 15, 2023 00:00 UTC - Aug 28, 2024 00:00 UTC, in a deliverable repository maintained by the team which the PTL would lead, as well as the Extra ACs who are acknowledged by the TC for that specific team. The list of project teams can be found at https://governance.openstack.org/tc/reference/projects/ and their individual team pages include lists of corresponding Extra ACs. Please find below the timeline: PTL + TC nomination starts @ Aug 14, 2024 23:45 UTC PTL + TC nomination ends @ Aug 28, 2024 23:45 UTC TC campaigning starts @ Aug 28, 2024 23:45 UTC TC campaigning ends @ Sep 04, 2024 23:45 UTC PTL + TC elections start @ Sep 04, 2024 23:45 UTC PTL + TC elections end @ Sep 18, 2024 23:45 UTC Shortly after election officials approve candidates, they will be listed on the https://governance.openstack.org/election/ page. The electorate is requested to confirm their email addresses in Gerrit prior to 2024-08-28 00:00:00+00:00, so that the emailed ballots are sent to the correct email address. This email address should match one which was provided in your foundation member profile as well. Gerrit account information and OSF member profiles can be updated at https://review.openstack.org/#/settings/contact and https://www.openstack.org/profile/ accordingly. If you have any questions please be sure to either ask them on the mailing list or to the elections officials: https://governance.openstack.org/election/#election-officials -- Best regards Slawek Kaplonski skaplons(a)redhat.com

5 5

[all][elections][ptl][tc] Combined PTL/TC 2024.2 cycle Election Nominations progress
by Sławek Kapłoński 22 Aug '24

22 Aug '24

Hi, Here is report about PTL/TC 2024.2 cycle nomination progress so far. If you consider to nominate yourself for the PTL or TC role, please do so in next 6 days. Nominations started @ 2024-08-14 23:45:00 UTC Nominations end @ 2024-08-28 23:45:00 UTC Nominations duration : 14 days, 0:00:00 Nominations remaining : 6 days, 11:07:57 Nominations progress : 53.83% --------------------------------------------------- Projects[1] : 41 Projects with candidates : 8 ( 19.51%) Projects with election : 0 ( 0.00%) --------------------------------------------------- Need election : 0 () Need appointment : 33 (Adjutant Barbican Cinder Cloudkitty Cyborg Designate Glance Heat Keystone Kury r Magnum Manila Masakari Mistral Monasca Neutron Nova Octavia OpenStackAnsible OpenStackSDK Puppet_OpenStack Q uality_Assurance Rally Skyline Storlets Swift Tacker Telemetry Trove Venus Vitrage Watcher Zun) --------------------------------------------------- TC Candidates : 3 (4 Required) =================================================== Stats gathered @ 2024-08-22 12:37:03 UTC -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

Troubleshooting ERROR status after starting instance from Ubuntu Image
by klaus.becker＠webmen.de 22 Aug '24

22 Aug '24

Hi OpenStackers! After a vacation I continue to get my first OpenStack work (one node guided install). What works: I managed to install OpenStack on Ubuntu 22.04.4 LTS on an older DELL server with plenty of RAM and Disks I used Snap Install I can use horizon to create my VMs Problem: After I made a few instances from the Ubuntu templates with different flavors, I am Getting the status „ERROR“ for each of these instances. Trying to shutdown an instance I am getting an Error: Questions: What is the best practice for troubleshooting regarding my installation type (single node, snap)? Where do I find log-files? Google brings different solutions, but the paths and files do not exist for my installation? I tried a few suggestions: Openstack instance generating status error – Fix it now <https://bobcares.com/blog/openstack-instance-generating-status-error/> ( /var/lib/nova/ and /var/lib/nova and /var/log/libvirt/qemu do not exist ) I found: ~snap/juju ~snap/openstack ~/snap/openstack$ ls -ahl total 20K drwxr-xr-x 5 webmen webmen 4.0K Aug 13 10:54 . drwx------ 4 webmen webmen 4.0K Jul 24 13:59 .. drwxr-xr-x 4 webmen webmen 4.0K Jul 19 12:53 576 drwxr-xr-x 4 webmen webmen 4.0K Jul 19 12:53 577 drwxr-xr-x 4 webmen webmen 4.0K Jul 17 12:16 common lrwxrwxrwx 1 webmen webmen 3 Jul 24 10:04 current -> 560 The link "current —> 560 has no target here! In „common", there are sunbeam-logs Maybe there is a misconfiguration with the soft-network. I had a hard time understanding the different networks I need to access VMs from the LAN. Finaly I found then logs using "find -name nova“ /var/snap/openstack-hypervisor/common/log webmen@ada:/var/snap/openstack-hypervisor/common/log$ ls -ahl total 92K drwxr-xr-x 7 root root 4.0K Jul 17 12:39 . drwxr-xr-x 8 root root 4.0K Jul 24 12:54 .. drwxr-xr-x 3 root root 4.0K Jul 17 12:39 libvirt drwxr-xr-x 2 root root 4.0K Jul 17 12:39 neutron -rw-r--r-- 1 root root 58K Aug 16 13:06 neutron.log drwxr-xr-x 2 root root 4.0K Jul 17 12:39 nova drwxr-xr-x 2 root root 4.0K Jul 17 12:39 openvswitch drwxr-xr-x 2 root root 4.0K Jul 17 12:39 ovn webmen@ada:/var/snap/openstack-hypervisor/common/log$ But most dirs are empty webmen@ada:/var/snap/openstack-hypervisor/common/log$ tree . ├── libvirt │ └── qemu ├── neutron ├── neutron.log ├── nova ├── openvswitch │ ├── ovsdb-server.log │ └── ovs-vswitchd.log └── ovn └── ovn-controller.log But I did not find the logs of glance webmen@ada:/$ sudo find -name glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/995/fs/var/lib/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/995/fs/etc/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1192/fs/var/lib/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1192/fs/etc/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/915/fs/usr/bin/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/894/fs/usr/bin/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/891/fs/usr/bin/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/205/fs/usr/bin/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/234/fs/var/log/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/234/fs/var/lib/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/234/fs/usr/share/doc/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/234/fs/usr/bin/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/234/fs/usr/lib/python3/dist-packages/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/234/fs/etc/glance ./snap/openstack-hypervisor/153/usr/bin/glance ./snap/openstack/577/bin/glance ./snap/openstack/577/lib/python3.10/site-packages/muranoclient/glance ./snap/openstack/576/bin/glance ./snap/openstack/576/lib/python3.10/site-packages/muranoclient/glance find: ‘./proc/3340174’: No such file or directory find: ‘./proc/3361710’: No such file or directory find: ‘./proc/3361712’: No such file or directory find: ‘./proc/3361763’: No such file or directory find: ‘./proc/3364581’: No such file or directory find: ‘./proc/3364766’: No such file or directory find: ‘./proc/3364791’: No such file or directory htop: server lose very high The server still has a load average of 4.68 running for 4 days by now. The hungry processes are: 5 processes of /charm/bin/pebble run —create-dies —hold —http :38813 —verbose /snap/micro8s/7038/kubelite/… Here a copy of top (htop did not let me copy the list) PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1272 root 20 0 11.4g 238528 59048 S 83.3 0.1 3596:35 containerd 21051 root 20 0 1638280 260508 7088 S 72.5 0.1 266:36.35 pebble 21945 root 20 0 1305356 52048 7088 S 71.8 0.0 91:01.52 pebble 20543 root 20 0 1704412 220848 7088 S 69.5 0.1 265:03.19 pebble 2994 root 20 0 1854984 606616 104600 S 37.0 0.3 2957:41 kubelite 1275 root 20 0 3586472 340048 19788 S 23.6 0.2 1822:17 k8s-dqlite 3340174 root 20 0 2135364 31952 19516 S 6.2 0.0 0:00.19 alertmanager 1749555 snap_da+ 20 0 3242712 924312 42656 S 5.9 0.5 67:37.06 mysqld 78926 snap_da+ 20 0 3490924 1.1g 42428 S 5.6 0.6 91:53.33 mysqld 1267 root 20 0 4276408 2.3g 55224 S 5.2 1.2 529:17.09 mongod So… how do I get rid of that status ERROR in Horizon? Best regards Klaus Becker Webmen Internet GmbH Faulenstraße 12 28195 Bremen +49 421 24 34 94-0 info(a)webmen.de www.webmen.de Geschäftsführerin: Christiane Niebuhr-Redder Prokuristin: Kathleen Marx-Colonius Amtsgericht Bremen HRB 16685 USt-ID DE 179413676 Facebook Instagram LinkedIn

2 2

[glance][cinder] No format check on glance image uploading vs. thorough image format checks on cinder
by Florent Le Lain 21 Aug '24

21 Aug '24

Hello folks, Since resolution of vmdk related CVE<https://bugs.launchpad.net/nova/+bug/1996188>, Cinder performs extra checks on format of images used for specific actions, like creating a volume out of an image for instance. See related cinder commit: https://review.opendev.org/c/openstack/cinder/+/871625 Those new checks first led to failures on our tempest tests, on tests that were passing before. E.g.: tempest.api.compute.servers.test_create_server.ServersTestBootFromVolume It turned out that tests were failing because of a discrepancy between actual format of the image (qcow2 here) and declared format in glance db (raw). To us, this raises the question of format checks responsibility throughout images lifecycle, between Glance and Cinder. Today, afaik, Glance will: - tag as 'raw' any image uploaded without any format specified - tag the image with the format provided by the customer, whatever the actual format of the image is whereas Cinder is now more picky, refusing to deal with an image that claims to be of X format but is actually Y. If you do confirm these observations, in your opinion, what would be the best option among: 1. leave it as is, after all openstack customer is responsible for the accuracy of the information provided on image upload and should not be surprised by downstream mismatches when using the image 2. strengthen customer's responsibility of bullet 1. by making image format declaration mandatory on image upload 3. enforce format checks on glance on image upload 4. other way you may think of? something upstream I may have missed? Thank you for your opinion! Regards, Florent Le Lain SRE Team Lead – Public Cloud OVHcloud Rennes [1672217009737]

2 1

Openstack Backup and Restore
by meteccinar＠gmail.com 21 Aug '24

21 Aug '24

Hi everyone, we make cloud region for our company. Using Openstack with Ceph (as Block Storage). We configured backup driver field as BackupDriverCeph in the cinder.conf file. Also created volume and backups pools in Ceph Rados. We have a few problems/questions. We have developed an API that regularly backs up VMs (instances) that work with volumes. The process works as follows: First, we use the create_server_image() method from the OpenStack API to create an image of the instance. This results in an image of the instance being automatically created in OpenStack with a Snapshot type (including metadata such as direct_url, stores, schema, etc.). Simultaneously, a snapshot is also automatically created using with the create_server() command. We then take backups of all volumes attached to the instance. In our restore API, we use the image created from the instance backup mentioned above. We change block_device_mapping field and add the IDs of the volume backups to the block_device_mapping field of the image and use the create_server(image_name, volume_from_backed_volume) method in the OpenStack API to create a new instance. Then new volumes are created from volume backups and attach new volumes in new restored instance. The problem we're encountering is that when using the create_server command, the image created from the instance backup is tied to the direct_url, which points to the RBD URL of the snapshot. As a result, the instance does not running with new volume. Note: If we create a new VM with new volume from volume backup in Horizon. Instance running successfully. What steps should we take to resolve this? Is there a flaw in our approach when designing the API? What is the best practice we should follow when creating an API for backup and restore (especially for instances that boot from volumes)? Is there a configuration we need to adjust in Ceph? We would appreciate your advice on the best practices for backup and restore, particularly for instances that boot from volumes. Kind regards,

3 6

[neutron] Moving CI Meeting to Monday 1400 UTC
by Yatin Karel 21 Aug '24

21 Aug '24

Hello Neutron Team !! As discussed/agreed over the last couple of CI meetings, pushed the changes[1][2] for the new timing. [1] https://review.opendev.org/c/opendev/irc-meetings/+/926763 [2] https://etherpad.opendev.org/p/neutron-ci-meetings -- Thanks and Regards Yatin Karel

1 0

Magnum Caracal: cluster creation goes through state CREATION_FAILED before CREATION_COMPLETE
by Michel Jouvin 21 Aug '24

21 Aug '24

Hi, I continue my tests with Magnum Caracal and found another strange behaviour during cluster creation. It goes through state CREATION_FAILED before reaching the state CREATION_COMPLETE (HEALTHY). It is very confusing. Looking at Heat engine logs, it happens during load balancer creation, at the time the LB amphora has been created and Octavia is waiting to be able to connect it (typically one minue). During this period, Heat logs the following error: ----------- 2024-08-21 09:54:52.194 3200318 INFO heat.engine.resource [None req-610b24bd-7511-4d31-9f8c-b922162a7648 jouvin - - - - -] CREATE: LoadBalancer "loadbalancer" [670985ef-5808-4226-ad1b-b1244871de16] Stack "test-mj-1-30-m1n1-bis-ipufjomqmnnf-etcd_lb-d5fuqygvpp2j" [5c5905bf-f538-4a42-93ce-38d96b7c6603] 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource Traceback (most recent call last): 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resource.py", line 922, in _action_recorder 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource yield 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resource.py", line 1034, in _do_action 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource yield from self.action_handler_task(action, args=handler_args) 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resource.py", line 984, in action_handler_task 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource done = check(handler_data) 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resources/openstack/octavia/loadbalancer.py", line 169, in check_create_complete 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource return self._check_status() 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resources/openstack/octavia/octavia_base.py", line 29, in _check_status 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource raise exception.ResourceInError(resource_status=status) 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource heat.common.exception.ResourceInError: Went to status ERROR due to "Unknown" 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource 2024-08-21 09:54:58.804 3200319 INFO heat.engine.scheduler [None req-610b24bd-7511-4d31-9f8c-b922162a7648 jouvin - - - - -] Task pause timed out 2024-08-21 09:54:59.340 3200318 INFO heat.engine.scheduler [None req-610b24bd-7511-4d31-9f8c-b922162a7648 jouvin - - - - -] Task pause timed out ----- that results in Magnum setting the state to CREATE_FAILED despite it is a transient state from Heat point of view and despite Heat stack creation continues successfully once the transient state is cleared and Magnum later updates the status to CREATE_COMPLETED. Unfortunately magnum-conductor log has nothing related to these events (it is filled with barbican_client related messages mainly). Any idea what could cause this behaviour? Best regards, Michel

2 3