13 Jun
2022
13 Jun
'22
10:03 p.m.
Hi, I don't have a containerized deployment so I don't have anything to compare, but my compute nodes have a ceph.conf and all required keyrings in /etc/ceph/, maybe someone else can confirm that it's missing and should be present in a kolla-ansible deployment. In the meantime you could provide those files to the container and retry the live snapshot. Zitat von Parsa Aminian <p.aminian.server@gmail.com>: > snapshot_image_format doesn't exist on compute or controller nova.conf > For the manual snapshot on the previous reply I ran it directly from ceph, > but it's interesting from nova compute there is an error about connecting > to CEPH cluster !!! also ceph -s is showing similar error > [root@R3SG4 ~]# docker exec -u root -it nova_compute rbd snap create > vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk@test-snap2022-06-13 > 13:42:18.061 7f03dac23200 -1 auth: unable to find a keyring on > /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > (2) No such file or directory2022-06-13 13:42:18.061 7f03dac23200 -1 > AuthRegistry(0x561d7014a790) no keyring found at > /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, > disabling ceph2022-06-13 13:42:18.065 7f03dac23200 -1 monclient: > authenticate NOTE: no keyring found; disabled cephx authentication > rbd: couldn't connect to the cluster! > > On Mon, Jun 13, 2022 at 12:15 PM Eugen Block <eblock@nde.ag> wrote: > >> Could you share your whole nova.conf (only the uncommented options)? >> Is this option set in your env? >> >> #snapshot_image_format = <None> >> >> Also when you manually created the snapshot did you do it as the nova >> user on the compute node? If not, could you retry? >> >> >> Zitat von Parsa Aminian <p.aminian.server@gmail.com>: >> >> > Hi >> > kolla-ansible victoria version with ceph backend . >> > rbd info output : >> > rbd image '25c8d676-e20a-4238-a45c-d51daa62b941_disk': >> > size 20 GiB in 5120 objects >> > order 22 (4 MiB objects) >> > snapshot_count: 0 >> > id: b69aaf907284da >> > block_name_prefix: rbd_data.b69aaf907284da >> > format: 2 >> > features: layering, exclusive-lock, object-map, fast-diff, >> > deep-flatten >> > op_features: >> > flags: >> > create_timestamp: Fri May 20 00:04:47 2022 >> > access_timestamp: Sun Jun 12 16:26:02 2022 >> > --------------- >> > also live snapshot seems to work correctly without any error or any >> > downtime : >> > docker exec -u root -it ceph-mgr-cephosd01 rbd snap ls >> > vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk >> > SNAPID NAME SIZE PROTECTED TIMESTAMP >> > 344 test-snap 20 GiB Sun Jun 12 23:48:39 2022 >> > >> > also on compute nova.conf, images_type is set on rbd . >> > >> > On Sun, Jun 12, 2022 at 5:55 PM Eugen Block <eblock@nde.ag> wrote: >> > >> >> You should respond to the list so other users can try to support you. >> >> >> >> So nova is trying to live snapshot the instance: >> >> >> >> > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> > 4dbffaa9c14e401c8c210e23ebe0ab7b ef940663426b4c87a751afaf13b758e0 - >> >> > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] >> >> > instance snapshotting >> >> > [...] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning >> >> > live snapshot process >> >> >> >> But I don't see any 'rbd snap create' command. Either the rbd image >> >> doesn't support it or there is some setting to keep all rbd images >> >> "flat". Can you check any relevant configs you might have in nova? >> >> Also can you show the output of 'rbd info >> >> <pool>/25c8d676-e20a-4238-a45c-d51daa62b941_disk' ? Then to test if >> >> the underlying rbd functions work as expected you could try to create >> >> a live snapshot manually: >> >> >> >> rbd snap create >> <pool>/25c8d676-e20a-4238-a45c-d51daa62b941_disk@test-snap >> >> >> >> And paste any relevant output here. >> >> >> >> Zitat von Parsa Aminian <p.aminian.server@gmail.com>: >> >> >> >> > Its not working for any instances and all of them are paused . I >> enable >> >> > debug logs please check the logs : >> >> > >> >> > 2022-06-12 16:16:13.478 7 DEBUG nova.compute.manager >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync >> for >> >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 >> >> > 2022-06-12 16:16:13.506 7 DEBUG oslo_concurrency.lockutils [-] Lock >> >> > "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by >> >> > >> >> >> "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" >> >> > :: waited 0.000s inner >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 >> >> > 2022-06-12 16:16:43.562 7 DEBUG nova.compute.resource_tracker >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this compute >> >> host >> >> > and has allocations in placement: {'resources': {'VCPU': 1, >> 'MEMORY_MB': >> >> > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 >> >> > 2022-06-12 16:25:55.104 7 DEBUG nova.compute.manager >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state _get_power_state >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 >> >> > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting >> >> > 63426b4c87a751afaf13b758e0 - default default] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning live snapshot process >> >> > default default] Lazy-loading 'pci_devices' on Instance uuid >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941 obj_load_attr >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 >> >> > 2022-06-12 16:25:57.250 7 DEBUG nova.objects.instance >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> >> > ef940663426b4c87a751afaf13b758e0 - default default] Lazy-loading >> >> > 'pci_devices' on Instance uuid 25c8d676-e20a-4238-a45c-d51daa62b941 >> >> > obj_load_attr >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101 >> >> > 2022-06-12 16:25:57.317 7 DEBUG nova.virt.driver [-] Emitting event >> >> > <LifecycleEvent: 1655034957.3158934, >> 25c8d676-e20a-4238-a45c-d51daa62b941 >> >> > => Paused> emit_event >> >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 >> >> > 2022-06-12 16:25:57.318 7 INFO nova.compute.manager [-] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event) >> >> > 2022-06-12 16:25:57.389 7 DEBUG nova.compute.manager >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state _get_power_state >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 >> >> > 2022-06-12 16:25:57.395 7 DEBUG nova.compute.manager >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power >> state >> >> > after lifecycle event "Paused"; current vm_state: active, current >> >> > task_state: image_pending_upload, current DB power_state: 1, VM >> >> > power_state: 3 handle_lifecycle_event >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 >> >> > 2022-06-12 16:25:57.487 7 INFO nova.compute.manager >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the >> >> instance >> >> > has a pending task (image_pending_upload). Skip. >> >> > 2022-06-12 16:26:02.039 7 DEBUG oslo_concurrency.processutils >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> >> > ef940663426b4c87a751afaf13b758e0 - default default] Running cmd >> >> > (subprocess): qemu-img convert -t none -O raw -f raw >> >> > >> >> >> rbd:vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk:id=cinder:conf=/etc/ceph/ceph.conf >> >> > >> >> >> /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad >> >> > execute >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384 >> >> > 2022-06-12 16:26:17.075 7 DEBUG nova.virt.driver [-] Emitting event >> >> > <LifecycleEvent: 1655034962.0316682, >> 25c8d676-e20a-4238-a45c-d51daa62b941 >> >> > => Stopped> emit_event >> >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 >> >> > INFO nova.compute.manager [-] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event) >> >> > DEBUG nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa >> - - >> >> - >> >> > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state >> >> > _get_power_state >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 >> >> > DEBUG nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa >> - - >> >> - >> >> > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing >> >> > instance power state after lifecycle event "Stopped"; current >> vm_state: >> >> > active, current task_state: image_pending_upload, current DB >> power_state: >> >> > 1, VM power_state: 4 handle_lifecycle_event >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 >> >> > INFO nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - >> - >> >> - - >> >> > -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] During >> >> sync_power_state >> >> > the instance has a pending task (image_pending_upload). Skip. >> >> > 2022-06-12 16:26:18.539 7 DEBUG nova.compute.manager >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync >> for >> >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 >> >> > 2022-06-12 16:26:18.565 7 DEBUG oslo_concurrency.lockutils [-] Lock >> >> > "25c8d676-e20a-4238 >> >> > -a45c-d51daa62b941" acquired by >> >> > >> >> >> "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" >> >> > :: waited 0.000s inner >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 >> >> > 2022-06-12 16:26:18.566 7 INFO nova.compute.manager [-] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the >> >> instance >> >> > has a pending task (image_pending_upload). Skip. >> >> > 2022-06-12 16:26:18.566 7 DEBUG oslo_concurrency.lockutils [-] Lock >> >> > "25c8d676-e20a-4238-a45c-d51daa62b941" released by >> >> > >> >> >> "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" >> >> > :: held 0.001s inner >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371 >> >> > 2022-06-12 16:26:25.769 7 DEBUG oslo_concurrency.processutils >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> >> > ef940663426b4c87a751afaf13b758e0 - default default] CMD "qemu-img >> convert >> >> > -t none -O raw -f raw >> >> > >> >> >> rbd:vms/25c8d676-e20a-4238-a45c-d51daa6b941_disk:id=cinder:conf=/etc/ceph/ceph.conf >> >> > >> >> >> /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad" >> >> > returned: 0 in 23.730s execute >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423 >> >> > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] >> >> Snapshot >> >> > extracted, beginning image upload >> >> > 2022-06-12 16:26:27.981 7 DEBUG nova.virt.driver >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] Emitting event >> >> > <LifecycleEvent: 1655034987.9807608, >> 25c8d676-e20a-4238-a45c-d51daa62b941 >> >> > => Started> emit_event >> >> > >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704 >> >> > 2022-06-12 16:26:27.983 7 INFO nova.compute.manager >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event) >> >> > [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state >> >> > _get_power_state >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569 >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power >> state >> >> > after lifecycle event "Started"; current vm_state: active, current >> >> > task_state: image_pending_upload, current DB power_state: 1, VM >> >> > power_state: 1 handle_lifecycle_event >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299 >> >> > 2022-06-12 16:26:28.173 7 INFO nova.compute.manager >> >> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Resumed (Lifecycle Event >> >> > 2022-06-12 16:29:00.859 7 DEBUG oslo_concurrency.lockutils >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Acquired lock >> >> > "refresh_cache-25c8d676-e20a-4238-a45c-d51daa62b941" lock >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:266 >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Forcefully refreshing network >> info >> >> > cache for instance _get_instance_nw_info >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:1833 >> >> > 2022-06-12 16:29:03.278 7 DEBUG nova.network.neutron >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Updating instance_info_cache >> with >> >> > network_info: [{"id": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", >> "address": >> >> > "fa:16:3e:ca:00:d9", "network": {"id": >> >> > "b86c8304-a9bd-4b39-b7fc-f70ffe76f2a8", "bridge": "br-int", "label": >> >> > "External_Network", "subnets": [{"cidr": "141.11.42.0/24", "dns": >> >> > [{"address": "8.8.8.8", "type": "dns", "version": 4, "meta": {}}, >> >> > {"address": "217.218.127.127", "type": "dns", "version": 4, "meta": >> {}}], >> >> > "gateway": {"address": "141.11.42.1", "type": "gateway", "version": 4, >> >> > "meta": {}}, "ips": [{"address": "141.11.42.37", "type": "fixed", >> >> > "version": 4, "meta": {}, "floating_ips": []}], "routes": [], >> "version": >> >> 4, >> >> > "meta": {}}], "meta": {"injected": true, "tenant_id": >> >> > "ef940663426b4c87a751afaf13b758e0", "mtu": 1500, "physical_network": >> >> > "physnet1", "tunneled": false}}, "type": "ovs", "details": >> >> {"connectivity": >> >> > "l2", "port_filter": true, "ovs_hybrid_plug": true, "datapath_type": >> >> > "system", "bridge_name": "br-int"}, "devname": "tapaa2fdd7d-ad", >> >> > "ovs_interfaceid": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", >> "qbh_params": >> >> > null, "qbg_params": null, "active": true, "vnic_type": "normal", >> >> "profile": >> >> > {}, "preserve_on_delete": false, "meta": {}}] >> >> > update_instance_cache_with_nw_info >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:117 >> >> > nstance 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this >> >> > compute host and has allocations in placement: {'resources': {'VCPU': >> 1, >> >> > 'MEMORY_MB': 1024, 'DISK_GB': 20}}. >> _remove_deleted_instances_allocations >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 >> >> > 2022-06-12 16:33:37.595 7 INFO nova.compute.manager >> >> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 >> >> 4dbffaa9c14e401c8c210e23ebe0ab7b >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Took 461.98 seconds to snapshot >> the >> >> > instance on the hypervisor. >> >> > 2022-06-12 16:36:16.459 7 DEBUG nova.compute.manager >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync >> for >> >> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693 >> >> > Lock "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by >> >> > >> >> >> "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync" >> >> > :: waited 0.000s inner >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359 >> >> > 2022-06-12 16:37:05.365 7 DEBUG nova.compute.resource_tracker >> >> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this compute >> >> host >> >> > and has allocations in placement: {'resources': {'VCPU': 1, >> 'MEMORY_MB': >> >> > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations >> >> > >> >> >> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539 >> >> > 2022-06-12 09:42:32.687 7 INFO nova.compute.manager >> >> > [req-e79e4177-4712-4795-91da-853bc524fac0 >> >> 93fb420b3c604d4fae408b81135b58e9 >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting >> >> > >> >> > On Sun, Jun 12, 2022 at 3:36 PM Eugen Block <eblock@nde.ag> wrote: >> >> > >> >> >> Have you tried with debug logs? Has it worked with live snapshots >> >> >> before for other instances or did it never work and all snapshots >> were >> >> >> "cold"? >> >> >> >> >> >> Zitat von Parsa Aminian <p.aminian.server@gmail.com>: >> >> >> >> >> >> > Hi >> >> >> > kolla-ansible victoria version with ceph backend without volume >> >> >> > >> >> >> > On Sun, Jun 12, 2022 at 12:45 PM Eugen Block <eblock@nde.ag> >> wrote: >> >> >> > >> >> >> >> Hi, >> >> >> >> >> >> >> >> can you share more details about your environment? Which openstack >> >> >> >> version is it? What is the storage backend? In earlier releases >> there >> >> >> >> was an option: >> >> >> >> >> >> >> >> #disable_libvirt_livesnapshot = false >> >> >> >> >> >> >> >> but this option has been deprecated. But if you're on an older >> >> version >> >> >> >> that could explain it. >> >> >> >> >> >> >> >> Zitat von Parsa Aminian <p.aminian.server@gmail.com>: >> >> >> >> >> >> >> >> > When I snapshot from the instance , server will gone away and >> its >> >> not >> >> >> >> > reachable until the snapshot is complete here is the logs : >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting >> >> >> >> > 2022-06-12 09:42:34.755 7 INFO nova.compute.manager >> >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle >> Event) >> >> >> >> > 2022-06-12 09:42:34.995 7 INFO nova.compute.manager >> >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state >> the >> >> >> >> instance >> >> >> >> > has a pending task (image_pending_upload). Skip. >> >> >> >> > 2022-06-12 09:42:57.749 7 INFO nova.compute.manager [-] >> [instance: >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle >> Event) >> >> >> >> > 2022-06-12 09:43:06.102 7 INFO nova.virt.libvirt.driver >> >> >> >> > [req-e79e4177-4712-4795-91da-853bc524fac0 >> >> >> >> 93fb420b3c604d4fae408b81135b58e9 >> >> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance: >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Snapshot extracted, >> beginning >> >> >> image >> >> >> >> > upload >> >> >> >> > 2022-06-12 09:43:08.778 7 INFO nova.compute.manager >> >> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance: >> >> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle >> Event) >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>