<div dir="ltr"><div>Hi</div>kolla-ansible victoria version with ceph backend . <div><div>rbd info output :</div><div>rbd image '25c8d676-e20a-4238-a45c-d51daa62b941_disk':<br> size 20 GiB in 5120 objects<br> order 22 (4 MiB objects)<br> snapshot_count: 0<br> id: b69aaf907284da<br> block_name_prefix: rbd_data.b69aaf907284da<br> format: 2<br> features: layering, exclusive-lock, object-map, fast-diff, deep-flatten<br> op_features:<br> flags:<br> create_timestamp: Fri May 20 00:04:47 2022<br> access_timestamp: Sun Jun 12 16:26:02 2022<br></div><div>---------------</div><div>also live snapshot seems to work correctly without any error or any downtime :</div><div>docker exec -u root -it ceph-mgr-cephosd01 rbd snap ls vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk<br>SNAPID NAME SIZE PROTECTED TIMESTAMP<br> 344 test-snap 20 GiB Sun Jun 12 23:48:39 2022</div></div><div><br></div><div>also on compute nova.conf, images_type is set on rbd .</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jun 12, 2022 at 5:55 PM Eugen Block <<a href="mailto:eblock@nde.ag">eblock@nde.ag</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">You should respond to the list so other users can try to support you.<br>
<br>
So nova is trying to live snapshot the instance:<br>
<br>
> 2022-06-12 16:25:55.603 7 INFO nova.compute.manager <br>
> [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 <br>
> 4dbffaa9c14e401c8c210e23ebe0ab7b ef940663426b4c87a751afaf13b758e0 - <br>
> default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] <br>
> instance snapshotting<br>
> [...] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning <br>
> live snapshot process<br>
<br>
But I don't see any 'rbd snap create' command. Either the rbd image <br>
doesn't support it or there is some setting to keep all rbd images <br>
"flat". Can you check any relevant configs you might have in nova? <br>
Also can you show the output of 'rbd info <br>
<pool>/25c8d676-e20a-4238-a45c-d51daa62b941_disk' ? Then to test if <br>
the underlying rbd functions work as expected you could try to create <br>
a live snapshot manually:<br>
<br>
rbd snap create <pool>/25c8d676-e20a-4238-a45c-d51daa62b941_disk@test-snap<br>
<br>
And paste any relevant output here.<br>
<br>
Zitat von Parsa Aminian <<a href="mailto:p.aminian.server@gmail.com" target="_blank">p.aminian.server@gmail.com</a>>:<br>
<br>
> Its not working for any instances and all of them are paused . I enable<br>
> debug logs please check the logs :<br>
><br>
> 2022-06-12 16:16:13.478 7 DEBUG nova.compute.manager<br>
> [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync for<br>
> uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693<br>
> 2022-06-12 16:16:13.506 7 DEBUG oslo_concurrency.lockutils [-] Lock<br>
> "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by<br>
> "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync"<br>
> :: waited 0.000s inner<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359<br>
> 2022-06-12 16:16:43.562 7 DEBUG nova.compute.resource_tracker<br>
> [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this compute host<br>
> and has allocations in placement: {'resources': {'VCPU': 1, 'MEMORY_MB':<br>
> 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539<br>
> 2022-06-12 16:25:55.104 7 DEBUG nova.compute.manager<br>
> [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 4dbffaa9c14e401c8c210e23ebe0ab7b<br>
> ef940663426b4c87a751afaf13b758e0 - default default] [instance:<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state _get_power_state<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569<br>
> 2022-06-12 16:25:55.603 7 INFO nova.compute.manager<br>
> [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 4dbffaa9c14e401c8c210e23ebe0ab7b<br>
> ef940663426b4c87a751afaf13b758e0 - default default] [instance:<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting<br>
> 63426b4c87a751afaf13b758e0 - default default] [instance:<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning live snapshot process<br>
> default default] Lazy-loading 'pci_devices' on Instance uuid<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941 obj_load_attr<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101<br>
> 2022-06-12 16:25:57.250 7 DEBUG nova.objects.instance<br>
> [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 4dbffaa9c14e401c8c210e23ebe0ab7b<br>
> ef940663426b4c87a751afaf13b758e0 - default default] Lazy-loading<br>
> 'pci_devices' on Instance uuid 25c8d676-e20a-4238-a45c-d51daa62b941<br>
> obj_load_attr<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101<br>
> 2022-06-12 16:25:57.317 7 DEBUG nova.virt.driver [-] Emitting event<br>
> <LifecycleEvent: 1655034957.3158934, 25c8d676-e20a-4238-a45c-d51daa62b941<br>
> => Paused> emit_event<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704<br>
> 2022-06-12 16:25:57.318 7 INFO nova.compute.manager [-] [instance:<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event)<br>
> 2022-06-12 16:25:57.389 7 DEBUG nova.compute.manager<br>
> [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance:<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state _get_power_state<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569<br>
> 2022-06-12 16:25:57.395 7 DEBUG nova.compute.manager<br>
> [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance:<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power state<br>
> after lifecycle event "Paused"; current vm_state: active, current<br>
> task_state: image_pending_upload, current DB power_state: 1, VM<br>
> power_state: 3 handle_lifecycle_event<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299<br>
> 2022-06-12 16:25:57.487 7 INFO nova.compute.manager<br>
> [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance:<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the instance<br>
> has a pending task (image_pending_upload). Skip.<br>
> 2022-06-12 16:26:02.039 7 DEBUG oslo_concurrency.processutils<br>
> [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 4dbffaa9c14e401c8c210e23ebe0ab7b<br>
> ef940663426b4c87a751afaf13b758e0 - default default] Running cmd<br>
> (subprocess): qemu-img convert -t none -O raw -f raw<br>
> rbd:vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk:id=cinder:conf=/etc/ceph/ceph.conf<br>
> /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad<br>
> execute<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384<br>
> 2022-06-12 16:26:17.075 7 DEBUG nova.virt.driver [-] Emitting event<br>
> <LifecycleEvent: 1655034962.0316682, 25c8d676-e20a-4238-a45c-d51daa62b941<br>
> => Stopped> emit_event<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704<br>
> INFO nova.compute.manager [-] [instance:<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event)<br>
> DEBUG nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - - -<br>
> - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state<br>
> _get_power_state<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569<br>
> DEBUG nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - - -<br>
> - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing<br>
> instance power state after lifecycle event "Stopped"; current vm_state:<br>
> active, current task_state: image_pending_upload, current DB power_state:<br>
> 1, VM power_state: 4 handle_lifecycle_event<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299<br>
> INFO nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - - - -<br>
> -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state<br>
> the instance has a pending task (image_pending_upload). Skip.<br>
> 2022-06-12 16:26:18.539 7 DEBUG nova.compute.manager<br>
> [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync for<br>
> uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693<br>
> 2022-06-12 16:26:18.565 7 DEBUG oslo_concurrency.lockutils [-] Lock<br>
> "25c8d676-e20a-4238<br>
> -a45c-d51daa62b941" acquired by<br>
> "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync"<br>
> :: waited 0.000s inner<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359<br>
> 2022-06-12 16:26:18.566 7 INFO nova.compute.manager [-] [instance:<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the instance<br>
> has a pending task (image_pending_upload). Skip.<br>
> 2022-06-12 16:26:18.566 7 DEBUG oslo_concurrency.lockutils [-] Lock<br>
> "25c8d676-e20a-4238-a45c-d51daa62b941" released by<br>
> "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync"<br>
> :: held 0.001s inner<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371<br>
> 2022-06-12 16:26:25.769 7 DEBUG oslo_concurrency.processutils<br>
> [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 4dbffaa9c14e401c8c210e23ebe0ab7b<br>
> ef940663426b4c87a751afaf13b758e0 - default default] CMD "qemu-img convert<br>
> -t none -O raw -f raw<br>
> rbd:vms/25c8d676-e20a-4238-a45c-d51daa6b941_disk:id=cinder:conf=/etc/ceph/ceph.conf<br>
> /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad"<br>
> returned: 0 in 23.730s execute<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423<br>
> default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Snapshot<br>
> extracted, beginning image upload<br>
> 2022-06-12 16:26:27.981 7 DEBUG nova.virt.driver<br>
> [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] Emitting event<br>
> <LifecycleEvent: 1655034987.9807608, 25c8d676-e20a-4238-a45c-d51daa62b941<br>
> => Started> emit_event<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704<br>
> 2022-06-12 16:26:27.983 7 INFO nova.compute.manager<br>
> [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance:<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event)<br>
> [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state<br>
> _get_power_state<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power state<br>
> after lifecycle event "Started"; current vm_state: active, current<br>
> task_state: image_pending_upload, current DB power_state: 1, VM<br>
> power_state: 1 handle_lifecycle_event<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299<br>
> 2022-06-12 16:26:28.173 7 INFO nova.compute.manager<br>
> [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance:<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941] VM Resumed (Lifecycle Event<br>
> 2022-06-12 16:29:00.859 7 DEBUG oslo_concurrency.lockutils<br>
> [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Acquired lock<br>
> "refresh_cache-25c8d676-e20a-4238-a45c-d51daa62b941" lock<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:266<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941] Forcefully refreshing network info<br>
> cache for instance _get_instance_nw_info<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:1833<br>
> 2022-06-12 16:29:03.278 7 DEBUG nova.network.neutron<br>
> [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] [instance:<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941] Updating instance_info_cache with<br>
> network_info: [{"id": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", "address":<br>
> "fa:16:3e:ca:00:d9", "network": {"id":<br>
> "b86c8304-a9bd-4b39-b7fc-f70ffe76f2a8", "bridge": "br-int", "label":<br>
> "External_Network", "subnets": [{"cidr": "<a href="http://141.11.42.0/24" rel="noreferrer" target="_blank">141.11.42.0/24</a>", "dns":<br>
> [{"address": "8.8.8.8", "type": "dns", "version": 4, "meta": {}},<br>
> {"address": "217.218.127.127", "type": "dns", "version": 4, "meta": {}}],<br>
> "gateway": {"address": "141.11.42.1", "type": "gateway", "version": 4,<br>
> "meta": {}}, "ips": [{"address": "141.11.42.37", "type": "fixed",<br>
> "version": 4, "meta": {}, "floating_ips": []}], "routes": [], "version": 4,<br>
> "meta": {}}], "meta": {"injected": true, "tenant_id":<br>
> "ef940663426b4c87a751afaf13b758e0", "mtu": 1500, "physical_network":<br>
> "physnet1", "tunneled": false}}, "type": "ovs", "details": {"connectivity":<br>
> "l2", "port_filter": true, "ovs_hybrid_plug": true, "datapath_type":<br>
> "system", "bridge_name": "br-int"}, "devname": "tapaa2fdd7d-ad",<br>
> "ovs_interfaceid": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", "qbh_params":<br>
> null, "qbg_params": null, "active": true, "vnic_type": "normal", "profile":<br>
> {}, "preserve_on_delete": false, "meta": {}}]<br>
> update_instance_cache_with_nw_info<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:117<br>
> nstance 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this<br>
> compute host and has allocations in placement: {'resources': {'VCPU': 1,<br>
> 'MEMORY_MB': 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539<br>
> 2022-06-12 16:33:37.595 7 INFO nova.compute.manager<br>
> [req-5ecfdf74-7cf3-481a-aa12-140deae202f7 4dbffaa9c14e401c8c210e23ebe0ab7b<br>
> ef940663426b4c87a751afaf13b758e0 - default default] [instance:<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941] Took 461.98 seconds to snapshot the<br>
> instance on the hypervisor.<br>
> 2022-06-12 16:36:16.459 7 DEBUG nova.compute.manager<br>
> [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync for<br>
> uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693<br>
> Lock "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by<br>
> "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync"<br>
> :: waited 0.000s inner<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359<br>
> 2022-06-12 16:37:05.365 7 DEBUG nova.compute.resource_tracker<br>
> [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this compute host<br>
> and has allocations in placement: {'resources': {'VCPU': 1, 'MEMORY_MB':<br>
> 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations<br>
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539<br>
> 2022-06-12 09:42:32.687 7 INFO nova.compute.manager<br>
> [req-e79e4177-4712-4795-91da-853bc524fac0 93fb420b3c604d4fae408b81135b58e9<br>
> ef940663426b4c87a751afaf13b758e0 - default default] [instance:<br>
> 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting<br>
><br>
> On Sun, Jun 12, 2022 at 3:36 PM Eugen Block <<a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a>> wrote:<br>
><br>
>> Have you tried with debug logs? Has it worked with live snapshots<br>
>> before for other instances or did it never work and all snapshots were<br>
>> "cold"?<br>
>><br>
>> Zitat von Parsa Aminian <<a href="mailto:p.aminian.server@gmail.com" target="_blank">p.aminian.server@gmail.com</a>>:<br>
>><br>
>> > Hi<br>
>> > kolla-ansible victoria version with ceph backend without volume<br>
>> ><br>
>> > On Sun, Jun 12, 2022 at 12:45 PM Eugen Block <<a href="mailto:eblock@nde.ag" target="_blank">eblock@nde.ag</a>> wrote:<br>
>> ><br>
>> >> Hi,<br>
>> >><br>
>> >> can you share more details about your environment? Which openstack<br>
>> >> version is it? What is the storage backend? In earlier releases there<br>
>> >> was an option:<br>
>> >><br>
>> >> #disable_libvirt_livesnapshot = false<br>
>> >><br>
>> >> but this option has been deprecated. But if you're on an older version<br>
>> >> that could explain it.<br>
>> >><br>
>> >> Zitat von Parsa Aminian <<a href="mailto:p.aminian.server@gmail.com" target="_blank">p.aminian.server@gmail.com</a>>:<br>
>> >><br>
>> >> > When I snapshot from the instance , server will gone away and its not<br>
>> >> > reachable until the snapshot is complete here is the logs :<br>
>> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting<br>
>> >> > 2022-06-12 09:42:34.755 7 INFO nova.compute.manager<br>
>> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance:<br>
>> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event)<br>
>> >> > 2022-06-12 09:42:34.995 7 INFO nova.compute.manager<br>
>> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance:<br>
>> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the<br>
>> >> instance<br>
>> >> > has a pending task (image_pending_upload). Skip.<br>
>> >> > 2022-06-12 09:42:57.749 7 INFO nova.compute.manager [-] [instance:<br>
>> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event)<br>
>> >> > 2022-06-12 09:43:06.102 7 INFO nova.virt.libvirt.driver<br>
>> >> > [req-e79e4177-4712-4795-91da-853bc524fac0<br>
>> >> 93fb420b3c604d4fae408b81135b58e9<br>
>> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance:<br>
>> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Snapshot extracted, beginning<br>
>> image<br>
>> >> > upload<br>
>> >> > 2022-06-12 09:43:08.778 7 INFO nova.compute.manager<br>
>> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance:<br>
>> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event)<br>
>> >><br>
>> >><br>
>> >><br>
>> >><br>
>> >><br>
>><br>
>><br>
>><br>
>><br>
>><br>
<br>
<br>
<br>
<br>
</blockquote></div>