13 Jun
2022
13 Jun
'22
12:33 a.m.
Hi
kolla-ansible victoria version with ceph backend .
rbd info output :
rbd image '25c8d676-e20a-4238-a45c-d51daa62b941_disk':
size 20 GiB in 5120 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: b69aaf907284da
block_name_prefix: rbd_data.b69aaf907284da
format: 2
features: layering, exclusive-lock, object-map, fast-diff,
deep-flatten
op_features:
flags:
create_timestamp: Fri May 20 00:04:47 2022
access_timestamp: Sun Jun 12 16:26:02 2022
---------------
also live snapshot seems to work correctly without any error or any
downtime :
docker exec -u root -it ceph-mgr-cephosd01 rbd snap ls
vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk
SNAPID NAME SIZE PROTECTED TIMESTAMP
344 test-snap 20 GiB Sun Jun 12 23:48:39 2022
also on compute nova.conf, images_type is set on rbd .
On Sun, Jun 12, 2022 at 5:55 PM Eugen Block <eblock@nde.ag> wrote:
> You should respond to the list so other users can try to support you.
>
> So nova is trying to live snapshot the instance:
>
> > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager
> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7
> > 4dbffaa9c14e401c8c210e23ebe0ab7b ef940663426b4c87a751afaf13b758e0 -
> > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941]
> > instance snapshotting
> > [...] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning
> > live snapshot process
>
> But I don't see any 'rbd snap create' command. Either the rbd image
> doesn't support it or there is some setting to keep all rbd images
> "flat". Can you check any relevant configs you might have in nova?
> Also can you show the output of 'rbd info
> <pool>/25c8d676-e20a-4238-a45c-d51daa62b941_disk' ? Then to test if
> the underlying rbd functions work as expected you could try to create
> a live snapshot manually:
>
> rbd snap create <pool>/25c8d676-e20a-4238-a45c-d51daa62b941_disk@test-snap
>
> And paste any relevant output here.
>
> Zitat von Parsa Aminian <p.aminian.server@gmail.com>:
>
> > Its not working for any instances and all of them are paused . I enable
> > debug logs please check the logs :
> >
> > 2022-06-12 16:16:13.478 7 DEBUG nova.compute.manager
> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync for
> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693
> > 2022-06-12 16:16:13.506 7 DEBUG oslo_concurrency.lockutils [-] Lock
> > "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by
> >
> "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync"
> > :: waited 0.000s inner
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
> > 2022-06-12 16:16:43.562 7 DEBUG nova.compute.resource_tracker
> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance
> > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this compute
> host
> > and has allocations in placement: {'resources': {'VCPU': 1, 'MEMORY_MB':
> > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539
> > 2022-06-12 16:25:55.104 7 DEBUG nova.compute.manager
> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7
> 4dbffaa9c14e401c8c210e23ebe0ab7b
> > ef940663426b4c87a751afaf13b758e0 - default default] [instance:
> > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state _get_power_state
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569
> > 2022-06-12 16:25:55.603 7 INFO nova.compute.manager
> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7
> 4dbffaa9c14e401c8c210e23ebe0ab7b
> > ef940663426b4c87a751afaf13b758e0 - default default] [instance:
> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting
> > 63426b4c87a751afaf13b758e0 - default default] [instance:
> > 25c8d676-e20a-4238-a45c-d51daa62b941] Beginning live snapshot process
> > default default] Lazy-loading 'pci_devices' on Instance uuid
> > 25c8d676-e20a-4238-a45c-d51daa62b941 obj_load_attr
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101
> > 2022-06-12 16:25:57.250 7 DEBUG nova.objects.instance
> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7
> 4dbffaa9c14e401c8c210e23ebe0ab7b
> > ef940663426b4c87a751afaf13b758e0 - default default] Lazy-loading
> > 'pci_devices' on Instance uuid 25c8d676-e20a-4238-a45c-d51daa62b941
> > obj_load_attr
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/instance.py:1101
> > 2022-06-12 16:25:57.317 7 DEBUG nova.virt.driver [-] Emitting event
> > <LifecycleEvent: 1655034957.3158934, 25c8d676-e20a-4238-a45c-d51daa62b941
> > => Paused> emit_event
> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704
> > 2022-06-12 16:25:57.318 7 INFO nova.compute.manager [-] [instance:
> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event)
> > 2022-06-12 16:25:57.389 7 DEBUG nova.compute.manager
> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance:
> > 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state _get_power_state
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569
> > 2022-06-12 16:25:57.395 7 DEBUG nova.compute.manager
> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance:
> > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power state
> > after lifecycle event "Paused"; current vm_state: active, current
> > task_state: image_pending_upload, current DB power_state: 1, VM
> > power_state: 3 handle_lifecycle_event
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299
> > 2022-06-12 16:25:57.487 7 INFO nova.compute.manager
> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance:
> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the
> instance
> > has a pending task (image_pending_upload). Skip.
> > 2022-06-12 16:26:02.039 7 DEBUG oslo_concurrency.processutils
> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7
> 4dbffaa9c14e401c8c210e23ebe0ab7b
> > ef940663426b4c87a751afaf13b758e0 - default default] Running cmd
> > (subprocess): qemu-img convert -t none -O raw -f raw
> >
> rbd:vms/25c8d676-e20a-4238-a45c-d51daa62b941_disk:id=cinder:conf=/etc/ceph/ceph.conf
> >
> /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad
> > execute
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:384
> > 2022-06-12 16:26:17.075 7 DEBUG nova.virt.driver [-] Emitting event
> > <LifecycleEvent: 1655034962.0316682, 25c8d676-e20a-4238-a45c-d51daa62b941
> > => Stopped> emit_event
> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704
> > INFO nova.compute.manager [-] [instance:
> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event)
> > DEBUG nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - -
> -
> > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state
> > _get_power_state
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569
> > DEBUG nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - -
> -
> > - -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing
> > instance power state after lifecycle event "Stopped"; current vm_state:
> > active, current task_state: image_pending_upload, current DB power_state:
> > 1, VM power_state: 4 handle_lifecycle_event
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299
> > INFO nova.compute.manager [req-f9f8cbf5-6208-4dca-aca6-48dee87f38fa - -
> - -
> > -] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] During
> sync_power_state
> > the instance has a pending task (image_pending_upload). Skip.
> > 2022-06-12 16:26:18.539 7 DEBUG nova.compute.manager
> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync for
> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693
> > 2022-06-12 16:26:18.565 7 DEBUG oslo_concurrency.lockutils [-] Lock
> > "25c8d676-e20a-4238
> > -a45c-d51daa62b941" acquired by
> >
> "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync"
> > :: waited 0.000s inner
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
> > 2022-06-12 16:26:18.566 7 INFO nova.compute.manager [-] [instance:
> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the
> instance
> > has a pending task (image_pending_upload). Skip.
> > 2022-06-12 16:26:18.566 7 DEBUG oslo_concurrency.lockutils [-] Lock
> > "25c8d676-e20a-4238-a45c-d51daa62b941" released by
> >
> "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync"
> > :: held 0.001s inner
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:371
> > 2022-06-12 16:26:25.769 7 DEBUG oslo_concurrency.processutils
> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7
> 4dbffaa9c14e401c8c210e23ebe0ab7b
> > ef940663426b4c87a751afaf13b758e0 - default default] CMD "qemu-img convert
> > -t none -O raw -f raw
> >
> rbd:vms/25c8d676-e20a-4238-a45c-d51daa6b941_disk:id=cinder:conf=/etc/ceph/ceph.conf
> >
> /var/lib/nova/instances/snapshots/tmpv21b_i59/8717dec4c99c4ef7bac752e2a48690ad"
> > returned: 0 in 23.730s execute
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:423
> > default default] [instance: 25c8d676-e20a-4238-a45c-d51daa62b941]
> Snapshot
> > extracted, beginning image upload
> > 2022-06-12 16:26:27.981 7 DEBUG nova.virt.driver
> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] Emitting event
> > <LifecycleEvent: 1655034987.9807608, 25c8d676-e20a-4238-a45c-d51daa62b941
> > => Started> emit_event
> > /var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/driver.py:1704
> > 2022-06-12 16:26:27.983 7 INFO nova.compute.manager
> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance:
> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event)
> > [instance: 25c8d676-e20a-4238-a45c-d51daa62b941] Checking state
> > _get_power_state
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1569
> > 25c8d676-e20a-4238-a45c-d51daa62b941] Synchronizing instance power state
> > after lifecycle event "Started"; current vm_state: active, current
> > task_state: image_pending_upload, current DB power_state: 1, VM
> > power_state: 1 handle_lifecycle_event
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:1299
> > 2022-06-12 16:26:28.173 7 INFO nova.compute.manager
> > [req-40444d74-f2fa-4569-87dd-375139938e81 - - - - -] [instance:
> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Resumed (Lifecycle Event
> > 2022-06-12 16:29:00.859 7 DEBUG oslo_concurrency.lockutils
> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Acquired lock
> > "refresh_cache-25c8d676-e20a-4238-a45c-d51daa62b941" lock
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:266
> > 25c8d676-e20a-4238-a45c-d51daa62b941] Forcefully refreshing network info
> > cache for instance _get_instance_nw_info
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:1833
> > 2022-06-12 16:29:03.278 7 DEBUG nova.network.neutron
> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] [instance:
> > 25c8d676-e20a-4238-a45c-d51daa62b941] Updating instance_info_cache with
> > network_info: [{"id": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", "address":
> > "fa:16:3e:ca:00:d9", "network": {"id":
> > "b86c8304-a9bd-4b39-b7fc-f70ffe76f2a8", "bridge": "br-int", "label":
> > "External_Network", "subnets": [{"cidr": "141.11.42.0/24", "dns":
> > [{"address": "8.8.8.8", "type": "dns", "version": 4, "meta": {}},
> > {"address": "217.218.127.127", "type": "dns", "version": 4, "meta": {}}],
> > "gateway": {"address": "141.11.42.1", "type": "gateway", "version": 4,
> > "meta": {}}, "ips": [{"address": "141.11.42.37", "type": "fixed",
> > "version": 4, "meta": {}, "floating_ips": []}], "routes": [], "version":
> 4,
> > "meta": {}}], "meta": {"injected": true, "tenant_id":
> > "ef940663426b4c87a751afaf13b758e0", "mtu": 1500, "physical_network":
> > "physnet1", "tunneled": false}}, "type": "ovs", "details":
> {"connectivity":
> > "l2", "port_filter": true, "ovs_hybrid_plug": true, "datapath_type":
> > "system", "bridge_name": "br-int"}, "devname": "tapaa2fdd7d-ad",
> > "ovs_interfaceid": "aa2fdd7d-ad18-4890-ad57-14bf9888d2c1", "qbh_params":
> > null, "qbg_params": null, "active": true, "vnic_type": "normal",
> "profile":
> > {}, "preserve_on_delete": false, "meta": {}}]
> > update_instance_cache_with_nw_info
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/network/neutron.py:117
> > nstance 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this
> > compute host and has allocations in placement: {'resources': {'VCPU': 1,
> > 'MEMORY_MB': 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539
> > 2022-06-12 16:33:37.595 7 INFO nova.compute.manager
> > [req-5ecfdf74-7cf3-481a-aa12-140deae202f7
> 4dbffaa9c14e401c8c210e23ebe0ab7b
> > ef940663426b4c87a751afaf13b758e0 - default default] [instance:
> > 25c8d676-e20a-4238-a45c-d51daa62b941] Took 461.98 seconds to snapshot the
> > instance on the hypervisor.
> > 2022-06-12 16:36:16.459 7 DEBUG nova.compute.manager
> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Triggering sync for
> > uuid 25c8d676-e20a-4238-a45c-d51daa62b941 _sync_power_states
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:9693
> > Lock "25c8d676-e20a-4238-a45c-d51daa62b941" acquired by
> >
> "nova.compute.manager.ComputeManager._sync_power_states.<locals>._sync.<locals>.query_driver_power_state_and_sync"
> > :: waited 0.000s inner
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:359
> > 2022-06-12 16:37:05.365 7 DEBUG nova.compute.resource_tracker
> > [req-2ecf34c3-72e7-4f33-89cb-9b250cd6d223 - - - - -] Instance
> > 25c8d676-e20a-4238-a45c-d51daa62b941 actively managed on this compute
> host
> > and has allocations in placement: {'resources': {'VCPU': 1, 'MEMORY_MB':
> > 1024, 'DISK_GB': 20}}. _remove_deleted_instances_allocations
> >
> /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py:1539
> > 2022-06-12 09:42:32.687 7 INFO nova.compute.manager
> > [req-e79e4177-4712-4795-91da-853bc524fac0
> 93fb420b3c604d4fae408b81135b58e9
> > ef940663426b4c87a751afaf13b758e0 - default default] [instance:
> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting
> >
> > On Sun, Jun 12, 2022 at 3:36 PM Eugen Block <eblock@nde.ag> wrote:
> >
> >> Have you tried with debug logs? Has it worked with live snapshots
> >> before for other instances or did it never work and all snapshots were
> >> "cold"?
> >>
> >> Zitat von Parsa Aminian <p.aminian.server@gmail.com>:
> >>
> >> > Hi
> >> > kolla-ansible victoria version with ceph backend without volume
> >> >
> >> > On Sun, Jun 12, 2022 at 12:45 PM Eugen Block <eblock@nde.ag> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> can you share more details about your environment? Which openstack
> >> >> version is it? What is the storage backend? In earlier releases there
> >> >> was an option:
> >> >>
> >> >> #disable_libvirt_livesnapshot = false
> >> >>
> >> >> but this option has been deprecated. But if you're on an older
> version
> >> >> that could explain it.
> >> >>
> >> >> Zitat von Parsa Aminian <p.aminian.server@gmail.com>:
> >> >>
> >> >> > When I snapshot from the instance , server will gone away and its
> not
> >> >> > reachable until the snapshot is complete here is the logs :
> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] instance snapshotting
> >> >> > 2022-06-12 09:42:34.755 7 INFO nova.compute.manager
> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance:
> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Paused (Lifecycle Event)
> >> >> > 2022-06-12 09:42:34.995 7 INFO nova.compute.manager
> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance:
> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] During sync_power_state the
> >> >> instance
> >> >> > has a pending task (image_pending_upload). Skip.
> >> >> > 2022-06-12 09:42:57.749 7 INFO nova.compute.manager [-] [instance:
> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Stopped (Lifecycle Event)
> >> >> > 2022-06-12 09:43:06.102 7 INFO nova.virt.libvirt.driver
> >> >> > [req-e79e4177-4712-4795-91da-853bc524fac0
> >> >> 93fb420b3c604d4fae408b81135b58e9
> >> >> > ef940663426b4c87a751afaf13b758e0 - default default] [instance:
> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] Snapshot extracted, beginning
> >> image
> >> >> > upload
> >> >> > 2022-06-12 09:43:08.778 7 INFO nova.compute.manager
> >> >> > [req-786946b1-3d22-489c-bf4d-8b1375b09ecb - - - - -] [instance:
> >> >> > 25c8d676-e20a-4238-a45c-d51daa62b941] VM Started (Lifecycle Event)
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >>
> >>
> >>
> >>
> >>
>
>
>
>
>