[Nova] migration issue
Eugen Block
eblock at nde.ag
Wed Jul 24 19:37:32 UTC 2019
Hi,
is this the first migration test or have there already been some
successful migrations? If it's the first I would suggest to check if
the nova user can access the other node via ssh without password. Is
/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 present on the source node? Can you run the 'scp' command manually as nova
user?
Regards,
Eugen
Zitat von Budai Laszlo <laszlo.budai at gmail.com>:
> Dear all,
>
> we are testing the cold migratoin of instances that are using local
> ephemeral storage and it fails with the following error:
>
>
> 2019-07-24 13:47:13.115 58902 INFO nova.virt.libvirt.driver
> [req-356fce31-5e98-425f-89d6-8a98664d31ad
> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 -
> default default] [instance: 5a6dae
> 58-00d6-4317-b635-909fdf09ac49] Instance shutdown successfully after
> 3 seconds.
> 2019-07-24 13:47:13.126 58902 INFO nova.virt.libvirt.driver [-]
> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Instance destroyed
> successfully.
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager
> [req-356fce31-5e98-425f-89d6-8a98664d31ad
> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 -
> default default] [instance: 5a6dae58-
> 00d6-4317-b635-909fdf09ac49] Setting instance vm_state to ERROR:
> ProcessExecutionError: Unexpected error while running command.
> Command: scp -r
> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0
> 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph
>
> Exit code: 1
> Stdout: u''
> Stderr:
> u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured
> syst
> em and your actions will be logged along *\n* with identifying
> information. Disconnect immediately if you are not an *\n*
> authorized user of this system.
> *
> \n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No
> suc
> h file or directory\n'
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] Traceback (most recent call
> last):
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line
> 75
> 18, in _error_out_instance_on_exception
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] yield
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line
> 42
> 75, in _resize_instance
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] timeout, retry_interval)
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
> lin
> e 8200, in migrate_disk_and_power_off
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] shared_storage)
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line
> 220
> , in __exit__
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] self.force_reraise()
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line
> 196
> , in force_reraise
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] six.reraise(self.type_,
> self.value, self.tb)
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
> lin
> e 8185, in migrate_disk_and_power_off
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] compression=compression)
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py",
> line
> 226, in copy_image
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] compression=compression)
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs
> .py", line 110, in copy_file
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] compression=compression)
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs
> .py", line 196, in copy_file
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] on_execute=on_execute,
> on_completion=on_completion)
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py",
> line 231, in exec
> ute
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] return
> processutils.execute(*cmd, **kwargs)
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py"
> , line 424, in execute
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] cmd=sanitized_cmd)
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] ProcessExecutionError:
> Unexpected error while running command.
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] Command: scp -r
> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0
> 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] Exit code: 1
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] Stdout: u''
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] Stderr:
> u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or
> directory\n'
> 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49]
> 2019-07-24 13:47:14.380 58902 INFO nova.compute.manager
> [req-356fce31-5e98-425f-89d6-8a98664d31ad
> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 -
> default default] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]
> Swapping old allocation on 3da11b23-5ef8-4471-a3c9-5f743b0bbfd7 held
> by migration da66c416-2636-421b-99cb-470bd07dad4c for instance
> 2019-07-24 13:47:14.979 58902 INFO nova.compute.manager
> [req-356fce31-5e98-425f-89d6-8a98664d31ad
> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 -
> default default] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49]
> Successfully reverted task state from resize_migrating on failure
> for instance.
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> [req-356fce31-5e98-425f-89d6-8a98664d31ad
> 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 -
> default default] Exception during message handling:
> ProcessExecutionError: Unexpected error while running command.
> Command: scp -r
> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0
> 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0
> Exit code: 1
> Stdout: u''
> Stderr:
> u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or
> directory\n'
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> Traceback (most recent call last):
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in
> _process_incoming
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> res = self.dispatcher.dispatch(message)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in
> dispatch
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> return self._do_dispatch(endpoint, method, ctxt, args)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in
> _do_dispatch
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> result = func(ctxt, **new_args)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in
> wrapped
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> function_name, call_dict, binary)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> self.force_reraise()
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> six.reraise(self.type_, self.value, self.tb)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in
> wrapped
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> return f(self, context, *args, **kw)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 186, in
> decorated_function
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> "Error: %s", e, instance=instance)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> self.force_reraise()
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> six.reraise(self.type_, self.value, self.tb)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 156, in
> decorated_function
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> return function(self, context, *args, **kwargs)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/utils.py", line 977, in
> decorated_function
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> return function(self, context, *args, **kwargs)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in
> decorated_function
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> kwargs['instance'], e, sys.exc_info())
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> self.force_reraise()
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> six.reraise(self.type_, self.value, self.tb)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in
> decorated_function
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> return function(self, context, *args, **kwargs)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4240, in
> resize_instance
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> self._revert_allocation(context, instance, migration)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> self.force_reraise()
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> six.reraise(self.type_, self.value, self.tb)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4237, in
> resize_instance
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> instance_type, clean_shutdown)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4275, in
> _resize_instance
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> timeout, retry_interval)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8200, in
> migrate_disk_and_power_off
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> shared_storage)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> self.force_reraise()
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> six.reraise(self.type_, self.value, self.tb)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8185, in
> migrate_disk_and_power_off
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> compression=compression)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 226, in
> copy_image
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> compression=compression)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 110, in
> copy_file
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> compression=compression)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 196, in
> copy_file
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> on_execute=on_execute, on_completion=on_completion)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py",
> line 231, in execute
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> return processutils.execute(*cmd, **kwargs)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File
> "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in
> execute
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> cmd=sanitized_cmd)
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> ProcessExecutionError: Unexpected error while running command.
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> Command: scp -r
> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0
> 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Exit code: 1
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Stdout: u''
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> Stderr:
> u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or
> directory\n'
> 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server
> 2019-07-24 13:47:17.065 58902 INFO nova.virt.libvirt.imagecache
> [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] image
> ca89475a-cf15-46d1-ab2d-c2b96c403248 at
> (/var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07):
> checking
> 2019-07-24 13:47:17.066 58902 INFO nova.virt.libvirt.imagecache
> [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] Active base
> files:
> /var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07
> 2019-07-24 13:47:27.768 58902 INFO nova.compute.manager [-]
> [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] VM Stopped
> (Lifecycle Event)
> 2019-07-24 13:47:27.918 58902 INFO nova.compute.manager
> [req-e53e928d-81fe-4185-9e62-abc2113e838c - - - - -] [instance:
> 5a6dae58-00d6-4317-b635-909fdf09ac49] During
> _sync_instance_power_state the DB power_state (1) does not match the
> vm_power_state from the hypervisor (4). Updating power_state in the
> DB to match the hypervisor.
>
>
> The disk has its root FS n an RBD device (a volume created by
> cinder) and it has its ephemeral disk on a local LVM volume created
> by nova.
>
> It seems that the
> /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 is not there when scp is trying to copy it to the other
> side.
>
> Any ideas how to fix this?
>
> The nova config on the nodes looks like this:
>
> # Ansible managed
> [DEFAULT]
> allow_resize_to_same_host = True
> # Compute
> compute_driver = libvirt.LibvirtDriver
> # Scheduler
> cpu_allocation_ratio = 2.0
> # Logs / State
> debug = False
> # Hypervisor
> default_ephemeral_format = ext4
> disk_allocation_ratio = 1.0
> # Api's
> enabled_apis = osapi_compute,metadata
> executor_thread_pool_size = 64
> fatal_deprecations = False
> # Configdrive
> force_config_drive = False
> host = compute8.mgmt.lab.mydomain.com
> image_cache_manager_interval = 0
> instance_name_template = instance-%08x
> # Ceilometer notification configurations
> instance_usage_audit = True
> instance_usage_audit_period = hour
> instances_path = /var/lib/nova/instances
> ## Vif
> libvirt_vif_type = ethernet
> log_dir = /var/log/nova
> # Metadata
> metadata_workers = 16
> # Network
> my_ip = 192.168.56.47
> notify_on_state_change = vm_and_task_state
> osapi_compute_workers = 16
> ram_allocation_ratio = 1.0
> reserved_host_disk_mb = 0
> reserved_host_memory_mb = 2048
> resume_guests_state_on_host_boot = False
> rootwrap_config = /etc/nova/rootwrap.conf
> rpc_response_timeout = 60
> service_down_time = 120
> state_path = /var/lib/nova
> # Rpc all
> transport_url =
> rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7@192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7@192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e
> 8cf8bd7 at 192.168.56.179:5671//nova
> # Disable stderr logging
> use_stderr = False
> vif_plugging_is_fatal = True
> vif_plugging_timeout = 30
>
> [api]
> auth_strategy = keystone
> enable_instance_password = True
> use_forwarded_for = False
> vendordata_jsonfile_path = /etc/nova/vendor_data.json
>
> # Cache
> [cache]
> backend = oslo_cache.memcache_pool
> enabled = true
> memcache_servers =
> 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211
>
> # Cinder
> [cinder]
> cafile = /etc/ssl/certs/ca-certificates.crt
> catalog_info = volumev3:cinderv3:internalURL
> cross_az_attach = True
> os_region_name = A_Lab
>
> [conductor]
> workers = 16
>
> [filter_scheduler]
> available_filters = nova.scheduler.filters.all_filters
> enabled_filters = RetryFilter, AvailabilityZoneFilter, RamFilter,
> AggregateRamFilter, ComputeFilter, AggregateCoreFilter, DiskFilter,
> AggregateDiskFilter, AggregateNumInstancesFilter,
> AggregateIoOpsFilter, ComputeCapabilitiesFilter,
> ImagePropertiesFilter, ServerGroupAntiAffinityFilter,
> ServerGroupAffinityFilter, NUMATopologyFilter, SameHostFilter,
> DifferentHostFilter
> host_subset_size = 10
> max_instances_per_host = 50
> max_io_ops_per_host = 10
> ram_weight_multiplier = 5.0
> tracks_instance_changes = True
> weight_classes = nova.scheduler.weights.all_weighers
>
> # Glance
> [glance]
> api_servers = https://vip.mgmt.lab.mydomain.com:9292
> cafile = /etc/ssl/certs/ca-certificates.crt
>
> [key_manager]
> fixed_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
>
> [keystone_authtoken]
> auth_type = password
> auth_uri = https://vip.mgmt.lab.mydomain.com:5000
> auth_url = https://vip.mgmt.lab.mydomain.com:35357
> cafile = /etc/ssl/certs/ca-certificates.crt
> insecure = False
> memcache_secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> # if your memcached server is shared, use these settings to avoid
> cache poisoning
> memcache_security_strategy = ENCRYPT
> memcached_servers =
> 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211
> password = xxxxxxxxxxxxxxxxxxxxxxx
>
> project_domain_id = default
> project_name = service
> region_name = A_Lab
> token_cache_time = 300
> user_domain_id = default
> username = nova
>
> [libvirt]
> disk_cachemodes = network=writeback
> hw_disk_discard = unmap
> images_rbd_ceph_conf = /etc/ceph/ceph.conf
> images_rbd_pool = vms
> images_type = lvm
> images_volume_group = ssd_nova
> inject_key = False
> inject_partition = -2
> inject_password = False
> live_migration_tunnelled = True
> live_migration_uri =
> "qemu+ssh://nova@%s/system?no_verify=1&keyfile=/var/lib/nova/.ssh/id_rsa"
> rbd_secret_uuid = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> # ceph rbd support
> rbd_user = cinder
> remove_unused_resized_minimum_age_seconds = 3600
> use_virtio_for_bridges = True
> virt_type = kvm
>
> # Neutron
> [neutron]
> auth_type = password
> # Keystone client plugin authentication URL option
> auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3
> cafile = /etc/ssl/certs/ca-certificates.crt
> default_floating_pool = public
> insecure = False
> metadata_proxy_shared_secret = xxxxxxxxxxxxxxxxxxxxxxxxxxxx
> # Keystone client plugin password option
> password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> project_domain_name = Default
> project_name = service
> region_name = A_Lab
> service_metadata_proxy = True
> url = https://vip.mgmt.lab.mydomain.com:9696
> user_domain_name = Default
> # Keystone client plugin username option
> username = neutron
>
> [oslo_concurrency]
> lock_path = /var/lock/nova
> # Notifications
> [oslo_messaging_notifications]
> driver = messagingv2
> notification_topics = notifications
> transport_url =
> rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7@192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7@192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7@192.168.56.179:5671//nova
>
> [oslo_messaging_rabbit]
> rpc_conn_pool_size = 30
> ssl = True
>
> # Placement
> [placement]
> auth_type = "password"
> auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3
> cafile = /etc/ssl/certs/ca-certificates.crt
> insecure = False
> os_interface = internal
> os_region_name = A_Lab
> password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> project_domain_name = Default
> project_name = service
> user_domain_name = Default
> username = placement
>
> [quota]
> cores = 20
> injected_file_content_bytes = 10240
> injected_file_path_length = 255
> injected_files = 5
> instances = 10
> key_pairs = 100
> max_age = 0
> metadata_items = 128
> ram = 32768
> server_group_members = 10
> server_groups = 10
>
> [scheduler]
> discover_hosts_in_cells_interval = 60
> host_manager = host_manager
> max_attempts = 5
> periodic_task_interval = 60
> scheduler_driver = filter_scheduler
>
> [spice]
> agent_enabled = True
> enabled = True
> # Console Url and binds
> html5proxy_base_url =
> https://dashboard.iaas.lab.mydomain.com:6080/spice_auto.html
> server_listen = 192.168.56.47
> server_proxyclient_address = 192.168.56.47
>
> [upgrade_levels]
> compute = auto
>
> [vnc]
> enabled = False
>
> [wsgi]
> api_paste_config = /etc/nova/api-paste.ini
> secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO
>
>
> Thank you in advance for any suggestions.
>
> Kind regards,
> Laszlo
More information about the openstack-discuss
mailing list