July 2019 - openstack-discuss - lists.openstack.org

[keystone] Need clarification about Stein policies
by Bernd Bausch 25 Jul '19

25 Jul '19

The Keystone policy.json file I created with oslo-policy-generator contains lines I don't understand. For example /list_users/. The comment says: # DEPRECATED "identity:list_users":"rule:admin_required" has been # deprecated since S in favor of "identity:list_users":"(role:reader # and system_scope:all) or (role:reader and # domain_id:%(target.domain_id)s)". I do understand the expression starting with (role:reader .... , but contrarily to the comment, the policy is "identity:list_users": "rule:identity:list_users" This looks like a circular definition, and in any case, nowhere do I seerule:identity:list_users defined. Can someone in the know explain how this policy is processed? Thanks much, Bernd

2 1

[dev][nova][glance][cinder] introducing 'compressed' image container_format
by Ma, Zheng 25 Jul '19

25 Jul '19

Hello Nova developers, Cinder is working on a feature that will allow it to compress volumes uploaded to Glance [0,1]. We are proposing a new container_format for Glance named 'compressed' [2]. The idea is that a consumer of a 'compressed' image would read the file to determine the kind of compression and take appropriate action (either successfully decompress to find a payload that matches the image's disk_format, or fail with 'unsupported container_format'). Our preliminary testing with a raw image with container_format == compressed, disk_format == raw is that Nova boots a VM from the image but the VM appears to be unusable. Nova seems taking it as a real RAW image as disk_format indicates. So we want to open a discussion about handling this. Thanks! [0] cinder spec patch https://review.opendev.org/#/c/652275/ [1] cinder code implementation https://review.opendev.org/#/c/668825 [2] glance spec patch https://review.opendev.org/#/c/670454/

1 0

[Nova] migration issue
by Budai Laszlo 25 Jul '19

25 Jul '19

Dear all, we are testing the cold migratoin of instances that are using local ephemeral storage and it fails with the following error: 2019-07-24 13:47:13.115 58902 INFO nova.virt.libvirt.driver [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae 58-00d6-4317-b635-909fdf09ac49] Instance shutdown successfully after 3 seconds. 2019-07-24 13:47:13.126 58902 INFO nova.virt.libvirt.driver [-] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Instance destroyed successfully. 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae58- 00d6-4317-b635-909fdf09ac49] Setting instance vm_state to ERROR: ProcessExecutionError: Unexpected error while running command. Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph 0 Exit code: 1 Stdout: u'' Stderr: u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured syst em and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. * \n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No suc h file or directory\n' 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Traceback (most recent call last): 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 75 18, in _error_out_instance_on_exception 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] yield 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 42 75, in _resize_instance 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] timeout, retry_interval) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", lin e 8200, in migrate_disk_and_power_off 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] shared_storage) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220 , in __exit__ 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] self.force_reraise() 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196 , in force_reraise 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] six.reraise(self.type_, self.value, self.tb) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", lin e 8185, in migrate_disk_and_power_off 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] compression=compression) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 226, in copy_image 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] compression=compression) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs .py", line 110, in copy_file 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] compression=compression) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs .py", line 196, in copy_file 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] on_execute=on_execute, on_completion=on_completion) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py", line 231, in exec ute 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] return processutils.execute(*cmd, **kwargs) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py" , line 424, in execute 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] cmd=sanitized_cmd) 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] ProcessExecutionError: Unexpected error while running command. 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Exit code: 1 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Stdout: u'' 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Stderr: u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or directory\n' 2019-07-24 13:47:14.008 58902 ERROR nova.compute.manager [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] 2019-07-24 13:47:14.380 58902 INFO nova.compute.manager [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Swapping old allocation on 3da11b23-5ef8-4471-a3c9-5f743b0bbfd7 held by migration da66c416-2636-421b-99cb-470bd07dad4c for instance 2019-07-24 13:47:14.979 58902 INFO nova.compute.manager [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] Successfully reverted task state from resize_migrating on failure for instance. 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server [req-356fce31-5e98-425f-89d6-8a98664d31ad 122b0950c0cc47bdbb78e63724d65105 f713e44c723e491aa67352e12f83e0d7 - default default] Exception during message handling: ProcessExecutionError: Unexpected error while running command. Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 Exit code: 1 Stdout: u'' Stderr: u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or directory\n' 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server function_name, call_dict, binary) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 186, in decorated_function 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server "Error: %s", e, instance=instance) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 156, in decorated_function 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/utils.py", line 977, in decorated_function 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in decorated_function 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in decorated_function 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4240, in resize_instance 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server self._revert_allocation(context, instance, migration) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4237, in resize_instance 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server instance_type, clean_shutdown) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/compute/manager.py", line 4275, in _resize_instance 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server timeout, retry_interval) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8200, in migrate_disk_and_power_off 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server shared_storage) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8185, in migrate_disk_and_power_off 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server compression=compression) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 226, in copy_image 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server compression=compression) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 110, in copy_file 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server compression=compression) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/virt/libvirt/volume/remotefs.py", line 196, in copy_file 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server on_execute=on_execute, on_completion=on_completion) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/nova/utils.py", line 231, in execute 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server return processutils.execute(*cmd, **kwargs) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-17.1.3/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in execute 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server cmd=sanitized_cmd) 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command. 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Command: scp -r /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 192.168.56.46:/dev/ssd_nova/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Exit code: 1 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Stdout: u'' 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server Stderr: u'------------------------------------------------------------------------------\n* WARNING *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\n* authorized user of this system. *\n------------------------------------------------------------------------------\n/var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0: No such file or directory\n' 2019-07-24 13:47:15.000 58902 ERROR oslo_messaging.rpc.server 2019-07-24 13:47:17.065 58902 INFO nova.virt.libvirt.imagecache [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] image ca89475a-cf15-46d1-ab2d-c2b96c403248 at (/var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07): checking 2019-07-24 13:47:17.066 58902 INFO nova.virt.libvirt.imagecache [req-e73474d0-6857-4c52-9c93-73f3bae247b5 - - - - -] Active base files: /var/lib/nova/instances/_base/ab9f9dd0888a4950702b9c993c82f7ab36a3ed07 2019-07-24 13:47:27.768 58902 INFO nova.compute.manager [-] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] VM Stopped (Lifecycle Event) 2019-07-24 13:47:27.918 58902 INFO nova.compute.manager [req-e53e928d-81fe-4185-9e62-abc2113e838c - - - - -] [instance: 5a6dae58-00d6-4317-b635-909fdf09ac49] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. The disk has its root FS n an RBD device (a volume created by cinder) and it has its ephemeral disk on a local LVM volume created by nova. It seems that the /var/lib/nova/instances/5a6dae58-00d6-4317-b635-909fdf09ac49_resize/5a6dae58-00d6-4317-b635-909fdf09ac49_disk.eph0 is not there when scp is trying to copy it to the other side. Any ideas how to fix this? The nova config on the nodes looks like this: # Ansible managed [DEFAULT] allow_resize_to_same_host = True # Compute compute_driver = libvirt.LibvirtDriver # Scheduler cpu_allocation_ratio = 2.0 # Logs / State debug = False # Hypervisor default_ephemeral_format = ext4 disk_allocation_ratio = 1.0 # Api's enabled_apis = osapi_compute,metadata executor_thread_pool_size = 64 fatal_deprecations = False # Configdrive force_config_drive = False host = compute8.mgmt.lab.mydomain.com image_cache_manager_interval = 0 instance_name_template = instance-%08x # Ceilometer notification configurations instance_usage_audit = True instance_usage_audit_period = hour instances_path = /var/lib/nova/instances ## Vif libvirt_vif_type = ethernet log_dir = /var/log/nova # Metadata metadata_workers = 16 # Network my_ip = 192.168.56.47 notify_on_state_change = vm_and_task_state osapi_compute_workers = 16 ram_allocation_ratio = 1.0 reserved_host_disk_mb = 0 reserved_host_memory_mb = 2048 resume_guests_state_on_host_boot = False rootwrap_config = /etc/nova/rootwrap.conf rpc_response_timeout = 60 service_down_time = 120 state_path = /var/lib/nova # Rpc all transport_url = rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7@192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7@192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e 8cf8bd7@192.168.56.179:5671//nova # Disable stderr logging use_stderr = False vif_plugging_is_fatal = True vif_plugging_timeout = 30 [api] auth_strategy = keystone enable_instance_password = True use_forwarded_for = False vendordata_jsonfile_path = /etc/nova/vendor_data.json # Cache [cache] backend = oslo_cache.memcache_pool enabled = true memcache_servers = 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211 # Cinder [cinder] cafile = /etc/ssl/certs/ca-certificates.crt catalog_info = volumev3:cinderv3:internalURL cross_az_attach = True os_region_name = A_Lab [conductor] workers = 16 [filter_scheduler] available_filters = nova.scheduler.filters.all_filters enabled_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, AggregateRamFilter, ComputeFilter, AggregateCoreFilter, DiskFilter, AggregateDiskFilter, AggregateNumInstancesFilter, AggregateIoOpsFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, NUMATopologyFilter, SameHostFilter, DifferentHostFilter host_subset_size = 10 max_instances_per_host = 50 max_io_ops_per_host = 10 ram_weight_multiplier = 5.0 tracks_instance_changes = True weight_classes = nova.scheduler.weights.all_weighers # Glance [glance] api_servers = https://vip.mgmt.lab.mydomain.com:9292 cafile = /etc/ssl/certs/ca-certificates.crt [key_manager] fixed_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx [keystone_authtoken] auth_type = password auth_uri = https://vip.mgmt.lab.mydomain.com:5000 auth_url = https://vip.mgmt.lab.mydomain.com:35357 cafile = /etc/ssl/certs/ca-certificates.crt insecure = False memcache_secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx # if your memcached server is shared, use these settings to avoid cache poisoning memcache_security_strategy = ENCRYPT memcached_servers = 192.168.56.68:11211,192.168.56.85:11211,192.168.56.170:11211 password = xxxxxxxxxxxxxxxxxxxxxxx project_domain_id = default project_name = service region_name = A_Lab token_cache_time = 300 user_domain_id = default username = nova [libvirt] disk_cachemodes = network=writeback hw_disk_discard = unmap images_rbd_ceph_conf = /etc/ceph/ceph.conf images_rbd_pool = vms images_type = lvm images_volume_group = ssd_nova inject_key = False inject_partition = -2 inject_password = False live_migration_tunnelled = True live_migration_uri = "qemu+ssh://nova@%s/system?no_verify=1&keyfile=/var/lib/nova/.ssh/id_rsa" rbd_secret_uuid = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx # ceph rbd support rbd_user = cinder remove_unused_resized_minimum_age_seconds = 3600 use_virtio_for_bridges = True virt_type = kvm # Neutron [neutron] auth_type = password # Keystone client plugin authentication URL option auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3 cafile = /etc/ssl/certs/ca-certificates.crt default_floating_pool = public insecure = False metadata_proxy_shared_secret = xxxxxxxxxxxxxxxxxxxxxxxxxxxx # Keystone client plugin password option password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx project_domain_name = Default project_name = service region_name = A_Lab service_metadata_proxy = True url = https://vip.mgmt.lab.mydomain.com:9696 user_domain_name = Default # Keystone client plugin username option username = neutron [oslo_concurrency] lock_path = /var/lock/nova # Notifications [oslo_messaging_notifications] driver = messagingv2 notification_topics = notifications transport_url = rabbit://nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7@192.168.56.194:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7@192.168.56.209:5671,nova:09c23ad7680fc920ab466c5ad5f6c88c2e2e8cf8bd7@192.168.56.179:5671//nova [oslo_messaging_rabbit] rpc_conn_pool_size = 30 ssl = True # Placement [placement] auth_type = "password" auth_url = https://vip.mgmt.lab.mydomain.com:35357/v3 cafile = /etc/ssl/certs/ca-certificates.crt insecure = False os_interface = internal os_region_name = A_Lab password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx project_domain_name = Default project_name = service user_domain_name = Default username = placement [quota] cores = 20 injected_file_content_bytes = 10240 injected_file_path_length = 255 injected_files = 5 instances = 10 key_pairs = 100 max_age = 0 metadata_items = 128 ram = 32768 server_group_members = 10 server_groups = 10 [scheduler] discover_hosts_in_cells_interval = 60 host_manager = host_manager max_attempts = 5 periodic_task_interval = 60 scheduler_driver = filter_scheduler [spice] agent_enabled = True enabled = True # Console Url and binds html5proxy_base_url = https://dashboard.iaas.lab.mydomain.com:6080/spice_auto.html server_listen = 192.168.56.47 server_proxyclient_address = 192.168.56.47 [upgrade_levels] compute = auto [vnc] enabled = False [wsgi] api_paste_config = /etc/nova/api-paste.ini secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO Thank you in advance for any suggestions. Kind regards, Laszlo

5 10

[nova] Train Spec Freeze
by Eric Fried 25 Jul '19

25 Jul '19

Hi Nova contributors and maintainers. Spec freeze for Nova is Thursday, July 25th. This means that any specs not approved by EOD [1] become drastically less likely to land in Train. For your convenience, here is a search that's intended to catch open Train specs while filtering out things like updates to existing specs: https://review.opendev.org/#/q/project:openstack/nova-specs+status:open+pat… At the time of this writing, there are 15 on that list. Of those, IMO two are "close": Add Unified Limits Spec - https://review.opendev.org/602201 - Two +1s, needs review from "Nova API subteam" Spec for the Nova part of Image Encryption - https://review.opendev.org/608696 - +3, needs a second core If you own a spec you think should be able to land [2], please: - Check it to make sure any comments have been addressed (even if there's no -1). - Explicitly ask for reviews, here on the mailing list and/or in IRC in #openstack-nova. Reviewers, please make an effort to ensure that all the specs on that list have at least one vote. (Currently there are six with no vote.) I'll send an update Friday or Monday with a final tally. Thanks, efried [1] i.e. by the time cores have signed off for the day, wherever they are [2] meaning it has already been through some reviews and seems to have support. If you just proposed your spec this week, it's not reasonable to expect it to land.

2 1

Heat resource types - aodh and ceilometer
by Dhairyasheel Ghadge 25 Jul '19

25 Jul '19

Hello all, I have installed heat engine along with heat dashboard for orchestration in openstack cloud. I am creating a template to autoscale a web server pool and spin a new VM when load on other one increases. For the autoscaling purpose, I require to have an alarming system which will monitor the cp meter of the instance, which is provided by ceilometer and aodh services of openstack. I cannot find resource types of aodh and ceilometer on the heat dashboard. These same resource types are present in the heat openstack resources directory. Are there any steps to be carried out to make it available on the dashboard? Thanks & Regards Dhairyasheel

1 0

[nova] [horizon] Horizon and nova-cli inconsistency in live migration parameters
by Mary Atakova 25 Jul '19

25 Jul '19

Hi everyone, I think I have found an inconsistency between nova cli and Horizon in the way they send live migration commands to nova API. I'm not sure where this needs to be fixed, so please advise on that. The situation: I try to live-migrate a VM, and I do it through nova cli and through Horizon, and nova-api logs contain different data depending on the way I do it. 1. nova cli I run the command: nova live-migration 17359460-d23c-4acc-a9b1-5cf117b54430 and it shows up in nova-api logs as: {"os-migrateLive": {"block_migration": "auto", "host": null}} 2. horizon I run live migration via Horizon with unchecked Disk Overcommit and unchecked Block Migration. It shows up in nova-api logs as: {"os-migrateLive": {"disk_over_commit": false, "block_migration": false, "host": null}} This difference can lead to different parameters provided for live migration when block_migration: auto is translated into block_migration: true. My environment: Openstack Stein CentOS Linux release 7.6.1810 nova versions: # yum list installed | grep nova openstack-nova-api.noarch 1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-common.noarch 1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-conductor.noarch 1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-console.noarch 1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-novncproxy.noarch openstack-nova-placement-api.noarch openstack-nova-scheduler.noarch 1:19.0.1-1.el7 @centos-openstack-stein python2-nova.noarch 1:19.0.1-1.el7 @centos-openstack-stein python2-novaclient.noarch 1:13.0.0-1.el7 @centos-openstack-stein horizon version: # yum list installed | grep dashboard openstack-dashboard.noarch 1:15.1.0-1.el7 @centos-openstack-stein openstack-dashboard-theme.noarch 1:15.1.0-1.el7 @centos-openstack-stein Thank you in advance for your attention Best regards, Mary

2 2

DuplicateMessageError after restart of a control node
by Pawel Konczalski 25 Jul '19

25 Jul '19

Hello everybody, after restart of one of the control nodes i see this error message every few seconds int the logs: @timestamp July 23rd 2019, 16:44:08.927 t Hostname ctlX t Logger openstack.nova t Payload Failed to process message ... skipping it.: DuplicateMessageError: Found duplicate message(f2178cffb5d249f3a1d2df1c0322f18c). Skipping it. 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 361, in _callback 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit self.callback(RabbitMessage(message)) 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 251, in __call__ 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit unique_id = self.msg_id_cache.check_duplicate_message(message) 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py", line 122, in check_duplicate_message 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit raise rpc_common.DuplicateMessageError(msg_id=msg_id) 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit DuplicateMessageError: Found duplicate message(f2178cffb5d249f3a1d2df1c0322f18c). Skipping it. 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit t Pid 82 t Timestamp 2019-07-23 16:44:08.927 t _id AWwfSjEtsndUu9YeotwF t _index flog-2019.07.23 # _score - t _type fluentd t log_level ERROR t programname nova-scheduler t python_module oslo.messaging._drivers.impl_rabbit As soon as the restarted node is taken out of the cluster, the errors disappear. Any idea how to fix this? Thank you Pawel

2 2

[release] (Belated) Release countdown for week R-12, July 22-26
by Kendall Nelson 24 Jul '19

24 Jul '19

Hello Everyone, Development Focus ----------------- The Train-2 milestone is July 25! Train-related specs should now be finalized so that teams can move to implementation ASAP. Some teams observe specific deadlines on the second milestone (mostly spec freezes): please refer to https://releases.openstack.org/train/schedule.html for details. General Information ------------------- Libraries need to be released at least once per milestone period. This week, the release team will propose releases for any library that has not been otherwise released since milestone 1. See list at: http://paste.openstack.org/show/754817/. PTL's and release liaisons, please watch for these and give a +1 to acknowledge them. If there is some reason to hold off on a release, let us know that as well. A +1 would be appreciated, but if we do not hear anything at all by the end of the week, we will assume things are OK to proceed. Remember that non-library deliverables that follow the cycle-with-intermediary release model should have an intermediary release before milestone-2. Those who haven't will be proposed to switch to the cycle-with-rc model, which is more suited to deliverables that are released only once per cycle. This week is also the deadline to freeze the contents of the final release. All new 'Train' deliverables need to have a deliverable file ( https://opendev.org/openstack/releases/src/branch/master/deliverables/train) and need to have done a release by milestone-2. The following new deliverables have not had a release yet (or not been already discussed in the etherpad[1]), and will not be included in Train unless a release is requested for them in the coming week: python-adjutantclient, cyborg-tempest-plugin, neutron-interconnection, openstack-helm, openstack-helm-addons, openstack-help-docs, openstack-helm-images, openstack-helm-infra, and compute-hyperv. The full list of currently deliverable-less deliverables can be found here[2]. Upcoming Deadlines & Dates -------------------------- Train-2 Milestone: July 25 (R-12 week) Thanks! -Kendall Nelson(diablo_rojo) + the Release Management Team [1]https://etherpad.openstack.org/p/train-membership-freeze [2] http://paste.openstack.org/show/754816/

1 0

[oslo] PTO next week
by Ben Nemec 24 Jul '19

24 Jul '19

Just a heads up that I will be on PTO next week and incommunicado pretty much the whole time. Moises (moguimar) volunteered to run the Oslo meeting so that will still be happening. Herve (hberaud) and Doug (dhellmann) are our release liaisons so if you need a release of any Oslo libraries next week, they're the ones to talk to. Otherwise, things have been pretty quiet in Oslo-land (which would be Norway, I guess), so I expect all the things to break next week while I'm out. Good luck. ;-) Thanks. -Ben

1 0

[ops] [nova] How to change/remove a property from an instance ?
by Massimo Sgaravatto 24 Jul '19

24 Jul '19

In my OpenStack Rocky cloud I have some instances that were created (before updating to Rocky) using images with the property hypervisor_type='qemu'. Since in my compute nodes [libvirt]/virt_type is "kvm", after having updated to Rocky I had to change these images removing that property of changing its value (qemu --> kvm): this is discussed in the thread: http://lists.openstack.org/pipermail/openstack-discuss/2019-July/thread.htm… Now I am not able to migrate anymore the instances that were created using those images (ImagePropertiesFilter filters out all compute nodes) before fixing the property. As suggested by Matt I tried to play with the instance_system_metadata table of the nova database. I first tried to change the value of that property (qemu --> kvm) [*] I also tried to remove the property [**] but the nova migrate keeps failing. In the request I keep seeing that Instance contains properties ImageMetaProps...img_hv_type='qemu' What else needs to be modified? Thanks, Massimo [*] mysql> select * from instance_system_metadata where instance_uuid='57dcdc2f-ef30-4512-beff-128c1621c5c6'; +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ | created_at | updated_at | deleted_at | id | instance_uuid | key | value | deleted | +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ | 2017-05-15 12:32:57 | NULL | NULL | 9868217 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_hypervisor_type | kvm | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868220 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_disk_format | qcow2 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868223 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_container_format | bare | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868226 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_min_ram | 512 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868229 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_min_disk | 20 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868232 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_base_image_ref | c2d3e37c-b14f-4472-9bc2-d75f26ab900d | 0 | +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ [**] mysql> delete from instance_system_metadata where id='9868217'; Query OK, 1 row affected (0.01 sec) mysql> select * from instance_system_metadata where instance_uuid='57dcdc2f-ef30-4512-beff-128c1621c5c6'; +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ | created_at | updated_at | deleted_at | id | instance_uuid | key | value | deleted | +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ | 2017-05-15 12:32:57 | NULL | NULL | 9868220 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_disk_format | qcow2 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868223 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_container_format | bare | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868226 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_min_ram | 512 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868229 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_min_disk | 20 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868232 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_base_image_ref | c2d3e37c-b14f-4472-9bc2-d75f26ab900d | 0 | +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ 5 rows in set (0.00 sec)

2 1

[opendev] git clone fails: "The TLS connection was non-properly terminated"
by Bernd Bausch 24 Jul '19

24 Jul '19

There seems to be something wrong with the opendev server. Is this a glitch, a daily occurrence (backup?), or a deeper problem? This happens around midnight between July 22 and 23 UTC. $ git clone https://opendev.org/openstack/devstack.git -b stable/stein Cloning into 'devstack'... fatal: unable to access 'https://opendev.org/openstack/devstack.git/': gnutls_handshake() failed: The TLS connection was non-properly terminated. Same result when I try to clone any other random repo. I can get Devstack from Github, but when setting up the cloud, stack.sh wants to clone everything from git.openstack.org, which fails again. I could replace all the hardcoded /git.openstack.org/ strings with the equivalent /github.com/ URL, but I am not sure if I can trust that all Github repos are up to date. Bernd.

4 7

[devstack] Devstack fails with requirement pyparsing===2.4.1
by Bernd Bausch 24 Jul '19

24 Jul '19

Trying to install Devstack Stein on freshly installed Ubuntu 18.04. stack.sh fails with this: Downloading https://files.pythonhosted.org/packages/6f/7c/ebba93a2d3e09fd0220e1adfdb823… (131kB) Collecting pyparsing===2.4.1 (from -c https://releases.openstack.org/constraints/upper/master (line 492)) Could not find a version that satisfies the requirement pyparsing===2.4.1 (from -c https://releases.openstack.org/constraints/upper/master (line 492)) (from versions: 1.4.6, 1.4.7, 1.4.8, 1.4.11, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.5.7, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.0.6, 2.0.7, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.6, 2.1.7, 2.1.8, 2.1.9, 2.1.10, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0) No matching distribution found for *pyparsing===2.4.1* (from -c https://releases.openstack.org/constraints/upper/master (line 492)) (red highlight is mine) The most recent version of pyparsing <https://pypi.org/project/pyparsing/>is 2.4.0. I would file a bug, but I don't know against which project. Who owns https://releases.openstack.org/constraints/upper/master? Or is there a misunderstanding somewhere? My local.conf is below - it's based on a Rocky local.conf (which does work on Rocky) and might be incorrect. Bernd. [[local|localrc]] ########################################################################### ### Passwords ADMIN_PASSWORD=MYPASSWORD DATABASE_PASSWORD=$ADMIN_PASSWORD RABBIT_PASSWORD=$ADMIN_PASSWORD SERVICE_PASSWORD=$ADMIN_PASSWORD ########################################################################### ### Logging LOG_COLOR=False LOGFILE=$DEST/logs/stack.sh.log ########################################################################### ### Networking IP_VERSION=4 HOST_IP=192.168.1.200 ########################################################################### ### Software sources # use https since the vlabs proxy seems to block the git:// scheme GIT_BASE=https://git.openstack.org # Ensure the latest software is installed PIP_UPGRADE=True ########################################################################### ### Glance DOWNLOAD_DEFAULT_IMAGES=False IMAGE_URLS="http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img" # Glance's image import doesn't work when Glance runs as a UWSGI server # For this setting to take effect, devstack/lib/glance must be patched GLANCE_USE_UWSGI=False ########################################################################### ### Swift # by default, Devstack interactively asks for a hash value. Avoid this. SWIFT_HASH=123 enable_service swift ########################################################################### ### Heat enable_plugin heat https://git.openstack.org/openstack/heat stable/stein enable_service h-eng h-api h-api-cfn h-api-cw enable_plugin heat-dashboard https://git.openstack.org/openstack/heat-dashboard stable/stein enable_service heat-dashboard ########################################################################### ### Cinder: Enable backup enable_service c-bak ########################################################################### ### Telemetry: Use Gnocchi as sample repository ### and configure medium archive policy CEILOMETER_BACKEND=gnocchi GNOCCHI_ARCHIVE_POLICY=medium enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer stable/stein enable_plugin aodh https://git.openstack.org/openstack/aodh stable/stein ########################################################################### ### Neutron: Enable internal DNS resolution in ml2_conf.ini Q_ML2_PLUGIN_EXT_DRIVERS=port_security,dns_domain_ports ########################################################################### ### Post config for Neutron (DNS) and Nova (allow KVM in Vmware machine) [[post-config|$NEUTRON_CONF]] [DEFAULT] dns_domain = devstack.org. [[post-config|/etc/nova/nova-cpu.conf]] [libvirt] # set machine type to work around bug # https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1691109 hw_machine_type = x86_64=pc-i440fx-xenial

5 4

[mistral] Office hour in ~5 min
by Renat Akhmerov 24 Jul '19

24 Jul '19

Hi, We’ll have an office hour in about 5 mins. Join us at #openstack-mistral if you have anything to discuss. Thanks Renat Akhmerov @Nokia

1 0

[Kolla][Keystone] No "Domains" found in identity area.
by Eddie Yen 24 Jul '19

24 Jul '19

Hi, I'm using kolla-ansible stable/rocky release to deploy Openstack Try to test with multi-domains requirements. I'm do nothing but add "horizon_keystone_multidomain: True" into globals.yml After deployment, it works because I have to enter the domain name at login screen. Logged in with default domain and admin account. But I cannot found any "Domains" section in identity block. Is there something I miss?

1 2

[keystone] Keystone Train Midcycle Report
by Colleen Murphy 24 Jul '19

24 Jul '19

This week the keystone team conducted a two-day virtual midcycle. Thanks to everyone who participated and contributed to making this a successful and productive meeting! This email will be a short summary of the key takeaways from the last two days. The full agenda and meeting notes can be found on this etherpad: https://etherpad.openstack.org/p/keystone-train-midcycle-topics # Caching Guide We've long been proposing to document best practices for configuring caching for keystone. We began the meeting with a discussion about what exactly should be contained in such a guide, starting with anecdotes from the team about our own operational experience with configuring memcached for keystone. We quickly realized the guide needed at least two parts, one for keystone itself and one for keystonemiddleware. We also made two technical observations, first that the default value for the memcache_socket_timeout option in oslo.cache could be reduced[1] and second that the memcache_socket_timeout option exposed in keystone is not actually used by keystone and needs to be removed[2]. We talked about security strategies for memcached: oslo.cache exposes encryption capabilities which is required for organizations that consider the data stored in memcached to be "at rest", but it also hurts performance. An alternative strategy that could be recommended in the guide is to set up TLS tunneling to secure in-flight data without impeding fetching performance. The initial caching guide update was proposed here[3]. This guide will be a living document and will continue to grow as we gather real-life data from operators. After this initial update, we'll start taking steps to ensure that keystone requires caching be set up in order to operate. [1] https://review.opendev.org/672063 [2] https://bugs.launchpad.net/keystone/+bug/1837407 [3] https://review.opendev.org/672120 # Community Goals We then had a short status update session on the various community goals we need to meet. The mutable config goal from a few cycles ago is not yet complete but has an implementation proposed[4] which needs more eyes and testing. The PDF generation goal implementation is underway[5] but there are some issues already being tracked by the community[6]. We will need to work to ensure the issues we're seeing are noted and addressed. The last of the Python3 Train job changes for keystone repos were merged, so I think we've completed our end of that goal. We also already have a patch proposed to add the ipv6-only job[7] which is just waiting on the base job to merge in tempest. [4] https://review.opendev.org/585417 [5] https://review.opendev.org/669982 [6] https://etherpad.openstack.org/p/pdf-goal-train-common-problems [7] https://review.opendev.org/671903 # Milestone 2 Train Retrospective We closed out the first day with a mid-cycle retrospective[8]. Some key areas for improvement from this were: * Do more video conferencing - The real-time face-to-face communication is really effective for getting to the heart of discussions and making decisions quickly - Since it is looking likely that most of us will not be at the next PTG in Shanghai, we can set up a pre- and post- PTG video meeting to plan the cycle and catch people up after the IRL PTG - Milestone-ly video meetings are working for helping us keep pace and sync up - We can keep a queue of potential topics and make sure we have a meeting when there are enough topics to discuss - We'll try using video conferencing for office hours - We'll try Jitsi again even though there were some technical issues last time since it is OSS and supports recording * Improve office hours - Office hours still don't feel useful and haven't happened regularly, but we still think they have potential - We'll use the weekly newsletter to commit to the topic the week before, which gives the team a better opportunity to prepare for it and can also attract interest from people outside the keystone team - Use video conferencing as mentioned above * Improve the weekly newsletter - We'll use the weekly newsletter to call out action items from the last meeting - In addition to providing statistics about the number of open reviews, we'll highlight certain reviews that need attention based on various criteria such as age, priority with regard to cycle commitments, community goals, and specific review requests. - We'll announce the following week's office hours topic in the newsletter [8] https://etherpad.openstack.org/p/keystone-train-midcycle-retrospective # System Scope and Default Roles The second day of the midcycle was used to go through our list of open system scope[9] and default roles[10] bugs and work through what scopes and roles each API should support. The notes in the etherpad largely capture every decision fairly well, but a few issues worth highlighting are: * Implied roles[11]: the regular roles API is currently system-scope only, even though it's possible to have domain-specific roles, because for the most part for a role to be useful the role's creator must also have the ability to configure policy, which domain admins do not. Implied roles still require access to policy configuration except in the case where you are only using implied roles for aliasing, in which case it may make sense for a domain admin to have the ability to create an implied role relationship. We decided to start with making the implied roles API system-scope only for now and to open a separate bug to discuss how to deal with domain-specific roles and implied roles. * Revocation list[12]: we're currently not enforcing policy on this API at all because it can only ever return a 410 Gone or a 403 Forbidden, never a success, so we will just deprecate the policy entirely rather than updating it to work with system scope or default roles. We reevaluated the priority on some of the bugs in the list. However, we noted that while some APIs are more heavily used than others, we can't change enforce_scope to True until all of them are completed - picking and choosing the most important ones and letting the others slide is not really an option. [9] https://bugs.launchpad.net/keystone/+bugs?field.tag=system-scope [10] https://bugs.launchpad.net/keystone/+bugs?field.tag=default-roles [11] https://bugs.launchpad.net/keystone/+bug/1805371 [12] https://bugs.launchpad.net/keystone/+bug/1818845 If I've left out any important details, don't hesitate to add them to this thread. Thanks again to everyone who was able to join the meeting! Thanks also to Julia Kreger for her advice on how to hold one of these virtual midcycles, it proved to be very useful and this first-timer really appreciated it :) Colleen

1 0

[tripleo] CI scenario001-standalone broken
by John Fulton 23 Jul '19

23 Jul '19

As per $subject [0]. scenario010 is affected too. This correlates with the promotion of the ceph-ansible 4rc11 RPM [1]. 001 jobs using rc9 do not have this problem. I'm looking into a fix, e.g. removing the storage7-ceph-nautilus-release tag from the rc11 package as TripleO CI uses the version of ceph-ansible which is tagged for release [2]. I'll keep this thread updated with the status and when it's fixed. John [0] http://logs.openstack.org/74/670574/4/check/tripleo-ci-centos-7-scenario001… [1] https://cbs.centos.org/koji/buildinfo?buildID=26343 [2] https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/r…

3 4

[Diversity] Grace Hopper Lead Co-Mentor(s) needed
by Amy Marrich 23 Jul '19

23 Jul '19

OpenStack has participated in Open Source Day at the Grace Hopper Conference for the last 5 years and this year it is in Orlando on October 3rd. Due to unforeseen circumstances we are one lead mentor short to participate this year and are looking for someone willing to help lead the session. The Anita Borg Institute will provide a ticket for the entire conference, the downside is you or your company will be responsible for your travel, lodging and food. This year CityNetwork is supplying the VM infrastructure for us to use. The session submitted is as follows: Participants will sign up for the necessary accounts needed to contribute to OpenStack and install Git and Gerrit. They will configure their systems and learn how to file bugs and review code within the Open Dev infrastructure. Due to time and resource requirements the groups will be given a virtual machine running Devstack in which to create their own theme for the Horizon Dashboard. The group will decide on which humanitarian effort they wish to support but suggestions will be provided for those groups that can not decide. The group will also work as a team to locate the necessary documentation on the OpenStack site if they have not done so in preparation. The selected Project Manager will lead these discussions as well as making sure efforts are staying on track to meet the deadline of the demonstrations. The Developers in the group will work to locate the necessary files and directories on the VM in order to complete their task. They will be in charge of creating/changing the necessary files to create a new theme utilizing SCSS and restarting any services needed to implement. The Graphic Designer if present will create any necessary graphics for the theme and will either scp them themselves to the virtual machine or provide to the developer. If there is no designated graphic designer the PM and devs will divide this duty up. After the themes are designed relevant files will be scped back down to the participants machines and they will be able to commit their code for review in a sandbox environment for other groups to review. If you are interested in participating please let me know by July 20th as we need to send additional information to the OSD folks on July 21st. Any questions please reach out on the lists, IRC, or privately! Thanks, Amy (spotz)

1 1

[Octavia]-Seeking performance numbers on Octavia
by Singh, Prabhjit 23 Jul '19

23 Jul '19

Hi I have been trying to test Octavia with some traffic generators and my tests are inconclusive. Appreciate your inputs on the following * It would be really nice to have some performance numbers that you guys have been able to achieve for this to be termed as carrier grade. * Would also appreciate if you could share any inputs on performance tuning Octavia * Any recommended flavor sizes for spinning up Amphorae, the default size of 1 core, 2 Gb disk and 1 Gig RAM does not seem enough. * Also I noticed when the Amphorae are spun up, at one time only one master is talking to the backend servers and has one IP that its using, it has to run out of ports after 64000 TCP concurrent sessions, id there a way to add more IPs or is this the limitation * If I needed some help with Octavia and some guidance around performance tuning can someone from the community help Thanks & Regards Prabhjit Singh

3 5

[ironic] Using ironic in a devstack environment to deploy OS on bare-metal host.
by Guannan GN2 Sun 23 Jul '19

23 Jul '19

Hi team, I met some problem when using ironic in a devstack environment to deploy OS on bare-metal host. By now I can successfully boot into the tiny os and set ip, also I can ping the ip from my devstack server. However, when the tiny os running some configuration, it will failed after about 5 minutes. I found some log information in "/var/log/ironic-python-agent.log" in the tiny os that may relate to the problem: "DEBUG oslo_concurrency.processutils [-] u'iscsistart -f' failed. Not Retrying. execute /usr/local/lib/python2.7/site-packages/oslo_concurrency/processutils.py:457" "DEBUG root [-] No iscsi connection detected. Skipping iscsi. Error: [Errno 2] No such file or directory _check_for_iscsi /usr/local/lib/python2.7/site-packages/ironic_python_agent/hardware.py:107" "WARNING root [-] The root device was not detected in 27 seconds: DeviceNotFound: Error finding the disk or partition device to deploy the image onto: No suitable device was found for deployment - root device hints were not provided and all found block devices are smaller than 4294967296B." "WARNING root [-] Can't find field vendor for device usb0 in device class net: IOError: [Errno 2] No such file or directory: '/sys/class/net/usb0/device/vendor'" "WARNING root [-] Can't find field vendor for device tunl0 in device class net: IOError: [Errno 2] No such file or directory: '/sys/class/net/tunl0/device/vendor'" "DEBUG root [-] No Mellanox devices found evaluate_hardware_support /usr/local/lib/python2.7/site-packages/ironic_python_agent/hardware_managers/mlnx.py:84" And when it deploy faild and I run command "openstack baremetal node show server-51", it will show following information on 'last_error' field: "Asynchronous exception: Node failed to deploy. Exception: Connection to agent failed: Failed to connect to the agent running on node f8794658-bc69-40cb-9673-be75f78ced21 for invoking command iscsi.start_iscsi_target. Error: ('Connection aborted.', BadStatusLine("''",)) for node" I'm trying to debug, but have not find the root cause, please give me your advice if someone have experience on it. Thank you! Best Regards, Guannan

2 1

[dev][magnum] Magnum event notifications not having relevant content related to clusters
by Emnauel Andrecut 23 Jul '19

23 Jul '19

I'm writing about Magnum event notifications that do not contain any relevant information about the related Cluster (missing cluster id, other attributes..) As I've seen the PyCADF library receives no targetId and so the id (and mostly any other id) is randomly generated. Do you plan to add more information about the cluster on create success/pending events. Or would you accept a patch that does that?

3 2

[all][meta-sig][containers-sig] Creation of Containers SIG
by Jean-Philippe Evrard 23 Jul '19

23 Jul '19

Hello everyone, I have discussed with some of you about the fact OpenStack is lacking a a group interested in sharing the practices for the building/management of OpenStack application containers. Every container related project is doing its own way (and it's probably fine), but very few experiences are shared (and it's sad). I started an etherpad [1] to see areas where container projects/people could collaborate. I already received significant feedback on how to improve some of my practices, so I think it's already a positive experience for me to have this group started. Anyway, long story short, I believe we should create a SIG for this, so I proposed the addition of the containers-sig into governance [2]. We have enough work to started this group nowadays (with documentation, bindep shared practices, and improvement of projects). I suppose technologies will evolve over time, so I expect a SIG is the best tool to do this job of *for example* gathering experience, as it's not really time bound, and it _is_ a quite special interest. If you're interested by learning what is going on in the container building world for OpenStack, please add [containers-sig] to your openstack-discuss topics watch list. Regards, Jean-Philippe Evrard (evrardjp) [1]: https://etherpad.openstack.org/p/containers-sig [2]: https://review.opendev.org/#/c/666001/

1 0

[kolla] Stein release now available
by Mark Goddard 23 Jul '19

23 Jul '19

Hi, I'm pleased to announce that the first official Stein cycle releases for Kolla and Kolla Ansible are now available - both 8.0.0. A lot of work has gone into this release, including improving the CI tests, making it the most well tested release yet. Thanks to everyone who contributed. Release notes: https://docs.openstack.org/releasenotes/kolla/stein.html https://docs.openstack.org/releasenotes/kolla-ansible/stein.html Join us on #openstack-kolla to help make the Train cycle release even better. Cheers, Mark

1 0

[telemetry] Looking for new meeting time
by Trinh Nguyen 23 Jul '19

23 Jul '19

Hi team, I've changed job and could not afford doing the meeting at 02:00 UTC on Thursday anymore. My possible time slot is from 13:00-15:00 UTC Tuesday-Thursday. Please propose a better time frame for our team meeting. Sincerely, -- *Trinh Nguyen* *www.edlab.xyz <https://www.edlab.xyz>*

2 3

[nova] FYI to out-of-tree drivers: adding a positional argument to finish_revert_migration()
by Artom Lifshitz 22 Jul '19

22 Jul '19

[1] will most likely merge in its current shape or something very close to it. It adds a `migration` positional argument to the finish_revert_migration() virt driver method. `migration` is the, err, migration that's being reverted. If you have out of tree virt drivers, please update them to match this new method signature. Thanks! [1] https://review.opendev.org/#/c/668631/

1 0

[tripleo] TripleO CI Summary: Sprint 33
by Rafael Folco 22 Jul '19

22 Jul '19

Greetings, The TripleO CI team has just completed Sprint 33 / Unified Sprint 12 (Jun 27 thru Jul 17). The following is a summary of completed work during this sprint cycle: - Tested check jobs for periodic image/container builds on RHEL8 and resolved its dependencies on rhui repositories. - Enabled tripleo-repos to support RHEL8. - Addressed changes in kolla to support RHEL8. - Created a nodepool image on RDO for RHEL8 w/ elements and dependencies installed. - Improved container build script to consolidate kolla build logs and ease troubleshooting. - Manually pushed rhel containers to RDO registry and identified gaps in authentication workflow for the container build jobs using buildah tooling. - Modified scripts and post playbooks to upload RHEL8 overcloud images. - Created RHEL8 jobs to build a periodic pipeline in the RDO Software Factory and provide feedback for CentOS8 coverage. - Resumed scenario007 and featureset039 job updates upstream. - Promotion status: green on all branches at most of the sprint. The planned work for the next sprint [1] are: - Get RDO on RHEL8 build jobs working in periodic pipeline and bootstrap standalone and featureset001 jobs to consume RHEL8 images/containers. - Get featureset039 job to run novajoin tests and merge scenario007 job changes. - Review templates for branched and non-branched projects to have more control on what jobs are being run and reduce queue lengths. - Resume the design work for a staging environment to test changes in the promoter server for the multi-arch builds. The Ruck and Rover for this sprint are Marios Andreou (marios) and Rafael Folco (rfolco). Please direct questions or queries to them regarding CI status or issues in #tripleo, ideally to whomever has the ‘|ruck’ suffix on their nick. Ruck/rover notes are being tracked in etherpad [2]. Thanks, rfolco [1] https://tree.taiga.io/project/tripleo-ci-board/taskboard/unified-sprint-13 [2] https://etherpad.openstack.org/p/ruckroversprint13

1 0