July 2019 - openstack-discuss - lists.openstack.org

[nova] [horizon] Horizon and nova-cli inconsistency in live migration parameters
by Mary Atakova 25 Jul '19

25 Jul '19

Hi everyone, I think I have found an inconsistency between nova cli and Horizon in the way they send live migration commands to nova API. I'm not sure where this needs to be fixed, so please advise on that. The situation: I try to live-migrate a VM, and I do it through nova cli and through Horizon, and nova-api logs contain different data depending on the way I do it. 1. nova cli I run the command: nova live-migration 17359460-d23c-4acc-a9b1-5cf117b54430 and it shows up in nova-api logs as: {"os-migrateLive": {"block_migration": "auto", "host": null}} 2. horizon I run live migration via Horizon with unchecked Disk Overcommit and unchecked Block Migration. It shows up in nova-api logs as: {"os-migrateLive": {"disk_over_commit": false, "block_migration": false, "host": null}} This difference can lead to different parameters provided for live migration when block_migration: auto is translated into block_migration: true. My environment: Openstack Stein CentOS Linux release 7.6.1810 nova versions: # yum list installed | grep nova openstack-nova-api.noarch 1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-common.noarch 1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-conductor.noarch 1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-console.noarch 1:19.0.1-1.el7 @centos-openstack-stein openstack-nova-novncproxy.noarch openstack-nova-placement-api.noarch openstack-nova-scheduler.noarch 1:19.0.1-1.el7 @centos-openstack-stein python2-nova.noarch 1:19.0.1-1.el7 @centos-openstack-stein python2-novaclient.noarch 1:13.0.0-1.el7 @centos-openstack-stein horizon version: # yum list installed | grep dashboard openstack-dashboard.noarch 1:15.1.0-1.el7 @centos-openstack-stein openstack-dashboard-theme.noarch 1:15.1.0-1.el7 @centos-openstack-stein Thank you in advance for your attention Best regards, Mary

2 2

DuplicateMessageError after restart of a control node
by Pawel Konczalski 25 Jul '19

25 Jul '19

Hello everybody, after restart of one of the control nodes i see this error message every few seconds int the logs: @timestamp July 23rd 2019, 16:44:08.927 t Hostname ctlX t Logger openstack.nova t Payload Failed to process message ... skipping it.: DuplicateMessageError: Found duplicate message(f2178cffb5d249f3a1d2df1c0322f18c). Skipping it. 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 361, in _callback 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit self.callback(RabbitMessage(message)) 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 251, in __call__ 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit unique_id = self.msg_id_cache.check_duplicate_message(message) 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py", line 122, in check_duplicate_message 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit raise rpc_common.DuplicateMessageError(msg_id=msg_id) 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit DuplicateMessageError: Found duplicate message(f2178cffb5d249f3a1d2df1c0322f18c). Skipping it. 2019-07-23 16:44:08.927 82 ERROR oslo.messaging._drivers.impl_rabbit t Pid 82 t Timestamp 2019-07-23 16:44:08.927 t _id AWwfSjEtsndUu9YeotwF t _index flog-2019.07.23 # _score - t _type fluentd t log_level ERROR t programname nova-scheduler t python_module oslo.messaging._drivers.impl_rabbit As soon as the restarted node is taken out of the cluster, the errors disappear. Any idea how to fix this? Thank you Pawel

2 2

[release] (Belated) Release countdown for week R-12, July 22-26
by Kendall Nelson 24 Jul '19

24 Jul '19

Hello Everyone, Development Focus ----------------- The Train-2 milestone is July 25! Train-related specs should now be finalized so that teams can move to implementation ASAP. Some teams observe specific deadlines on the second milestone (mostly spec freezes): please refer to https://releases.openstack.org/train/schedule.html for details. General Information ------------------- Libraries need to be released at least once per milestone period. This week, the release team will propose releases for any library that has not been otherwise released since milestone 1. See list at: http://paste.openstack.org/show/754817/. PTL's and release liaisons, please watch for these and give a +1 to acknowledge them. If there is some reason to hold off on a release, let us know that as well. A +1 would be appreciated, but if we do not hear anything at all by the end of the week, we will assume things are OK to proceed. Remember that non-library deliverables that follow the cycle-with-intermediary release model should have an intermediary release before milestone-2. Those who haven't will be proposed to switch to the cycle-with-rc model, which is more suited to deliverables that are released only once per cycle. This week is also the deadline to freeze the contents of the final release. All new 'Train' deliverables need to have a deliverable file ( https://opendev.org/openstack/releases/src/branch/master/deliverables/train) and need to have done a release by milestone-2. The following new deliverables have not had a release yet (or not been already discussed in the etherpad[1]), and will not be included in Train unless a release is requested for them in the coming week: python-adjutantclient, cyborg-tempest-plugin, neutron-interconnection, openstack-helm, openstack-helm-addons, openstack-help-docs, openstack-helm-images, openstack-helm-infra, and compute-hyperv. The full list of currently deliverable-less deliverables can be found here[2]. Upcoming Deadlines & Dates -------------------------- Train-2 Milestone: July 25 (R-12 week) Thanks! -Kendall Nelson(diablo_rojo) + the Release Management Team [1]https://etherpad.openstack.org/p/train-membership-freeze [2] http://paste.openstack.org/show/754816/

1 0

[oslo] PTO next week
by Ben Nemec 24 Jul '19

24 Jul '19

Just a heads up that I will be on PTO next week and incommunicado pretty much the whole time. Moises (moguimar) volunteered to run the Oslo meeting so that will still be happening. Herve (hberaud) and Doug (dhellmann) are our release liaisons so if you need a release of any Oslo libraries next week, they're the ones to talk to. Otherwise, things have been pretty quiet in Oslo-land (which would be Norway, I guess), so I expect all the things to break next week while I'm out. Good luck. ;-) Thanks. -Ben

1 0

[ops] [nova] How to change/remove a property from an instance ?
by Massimo Sgaravatto 24 Jul '19

24 Jul '19

In my OpenStack Rocky cloud I have some instances that were created (before updating to Rocky) using images with the property hypervisor_type='qemu'. Since in my compute nodes [libvirt]/virt_type is "kvm", after having updated to Rocky I had to change these images removing that property of changing its value (qemu --> kvm): this is discussed in the thread: http://lists.openstack.org/pipermail/openstack-discuss/2019-July/thread.htm… Now I am not able to migrate anymore the instances that were created using those images (ImagePropertiesFilter filters out all compute nodes) before fixing the property. As suggested by Matt I tried to play with the instance_system_metadata table of the nova database. I first tried to change the value of that property (qemu --> kvm) [*] I also tried to remove the property [**] but the nova migrate keeps failing. In the request I keep seeing that Instance contains properties ImageMetaProps...img_hv_type='qemu' What else needs to be modified? Thanks, Massimo [*] mysql> select * from instance_system_metadata where instance_uuid='57dcdc2f-ef30-4512-beff-128c1621c5c6'; +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ | created_at | updated_at | deleted_at | id | instance_uuid | key | value | deleted | +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ | 2017-05-15 12:32:57 | NULL | NULL | 9868217 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_hypervisor_type | kvm | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868220 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_disk_format | qcow2 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868223 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_container_format | bare | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868226 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_min_ram | 512 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868229 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_min_disk | 20 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868232 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_base_image_ref | c2d3e37c-b14f-4472-9bc2-d75f26ab900d | 0 | +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ [**] mysql> delete from instance_system_metadata where id='9868217'; Query OK, 1 row affected (0.01 sec) mysql> select * from instance_system_metadata where instance_uuid='57dcdc2f-ef30-4512-beff-128c1621c5c6'; +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ | created_at | updated_at | deleted_at | id | instance_uuid | key | value | deleted | +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ | 2017-05-15 12:32:57 | NULL | NULL | 9868220 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_disk_format | qcow2 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868223 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_container_format | bare | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868226 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_min_ram | 512 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868229 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_min_disk | 20 | 0 | | 2017-05-15 12:32:57 | NULL | NULL | 9868232 | 57dcdc2f-ef30-4512-beff-128c1621c5c6 | image_base_image_ref | c2d3e37c-b14f-4472-9bc2-d75f26ab900d | 0 | +---------------------+------------+------------+---------+--------------------------------------+------------------------+--------------------------------------+---------+ 5 rows in set (0.00 sec)

2 1

[opendev] git clone fails: "The TLS connection was non-properly terminated"
by Bernd Bausch 24 Jul '19

24 Jul '19

There seems to be something wrong with the opendev server. Is this a glitch, a daily occurrence (backup?), or a deeper problem? This happens around midnight between July 22 and 23 UTC. $ git clone https://opendev.org/openstack/devstack.git -b stable/stein Cloning into 'devstack'... fatal: unable to access 'https://opendev.org/openstack/devstack.git/': gnutls_handshake() failed: The TLS connection was non-properly terminated. Same result when I try to clone any other random repo. I can get Devstack from Github, but when setting up the cloud, stack.sh wants to clone everything from git.openstack.org, which fails again. I could replace all the hardcoded /git.openstack.org/ strings with the equivalent /github.com/ URL, but I am not sure if I can trust that all Github repos are up to date. Bernd.

4 7

[devstack] Devstack fails with requirement pyparsing===2.4.1
by Bernd Bausch 24 Jul '19

24 Jul '19

Trying to install Devstack Stein on freshly installed Ubuntu 18.04. stack.sh fails with this: Downloading https://files.pythonhosted.org/packages/6f/7c/ebba93a2d3e09fd0220e1adfdb823… (131kB) Collecting pyparsing===2.4.1 (from -c https://releases.openstack.org/constraints/upper/master (line 492)) Could not find a version that satisfies the requirement pyparsing===2.4.1 (from -c https://releases.openstack.org/constraints/upper/master (line 492)) (from versions: 1.4.6, 1.4.7, 1.4.8, 1.4.11, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.5.7, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.0.6, 2.0.7, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.6, 2.1.7, 2.1.8, 2.1.9, 2.1.10, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0) No matching distribution found for *pyparsing===2.4.1* (from -c https://releases.openstack.org/constraints/upper/master (line 492)) (red highlight is mine) The most recent version of pyparsing <https://pypi.org/project/pyparsing/>is 2.4.0. I would file a bug, but I don't know against which project. Who owns https://releases.openstack.org/constraints/upper/master? Or is there a misunderstanding somewhere? My local.conf is below - it's based on a Rocky local.conf (which does work on Rocky) and might be incorrect. Bernd. [[local|localrc]] ########################################################################### ### Passwords ADMIN_PASSWORD=MYPASSWORD DATABASE_PASSWORD=$ADMIN_PASSWORD RABBIT_PASSWORD=$ADMIN_PASSWORD SERVICE_PASSWORD=$ADMIN_PASSWORD ########################################################################### ### Logging LOG_COLOR=False LOGFILE=$DEST/logs/stack.sh.log ########################################################################### ### Networking IP_VERSION=4 HOST_IP=192.168.1.200 ########################################################################### ### Software sources # use https since the vlabs proxy seems to block the git:// scheme GIT_BASE=https://git.openstack.org # Ensure the latest software is installed PIP_UPGRADE=True ########################################################################### ### Glance DOWNLOAD_DEFAULT_IMAGES=False IMAGE_URLS="http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img" # Glance's image import doesn't work when Glance runs as a UWSGI server # For this setting to take effect, devstack/lib/glance must be patched GLANCE_USE_UWSGI=False ########################################################################### ### Swift # by default, Devstack interactively asks for a hash value. Avoid this. SWIFT_HASH=123 enable_service swift ########################################################################### ### Heat enable_plugin heat https://git.openstack.org/openstack/heat stable/stein enable_service h-eng h-api h-api-cfn h-api-cw enable_plugin heat-dashboard https://git.openstack.org/openstack/heat-dashboard stable/stein enable_service heat-dashboard ########################################################################### ### Cinder: Enable backup enable_service c-bak ########################################################################### ### Telemetry: Use Gnocchi as sample repository ### and configure medium archive policy CEILOMETER_BACKEND=gnocchi GNOCCHI_ARCHIVE_POLICY=medium enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer stable/stein enable_plugin aodh https://git.openstack.org/openstack/aodh stable/stein ########################################################################### ### Neutron: Enable internal DNS resolution in ml2_conf.ini Q_ML2_PLUGIN_EXT_DRIVERS=port_security,dns_domain_ports ########################################################################### ### Post config for Neutron (DNS) and Nova (allow KVM in Vmware machine) [[post-config|$NEUTRON_CONF]] [DEFAULT] dns_domain = devstack.org. [[post-config|/etc/nova/nova-cpu.conf]] [libvirt] # set machine type to work around bug # https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1691109 hw_machine_type = x86_64=pc-i440fx-xenial

5 4

[mistral] Office hour in ~5 min
by Renat Akhmerov 24 Jul '19

24 Jul '19

Hi, We’ll have an office hour in about 5 mins. Join us at #openstack-mistral if you have anything to discuss. Thanks Renat Akhmerov @Nokia

1 0

[Kolla][Keystone] No "Domains" found in identity area.
by Eddie Yen 23 Jul '19

23 Jul '19

Hi, I'm using kolla-ansible stable/rocky release to deploy Openstack Try to test with multi-domains requirements. I'm do nothing but add "horizon_keystone_multidomain: True" into globals.yml After deployment, it works because I have to enter the domain name at login screen. Logged in with default domain and admin account. But I cannot found any "Domains" section in identity block. Is there something I miss?

1 2

[keystone] Keystone Train Midcycle Report
by Colleen Murphy 23 Jul '19

23 Jul '19

This week the keystone team conducted a two-day virtual midcycle. Thanks to everyone who participated and contributed to making this a successful and productive meeting! This email will be a short summary of the key takeaways from the last two days. The full agenda and meeting notes can be found on this etherpad: https://etherpad.openstack.org/p/keystone-train-midcycle-topics # Caching Guide We've long been proposing to document best practices for configuring caching for keystone. We began the meeting with a discussion about what exactly should be contained in such a guide, starting with anecdotes from the team about our own operational experience with configuring memcached for keystone. We quickly realized the guide needed at least two parts, one for keystone itself and one for keystonemiddleware. We also made two technical observations, first that the default value for the memcache_socket_timeout option in oslo.cache could be reduced[1] and second that the memcache_socket_timeout option exposed in keystone is not actually used by keystone and needs to be removed[2]. We talked about security strategies for memcached: oslo.cache exposes encryption capabilities which is required for organizations that consider the data stored in memcached to be "at rest", but it also hurts performance. An alternative strategy that could be recommended in the guide is to set up TLS tunneling to secure in-flight data without impeding fetching performance. The initial caching guide update was proposed here[3]. This guide will be a living document and will continue to grow as we gather real-life data from operators. After this initial update, we'll start taking steps to ensure that keystone requires caching be set up in order to operate. [1] https://review.opendev.org/672063 [2] https://bugs.launchpad.net/keystone/+bug/1837407 [3] https://review.opendev.org/672120 # Community Goals We then had a short status update session on the various community goals we need to meet. The mutable config goal from a few cycles ago is not yet complete but has an implementation proposed[4] which needs more eyes and testing. The PDF generation goal implementation is underway[5] but there are some issues already being tracked by the community[6]. We will need to work to ensure the issues we're seeing are noted and addressed. The last of the Python3 Train job changes for keystone repos were merged, so I think we've completed our end of that goal. We also already have a patch proposed to add the ipv6-only job[7] which is just waiting on the base job to merge in tempest. [4] https://review.opendev.org/585417 [5] https://review.opendev.org/669982 [6] https://etherpad.openstack.org/p/pdf-goal-train-common-problems [7] https://review.opendev.org/671903 # Milestone 2 Train Retrospective We closed out the first day with a mid-cycle retrospective[8]. Some key areas for improvement from this were: * Do more video conferencing - The real-time face-to-face communication is really effective for getting to the heart of discussions and making decisions quickly - Since it is looking likely that most of us will not be at the next PTG in Shanghai, we can set up a pre- and post- PTG video meeting to plan the cycle and catch people up after the IRL PTG - Milestone-ly video meetings are working for helping us keep pace and sync up - We can keep a queue of potential topics and make sure we have a meeting when there are enough topics to discuss - We'll try using video conferencing for office hours - We'll try Jitsi again even though there were some technical issues last time since it is OSS and supports recording * Improve office hours - Office hours still don't feel useful and haven't happened regularly, but we still think they have potential - We'll use the weekly newsletter to commit to the topic the week before, which gives the team a better opportunity to prepare for it and can also attract interest from people outside the keystone team - Use video conferencing as mentioned above * Improve the weekly newsletter - We'll use the weekly newsletter to call out action items from the last meeting - In addition to providing statistics about the number of open reviews, we'll highlight certain reviews that need attention based on various criteria such as age, priority with regard to cycle commitments, community goals, and specific review requests. - We'll announce the following week's office hours topic in the newsletter [8] https://etherpad.openstack.org/p/keystone-train-midcycle-retrospective # System Scope and Default Roles The second day of the midcycle was used to go through our list of open system scope[9] and default roles[10] bugs and work through what scopes and roles each API should support. The notes in the etherpad largely capture every decision fairly well, but a few issues worth highlighting are: * Implied roles[11]: the regular roles API is currently system-scope only, even though it's possible to have domain-specific roles, because for the most part for a role to be useful the role's creator must also have the ability to configure policy, which domain admins do not. Implied roles still require access to policy configuration except in the case where you are only using implied roles for aliasing, in which case it may make sense for a domain admin to have the ability to create an implied role relationship. We decided to start with making the implied roles API system-scope only for now and to open a separate bug to discuss how to deal with domain-specific roles and implied roles. * Revocation list[12]: we're currently not enforcing policy on this API at all because it can only ever return a 410 Gone or a 403 Forbidden, never a success, so we will just deprecate the policy entirely rather than updating it to work with system scope or default roles. We reevaluated the priority on some of the bugs in the list. However, we noted that while some APIs are more heavily used than others, we can't change enforce_scope to True until all of them are completed - picking and choosing the most important ones and letting the others slide is not really an option. [9] https://bugs.launchpad.net/keystone/+bugs?field.tag=system-scope [10] https://bugs.launchpad.net/keystone/+bugs?field.tag=default-roles [11] https://bugs.launchpad.net/keystone/+bug/1805371 [12] https://bugs.launchpad.net/keystone/+bug/1818845 If I've left out any important details, don't hesitate to add them to this thread. Thanks again to everyone who was able to join the meeting! Thanks also to Julia Kreger for her advice on how to hold one of these virtual midcycles, it proved to be very useful and this first-timer really appreciated it :) Colleen

1 0