- openstack-discuss - lists.openstack.org

[tripleo][ci] ovb featureset01/35 and other jobs are down
by Wesley Hayutin 19 Mar '21

19 Mar '21

Greetings, ATM, the RDO team is working w/ Vexxhost to diagnose an outage in vexxhost that has caused various problems w/ heat stacks and now overcloud node registration and introspection. The issue is tracked w/ vexx and https://bugs.launchpad.net/tripleo/+bug/1920101. The third party check jobs are not expected to pass at this time. We'll follow up when we have more information. Thank you

1 0

[tripleo] Nominate David J. Peacock (dpeacock) for Validation Framework Core
by Gaël Chamoulaud 19 Mar '21

19 Mar '21

Hi TripleO Devs, David is already a key member of our team since a long time now, he provided all the needed ansible roles for the Validation Framework into tripleo-ansible-operator. He continuously provides excellent code reviews and he is a source of great ideas for the future of the Validation Framework. That's why we would highly benefit from his addition to the core reviewer team. Assuming that there are no objections, we will add David to the core team next week. Thanks, David, for your excellent work! -- Gaël Chamoulaud - (He/Him/His) .::. Red Hat .::. OpenStack .::. .::. DFG:DF Squad:VF .::.

9 12

[Cinder] Review pretty please
by Sam Morrison 19 Mar '21

19 Mar '21

Hi Cinder cores, I've had a simple review [1] waiting for several months, could I please get some feedback on this. Thanks! Sam [1] https://review.opendev.org/c/openstack/cinder/+/764875

3 2

[ops] Bandwidth problem on computes
by Jahson Babel 19 Mar '21

19 Mar '21

Hello everyone, I have a bandwidth problem between the computes nodes of an openstack cluster. This cluster runs on Rocky version with OpenVSwitch. To simplify I'll just pick 3 servers, one controller and two computes nodes all connected to the same switch. Every server is configured with two 10G links. Those links are configured in LACP /teaming. From what I understand of teaming and this configuration I should be able to get 10Gbps between all three nodes. But if I iperf we are way below this : compute1 # sudo iperf3 -c compute2 -p 5201 Connecting to host compute2, port 5201 [ 4] local X.X.X.X port 44946 connected to X.X.X.X port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 342 MBytes 2.87 Gbits/sec 137 683 KBytes [ 4] 1.00-2.00 sec 335 MBytes 2.81 Gbits/sec 8 501 KBytes Plus the problem seems to be only present with incoming traffic. Which mean I can almost get the full 10gbps if I iperf from a compute to the controller. compute1 # sudo iperf3 -c controller -p 5201 Connecting to host controller, port 5201 [ 4] local X.X.X.X port 39008 connected to X.X.X.X port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.10 GBytes 9.41 Gbits/sec 0 691 KBytes [ 4] 1.00-2.00 sec 1.09 GBytes 9.38 Gbits/sec 0 803 KBytes If I do the opposite I get the same results I was getting between the 2 computes. From the tests we've done it seems related to the openstack's services, specifically neutron or OpenVSwitch. From the time those services are running we can't get the full bandwidth. Stopping the services won't fix the issue, in our case removing the packages and rebooting is the only way to obtain the full bandwidth between computes. I voluntarily didn't mention VMs to simplify the question but of course this behavior can also be observed in VMs Knowing that we can achieve 10Gbps it doesn't seems related to the hardware nor the OS. That why we suspect OpenStack's services. But I couldn't find any evidence or misconfiguration that could confirm that. So if anyone got some hints about that kind of setup and/or how mitigate bandwidth decrease I would appreciate. Let know if you need more info. Thanks in advance, Jahson

2 2

Re: [nova] Train: how to use newer API microversion
by Eugen Block 19 Mar '21

19 Mar '21

Alright, I made it work it seems. After I added the new endpoints (v2.1) and disabled the old ones (v2) I restarted nova-api service. This allowed me to use microversion 2.37 to use the option "--nic none", so that works as expected now. I just don't know yet if there are any clients that require v2 so for now I let them in "disabled" state. Thanks, Eugen Zitat von Eugen Block <eblock(a)nde.ag>: >> If you (or your end users) rely on the legacy APIs, you can have both >> like this (192.0.2.40 is the example IP address of the API host; the >> /compute could be :some_port if you prefer that): > > I started with disabling the old endpoints but then I couldn't use > compute at all to create a new instance. There must be something > still relying on the old endpoint but I can't find it right now. > I'll give it another shot tomorrow. Thanks for your input. > > > Zitat von Radosław Piliszek <radoslaw.piliszek(a)gmail.com>: > >> On Thu, Mar 18, 2021 at 3:50 PM Eugen Block <eblock(a)nde.ag> wrote: >>> I also found that already, but it doesn't describe how I can move from >>> v2 to v2.1 in order to use only the newer API. Do you have any >>> information about that? >> >> If you (or your end users) rely on the legacy APIs, you can have both >> like this (192.0.2.40 is the example IP address of the API host; the >> /compute could be :some_port if you prefer that): >> >> +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ >> | ID | Region | Service Name | Service Type | Enabled | >> Interface | URL | >> +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ >> | xxx | RegionOne | nova | compute | True | public >> | http://192.0.2.40/compute/v2.1 | >> | xxx | RegionOne | nova_legacy | compute_legacy | True | public >> | http://192.0.2.40/compute/v2/$(project_id)s | >> +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ >> >> If you don't need the legacy APIs, you can delete their endpoints. >> >> -yoctozepto

1 0

[keystone][kolla][horizon] versioned or unversioned keystone endpoints?
by Mark Goddard 19 Mar '21

19 Mar '21

Hi, There seems to be some inconsistency around whether Keystone endpoints in the catalog should be versioned or not. Specifically, should they have a /v3 suffix? I did a survey of code using keystone-manage bootstrap via codesearch: * Kolla Ansible: unversioned * DevStack: unversioned * puppet-keystone: unversioned * Keystone install guides: versioned * OSA: unversioned I imagine that when v2 was around, the endpoint needed to be unversioned. But was there a point when it became recommended to add a version? I'm asking because of an issue in horizon when downloading a clouds.yaml file for application credentials. If the auth_url field is unversioned, then some older client libraries will fail with a Not Found error during e.g. openstack server list. I tested using Stein and Train U-C. The former fails, the latter works. If I use Stein U-C, then update keystoneauth1 to 3.17.3 (from Train U-C), it works. Thanks, Mark

1 0

[neutron] Drivers meeting agenda - 19.03.2021
by Slawek Kaplonski 19 Mar '21

19 Mar '21

Hi, First of all sorry for sending it so late - I simply forgot to send this email yesterday :) For today's drivers meeting we have 1 RFE to discuss: * https://bugs.launchpad.net/neutron/+bug/1904559 - it is continuation of the discussion from last week. Michael Johnson from Designate team provided his feedack on it so I think that based on that we can make final decision about that RFE on our today's meeting. -- Slawek Kaplonski Principal Software Engineer Red Hat

1 1

[release] Release countdown for week R-3 Mar 22 - Mar 26
by Herve Beraud 18 Mar '21

18 Mar '21

Development Focus ----------------- The Release Candidate (RC) deadline is next Thursday, 25 March, 2021. Work should be focused on fixing any release-critical bugs. General Information ------------------- All deliverables released under a cycle-with-rc model should have a first release candidate by the end of the week, from which a stable/wallaby branch will be cut. This branch will track the Wallaby release. Once stable/wallaby has been created, the master branch will be ready to switch to Xena development. While the master branch will no longer be feature-frozen, please prioritize any work necessary for completing Wallaby plans. Release-critical bugfixes will need to be merged in the master branch first, then backported to the stable/wallaby branch before a new release candidate can be proposed. Actions ------- Early in the week, the release team will be proposing RC1 patches for all cycle-with-rc projects, using the latest commit from the master branch. If your team is ready to go for cutting RC1, please let us know by leaving a +1 on these patches. If there are still a few more patches needed before RC1, you can -1 the patch and update it later in the week with the new commit hash you would like to use. Remember, stable/wallaby branches will be created with this, so you will want to make sure you have what you need included to avoid needing to backport changes from the master branch (which will technically then be Xena) to this stable branch for any additional RCs before the final release. The release team will also be proposing releases for any deliverable following a cycle-with-intermediary model that has not produced any Wallaby release so far. Finally, now is a good time to finalize release highlights. Release highlights help shape the messaging around the release and make sure that your work is properly represented. Upcoming Deadlines & Dates -------------------------- RC1 deadline: 25 March, 2021 (R-3 week) Final RC deadline: 08 April, 2021 (R-1 week) Final Wallaby release: 14 April, 2021 Xena virtual PTG: 19 - 23 April, 2021 Thanks for your reading -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

1 0

OSP13 - Queens - Inconsistent data between in DB between ipamallocation and ipallocation
by Laurent Dumont 18 Mar '21

18 Mar '21

Hey everyone, We just troubleshooted (and fixed) a weird issue with ipamallocation. It looked like ipallocations we're not properly removed from the ipamallocation table. This caused port creations to fail with an error about the IP being already allocated (when there wasn't an actual port using the IP). We found this old issue https://bugs.launchpad.net/neutron/+bug/1884532 but not a whole lot of details in there. Has anyone seen this before? Thanks!

2 2

[neutron] Unit test jobs broken
by Slawek Kaplonski 18 Mar '21

18 Mar '21

Hi, Our UT gate jobs are now broken due to new neutron-lib release. Fix for that is already proposed in [1]. So if Your UT jobs are failing like e.g. in [2] please don't recheck Your patch before [1] will be merged. [1] https://review.opendev.org/c/openstack/neutron/+/780802 [2] https://8ee33ec1a424812a1857-16390f05fde22eb723929856dcc38fcd.ssl.cf5.rackc… -- Slawek Kaplonski Principal Software Engineer Red Hat

1 1

[Neutron] [Designate] Private / Internal DNS Zones with custom records for i.e. service discovery
by Christian Rohmann 18 Mar '21

18 Mar '21

Hey Openstack-Discuss, apart from the standardized and auto-created records for ports / floating-ips and instances (https://docs.openstack.org/neutron/latest/admin/config-dns-int.html) - is there any way to allow users to add their own records which then only resolve internally? The Neutron API (https://docs.openstack.org/api-ref/network/v2/#id52) seems to be all about the resources it manages, so no additional or custom records there. Looking at the Designate API https://docs.openstack.org/api-ref/dns/?expanded=create-zone-detail#create-… is does not seem to be an option to mark a zone as "internal" or "private". But maybe there is another way to add records to the internal zone? I am thinking of an only internally resolvable / valid DNS zone carrying records for i.e. service discovery / cluster forming. There are more and more tools just looking up a DNS records to find cluster members ... * ElasticSearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-dis… * Hazelcast: https://github.com/hazelcast/hazelcast-kubernetes#understanding-discovery-m… * HiveMQ: https://github.com/hivemq/hivemq-dns-cluster-discovery-extension/blob/maste… * RabbitMQ: https://www.rabbitmq.com/cluster-formation.html#peer-discovery-dns [...] and with Kubernetes and the headless service concept there are more tools (ab)using DNS for this every week. So having internal dns zones which only resolve within the project would be really helpful. The hyperscalers call this feature * AWS "Private hosted zones" (https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-priv…) * Azure "Private DNS" (https://medium.com/azure-architects/exploring-azure-private-dns-be65de08f780) * GCP "Private zone" (https://cloud.google.com/blog/products/networking/introducing-private-dns-z…) * Alibaba Cloud "DNS PrivateZone" (https://www.alibabacloud.com/product/private-zone) Regards Christian

3 2

[tripleo] launchpad wallaby-3 milestone closed use wallaby-rc1
by Marios Andreou 18 Mar '21

18 Mar '21

Hi tripleo o/ as just mentioned on irc #tripleo, I closed out the wallaby-3 launchpad milestone yesterday. If you want to set a milestone in bugs please use wallaby-rc1 (wallaby-3 is no longer in the "assign milestone" list). I moved the wallaby-3 unclosed bugs to wallaby-rc1 with our helper script [1] results at [2]. Please try and keep bug status updated. Things like moving it to fix-released once we're done merging fixes - it is easy to forget about that. We have 155 bugs triaged at https://launchpad.net/tripleo/+milestone/wallaby-rc1 at least some of those can be moved to fixed/closed nofix etc. Yes I know, #cool story bro thank you for reading ;) and for your help on bug status regards, marios [1] https://opendev.org/openstack/tripleo-ci/src/branch/master/scripts/move_bug… [2] https://gist.github.com/marios/b3155fe3b1318cc26bfa4bc15c764a26#gistcomment…

1 0

Re: [nova] Train: how to use newer API microversion
by Radosław Piliszek 18 Mar '21

18 Mar '21

On Thu, Mar 18, 2021 at 3:50 PM Eugen Block <eblock(a)nde.ag> wrote: > I also found that already, but it doesn't describe how I can move from > v2 to v2.1 in order to use only the newer API. Do you have any > information about that? If you (or your end users) rely on the legacy APIs, you can have both like this (192.0.2.40 is the example IP address of the API host; the /compute could be :some_port if you prefer that): +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ | xxx | RegionOne | nova | compute | True | public | http://192.0.2.40/compute/v2.1 | | xxx | RegionOne | nova_legacy | compute_legacy | True | public | http://192.0.2.40/compute/v2/$(project_id)s | +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ If you don't need the legacy APIs, you can delete their endpoints. -yoctozepto

1 0

[ironic] [infra] Making Glean work with IPA for static IP assignment
by Dmitry Tantsur 18 Mar '21

18 Mar '21

Hi Ian, Given our timezone difference, I've decided to send you an email, adding openstack-discuss in CC for greater exposure. We're trying to make IPA (our agent ramdisk) to work without relying on DHCP for cases like Edge deployments. We've settled on the network_data.json format for the API side and wanted to use Glean on the ramdisk to apply it. You can read more details in [1]. The problem is, I cannot make Glean work with any ramdisk I build. The crux of the problem seems to be that NetworkManager (used by default in RHEL, CentOS, Fedora and Debian at least) starts very early, creates the default connection and ignores whatever files Glean happens to write afterwards. On Debian running `systemctl restart networking` actually helped to pick the new configuration, but I'm not sure we want to do that in Glean. I haven't been able to make NetworkManager pick up the changes on RH systems so far. I build ramdisks using IPA-builder [2] by adding the simple-init element. I've tried removing dhcp-all-interfaces (which we depend on by default) to no effect. I've tried disabling the DHCP server, ended up with no IP connectivity at all. I haven't tried to shutdown and restart a connection as recommended in [3] since it's not trivial to do via SSH. Do you maybe have any hints how to proceed? I'd be curious to know how static IP assignment works in the infra setup. Do you have images with NetworkManager there? Do you use the simple-init element? Any help is very appreciated. Dmitry [1] https://specs.openstack.org/openstack/ironic-specs/specs/not-implemented/L3… [2] https://opendev.org/openstack/ironic-python-agent-builder [3] https://mail.gnome.org/archives/networkmanager-list/2014-January/msg00032.h… -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

4 9

[nova] Train: how to use newer API microversion
by Eugen Block 18 Mar '21

18 Mar '21

Hi *, I'm a little confused and could use some guidance regarding endpoints and microversions. Just recently I did a lot of maintenance in our old cloud, I believe it was created with Kilo and is now Train. Anyway, I wanted to use the "server create" option "--nic none" but couldn't because of the v2 endpoint for the nova service, right? According to the cli help this option is available from microversion 2.37+. This was the first attempt before adding new endpoints: controller:~ # openstack server create --image <UUID> --nic none --flavor 3 test-nic-none nics must be a list or a tuple, not <class 'str'> These were the endpoints before I added new ones: controller:~ # openstack endpoint list | grep nova | <UUID> | RegionOne | nova | compute | True | public | http://controller:<PORT>/v2/%(tenant_id)s | | <UUID> | RegionOne | nova | compute | True | internal | http://controller:<PORT>/v2/%(tenant_id)s | | <UUID> | RegionOne | nova | compute | True | admin | http://controller:<PORT>/v2/%(tenant_id)s | I checked with the Train install guide and created new endpoints for nova: controller:~ # openstack endpoint create --region RegionOne compute public http://controller:<PORT>/v2.1 controller:~ # openstack endpoint create --region RegionOne compute internal http://controller:<PORT>/v2.1 controller:~ # openstack endpoint create --region RegionOne compute admin http://controller:<PORT>/v2.1 I should mention that I haven't restarted any services yet, I wanted to check each step carefully since people are working in that cloud. In the debug output from creating a new instance I saw that it's still trying to use v2 api, so I disabled it, resulting in error messages that there's no compute service available. So I reenabled the old endpoints and now have both versions active. Then I tried this: controller:~ # openstack --os-compute-api-version 2.37 server create --image <UUID> --nic none --flavor 3 test-nic-none Invalid input for field/attribute networks. Value: none. 'none' is not of type 'array' (HTTP 400) (Request-ID: req-b3c53ab2-8040-42cf-bf07-d03e8f46cfd5) Now I'm running out of ideas how to upgrade our api endpoints so the services will actually be able to use them properly. I read Train release notes (we're planning to upgrade to Ussuri soon) and multiple docs about microversions but nothing seems to cover my issue. Could anyone shed some light? I must admit, during the upgrade cycles in the past years I just hoped everything would keep working and it did, I didn't put too much attention in the api versions yet. Any help is appreciated! Thanks and best regards, Eugen

2 2

[glance][openstack-ansible] Snapshots disappear during saving
by Oliver Wenz 18 Mar '21

18 Mar '21

Hi! We are currently experiencing problems with our OpenStack Ansible Victoria cloud when trying to create snapshots from instances. In some cases, everything works but often the pending snapshots just disappear. When this happens, the following glance-api.service errors show up: Mar 17 08:33:12 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:33:11.975 85 INFO glance.api.v2.image_data [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token. Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.283 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] REQ: curl -i http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… -X PUT -H "X-Auth-Token: gAAAAABgUb7IBvZ_..." Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.285 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP STATUS: 504 Gateway Time-out Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.285 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP HEADERS: {'content-length': '92', 'cache-control': 'no-cache', 'content-type': 'text/html', 'connection': 'close'} Mar 17 08:35:15 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:15.286 85 INFO swiftclient [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP BODY: b"<html><body><h1>504 Gateway Time-out</h1>\nThe server didn't respond in time.\n</body></html>\n" Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.306 85 ERROR glance_store._drivers.swift.store [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Error during chunked upload to backend, deleting stale chunks.: swiftclient.exceptions.ClientException: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload. Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.320 85 ERROR glance_store._drivers.swift.store [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to add object to Swift. Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload..: swiftclient.exceptions.ClientException: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload. Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.327 85 ERROR glance.api.v2.image_data [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to upload image data due to internal error: glance_store.exceptions.BackendException: Failed to add object to Swift. Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Caught error: Failed to add object to Swift. Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload..: glance_store.exceptions.BackendException: Failed to add object to Swift. Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload.. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1014, in add 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self._delete_stale_chunks( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise() 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise value 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1003, in add 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi manager.get_connection().put_object( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1960, in put_object 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return self._retry(reset_func, put_object, container, obj, contents, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1843, in _retry 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi reset_func(func, *args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1940, in _default_reset 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise ClientException('put_object(%r, %r, ...) failure and no ' 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi swiftclient.exceptions.ClientException: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi During handling of the above exception, another exception occurred: 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1347, in __call__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1391, in dispatch 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return method(*args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py", line 416, in wrapped 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 298, in upload 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self._restore(image_repo, image) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise() 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise value 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 163, in upload 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py", line 208, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 501, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.force_reraise() 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise value 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 447, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py", line 198, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py", line 318, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend, 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 567, in set_data 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 458, in _upload_to_store 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 398, in add_with_multihash 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return store_add_to_backend_with_multihash( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 480, in store_add_to_backend_with_multihash 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add( 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py", line 279, in add_adapter 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py", line 176, in op_checker 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1082, in add 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi raise glance_store.BackendException(msg) 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi glance_store.exceptions.BackendException: Failed to add object to Swift. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Got error from Swift: put_object('glance_images', '1b9af05a-f7c9-4315-9354-e09f1df66321-00001', ...) failure and no ability to reset contents for reupload.. 2021-03-17 08:35:16.425 85 ERROR glance.common.wsgi Mar 17 08:35:16 infra1-glance-container-99614ac2 uwsgi[85]: Wed Mar 17 08:35:16 2021 - uwsgi_response_writev_headers_and_body_do(): Connection reset by peer [core/writer.c line 306] during PUT /v2/images/1b9af05a-f7c9-4315-9354-e09f1df66321/file (192.168.110.215) Mar 17 08:35:16 infra1-glance-container-99614ac2 glance-wsgi-api[85]: 2021-03-17 08:35:16.441 85 CRITICAL glance [req-32345bbf-a88f-450e-90b2-69c6a5804a7a 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unhandled error: OSError: write error 2021-03-17 08:35:16.441 85 ERROR glance OSError: write error 2021-03-17 08:35:16.441 85 ERROR glance It seems as if the problems occur more often with large instances, i.e. there are fewer problems with new Ubuntu 20.04 instances but after an 'apt-upgrade' on the instance, the error occurs every time a snapshot is taken. Any help is much appreciated! Kind regards, Oliver

2 1

[neutron][requirements] RFE requested for neutron-lib
by Slawek Kaplonski 18 Mar '21

18 Mar '21

Hi, I'm raising (again) RFE to ask if we can release, and include yet another new version of the neutron-lib in the Wallaby release. Recently we included version 2.10.0 which contained fix for bug related to the secure rbac policies but yesterday we found out one more bug related to the same policies [1]. Fix for that second bug is already merged [2] and new release patch is proposed at [3]. This time this new version don't bumps any other dependencies' versions or things like that. There is only one additional fix related to the way how context with new secure rbac policies is handled. Please let me know if I You need anything else regrarding this RFE. [1] https://launchpad.net/bugs/1919386 [2] https://review.opendev.org/c/openstack/neutron-lib/+/781075 [3] https://review.opendev.org/c/openstack/releases/+/781149 -- Slawek Kaplonski Principal Software Engineer Red Hat

3 4

[cinder] review priorities over the next week
by Brian Rosmaita 17 Mar '21

17 Mar '21

We're in the final push before RC-1 next week. Top priority are the FFE patches listed on the etherpad, which must be merged by Friday: https://etherpad.opendev.org/p/cinder-wallaby-features Other than that, work from the priority dashboard: http://tiny.cc/CinderPriorities If you have a follow-up patch requested during review of a feature (or if you have found a bug in a feature that needs to be fixed), please add it to the wallaby features etherpad (at the top) so a core reviewer can give it a +1 priority and make it show up in the priorities dashboard: https://etherpad.opendev.org/p/cinder-wallaby-features Finally, I think Rajat is still trying to get a release out from stable/train, so you can check here, too: https://etherpad.opendev.org/p/stable-releases-review-tracker-22-02-2021 And, don't forget about the Second Cinder Festival of XS Reviews on Friday 19 March 1400-1600 UTC. Happy Reviewing!

1 0

[cinder] Xena PTG Planning
by Brian Rosmaita 17 Mar '21

17 Mar '21

Hello Cinderinos, Some preliminary info about Cinder at the PTG: We'll meet for 3 hours each day on Tuesday 20 April through Friday 23 April from 1300-1600 UTC. - on Monday you are free to attend any PTG sessions you like - we'll have Happy Hour at 1500 UTC on Tuesday 20 April We'd like to have a "drivers day" devoted to discussion of issues impacting backend drivers. If you are a driver maintainer and have a preference for which day this is, please leave a note on the etherpad: https://etherpad.opendev.org/p/xena-ptg-cinder-planning Driver maintainers and anyone else who has something to discuss with the Cinder team, feel free to add a topic to the etherpad. Finally, don't forget to register for the PTG; info is on the etherpad. cheers, brian

1 0

[cinder] festival of XS reviews 19 March 2021
by Brian Rosmaita 17 Mar '21

17 Mar '21

Hello Cinder community members, Due to popular demand, we will hold the Second Cinder Festival of XS Reviews on Friday 19 March (yes, that's this Friday at the end of this very week). Apologies for the late notice, but with RC-1 being released next week, we need to give people time to revise and resubmit (if necessary) so the fixes can make it into the Wallaby release. what: The Second Cinder Festival of XS Reviews when: Friday 19 March 2021 from 1400-1600 UTC where: https://bluejeans.com/3228528973 See you there! brian

1 0

[blazar][ptg] Xena PTG
by Pierre Riteau 17 Mar '21

17 Mar '21

Hello, We have allocated two hours during the Xena PTG for the Blazar project. We will meet on Thursday, April 22, 2021 from 15:00 to 17:00 UTC in the Bexar room. The agenda will be decided over the coming weeks and kept at https://etherpad.opendev.org/p/xena-ptg-blazar Anyone can propose discussion topics through the Etherpad or in our IRC meeting. Feel free to join if you want to know more about Reservation as a Service. Cheers, Pierre Riteau (priteau)

1 0

How to detach cinder volume
by Md. Hejbul Tawhid MUNNA 17 Mar '21

17 Mar '21

Hi, We are using openstack rocky. One of our openstack vm was running with 3 volume , 1 is bootable and another 2 is normal volume . We have deleted 1 normal volume without properly detaching. So volume is deleted but instance is still showing 3 volume is attached. Now we can't snapshot the instance and facing some others issue. Please advise how we can detach the volume(deleted) from the instance Note: We have reset state volume attached status to detached and delete the volume. Regards, Munna

3 3

[kolla][ptg] Xena PTG slots
by Mark Goddard 17 Mar '21

17 Mar '21

Hi, The Xena PTG [1] is fast approaching, and it is time to decide on our time slots. The event will take place on 19th - 23rd April. For the last few events, we have settled on the following schedule: * Monday: 13:00 - 17:00 UTC (General & Kolla) * Tuesday: 13:00 - 17:00 UTC (Kolla Ansible) * Wednesday: 13:00 - 15:00 UTC (Kayobe) This has largely worked quite well, allowing the contributor base in Europe and the US to collaborate. This schedule does tend to exclude contributors in other timezones however, and I would like to explore ideas to increase our global coverage. I would be interested to hear from anyone who would be interested in joining one or more slots in the 6:00 - 8:00 UTC window on one or more days. Please respond here or in IRC if you are interested. PS remember to sign up [2]! Thanks, Mark [1] https://www.openstack.org/ptg/ [2] https://april2021-ptg.eventbrite.com/

2 1

[MURANO] Can not deploy environment on existing network
by İzzettin Erdem 17 Mar '21

17 Mar '21

Hi everyone, I am tryng to use Murano on OSA-stable/ussuri. I have encountered an error; when I try to deploy an environment on existing network, it creates the VM but do not attach floating IP to it. In murano-agent.log on the vm there is "rabbitmq queue not found" error -->http://paste.openstack.org/show/803667/ , rabbitmq logs ---> http://paste.openstack.org/show/803668/, murano-engine logs ---> http://paste.openstack.org/show/803669/, murano.conf on containers----> http://paste.openstack.org/show/803672/ I can attach floating IP to the VM after Murano fails and also I can deploy environment if I use "create new network" option. Do you have any suggestions about it? Thanks, İzzettin

1 0

[horizon] Welcome Tatiana Ovchinnikova in horizon-core team
by Ivan Kolodyazhny 17 Mar '21

17 Mar '21

Hi team, As agreed during the horizon meeting [1] last week, I've added Tatiana to horizon-core team. Tatiana, thanks for your contributions and welcome back to the team! [1] http://eavesdrop.openstack.org/meetings/horizon/2021/horizon.2021-03-10-15.… Regards, Ivan Kolodyazhny, http://blog.e0ne.info/

1 0

[kolla] Upgrading from Centos7 Train to Centos8 Ussuri
by Braden, Albert 17 Mar '21

17 Mar '21

I'm testing the Train7->Ussuri8 upgrade on a heat stack, following this document: OpenStack Docs: CentOS 8<https://docs.openstack.org/kolla-ansible/train/user/centos8.html#migrating-…> I used the instructions here to successfully remove a controller from the cluster: https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hos… Now I'm ready to add one, using the instructions here: https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hos… They say to run bootstrap-servers: kolla-ansible -i <inventory> bootstrap-servers [ --limit <limit> ] The question is, how to run it? My build server is running Centos 7 and has Python 2.7.5. Should I just download Ussuri to my existing build server, or do I need to build a new Centos 8 build server with Python 3? I read the "potential issues<https://docs.openstack.org/kolla-ansible/train/reference/deployment-and-boo…>" text but it's not obvious whether it is safe to run Ussuri bootstrap-servers on Train hosts. If I don't run bootstrap-servers on the Train controllers, then how does the new Ussuri controller get integrated (RMQ, MariaDB, etc.)? I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

2 4

[telemetry][ptg] Telemetry PTG planning
by Matthias Runge 17 Mar '21

17 Mar '21

Hi there, for Telemetry, we had a short planning meeting after PTG last year. If you are interested in Telemetry, or have something Telemetry related to talk about, please add yourself and the topic to the planning etherpad [1]. Matthias [1] https://etherpad.opendev.org/p/telemetry-xena-ptg

1 0

[manila][requirements] FFE requested for python-manilaclient
by Goutham Pacha Ravi 17 Mar '21

17 Mar '21

Hello, We were tracking one last feature inclusion into python-manilaclient for the wallaby release: https://review.opendev.org/c/openstack/python-manilaclient/+/775020 The change does not introduce any new requirements, but adds new SDK methods and shell implementations for end users to harness a feature addition in manila to be able to update their security services. The release team was very gracious in allowing us to wrap up code reviews pertaining to the above change over the weekend; however, the gate was uncooperative this morning by pointing us to lingering sporadic failures that are a combination of bad test case logic as well as occasional test environment setup issues. (Zuul is awesome, but it has a way to call out our procrastinations) No projects that depend on python-manilaclient need this change in their current form because it is net new feature functionality (existing SDK/CLI functionality is unaltered). However, I thought i should check here before requesting a re-release of python-manilaclient under the Wallaby cycle. Thanks, Goutham

3 5

[cinder] Bug deputy report for week of 2021-03-08
by Sofia Enriquez 16 Mar '21

16 Mar '21

Hello, This is a bug report from 2021-03-09 to 2021-03-16. Critical: - https://bugs.launchpad.net/cinder/+bug/1832164: "SADeprecationWarning: The joinedload_all() function is deprecated, and will be removed in a future release. Please use method chaining with joinedload() instead". Assigned to Gorka Eguileor. High: - https://bugs.launchpad.net/cinder/+bug/1919161: "Automatic quota refresh counting temporary volumes". Assigned to Gorka Eguileor. Medium: - https://bugs.launchpad.net/cinder/+bug/1918229: "Nimble free space calculation". Unassigned but looks like Ajitha Robert may be working on this. Low: - https://bugs.launchpad.net/cinder/+bug/1918932: "volume delete rejection message doesn't include awaiting-transfer state". (low-hanging-fruit) Unassigned. - https://bugs.launchpad.net/cinder/+bug/1918307: " api-ref: Make documentation of HTTP 403 consistent". Unassigned. Incomplete: - https://bugs.launchpad.net/cinder/+bug/1918449: " ensure_export model_update ignored ". Unassigned. - https://bugs.launchpad.net/cinder/+bug/1918879: "id of encryption for volume type not verified". Unassigned. Not a bug:- Feel free to reply/reach me if I missed something. Regards Sofi -- L. Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[charms] Trilio Charms 21.03 release is now available
by Peter Matulis 16 Mar '21

16 Mar '21

The 21.03 release of the Trilio charms is now available. Please see the Release Notes for full details: https://docs.openstack.org/charm-guide/latest/2103_Trilio.html == Highlights == * TrilioVault 4.1 The charms now support TrilioVault release 4.1. * OpenStack Ussuri The charms add support for OpenStack Ussuri on both Ubuntu 18.04 LTS (Bionic) and Ubuntu 20.04 LTS (Focal). This is in addition to current Bionic support for OpenStack Queens, Stein, and Train. == Thank you == Lots of thanks to the following people who contributed to this release via code changes, documentation updates and testing efforts. Alex Kavanagh Aurelien Lourot Chris MacNaughton David Ames Liam Young Marian Gasparovic Peter Matulis -- OpenStack Charms Team

1 0

[cinder] New Cinder Bug Squad Meeting
by Sofia Enriquez 16 Mar '21

16 Mar '21

Hello, As discussed in the last meeting, Cinder will have an additional mini meeting to discuss bugs. [1] Cinder Bug Squad Meeting - Weekly on Wednesday at 1500 UTC <http://www.timeanddate.com/worldclock/fixedtime.html?hour=15&min=00&sec=0> in #openstack-cinder (IRC webclient <https://webchat.freenode.net/?randomnick=1&channels=%23openstack-cinder&pro…>) - Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting Feel free to add your bugs to the Agenda if I haven't done so already. Cheers, Sofi [1] http://eavesdrop.openstack.org/#Cinder_Bug_Squad_Meeting -- L. Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[Horizon][Victoria][ops] - Scope based policies in Horizon not working
by Luke Camilleri 16 Mar '21

16 Mar '21

Hi Everyone, we had sent an email to this mailing list last week on an issue that we are having which we thought was being generated by keystone but on closer inspection it seems that this is coming actually from Horizon policy files. We would like to implement OpenStack using the scope based approach (v3) so that we could keep the main roles (reader,member and Admin) and use the scopes to differentiate the targets. We have implemented the below settings in /etc/keystone/keystone.conf: enforce_scope = true enforce_new_defaults = true All works well with the cli and the scopes except Horizon. We constantly receive errors in /var/log/keystone.log like: 2021-03-15 23:15:55.772 1611435 WARNING keystone.server.flask.application [req-a89277a2-8fb2-4fcb-b337-e76a73bb3af9 1405ecf1ff7f4e4188648dbd1fb84c07 c584121f9d1a4334a3853c61bb5b9d93 - default default] You are not authorized to perform the requested action: identity:validate_token.: keystone.exception.ForbiddenAction: You are not authorized to perform the requested action: identity:validate_token but the same command (for example an "openstack domain list") works well from the CLI. We could identify the issue coming from the policy files shipped with the dashboard at /etc/openstack-dashboard/<policies.json> as the rules within these policies are not scope based and in json format while according to the roadmap it seems that yaml is being used for the policy files and they should be scope based. When we try to convert the files to json to then upgrade them (to use the scope syntax) we receive the following errors: #oslopolicy-convert-json-to-yaml --config-file /etc/keystone/keystone.conf --namespace keystone --output-file keystone_policy.yaml --policy-file keystone_policy.json Traceback (most recent call last): File "/bin/oslopolicy-convert-json-to-yaml", line 10, in <module> sys.exit(convert_policy_json_to_yaml()) File "/usr/lib/python3.6/site-packages/oslo_policy/generator.py", line 599, in convert_policy_json_to_yaml conf.output_file) File "/usr/lib/python3.6/site-packages/oslo_policy/generator.py", line 451, in _convert_policy_json_to_yaml scope_types=default_rule.scope_types) File "/usr/lib/python3.6/site-packages/oslo_policy/policy.py", line 1239, in __init__ scope_types=scope_types File "/usr/lib/python3.6/site-packages/oslo_policy/policy.py", line 1154, in __init__ self.description = description File "/usr/lib/python3.6/site-packages/oslo_policy/policy.py", line 1251, in description raise InvalidRuleDefault('Description is required') oslo_policy.policy.InvalidRuleDefault: Invalid policy rule default: Description is required. We believe that converting such files to yaml and then upgrading them for scope based access (v3) should solve our issues based on the fact that if we disable the enforce _scope and enforce_new_defaults we can have a fully functional setup (but not scope based) which defies the initial objective Thanks in advance for anyone that may be able to assist with this issue

2 2

Properly consuming system-scope and operator workflows
by Lance Bragstad 16 Mar '21

16 Mar '21

Hey all, Several projects have made significant progress adopting default roles and system-scope during the Wallaby cycle. One part of that work is obviously testing and we've been stringing together various tempest jobs with the new policies enabled to see what falls out. Ironic was working on some of this last week and it led to an interesting discussion on Friday about how we should handle system-scope with the current assumptions about administrator (e.g., project-admin) workflows. The majority of the discussion took place in IRC, but I tried to consolidate as much of the context as I could into an etherpad [0]. I'm socializing the overall discussion so we can get feedback from users, operators, and developers, as well as summarize some comparisons to other systems. Today, by default system administrators are anyone with the 'admin' role on a project. These users are allowed to do things on behalf of users, like rebooting an instance. This mostly works because the administrator's token is project-scoped and the resource they're interacting with is owned by a project. This assumption isn't true in the new default policies. The question is, should system administrators, or users with system-scoped tokens, be allowed to use those tokens to make changes to project-specific resources (e.g., creating a private image in a project, rebooting an instance, deleting a snapshot, etc)? The general feeling was that, yes, system administrators should have the ability to interact with resources owned by projects in their deployment. The discussion then focused on how we actually make that happen. We came up with three different solutions, all of which are detailed in the etherpad [0], but I'll summarize them below. Option 1: Let system-administrators exchange system-scoped (god-mode) tokens for tokens scoped to a specific project without an explicit role assignment on the project Option 2: Require system-administrators to grant themselves authorization on the project, obtain a new token, and perform the operation Option 3: Allow system-administrators to use system-scoped tokens to operate on project-specific resources We're iterating through the pros and cons of each approach in the etherpad. If you haven't taken a look, please do so and add your thoughts. If you're an operator with an existing workflow, we want to hear your feedback. My request for additional feedback aside, several people in the discussion brought up comparison to other identity systems and frameworks (e.g., how do kerberos admins usually handle this sort of thing? how would we fix this if we were using OAuth 2.0/OIDC?) I spent some time Friday thinking through what an OAuth 2.0 flow would look like in OpenStack [1][2]. I don't think my findings were earth-shattering, and I'm sure there are other ways we could implement OAuth 2.0 flows. While adhering to a standard offers guidance and makes some things easier (like integrating with other systems), I don't think it would be the silver bullet for fixing our RBAC woes (at least not this late in the game). OpenStack APIs are massive, complex, and operate at different parts of the technology stack. Some of those APIs are meant for typical users just requesting resources. Others are written with the system administrator as the end user. OpenStack's explosive API and functionality growth out-paced its authorization tools to protect it, allowing project_id to bake itself into resources, services, and components (API code, middleware, database layers). Relaxing the assumption of an always-present project ID in every token is going to be hard and given OpenStack's distributed approach to policy enforcement, truly fixing it would require consistency across the entire ecosystem, regardless of the authentication or authorization framework. Either way, I thought that was an interesting exercise, and I'm curious if others have different opinions or perspectives on those findings. Thanks for reading, Lance [0] https://etherpad.opendev.org/p/consuming-system-scope [1] https://bit.ly/3cjUu6o My first few attempts at mapping this out were wrong, but I left links to the early drafts in the etherpad [2] I ignored a lot of assumptions about how clients interact with OpenStack, or how middleware works. This was purely a thought-experiment.

1 0

[cinder] docs job failures
by Brian Rosmaita 16 Mar '21

16 Mar '21

Note to reviewers, since all FFE patches need to be reviewed and merged by friday: Don't let a Zuul -1 for openstack-tox-docs stop you from reviewing patches. Gorka has a fix posted that we believe will address this: https://review.opendev.org/c/openstack/cinder/+/780907 but it needs to make its way through check and gate first, so it may be awhile. An example of the docs failure is: https://zuul.opendev.org/t/openstack/build/28c1d829e2d4469d82128a4f295f4dad It happens very early in the job, so even if there's a problem with the docs on a patch you're looking at, you aren't going to see it. If you want to check the docs on your patch locally, the change in this patch works for me, but YMMV: https://review.opendev.org/c/openstack/cinder/+/780696

1 1

[oslo] Xena PTG Etherpad
by Ben Nemec 16 Mar '21

16 Mar '21

Daniel created an etherpad for the Xena PTG[0]. If you have any topics to discuss please add them so we know whether we need to reserve a time. Thanks. -Ben 0: https://etherpad.opendev.org/p/oslo-xena-topics

1 0

[openstack-ansible] vPTG April 2021
by Dmitriy Rabotyagov 16 Mar '21

16 Mar '21

1 0

[live migration] [nova] [libvirt] [stein] Live migration of nested VMs fails
by Nagaraj Akkina 16 Mar '21

16 Mar '21

Hi Team, We are using live migration in our environment from some time now, recently there are nested VMs created and after that "Live migration of nested VMs fails" How to make live migration work on nested VMs? Regards, Akkina ReplyForward

2 1

[live migration] [libvirt] [nova]
by Nagaraj Akkina 16 Mar '21

16 Mar '21

Hi Team, We are using live migration in our environment from quite some time now, recently there are nested VMs created and after that "Live migration of nested VMs fails" Is there any documentation on this?. How to make live migration work on nested VMs? Regards, Akkina

2 1

[all] Help on Improving Software Documentation Through Code Reviews
by Felipe Ebert 16 Mar '21

16 Mar '21

Dear members of OpenStack, We are an international team of researchers, from the University of Bari - Italy, Federal University of Pernambuco - Brazil, University of Adelaide - Australia, and Eindhoven University of Technology - the Netherlands. The goal of our research is to improve software documentation and maintainability of open-source projects by providing automatic support of source code comments based on code review discussions. We would like to learn identifying relevant information from code review discussions such that tooling can be developed to improve software documentation and maintainability of open-source projects. Therefore, we would like to talk to open-source developers and interview them. If you are interested in talking about your experiences as an open-source reviewer then please schedule a meeting with us using YouCanBook.me here: [ https://tue-interview.youcanbook.me] We are interested in different experiences, therefore, we are looking to talk to people of all age-groups, genders, and people with different roles in open-source projects. So even whether you are a first-time contributor or a senior maintainer don’t hesitate to contact us. The interview itself will take no longer than 30 min and will be scheduled using Microsoft Teams. We will record the interview and the recording and transcript of the interview will be stored securely, and only made available to the researchers that are involved in this project. Additionally, we plan to use anonymized results of the interview in a scientific publication. Kind regards and thanks in advance, -- Nicole Novielli, University of Bari, Fernando Castor, Federal University of Pernambuco, Christoph Treude, University of Adelaide, Alexander Serebrenik, Eindhoven University of Technology, Felipe Ebert, Eindhoven University of Technology

1 0

[victoria][neutron] OVN Gateway Chassis Issue
by Ammad Syed 16 Mar '21

16 Mar '21

Hi, I have four compute nodes in my lab setup. Initially all the four compute nodes were acting as gateway chassis with priority 1, 2, 3 and 4. Then I have specifically marked two node as a gateway chassis with below command on compute nodes. ovs-vsctl set open . external-ids:ovn-cms-options="enable-chassis-as-gw" The command ovn-nbctl list gateway_chassis start showing two chassis. I have checked via tcpdump, the public traffic started flowing from both nodes. Look like its doing round robin to send packets. Then I tried to remove one chassis from gateway and used below command. ovs-vsctl remove open . external-ids ovn-cms-options=enable-chassis-as-gw The ovn-nbctl list gateway_chassis started showing one gateway chassis but I can see from tcpdump that public traffic still flows from both gateway chassis. Below is the current status of chassis. root@network:/etc/neutron# ovn-sbctl list chassis _uuid : 532bb9d0-6667-462c-9631-0cb5360bd4dc encaps : [358c4a59-0bca-459c-958c-524eb8c385ce] external_ids : {datapath-type=system, iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", "neutron:liveness_check_at"="2021-03-16T08:52:41.361302+00:00", "neutron:metadata_liveness_check_at"="2021-03-16T08:52:41.364928+00:00", "neutron:ovn-metadata-id"="2ac66785-d0c7-43ee-8c78-5fd6ed6ccc73", "neutron:ovn-metadata-sb-cfg"="6157", ovn-bridge-mappings="ext-net1:br-ext", ovn-chassis-mac-mappings="", ovn-cms-options=""} hostname : virtual-hv2 name : "fdfae005-7473-486a-b331-8a54c53c1279" nb_cfg : 6157 transport_zones : [] vtep_logical_switches: [] _uuid : a99ab389-96a5-4a58-a301-34618868450a encaps : [6e7490ce-3c58-4a1c-999d-ff1638c66feb] external_ids : {datapath-type=system, iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", neutron-metadata-proxy-networks="dc917847-f70f-4de0-9865-3e9594c65ef1", "neutron:liveness_check_at"="2021-03-16T08:52:41.368768+00:00", "neutron:metadata_liveness_check_at"="2021-03-16T08:52:41.372045+00:00", "neutron:ovn-metadata-id"="3441fc3c-ca43-4360-8210-8c9ebe4fc13d", "neutron:ovn-metadata-sb-cfg"="6157", ovn-bridge-mappings="ext-net1:br-ext", ovn-chassis-mac-mappings="", ovn-cms-options=""} hostname : kvm10-a1-khi01 name : "87504098-4474-40fc-9576-ac449c1c4448" nb_cfg : 6157 transport_zones : [] vtep_logical_switches: [] _uuid : b9bdfe12-fe27-4580-baee-159f871c442b encaps : [52a8f523-9740-4333-a4a4-69bf5e27117c] external_ids : {datapath-type=system, iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", neutron-metadata-proxy-networks="dc917847-f70f-4de0-9865-3e9594c65ef1", "neutron:liveness_check_at"="2021-03-16T08:52:41.326719+00:00", "neutron:metadata_liveness_check_at"="2021-03-16T08:52:41.342214+00:00", "neutron:ovn-metadata-id"="2a751610-97a8-4688-a719-df3616f4f770", "neutron:ovn-metadata-sb-cfg"="6157", ovn-bridge-mappings="ext-net1:br-ext", ovn-chassis-mac-mappings="", ovn-cms-options=""} hostname : kvm12-a1-khi01 name : "82630e57-668e-4f67-a3fb-a173f4da432a" nb_cfg : 6157 transport_zones : [] vtep_logical_switches: [] _uuid : 669d1ae3-7a5d-4ec3-869d-ca6240f9ae2c encaps : [ac8022b3-1ea5-45c7-a7e8-74db7b627df4] external_ids : {datapath-type=system, iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", "neutron:liveness_check_at"="2021-03-16T08:52:41.347144+00:00", "neutron:metadata_liveness_check_at"="2021-03-16T08:52:41.352021+00:00", "neutron:ovn-metadata-id"="2d5ce6fd-6a9f-4356-9406-6ca91601af43", "neutron:ovn-metadata-sb-cfg"="6157", ovn-bridge-mappings="ext-net1:br-ext", ovn-chassis-mac-mappings="", ovn-cms-options=enable-chassis-as-gw} hostname : virtual-hv1 name : "731e842a-3a69-4044-87e9-32b7517d4f07" nb_cfg : 6157 transport_zones : [] vtep_logical_switches: [] Need help how can I permanently remove a gateway chassis that it should stop serving public traffic ? also is it something to do with priority ? - Ammad

2 2

[largescale-sig] Next meeting: March 10, 15utc
by Thierry Carrez 16 Mar '21

16 Mar '21

Hi everyone, Our next Large Scale SIG meeting will be this Wednesday in #openstack-meeting-3 on IRC, at 15UTC. You can doublecheck how it translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210310T15 Belmiro Moreira will chair this meeting. A number of topics have already been added to the agenda, including discussing CentOS Stream, reflecting on last video meeting and pick a topic for the next one. Feel free to add other topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

4 3

[venus][ptg] Xena PTG
by Liye Pang(逄立业) 15 Mar '21

15 Mar '21

Hi, Venus is the OpenStack project that provides a one-stop solution for log collection, indexing, analysis, alerting, visualization, report generation and other needs. Additionally, this project plans to use machine learning algorithms to locate failures and find the root causes quickly, to improve operation and maintenance efficiency. If you are planning to attend venus sessions during the Xena PTG, please fill in the doodle[1] with time slots which are good for you before 23rd March so that we can book the best time slots for most of us. And, please add any topic which you want to discuss on the etherpad[2]. [1] https://doodle.com/poll/q72urs2mk7up45pf [2] https://etherpad.opendev.org/p/venus-xena-ptg

1 0

Re: [cinder] wallaby feature freeze
by Rajesh Ratnakaram 15 Mar '21

15 Mar '21

Hi, I have added missing features to Zadara cinder driver and posted: https://review.opendev.org/c/openstack/cinder/+/774463 The corresponding bp can be found at: https://blueprints.launchpad.net/cinder/+spec/zadara-wallaby-features I wanted to push the above changes to Wallaby release, hence requesting to add FFE for the above request. Thanks and Regards, Rajesh. Sent from Mail for Windows 10

2 1

Re: [cinder] wallaby feature freeze
by Ajitha Robert 15 Mar '21

15 Mar '21

Hi team, This patch https://review.opendev.org/c/openstack/cinder/+/778886 requires a feature freeze exception. I am working on the the minor revisions and the CI will be running tomorrow. Thank you. -- *Regards,Ajitha R*

2 1

Keystone enforced policy issues - Victoria
by Luke Camilleri 15 Mar '21

15 Mar '21

Hi there, we have been troubleshooting keystone policies for a couple of days now (Victoria) and would like to reach out to anyone with some experience on the keystone policies. Basically we would like to follow the standard admin, member and reader roles together with the scopes function and have therefore enabled the below two option in keystone.conf enforce_scope = true enforce_new_defaults = true once enabled we started seeing the below error related to token validation in keystone.log: 2021-03-11 19:50:12.009 1047463 WARNING keystone.server.flask.application [req-33cda154-1d54-447e-8563-0676dc5d8471 020bd854741e4ba69c87d4142cad97a5 c584121f9d1a4334a3853c61bb5b9d93 - default default] You are not authorized to perform the requested action: identity:validate_token.: keystone.exception.ForbiddenAction: You are not authorized to perform the requested action: identity:validate_token. The policy was previously setup as: identity:validate_token: rule:service_admin_or_token_subject but has now been implemented with the new scope format to: identity:validate_token: (role:reader and system_scope:all) or rule:service_role or rule:token_subject If we change the policy to the old one, we stop receiving the identity:validate_token exception. This only happens in horizon and running the same commands in the CLI (python-openstack-client) does not output any errors. To work around this behavior we place the old policy for validate_token rule in /usr/share/openstack-dashboard/openstack_dashboard/conf/keystone_policy.yaml and in the file /etc/keystone/keystone.conf A second way how we can solve the validate_token exception is to disable the option which has been enabled above "enforce_new_defaults = true" which will obviously allow the deprecated policy rules to become effective and hence has the same behavior as implementing the old policy as we did We would like to know if anyone have had this behavior and how it has been solved maybe someone can point us in the right direction to identify better what is going on. Last but not least, it seems that from Horizon, the admin user is being assigned a project scoped token instead of a system scoped token, while from the CLI the same admin user can successfully issue a system:all token and run commands across all resources. We would be very happy to receive any form of input related to the above issues we are facing Thanks in advance for any assistance

1 2

[barbican][ironic][neutron] [QA][OpenstackSDK][swift]
by Armstrong Foundjem 15 Mar '21

15 Mar '21

Hello! Quick reminder that for deliverables following the cycle-with-intermediary model, the release team will use the latest Wallaby release available on release week. The following deliverables have done a Wallaby release, but it was not refreshed in the last two months: ansible-role-lunasa-hsm ironic-inspector ironic-prometheus-exporter ironic-ui ovn-octavia-provider patrole python-openstackclient swift You should consider making a new one very soon, so that we don't use an outdated version for the final release. - Armstrong Foundjem (armstrong)

4 3

[neutron][requirements] RFE requested for neutron-lib
by Slawek Kaplonski 15 Mar '21

15 Mar '21

Hi, I'm raising this RFE to ask if we can release, and include new version of the neutron-lib in the Wallaby release. Neutron already migrated all our policy rules to use new personas like system-reader or system-admin. Recently Lance found bug [1] with those new personas. Fix for that is now merged in neutron-lib [2]. This fix required bump the oslo_context dependency in lower-constraints and that lead us to bump many other dependencies versions. But in fact all those new versions are now aligned with what we already have in the Neutron's requirements so in fact neutron-lib was tested with those new versions of the packages every time it was run with Neutron master branch. I already proposed release patch for neutron-lib [3]. Please let me know if I You need anything else regrarding this RFE. [1] https://launchpad.net/bugs/1918506 [2] https://review.opendev.org/c/openstack/neutron-lib/+/780204 [3] https://review.opendev.org/c/openstack/releases/+/780550 -- Slawek Kaplonski Principal Software Engineer Red Hat

3 6

[community/MLs] Measuring ML Success
by Jimmy McArthur 15 Mar '21

15 Mar '21

Hi All - Over the last six months or so, we've had feedback from people that feel their questions die on the ML or that are missing ask.openstack.org. I don't think we should open up the ask.openstack.org can of worms, by any means. However, I wanted to find out if there was any software out there we could use to track metrics on which questions go unanswered on the ML. Everything I've found is very focused on email marketing, which is not what we're after. Would love to try to get some numbers on individuals that are trying to reach out to the ML, but just aren't getting through to anyone. Assuming we get that, I feel like it would be an easy next step to do a monthly or bi-monthly check to reach out to these potential new contributors. I realize our community is busy and people are, by and large, volunteering their time to answer these questions. But as hard as that is, it's also tough to pose new questions to a community you're unfamiliar with and then hear crickets. Open to other ideas/thoughts :) Cheers! Jimmy

5 6

[PTLs][release] Wallaby Cycle Highlights
by Kendall Nelson 15 Mar '21

15 Mar '21

Hello Everyone! It's time to start thinking about calling out 'cycle-highlights' in your deliverables! I have no idea how we are here AGAIN ALREADY, alas, here we be. As PTLs, you probably get many pings towards the end of every release cycle by various parties (marketing, management, journalists, etc) asking for highlights of what is new and what significant changes are coming in the new release. By putting them all in the same place it makes them easy to reference because they get compiled into a pretty website like this from the last few releases: Stein[1], Train[2]. We don't need a fully fledged marketing message, just a few highlights (3-4 ideally), from each project team. Looking through your release notes might be a good place to start. *The deadline for cycle highlights is the end of the R-5 week [3] (next week) on March 12th.* How To Reminder: ------------------------- Simply add them to the deliverables/$RELEASE/$PROJECT.yaml in the openstack/releases repo like this: cycle-highlights: - Introduced new service to use unused host to mine bitcoin. The formatting options for this tag are the same as what you are probably used to with Reno release notes. Also, you can check on the formatting of the output by either running locally: tox -e docs And then checking the resulting doc/build/html/$RELEASE/highlights.html file or the output of the build-openstack-sphinx-docs job under html/$RELEASE/highlights.html. Feel free to add me as a reviewer on your patches. Can't wait to see you all have accomplished this release! Thanks :) -Kendall Nelson (diablo_rojo) [1] https://releases.openstack.org/stein/highlights.html [2] https://releases.openstack.org/train/highlights.html [3] htt <https://releases.openstack.org/train/schedule.html#t-cycle-highlights> https://releases.openstack.org/wallaby/schedule.html <https://releases.openstack.org/ussuri/schedule.html>

2 6

[queens][nova] live migration error
by Ignazio Cassano 15 Mar '21

15 Mar '21

Hello all, we are facing a problem while migrating some virtual machines on centos 7 queens. If we create a new virtual machine it migrates. Old virtual machines got the following: 37eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Increasing downtime to 50 ms after 0 sec elapsed time 2021-03-15 17:26:44.428 82382 INFO nova.virt.libvirt.driver [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Migration running for 0 secs, memory 100% remaining; (bytes processed=0, remaining=0, total=0) 2021-03-15 17:26:57.922 82382 INFO nova.compute.manager [req-83fd3d62-292e-4990-8f7a-47f404b557cc - - - - -] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] VM Paused (Lifecycle Event) 2021-03-15 17:26:58.617 82382 INFO nova.virt.libvirt.driver [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Migration operation has completed 2021-03-15 17:26:58.618 82382 INFO nova.compute.manager [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] _post_live_migration() is started.. 2021-03-15 17:26:58.658 82382 ERROR nova.virt.libvirt.driver [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Live Migration failure: operation failed: domain is not running: libvirtError: operation failed: domain is not running 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Post live migration at destination podto2-kvmae failed: InstanceNotFound_Remote: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found. InstanceNotFound: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found. 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Traceback (most recent call last): 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6505, in _post_live_migration 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] instance, block_migration, dest) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 783, in post_live_migration_at_destination 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] instance=instance, block_migration=block_migration) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] retry=self.retry) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] timeout=timeout, retry=retry) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] retry=retry) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 550, in _send 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] raise result 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] InstanceNotFound_Remote: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found. 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Traceback (most recent call last): 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] res = self.dispatcher.dispatch(message) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] return self._do_dispatch(endpoint, method, ctxt, args) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] result = func(ctxt, **new_args) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] function_name, call_dict, binary) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] self.force_reraise() 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] six.reraise(self.type_, self.value, self.tb) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] return f(self, context, *args, **kw) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1000, in decorated_function 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] return function(self, context, *args, **kwargs) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 203, in decorated_function 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] return function(self, context, *args, **kwargs) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6658, in post_live_migration_at_destination 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 'destination host.', instance=instance) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] self.force_reraise() 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] six.reraise(self.type_, self.value, self.tb) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6653, in post_live_migration_at_destination 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] block_device_info) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7844, in post_live_migration_at_destination 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] self._host.get_guest(instance) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 526, in get_guest 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] return libvirt_guest.Guest(self._get_domain(instance)) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 546, in _get_domain 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] raise exception.InstanceNotFound(instance_id=instance.uuid) 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] InstanceNotFound: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found. 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:01.259 82382 ERROR nova.compute.manager [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] 2021-03-15 17:27:05.179 82382 WARNING nova.compute.resource_tracker [req-2189a13d-097a-4eb0-9f9b-60767b9b0c68 66adb965bef64eaaab2af93ade87e2ca 85cace94dcc7484c85ff9337eb1d0c4c - default default] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] Instance not resizing, skipping migration.: InstanceNotFound_Remote: Instance 25fbb55c-5991-49b2-885f-26adebaeb572 could not be found. 2021-03-15 17:27:13.621 82382 INFO nova.compute.manager [-] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] VM Stopped (Lifecycle Event) 2021-03-15 17:27:13.711 82382 INFO nova.compute.manager [req-0dcd030c-d520-4223-91a6-f2fbc9f2444a - - - - -] [instance: 25fbb55c-5991-49b2-885f-26adebaeb572] During the sync_power process the instance has moved from host podto2-kvmae to host podto2-kvm01 The istances was on podto1-kvm01 and we try to migrate it on podto2-kvmae. It stops to migrate and goes in error state and stop to responding to ping requests. >From logs is seems it returned on podto2-kvm01 but it is in error state on podto2-kvmae. After hard rebooting it starts on podto2-kvmae and then we can migrate it where we want without errors. Please, any help ? Thanks Ignazio

1 1