March 2021 - openstack-discuss - lists.openstack.org

RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller
by Braden, Albert 31 Mar '21

31 Mar '21

Centos7: {rabbit,"RabbitMQ","3.7.24"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, Centos8: {rabbit,"RabbitMQ","3.7.28"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, When I deploy the first Centos8 controller, RMQ comes up with all 3 nodes active and seems to be working fine until I shut down the 2nd controller. The only hint of trouble when I replace the 1st node is this error message the first time I run the deployment: https://paste.ubuntu.com/p/h9HWdfwmrK/ and the crash dump that appears on control2: crash dump log: https://paste.ubuntu.com/p/MpZ8SwTJ2T/ First 1500 lines of the dump: https://paste.ubuntu.com/p/xkCyp2B8j8/ If I wait for a few minutes then RMQ recovers on control2 and the 2nd run of the deployment seems to work, and there is no trouble until I shut down control1. -----Original Message----- From: Mark Goddard <mark(a)stackhpc.com> Sent: Wednesday, March 31, 2021 4:14 AM To: Braden, Albert <C-Albert.Braden(a)charter.com> Cc: openstack-discuss(a)lists.openstack.org Subject: [EXTERNAL] Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. On Tue, 30 Mar 2021 at 13:41, Braden, Albert <C-Albert.Braden(a)charter.com> wrote: > > I’ve created a heat stack and installed Openstack Train to test the Centos7->8 upgrade following the document here: > > > > https://docs.openstack.org/kolla-ansible/train/user/centos8.html#migrating-… > > > > I used the instructions here to successfully remove and replace control0 with a Centos8 box > > > > https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hos… > > > > After this my RMQ admin page shows all 3 nodes up, including the new control0. The name of the cluster is rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-2 /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-2 ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-0-replace',[]}, > > {'rabbit@chrnc-void-testupgrade-control-1',[]}, > > {'rabbit@chrnc-void-testupgrade-control-2',[]}]}] > > > > After that I create a new VM to verify that the cluster is still working, and then perform the same procedure on control1. When I shut down services on control1, the ansible playbook finishes successfully: > > > > kolla-ansible -i ../multinode stop --yes-i-really-really-mean-it --limit control1 > > … > > control1 : ok=45 changed=22 unreachable=0 failed=0 skipped=105 rescued=0 ignored=0 > > > > After this my RMQ admin page stops responding. When I check RMQ on the new control0 and the existing control2, the container is still up but RMQ is not running: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'. > > > > If I start it on control0 and control2, then the cluster seems normal and the admin page starts working again, and cluster status looks normal: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-0-replace ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-2', > > 'rabbit@chrnc-void-testupgrade-control-0-replace']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-2',[]}, > > {'rabbit@chrnc-void-testupgrade-control-0-replace',[]}]}] > > > > But my hypervisors are down: > > > > (openstack) [root@chrnc-void-testupgrade-build kolla-ansible]# ohll > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | 3 | chrnc-void-testupgrade-compute-2.dev.chtrse.com | QEMU | 172.16.2.106 | down | 5 | 8 | 2560 | 30719 | > > | 6 | chrnc-void-testupgrade-compute-0.dev.chtrse.com | QEMU | 172.16.2.31 | down | 5 | 8 | 2560 | 30719 | > > | 9 | chrnc-void-testupgrade-compute-1.dev.chtrse.com | QEMU | 172.16.0.30 | down | 5 | 8 | 2560 | 30719 | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > > > When I look at the nova-compute.log on a compute node, I see RMQ failures every 10 seconds: > > > > 172.16.2.31 compute0 > > 2021-03-30 03:07:54.893 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > 2021-03-30 03:07:55.905 7 INFO oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] Reconnected to AMQP server on 172.16.1.132:5672 via [amqp] client with port 56422. > > 2021-03-30 03:08:05.915 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > > > In the RMQ logs I see this every 10 seconds: > > > > 172.16.1.132 control2 > > [root@chrnc-void-testupgrade-control-2 ~]# tail -f /var/log/kolla/rabbitmq/rabbit\(a)chrnc-void-testupgrade-control-2.log |grep 172.16.2.31 > > 2021-03-30 03:07:54.895 [warning] <0.13247.35> closing AMQP connection <0.13247.35> (172.16.2.31:56420 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > client unexpectedly closed TCP connection > > 2021-03-30 03:07:55.901 [info] <0.15288.35> accepting AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) > > 2021-03-30 03:07:55.903 [info] <0.15288.35> Connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) has a client-provided name: nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e > > 2021-03-30 03:07:55.904 [info] <0.15288.35> connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e): user 'openstack' authenticated and granted access to vhost '/' > > 2021-03-30 03:08:05.916 [warning] <0.15288.35> closing AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > > > Why does RMQ fail when I shut down the 2nd controller after successfully replacing the first one? Hi Albert, Could you share the versions of RabbitMQ and erlang in both versions of the container? When initially testing this setup, I think we had 3.7.24 on both sides. Perhaps the CentOS 8 version has moved on sufficiently to become incompatible? Mark > > > > I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails. > > > > The contents of this e-mail message and > any attachments are intended solely for the > addressee(s) and may contain confidential > and/or legally privileged information. If you > are not the intended recipient of this message > or if this message has been addressed to you > in error, please immediately alert the sender > by reply e-mail and then delete this message > and any attachments. If you are not the > intended recipient, you are notified that > any use, dissemination, distribution, copying, > or storage of this message or any attachment > is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

1 0

[qa][tempest] Announcing scenario.manager stable interface
by Martin Kopec 31 Mar '21

31 Mar '21

Hi all, we would like to announce that tempest.scenario.manager has been declared stable in Tempest 27.0.0 and it's ready to be consumed by tempest plugins. All related patches were tracked under 'tempest-scenario-manager-stable' blueprint [1] and the whole effort was tracked in [2]. A little history: Some time ago, tempest/scenario/manager.py got copied to most of the plugins and therefore, it diverged - every plugin's copy had slight differences. In the latest release, we pushed changes to unify the manager's methods and improved their APIs in order to have them easier to consume. I would also like to thank Soniya Vyas for an exceptional job moving this effort forward. [1] https://review.opendev.org/q/topic:bp/tempest-scenario-manager-stable [2] https://etherpad.opendev.org/p/tempest-scenario-manager Regards, -- Martin Kopec

5 4

[Swift] swift client querying capabilites fails and may affect Horizon's container management
by Stefan Kelber 31 Mar '21

31 Mar '21

Hello, i have a Kolla-Ansible AIO stack i added SWIFT to after the initial installation. CLI management of containers work on CLI, but Horizon is troubled: Clicking "container" throws "Error: Unable to fetch the policy details", and no horizon based container management is possible therefore. In an effort to find the root problem, i stumbled upon the following, which i assume to be related: it looks to me like there are issues with keystone authentication of SWIFT client (3.10.1) when attempting to fetch capabilities on OS Victoria Kolla-Ansible, but *only* when querying capabilities: <EXAMPLES> ## 1 (swift-container-server)[root@testhost ~]# swift --os-auth-url http://10.10.10.82:35357/v3 --auth-version 3 --os-project-name admin --os-project-domain-name Default --os-username admin --os-user-domain-name Default --os-password T-STRING-s6 stat Account: AUTH_4334f282aa3a498aac95d2cf9aa9fa91 Containers: 1 Objects: 2 Bytes: 0 Containers in policy "policy-0": 1 Objects in policy "policy-0": 2 Bytes in policy "policy-0": 0 Content-Type: text/plain; charset=utf-8 X-Timestamp: 1616964437.15820 Accept-Ranges: bytes X-Account-Project-Domain-Id: default Vary: Accept X-Trans-Id: tx2638af9bd98646ce92dfe-00606342a1 X-Openstack-Request-Id: tx2638af9bd98646ce92dfe-00606342a1 Transfer-Encoding: chunked ## 2 (swift-container-server)[root@testhost ~]# swift --os-auth-url http://10.10.10.82:35357/v3 --auth-version 3 --os-project-name admin --os-project-domain-name Default --os-username admin --os-user-domain-name Default --os-password T-STRING-s6 list mycontainer ## 3 (swift-container-server)[root@testhost ~]# swift --os-auth-url http://10.10.10.82:35357/v3 --auth-version 3 --os-project-name admin --os-project-domain-name Default --os-username admin --os-user-domain-name Default --os-password T-STRING-s6 capabilities Capabilities GET failed: http://10.10.10.82:8080/info 401 Unauthorized [first 60 chars of response] b'{"error": {"code": 401, "title": "Unauthorized", "message": ' Failed Transaction ID: txeb57fccc09e54e3fa2fa5-00606342af </EXAMPLES> Am i on the wrong track? Best Stefan Here is the entire debug output of the failing transaction: (swift-container-server)[root@testhost ~]# swift --os-auth-url http://10.10.10.82:35357/v3 --auth-version 3 --os-project-name admin --os-project-domain-name Default --os-username admin --os-user-domain-name Default --os-password T-STRING-s6 capabilities --debug DEBUG:keystoneclient.auth.identity.v3.base:Making authentication request to http://10.10.10.82:35357/v3/auth/tokens DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 10.10.10.82:35357 DEBUG:urllib3.connectionpool:http://10.10.10.82:35357 "POST /v3/auth/tokens HTTP/1.1" 201 7299 DEBUG:keystoneclient.auth.identity.v3.base:{"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "12940206c0a1464f96a47fe74d19c893", "name": "admin", "password_expires_at": null}, "audit_ids": ["WjAnZyn5TaGufxZGqnvW3Q"], "expires_at": "2021-03-31T14:22:13.000000Z", "issued_at": "2021-03-30T14:22:13.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "4334f282aa3a498aac95d2cf9aa9fa91", "name": "admin"}, "is_domain": false, "roles": [{"id": "b740e2667f034d02b6af2114ec06e20a", "name": "admin"}, {"id": "ceb68a7214c74a969cd83b9146e1fbda", "name": "heat_stack_owner"}, {"id": "6b2ae10793c34fb287dd994a8dd2e8a1", "name": "member"}, {"id": "edb76a75b2fc4af79bf09e38e07be5f1", "name": "reader"}], "catalog": [{"endpoints": [{"id": "05e0b6b67f094b64b1caa9cf78dd34b2", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8780", "region": "RegionOne"}, {"id": "9575e60e3a034ff282bbeda2287b3d01", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8780", "region": "RegionOne"}, {"id": "f5fb2fffde5d4b88b230397560ac6952", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8780", "region": "RegionOne"}], "id": "16edbecc93484568a2f5b46db9909293", "type": "placement", "name": "placement"}, {"endpoints": [{"id": "09275022c3de4405adb6bb25071a49ba", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:9292", "region": "RegionOne"}, {"id": "3a6fa026c7c9422f8cb013492f5ba0e0", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:9292", "region": "RegionOne"}, {"id": "a4fa0d097eed4c3b97ff4ed0f27096d6", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:9292", "region": "RegionOne"}], "id": "2a71f9f773734edb8ea6b711c844b334", "type": "image", "name": "glance"}, {"endpoints": [{"id": "a20fbce4737540ce91039175c7409811", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8774/v2.1", "region": "RegionOne"}, {"id": "b16bad7fdf1342329a59e231cf56851d", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8774/v2.1", "region": "RegionOne"}, {"id": "ce27e7239706458c80e42a68a13383b2", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8774/v2.1", "region": "RegionOne"}], "id": "2c56db642dde46a2ab2f766dbd0ba9c9", "type": "compute", "name": "nova"}, {"endpoints": [{"id": "8c183a6cd8244b2cb1630a662c3a292e", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8004/v1/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "958487dfae194f69b5f56f0b6e901163", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8004/v1/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "a8e0db3b2e1c49a99ed97ec18a64ece6", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8004/v1/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}], "id": "5428e2837ccf413886140c0603867ee3", "type": "orchestration", "name": "heat"}, {"endpoints": [{"id": "2e9a6a2fc78340b7bdc18041dec79a86", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8774/v2/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "894ab56c6c7140ebaac8e7ec502e9b03", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8774/v2/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "8e86543bfeb547d78cf707100d0ef7f0", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8774/v2/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}], "id": "5639441ae48441ed9c8163fc537f0c1a", "type": "compute_legacy", "name": "nova_legacy"}, {"endpoints": [{"id": "32092c9ac4634ece9d89edc2b146af54", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:9696", "region": "RegionOne"}, {"id": "dad19d0e0d8e420caa5b1276d82917e4", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:9696", "region": "RegionOne"}, {"id": "f085e59dae874005b8e6fb1df55db5d5", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:9696", "region": "RegionOne"}], "id": "56e63ef1e3b24fbca6a2efb7394eb77b", "type": "network", "name": "neutron"}, {"endpoints": [{"id": "0ccb4189837f482aac1ee67a0f5b69b7", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8000/v1", "region": "RegionOne"}, {"id": "be4fab7e4ae04786817702817ea09c34", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8000/v1", "region": "RegionOne"}, {"id": "eb8ccdc41ab94aaa9567e29e728e9109", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8000/v1", "region": "RegionOne"}], "id": "83cee8b745d74b1d9a3bc1e37d3e21bf", "type": "cloudformation", "name": "heat-cfn"}, {"endpoints": [{"id": "917c36b07fae4b278a2f63d3886933c3", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:5000", "region": "RegionOne"}, {"id": "9d40ecfea4a04327b64e5751ed8043ce", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:5000", "region": "RegionOne"}, {"id": "d4941b09a2cb40648f3b019bb1653120", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:35357", "region": "RegionOne"}], "id": "94dce13398fd42e4805ae1a7d9976e78", "type": "identity", "name": "keystone"}, {"endpoints": [{"id": "10297840af2b4370a2f63908aea3ff9c", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8776/v2/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "a1a617f8996d439b8de091a4f15a88ae", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8776/v2/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "a930062cb2da497bbeb727c9fec6bba0", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8776/v2/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}], "id": "d7f1cd74b2294126ac939b7d705369a3", "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"id": "1bb18181af9b4dd28bbf16491adb16fe", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8080/v1/AUTH_4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "686b82631d4d40b1b62ed1a7cb8fb1cf", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8080/v1", "region": "RegionOne"}, {"id": "e0cc80d07c6246668ca9cce9d736af62", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8080/v1/AUTH_4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}], "id": "eabcc4d34b3c4ce499025816e2302f34", "type": "object-store", "name": "swift"}, {"endpoints": [{"id": "4c2b1dbe8d3544c9add475d63b0492d4", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8776/v3/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "77634a04b3644d0eb2abab41f3325a1c", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8776/v3/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "7f11d1c28644413e8cfe5ff19b91ebd2", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8776/v3/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}], "id": "fda50e1a62564367bca385a64088334d", "type": "volumev3", "name": "cinderv3"}]}} DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 10.10.10.82:8080 DEBUG:urllib3.connectionpool:http://10.10.10.82:8080 "GET /info HTTP/1.1" 401 114 INFO:swiftclient:REQ: curl -i http://10.10.10.82:8080/info -X GET -H "Accept-Encoding: gzip" INFO:swiftclient:RESP STATUS: 401 Unauthorized INFO:swiftclient:RESP HEADERS: {'Content-Type': 'application/json', 'Content-Length': '114', 'WWW-Authenticate': 'Keystone uri="http://10.10.10.82:5000"', 'X-Trans-Id': 'txac041bfff30342729c2dd-0060633415', 'X-Openstack-Request-Id': 'txac041bfff30342729c2dd-0060633415', 'Date': 'Tue, 30 Mar 2021 14:22:13 GMT'} INFO:swiftclient:RESP BODY: b'{"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}' Capabilities GET failed: http://10.10.10.82:8080/info 401 Unauthorized [first 60 chars of response] b'{"error": {"code": 401, "title": "Unauthorized", "message": ' Failed Transaction ID: txac041bfff30342729c2dd-0060633415

1 1

[kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller
by Braden, Albert 31 Mar '21

31 Mar '21

I've created a heat stack and installed Openstack Train to test the Centos7->8 upgrade following the document here: https://docs.openstack.org/kolla-ansible/train/user/centos8.html#migrating-… I used the instructions here to successfully remove and replace control0 with a Centos8 box https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hos… After this my RMQ admin page shows all 3 nodes up, including the new control0. The name of the cluster is rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com<mailto:rabbit@chrnc-void-testupgrade-control-0.dev.chtrse.com> (rabbitmq)[root@chrnc-void-testupgrade-control-2 /]# rabbitmqctl cluster_status Cluster status of node rabbit@chrnc-void-testupgrade-control-2 ... [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', 'rabbit@chrnc-void-testupgrade-control-0-replace', 'rabbit@chrnc-void-testupgrade-control-1', 'rabbit@chrnc-void-testupgrade-control-2']}]}, {running_nodes,['rabbit@chrnc-void-testupgrade-control-0-replace', 'rabbit@chrnc-void-testupgrade-control-1', 'rabbit@chrnc-void-testupgrade-control-2']}, {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, {partitions,[]}, {alarms,[{'rabbit@chrnc-void-testupgrade-control-0-replace',[]}, {'rabbit@chrnc-void-testupgrade-control-1',[]}, {'rabbit@chrnc-void-testupgrade-control-2',[]}]}] After that I create a new VM to verify that the cluster is still working, and then perform the same procedure on control1. When I shut down services on control1, the ansible playbook finishes successfully: kolla-ansible -i ../multinode stop --yes-i-really-really-mean-it --limit control1 ... control1 : ok=45 changed=22 unreachable=0 failed=0 skipped=105 rescued=0 ignored=0 After this my RMQ admin page stops responding. When I check RMQ on the new control0 and the existing control2, the container is still up but RMQ is not running: (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'. If I start it on control0 and control2, then the cluster seems normal and the admin page starts working again, and cluster status looks normal: (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status Cluster status of node rabbit@chrnc-void-testupgrade-control-0-replace ... [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', 'rabbit@chrnc-void-testupgrade-control-0-replace', 'rabbit@chrnc-void-testupgrade-control-1', 'rabbit@chrnc-void-testupgrade-control-2']}]}, {running_nodes,['rabbit@chrnc-void-testupgrade-control-2', 'rabbit@chrnc-void-testupgrade-control-0-replace']}, {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, {partitions,[]}, {alarms,[{'rabbit@chrnc-void-testupgrade-control-2',[]}, {'rabbit@chrnc-void-testupgrade-control-0-replace',[]}]}] But my hypervisors are down: (openstack) [root@chrnc-void-testupgrade-build kolla-ansible]# ohll +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ | 3 | chrnc-void-testupgrade-compute-2.dev.chtrse.com | QEMU | 172.16.2.106 | down | 5 | 8 | 2560 | 30719 | | 6 | chrnc-void-testupgrade-compute-0.dev.chtrse.com | QEMU | 172.16.2.31 | down | 5 | 8 | 2560 | 30719 | | 9 | chrnc-void-testupgrade-compute-1.dev.chtrse.com | QEMU | 172.16.0.30 | down | 5 | 8 | 2560 | 30719 | +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ When I look at the nova-compute.log on a compute node, I see RMQ failures every 10 seconds: 172.16.2.31 compute0 2021-03-30 03:07:54.893 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out 2021-03-30 03:07:55.905 7 INFO oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] Reconnected to AMQP server on 172.16.1.132:5672 via [amqp] client with port 56422. 2021-03-30 03:08:05.915 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out In the RMQ logs I see this every 10 seconds: 172.16.1.132 control2 [root@chrnc-void-testupgrade-control-2 ~]# tail -f /var/log/kolla/rabbitmq/rabbit\(a)chrnc-void-testupgrade-control-2.log |grep 172.16.2.31 2021-03-30 03:07:54.895 [warning] <0.13247.35> closing AMQP connection <0.13247.35> (172.16.2.31:56420 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): client unexpectedly closed TCP connection 2021-03-30 03:07:55.901 [info] <0.15288.35> accepting AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) 2021-03-30 03:07:55.903 [info] <0.15288.35> Connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) has a client-provided name: nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e 2021-03-30 03:07:55.904 [info] <0.15288.35> connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e): user 'openstack' authenticated and granted access to vhost '/' 2021-03-30 03:08:05.916 [warning] <0.15288.35> closing AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): Why does RMQ fail when I shut down the 2nd controller after successfully replacing the first one? I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

2 1

Networking-midonet deprecated
by Slawek Kaplonski 30 Mar '21

30 Mar '21

Hi, As it was discussed and decided in the other thread [1] due to lack of maintainers and not good shape of the project, networking-midonet was deprecated and is not part of the official Neutron stadium. Almost all required patches from [2] are already merged. Also patch [3] is merged already so there will be no official Wallaby release of the networking-midonet project. Last stable, supported release is Victroria. Project can be revived in the x/ namespace if there will be someone who wants to maintain it. [1] http://lists.openstack.org/pipermail/openstack-discuss/2021-March/021400.ht… [2] https://review.opendev.org/q/topic:%22deprecate-networking-midonet%22+(stat… [3] https://review.opendev.org/c/openstack/releases/+/781713 -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

How to customize the xml used in libvirt from GUI/opestack command line?
by Evan Zhao 30 Mar '21

30 Mar '21

Hi there, I googled this question and found two major answers: 1. ssh to the compute node and use `virsh dumpxml` and `virsh undefine/define`, etc. 2. edit nova/virt/libvrit/config.py directly. However, it's trivial to ssh to each node and do the modification, and I prefer not to touch the nova source code, is there any better ways to achieve this? I expect to edit the namespace of a certain element and append an additional element <qemu:command> to the xml file. Any information will be appreciated.

1 0

[security sig] April meeting cancelled
by Gage Hugo 30 Mar '21

30 Mar '21

Hey team, The security sig meeting for April 1st is cancelled, we will meet again next month at the usual time. Thanks

1 0

[security sig] PTG Timeslot
by Gage Hugo 30 Mar '21

30 Mar '21

Hello, The security SIG has a spot at the PTG from 1500 to 1700 UTC on April 19th. The agenda[0] is up for anyone to add if they have something to discuss. Hope to see you all there. [0] https://etherpad.opendev.org/p/security-sig-ptg-xena

1 0

[oslo] fix flake8-hacking inconsistences on xena/wallaby
by Herve Beraud 30 Mar '21

30 Mar '21

Hello Osloers, As you surely recently observed patches proposed by the openstack bot to configure wallaby fail with a flake8/hacking issue. Here are some guideline to fix the problem the inconsistency: - Patch pre-commit on master (now xena) (here is the patch: https://gist.github.com/4383/6e96c836bb1b0e1e0e599a5106f43f1a) - Once Xena is patched cherry-pick and backport these changes on Wallaby - Rebase the openstack bot patches on the top of this cherry-pick (or wait for the merge of the previous one patch) You can copy the patch on oslo.db with the related commit message [1]. The root cause of the issue was that with the introduction of pre-commit we started to define the version of flake8 to use. Previously this version was defined by hacking's requirements. Indeed a few months ago we added pre-commit to allow us to run checks with git hooks and reduce the usage of our gates. These changes were standardized and spread on all the scope of oslo [2]. However, during the design of these changes [3] and after some discussion we decided to pin the version of flake8 to use, hence by doing this we short circuited hacking on its management of flake8. The solution to solve this issue is simply to trust hacking on its flake8 management. Hacking will pull the right version of flake8 and the inconsistency will disappear. flake8 provides a pre-commit hook so it could be seen and called as a local target. [1] https://review.opendev.org/c/openstack/oslo.db/+/781470 [2] https://review.opendev.org/q/topic:%22oslo-pre-commit%22 [3] https://review.opendev.org/q/topic:%22pre-commit%22+(status:open%20OR%20sta… -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

4 7

Re: [glance][openstack-ansible] Snapshots disappear during saving
by Oliver Wenz 30 Mar '21

30 Mar '21

> Oh, I have a guess what this might actually be. During snapshot upload process > user token that is used for the upload might get expired. If that's the case, > following changes in user_variables might help to resolve the issue: > > glance_glance_api_conf_overrides: > keystone_authtoken: > service_token_roles_required: True > service_token_roles: service I found out that after inserting the above the problems now occurs every time I try to take a snapshot of an instance. Below are my logs: For swift: Mar 29 12:50:37 infra1-swift-proxy-container-27169fa7 proxy-server[2049]: Could not autocreate account '/AUTH_024cc551782f41e395d3c9f13582ef7d' (txn: tx340e7873c3 ce42ab9808e-006061cd0f) Mar 29 12:50:37 infra1-swift-proxy-container-27169fa7 proxy-server[2049]: 192.168.110.106 192.168.110.211 29/Mar/2021/12/50/37 PUT /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images HTTP/1.0 50 3 - python-swiftclient-3.10.1 gAAAAABgYczvODdX... - 118 - tx340e7873c3ce42ab9808e-006061cd0f - 14.3464 - - 1617022223.371025085 1617022237.717381477 - Mar 29 12:50:40 infra1-swift-proxy-container-27169fa7 proxy-server[2053]: ERROR with Account server 10.0.3.212:6002/os-objects re: Trying to HEAD /v1/AUTH_024cc551782f41e395d 3c9f13582ef7d: Host unreachable (txn: tx44e98ea667f04751bf22e-006061cd1f) Mar 29 12:50:40 infra1-swift-proxy-container-27169fa7 proxy-server[2053]: Account HEAD returning 503 for [] (txn: tx44e98ea667f04751bf22e-006061cd1f) Mar 29 12:50:40 infra1-swift-proxy-container-27169fa7 proxy-server[2053]: - - 29/Mar/2021/12/50/40 HEAD /v1/AUTH_024cc551782f41e395d3c9f13582ef7d%3Fformat%3Djson HTTP/1.0 503 - Swift - - - - tx44e98 ea667f04751bf22e-006061cd1f - 1.0482 RL - 1617022239.738376379 1617022240.786619186 - and for glance: Mar 29 12:50:56 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:50:56.155 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP STATUS: 503 Service Unavailable Mar 29 12:50:56 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:50:56.156 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP HEADERS: {'content-type': 'text/html; charset=UTF-8', 'content-length': '118', 'x-trans-id': 'tx44e98ea667f04751bf22e-006061cd1f', 'x-openstack-request-id': 'tx44e98ea667f04751bf22e-006061cd1f', 'date': 'Mon, 29 Mar 2021 12:50:56 GMT', 'connection': 'close'} Mar 29 12:50:56 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:50:56.157 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP BODY: b'<html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html>' Mar 29 12:51:15 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:51:15.544 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] REQ: curl -i http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… -X PUT -H "X-Auth-Token: gAAAAABgYczvODdX..." -H "Content-Length: 0" Mar 29 12:51:15 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:51:15.545 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP STATUS: 503 Service Unavailable Mar 29 12:51:15 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:51:15.545 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP HEADERS: {'content-type': 'text/html; charset=UTF-8', 'content-length': '118', 'x-trans-id': 'tx01d5fb0cb5cb4f6dbff12-006061cd34', 'x-openstack-request-id': 'tx01d5fb0cb5cb4f6dbff12-006061cd34', 'date': 'Mon, 29 Mar 2021 12:51:15 GMT', 'connection': 'close'} Mar 29 12:51:15 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:51:15.545 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP BODY: b'<html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html>' Mar 29 12:51:42 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:51:42.007 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] REQ: curl -i http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… -X PUT -H "X-Auth-Token: gAAAAABgYczvODdX..." -H "Content-Length: 0" Mar 29 12:51:42 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:51:42.008 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP STATUS: 503 Service Unavailable Mar 29 12:51:42 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:51:42.009 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP HEADERS: {'content-type': 'text/html; charset=UTF-8', 'content-length': '118', 'x-trans-id': 'tx9cab391639a344f4aa548-006061cd4b', 'x-openstack-request-id': 'tx9cab391639a344f4aa548-006061cd4b', 'date': 'Mon, 29 Mar 2021 12:51:42 GMT', 'connection': 'close'} Mar 29 12:51:42 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:51:42.009 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP BODY: b'<html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html>' Mar 29 12:52:16 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:52:16.472 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] REQ: curl -i http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… -X PUT -H "X-Auth-Token: gAAAAABgYczvODdX..." -H "Content-Length: 0" Mar 29 12:52:16 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:52:16.473 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP STATUS: 503 Service Unavailable Mar 29 12:52:16 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:52:16.473 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP HEADERS: {'content-type': 'text/html; charset=UTF-8', 'content-length': '118', 'x-trans-id': 'tx926ae448b316485d93a68-006061cd6e', 'x-openstack-request-id': 'tx926ae448b316485d93a68-006061cd6e', 'date': 'Mon, 29 Mar 2021 12:52:16 GMT', 'connection': 'close'} Mar 29 12:52:16 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:52:16.474 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP BODY: b'<html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html>' Mar 29 12:52:16 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:52:16.476 2403 ERROR glance.api.v2.image_data [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to upload image data due to internal error: glance_store.exceptions.BackendException: Failed to add container to Swift. Mar 29 12:52:16 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Caught error: Failed to add container to Swift. Got error from Swift: Container PUT failed: http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… 503 Service Unavailable [first 60 chars of response] b'<html><h1>Service Unavailable</h1><p>The server is currently'.: glance_store.exceptions.BackendException: Failed to add container to Swift. Got error from Swift: Container PUT failed: http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… 503 Service Unavailable [first 60 chars of response] b'<html><h1>Service Unavailable</h1><p>The server is currently'. 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1154, in _create_container_if_missing 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi connection.head_container(container) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1870, in head_container 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi return self._retry(None, head_container, container, headers=headers) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1801, in _retry 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi rv = func(self.url, self.token, *args, 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1097, in head_container 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi raise ClientException.from_response( 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi swiftclient.exceptions.ClientException: Container HEAD failed: http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… 404 Not Found 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi During handling of the above exception, another exception occurred: 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1162, in _create_container_if_missing 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi connection.put_container(container) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1890, in put_container 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi return self._retry(None, put_container, container, headers=headers, 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1801, in _retry 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi rv = func(self.url, self.token, *args, 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1144, in put_container 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi raise ClientException.from_response(resp, 'Container PUT failed', body) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi swiftclient.exceptions.ClientException: Container PUT failed: http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… 503 Service Unavailable [first 60 chars of response] b'<html><h1>Service Unavailable</h1><p>The server is currently' 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi During handling of the above exception, another exception occurred: 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1347, in __call__ 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action, 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1391, in dispatch 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi return method(*args, **kwargs) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py", line 416, in wrapped 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 298, in upload 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi self._restore(image_repo, image) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi self.force_reraise() 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi raise value 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 163, in upload 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py", line 208, in set_data 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 501, in set_data 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi self.force_reraise() 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi raise value 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 447, in set_data 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend, 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py", line 198, in set_data 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py", line 318, in set_data 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend, 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 567, in set_data 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 458, in _upload_to_store 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash( 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 398, in add_with_multihash 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi return store_add_to_backend_with_multihash( 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 480, in store_add_to_backend_with_multihash 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add( 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py", line 279, in add_adapter 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py", line 176, in op_checker 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 942, in add 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi self._create_container_if_missing(location.container, 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1167, in _create_container_if_missing 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi raise glance_store.BackendException(msg) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi glance_store.exceptions.BackendException: Failed to add container to Swift. 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi Got error from Swift: Container PUT failed: http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… 503 Service Unavailable [first 60 chars of response] b'<html><h1>Service Unavailable</h1><p>The server is currently'. 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi Mar 29 12:52:16 infra1-glance-container-99614ac2 uwsgi[2403]: Mon Mar 29 12:52:16 2021 - uwsgi_response_writev_headers_and_body_do(): Connection reset by peer [core/writer.c line 306] during PUT /v2/images/15af8603-2428-4a9d-9202-eb7250cfea46/file (192.168.110.213) Mar 29 12:52:16 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:52:16.565 2403 CRITICAL glance [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unhandled error: OSError: write error 2021-03-29 12:52:16.565 2403 ERROR glance OSError: write error 2021-03-29 12:52:16.565 2403 ERROR glance Kind regards, Oliver

2 1