April 2021 - openstack-discuss - lists.openstack.org

[all][elections][tc] TC Vacancy Special Election Nominations end this week
by Jeremy Stanley 06 Apr '21

06 Apr '21

Just a reminder, nominations for one vacant OpenStack TC (Technical Committee) position are only open for three more days, until Apr 08, 2021 23:45 UTC. All nominations must be submitted as a text file to the openstack/election repository as explained at https://governance.openstack.org/election/#how-to-submit-a-candidacy Please make sure to follow the candidacy file naming convention: candidates/xena/<Project_Name or "TC">/<email_address> (for example, "candidates/xena/TC/stacker(a)example.org"). The name of the file should match an email address for your current OpenStack Foundation Individual Membership. Take this opportunity to ensure that your OSF member profile contains current information: https://www.openstack.org/profile/ Any OpenStack Foundation Individual Member can propose their candidacy for the vacant seat on the Technical Committee. This TC vacancy special election will be held from Apr 8, 2021 23:45 UTC through to Apr 15, 2021 23:45 UTC. The electorate for the TC election are the OpenStack Foundation Individual Members who have a code contribution to one of the official teams over the Victoria to Wallaby timeframe, Apr 24, 2020 00:00 UTC - Mar 08, 2021 00:00 UTC, as well as any Extra ATCs who are acknowledged by the TC. Note that the contribution qualifying period for this special election is being kept the same as what would have been used for the original TC election. The four already elected TC members for this term are listed as candidates in the special election, but will not appear on any resulting poll as they have already been officially elected. Only new candidates in addition to the four elected TC members for this term will appear on a subsequent poll for the TC vacancy special election. Please find below the timeline: nomination starts @ Mar 25, 2021 23:45 UTC nomination ends @ Apr 08, 2021 23:45 UTC elections start @ Apr 08, 2021 23:45 UTC elections end @ Apr 15, 2021 23:45 UTC Shortly after election officials approve candidates, they will be listed on the https://governance.openstack.org/election/ page. If you have any questions please be sure to either ask them on the mailing list or to the elections officials: https://governance.openstack.org/election/#election-officials -- Jeremy Stanley on behalf of the OpenStack Technical Elections Officials

1 0

[tripleo] centos-stream-8 and container-tools 3.0
by Wesley Hayutin 05 Apr '21

05 Apr '21

FYI.. The container-tools 3.0 module has been published by the centos team. We're seeing it land in: http://dashboard-ci.tripleo.org/d/jwDYSidGz/rpm-dependency-pipeline?viewPan… We will be moving ALL the upstream centos-stream-8 jobs to use container-tools 3.0 now. https://review.opendev.org/c/openstack/tripleo-quickstart/+/784770 https://review.opendev.org/c/openstack/tripleo-quickstart/+/784768 0/ happy monday

1 0

[TripleO][ussuri] undercloud install # fails on heat launch
by Ruslanas Gžibovskis 05 Apr '21

05 Apr '21

Hi all, I am trying to understand why undercloud install does not pass the heat step. I feel that it is related to undercloud.conf file, something wrong there? Or some special char which python do not want to understand? How to enable heat-launcher more verbose output? as last line is starting engine and that's it... VERY HELPFULLL!!! AND IT CANNOT CONNECT?! when I launch it manuallly it is running for long time... ?!>!???>?>?>?><>?!>?!!>!?!!>?!!?>!?! AAAAAA PAIN!!!!!! openstack undercloud install last lines: http://paste.openstack.org/show/RCUVMaHa4XwXwRM6IS67/ undercloud.conf: https://raw.githubusercontent.com/qw3r3wq/OSP-ussuri/master/undercloud-v3/u… heat.log: http://paste.openstack.org/show/C5yP3iO8VDk7ugLOHTo6/ -- Ruslanas Gžibovskis +370 6030 7030

2 4

can't build a instance successfully
by 朝阳未烈 05 Apr '21

05 Apr '21

Folks,   I am deploying OpenStack manually and have completed minimal development of the Ussuri. My controller node can find my compute node and confirm there are compute hosts in the database with the instruction:   ” openstack compute service list --service nova-compute”   But when I want to create an instance on the compute node, the status of the compute node just remains “build”. And I try to look for faults from the “/var/log/nova/nova-compute.log” of the compute node:   2021-04-03 00:59:43.379 1432 INFO os_vif [req-2ece8c1c-a96f-4d91-b704-5598c1166016 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] Successfully unplugged vif VIFBridge(active=True,address=fa:16:3e:55:a5:65,bridge_name='brq3169e77c-99',has_traffic_filtering=True,id=28248ef5-6ad6-44bf-b2ce-3fa7ac2371ef,network=Network(3169e77c-9945-454f-9562-6e9a55e1adce),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tap28248ef5-6a') 2021-04-03 00:59:43.380 1432 INFO nova.virt.libvirt.driver [req-2ece8c1c-a96f-4d91-b704-5598c1166016 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] [instance: 3de9dac7-6704-4df9-bd81-cfdadf3ba59a] Deletion of /var/lib/nova/instances/3de9dac7-6704-4df9-bd81-cfdadf3ba59a_del complete 2021-04-03 00:59:43.426 1432 INFO nova.compute.manager [req-2ece8c1c-a96f-4d91-b704-5598c1166016 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] [instance: 3de9dac7-6704-4df9-bd81-cfdadf3ba59a] Took 0.06 seconds to destroy the instance on the hypervisor. 2021-04-03 00:59:43.980 1432 INFO nova.compute.manager [-] [instance: 3de9dac7-6704-4df9-bd81-cfdadf3ba59a] Took 0.55 seconds to deallocate network for instance. 2021-04-03 00:59:44.139 1432 INFO nova.scheduler.client.report [req-2ece8c1c-a96f-4d91-b704-5598c1166016 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] Deleted allocation for instance 3de9dac7-6704-4df9-bd81-cfdadf3ba59a 2021-04-03 01:00:00.763 1432 INFO nova.compute.claims [req-b1aed986-ee9a-46ee-8408-d65a9357f430 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] [instance: 0f53309f-2196-4215-873b-ba05fb99de57] Claim successful on node compute1 2021-04-03 01:00:00.991 1432 INFO nova.virt.libvirt.driver [req-b1aed986-ee9a-46ee-8408-d65a9357f430 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] [instance: 0f53309f-2196-4215-873b-ba05fb99de57] Creating image 2021-04-03 01:00:01.421 1432 INFO oslo.privsep.daemon [req-b1aed986-ee9a-46ee-8408-d65a9357f430 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/nova/nova-dist.conf', '--config-file', '/etc/nova/nova.conf', '--privsep_context', 'nova.privsep.sys_admin_pctxt', '--privsep_sock_path', '/tmp/tmpqsoglfcp/privsep.sock'] 2021-04-03 01:00:02.056 1432 INFO oslo.privsep.daemon [req-b1aed986-ee9a-46ee-8408-d65a9357f430 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] Spawned new privsep daemon via rootwrap 2021-04-03 01:00:01.986 1861 INFO oslo.privsep.daemon [-] privsep daemon starting 2021-04-03 01:00:01.989 1861 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2021-04-03 01:00:01.991 1861 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_CHOWN|CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_FOWNER|CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_CHOWN|CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_FOWNER|CAP_NET_ADMIN|CAP_SYS_ADMIN/none 2021-04-03 01:00:01.991 1861 INFO oslo.privsep.daemon [-] privsep daemon running as pid 1861 2021-04-03 01:00:02.716 1432 INFO oslo.privsep.daemon [req-b1aed986-ee9a-46ee-8408-d65a9357f430 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/usr/share/nova/nova-dist.conf', '--config-file', '/etc/nova/nova.conf', '--privsep_context', 'vif_plug_linux_bridge.privsep.vif_plug', '--privsep_sock_path', '/tmp/tmp50wp4hs_/privsep.sock'] 2021-04-03 01:00:03.404 1432 INFO oslo.privsep.daemon [req-b1aed986-ee9a-46ee-8408-d65a9357f430 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] Spawned new privsep daemon via rootwrap 2021-04-03 01:00:03.324 1890 INFO oslo.privsep.daemon [-] privsep daemon starting 2021-04-03 01:00:03.326 1890 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0 2021-04-03 01:00:03.328 1890 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_NET_ADMIN/CAP_NET_ADMIN/none 2021-04-03 01:00:03.328 1890 INFO oslo.privsep.daemon [-] privsep daemon running as pid 1890 2021-04-03 01:00:03.620 1432 INFO os_vif [req-b1aed986-ee9a-46ee-8408-d65a9357f430 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] Successfully plugged vif VIFBridge(active=False,address=fa:16:3e:e6:b4:89,bridge_name='brq3169e77c-99',has_traffic_filtering=True,id=c18f70ac-31ac-40eb-90f0-72c4e559f712,network=Network(3169e77c-9945-454f-9562-6e9a55e1adce),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tapc18f70ac-31') 2021-04-03 01:00:04.177 1432 INFO nova.compute.manager [-] [instance: 0f53309f-2196-4215-873b-ba05fb99de57] VM 已开始 (Lifecycle Event) 2021-04-03 01:00:04.202 1432 INFO nova.compute.manager [req-f136de80-ad6d-497a-b190-4181d739c8ba - - - - -] [instance: 0f53309f-2196-4215-873b-ba05fb99de57] VM 已暂停 (Lifecycle Event) 2021-04-03 01:00:04.246 1432 INFO nova.compute.manager [req-f136de80-ad6d-497a-b190-4181d739c8ba - - - - -] [instance: 0f53309f-2196-4215-873b-ba05fb99de57] During sync_power_state the instance has a pending task (spawning). Skip. 2021-04-03 01:00:36.581 1432 INFO nova.compute.manager [req-eb71729b-0263-4577-8248-d8c9f69387ca - - - - -] Updating bandwidth usage cache 2021-04-03 01:00:51.700 1432 ERROR oslo.messaging._drivers.impl_rabbit [req-eb71729b-0263-4577-8248-d8c9f69387ca - - - - -] Connection failed: timed out (retrying in 2.0 seconds): socket.timeout: timed out 2021-04-03 01:01:08.908 1432 ERROR oslo.messaging._drivers.impl_rabbit [req-eb71729b-0263-4577-8248-d8c9f69387ca - - - - -] Connection failed: timed out (retrying in 4.0 seconds): socket.timeout: timed out 2021-04-03 01:01:13.358 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8ded131-047e-4acb-a226-881596889c26] AMQP server on controller:5672 is unreachable: timed out. Trying again in 1 seconds.: socket.timeout: timed out 2021-04-03 01:01:27.960 1432 ERROR oslo.messaging._drivers.impl_rabbit [req-eb71729b-0263-4577-8248-d8c9f69387ca - - - - -] Connection failed: timed out (retrying in 6.0 seconds): socket.timeout: timed out 2021-04-03 01:01:37.983 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8ded131-047e-4acb-a226-881596889c26] AMQP server on controller:5672 is unreachable: timed out. Trying again in 2 seconds.: socket.timeout: timed out 2021-04-03 01:01:55.240 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8ded131-047e-4acb-a226-881596889c26] AMQP server on controller:5672 is unreachable: timed out. Trying again in 4 seconds.: socket.timeout: timed out 2021-04-03 01:02:05.262 1432 ERROR oslo.messaging._drivers.impl_rabbit [req-eb71729b-0263-4577-8248-d8c9f69387ca - - - - -] Connection failed: timed out (retrying in 8.0 seconds): socket.timeout: timed out 2021-04-03 01:02:05.262 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [3f1dbf14-730b-4f61-94b2-8674c243c457] AMQP server on controller:5672 is unreachable: Too many heartbeats missed. Trying again in 1 seconds.: amqp.exceptions.ConnectionForced: Too many heartbeats missed 2021-04-03 01:02:05.263 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [03922bfe-3f9c-41dd-8219-edd7bfe33e5f] AMQP server on controller:5672 is unreachable: Too many heartbeats missed. Trying again in 1 seconds.: amqp.exceptions.ConnectionForced: Too many heartbeats missed 2021-04-03 01:02:21.319 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [3f1dbf14-730b-4f61-94b2-8674c243c457] AMQP server on controller:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error> 2021-04-03 01:02:31.335 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [03922bfe-3f9c-41dd-8219-edd7bfe33e5f] AMQP server on controller:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error> 2021-04-03 01:02:41.351 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8ded131-047e-4acb-a226-881596889c26] AMQP server on controller:5672 is unreachable: timed out. Trying again in 6 seconds.: socket.timeout: timed out 2021-04-03 01:02:56.387 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [3f1dbf14-730b-4f61-94b2-8674c243c457] AMQP server on controller:5672 is unreachable: timed out. Trying again in 2 seconds.: socket.timeout: timed out 2021-04-03 01:03:06.406 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [03922bfe-3f9c-41dd-8219-edd7bfe33e5f] AMQP server on controller:5672 is unreachable: timed out. Trying again in 2 seconds.: socket.timeout: timed out 2021-04-03 01:03:16.417 1432 ERROR oslo.messaging._drivers.impl_rabbit [req-eb71729b-0263-4577-8248-d8c9f69387ca - - - - -] Connection failed: timed out (retrying in 10.0 seconds): socket.timeout: timed out 2021-04-03 01:03:31.457 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8ded131-047e-4acb-a226-881596889c26] AMQP server on controller:5672 is unreachable: timed out. Trying again in 8 seconds.: socket.timeout: timed out 2021-04-03 01:03:41.475 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [3f1dbf14-730b-4f61-94b2-8674c243c457] AMQP server on controller:5672 is unreachable: timed out. Trying again in 4 seconds.: socket.timeout: timed out 2021-04-03 01:03:51.495 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [03922bfe-3f9c-41dd-8219-edd7bfe33e5f] AMQP server on controller:5672 is unreachable: timed out. Trying again in 4 seconds.: socket.timeout: timed out 2021-04-03 01:04:09.540 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [3f1dbf14-730b-4f61-94b2-8674c243c457] AMQP server on controller:5672 is unreachable: timed out. Trying again in 6 seconds.: socket.timeout: timed out 2021-04-03 01:04:19.558 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [03922bfe-3f9c-41dd-8219-edd7bfe33e5f] AMQP server on controller:5672 is unreachable: timed out. Trying again in 6 seconds.: socket.timeout: timed out 2021-04-03 01:04:29.565 1432 ERROR oslo.messaging._drivers.impl_rabbit [req-eb71729b-0263-4577-8248-d8c9f69387ca - - - - -] Connection failed: timed out (retrying in 12.0 seconds): socket.timeout: timed out 2021-04-03 01:04:39.570 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8ded131-047e-4acb-a226-881596889c26] AMQP server on controller:5672 is unreachable: timed out. Trying again in 10 seconds.: socket.timeout: timed out 2021-04-03 01:04:59.594 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [3f1dbf14-730b-4f61-94b2-8674c243c457] AMQP server on controller:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error> 2021-04-03 01:05:09.607 1432 ERROR oslo.messaging._drivers.impl_rabbit [-] [03922bfe-3f9c-41dd-8219-edd7bfe33e5f] AMQP server on controller:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error> 2021-04-03 01:05:09.610 1432 WARNING nova.virt.libvirt.driver [req-b1aed986-ee9a-46ee-8408-d65a9357f430 98049570d7a54e26b8af4eaec9e2eca2 8342df14fa614ad79a08e68f097e4487 - default default] [instance: 0f53309f-2196-4215-873b-ba05fb99de57] Timeout waiting for [('network-vif-plugged', 'c18f70ac-31ac-40eb-90f0-72c4e559f712')] for instance with vm_state building and task_state spawning.: eventlet.timeout.Timeout: 300 seconds   And then controller node couldn’t find the compute node I don’t know why these happen. I have checked the port of 5672 is open on controller node and Rabbitmq is running.  Please help me.

2 1

[Ironic][RFE] Enable support for espresso machines in Ironic
by Jacob Anders 04 Apr '21

04 Apr '21

Hi There, I was discussing this RFE with Julia and we decided it would be great to get some feedback on it from the wider community, ideally by the end of today. Apologies for the short notice. Here's the story: https://storyboard.openstack.org/#!/story/2008791 What are your thoughts on this? Please comment in the Story or just reply to thread. Thank you, Jacob

14 18

ML2/OVN DVR question
by Satish Patel 03 Apr '21

03 Apr '21

Folks, I have deployed openstack using ML2/OVN on 1 controller and 2 compute nodes so far everything is working fine, when i configured router it by default used L3HA and i can see active-backup router on both compute nodes. currently my all SNAT traffic going out using compute-1 I don't want bottleneck in network so i am looking for DVR deployment and after reading i found tenant VLAN doesn't support DVR https://bugzilla.redhat.com/show_bug.cgi?id=1704596 After doing more research i found that if i set manually external_mac using the following command then my vm using local compute node to send traffic in/out just like DVR instead of centralized design. root@os-infra-1-neutron-ovn-northd-container-24eea9c2:~# ovn-nbctl find NAT type=dnat_and_snat _uuid : 99bdd866-01ed-425d-853b-9362ae8572c9 external_ids : {"neutron:fip_external_mac"="fa:16:3e:2d:7e:fa", "neutron:fip_id"="025a912a-c0ee-4f36-98ad-8992bd825cfc", "neutron:fip_network_id"="9cccf39d-edba-4069-91ef-5f30afbb6604", "neutron:fip_port_id"="70ad361a-b42e-403b-a5c1-4ee39ddf5e31", "neutron:revision_number"="6", "neutron:router_name"=neutron-8af10b06-c8de-4166-9ab1-ca2f775b08a8} external_ip : "10.40.255.10" external_mac : [] logical_ip : "172.168.0.164" logical_port : "70ad361a-b42e-403b-a5c1-4ee39ddf5e31" options : {} type : dnat_and_snat _uuid : c438e7be-5ff4-472e-b053-8d6ed74cd4dc external_ids : {"neutron:fip_external_mac"="fa:16:3e:f5:9f:f0", "neutron:fip_id"="31e8cb44-0acd-453b-a4e6-39f6ab3a6da4", "neutron:fip_network_id"="9cccf39d-edba-4069-91ef-5f30afbb6604", "neutron:fip_port_id"="44a677c5-86ff-4b6b-a046-54e79f79c4cd", "neutron:revision_number"="2", "neutron:router_name"=neutron-8af10b06-c8de-4166-9ab1-ca2f775b08a8} external_ip : "10.40.255.5" external_mac : [] logical_ip : "172.168.0.67" logical_port : "44a677c5-86ff-4b6b-a046-54e79f79c4cd" options : {} type : dnat_and_snat This is how i set external mac from fip_external_mac"="fa:16:3e:2d:7e:fa" in above command. ovn-nbctl set NAT 99bdd866-01ed-425d-853b-9362ae8572c9 external_mac="fa\:16\:3e\:2d\:7e\:fa" How do i make this behavior default for every single VM, i don't want to do this manually to set the external mac address of each FIP?

1 1

launch VM on volume vs. image
by Tony Liu 03 Apr '21

03 Apr '21

Hi, With Ceph as the backend storage, launching a VM on volume takes much longer than launching on image. Why is that? Could anyone elaborate the high level workflow for those two cases? Thanks! Tony

3 9

[Requirements][all] Requirements branched, freeze lifted
by Matthew Thode 02 Apr '21

02 Apr '21

The requirements freeze is now lifted. If your project has not branched please be aware that master moves on (to xena). -- Matthew Thode

1 0

RE: [Interop][Refstack] No meeting this Friday meeting
by Kanevsky, Arkady 02 Apr '21

02 Apr '21

Team, No meeting this Friday. Happy Holiday! Will update Etherpad. Cheers, Arkady From: prakash RAMCHANDRAN <pramchan(a)yahoo.com> Sent: Thursday, April 1, 2021 1:12 PM To: Jimmy McArthur; Vida Haririan Cc: Ghanshyam Mann; Kanevsky, Arkady; openstack-discuss; Martin Kopec; Goutham Pacha Ravi Subject: Re: [Interop][Refstack] this Friday meeting [EXTERNAL EMAIL] Looks like we can skip this Friday call and sure Arkady - lets cancel it. If you have something urgent we can talk offline - Thanks Prakash On Thursday, April 1, 2021, 11:06:25 AM PDT, Vida Haririan <vhariria(a)redhat.com<mailto:vhariria@redhat.com>> wrote: Hi Arkady, Friday is a company holiday and I will be ooo. Thanks, Vida On Thu, Apr 1, 2021 at 11:10 AM Jimmy McArthur <jimmy(a)openstack.org<mailto:jimmy@openstack.org>> wrote: I forgot this is a holiday. Same on my side. Thanks, Jimmy > On Apr 1, 2021, at 9:25 AM, Ghanshyam Mann <gmann(a)ghanshyammann.com<mailto:gmann@ghanshyammann.com>> wrote: > > > ---- On Thu, 01 Apr 2021 09:15:21 -0500 Kanevsky, Arkady <Arkady.Kanevsky(a)dell.com<mailto:Arkady.Kanevsky@dell.com>> wrote ---- >> >> Team, >> This Friday is Good Friday and some people have a day off. >> Should we cancel this week meeting? >> Please, respond so we can see if we will have quorum. > > Thanks Arkady, > > I will be off from work and would not be able to join. > > -gmann > >> Thanks, >> Arkady >> >> Arkady Kanevsky, Ph.D. >> SP Chief Technologist & DE >> Dell Technologies office of CTO >> Dell Inc. One Dell Way, MS PS2-91 >> Round Rock, TX 78682, USA >> Phone: 512 7204955 >> >> >

2 1

RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller
by Braden, Albert 02 Apr '21

02 Apr '21

I opened a bug for this issue: https://bugs.launchpad.net/kolla-ansible/+bug/1922269 -----Original Message----- From: Braden, Albert Sent: Thursday, April 1, 2021 11:34 AM To: 'openstack-discuss(a)lists.openstack.org' <openstack-discuss(a)lists.openstack.org> Subject: RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller Sorry that was a typo. Stopping RMQ during the removal of the *second* controller is what causes the problem. Is there a way to tell Centos 8 Train to use RMQ 3.7.24 instead of 3.7.28? -----Original Message----- From: Braden, Albert Sent: Thursday, April 1, 2021 9:34 AM To: 'openstack-discuss(a)lists.openstack.org' <openstack-discuss(a)lists.openstack.org> Subject: RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller I did some experimenting and it looks like stopping RMQ during the removal of the first controller is what causes the problem. After deploying the first controller, stopping the RMQ container on any controller including the new centos8 controller will cause the entire cluster to stop. This crash dump appears on the controllers that stopped in sympathy: https://paste.ubuntu.com/p/ZDgFgKtQTB/ This appears in the RMQ log: https://paste.ubuntu.com/p/5D2Qjv3H8c/ -----Original Message----- From: Braden, Albert Sent: Wednesday, March 31, 2021 8:31 AM To: openstack-discuss(a)lists.openstack.org Subject: RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller Centos7: {rabbit,"RabbitMQ","3.7.24"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, Centos8: {rabbit,"RabbitMQ","3.7.28"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, When I deploy the first Centos8 controller, RMQ comes up with all 3 nodes active and seems to be working fine until I shut down the 2nd controller. The only hint of trouble when I replace the 1st node is this error message the first time I run the deployment: https://paste.ubuntu.com/p/h9HWdfwmrK/ and the crash dump that appears on control2: crash dump log: https://paste.ubuntu.com/p/MpZ8SwTJ2T/ First 1500 lines of the dump: https://paste.ubuntu.com/p/xkCyp2B8j8/ If I wait for a few minutes then RMQ recovers on control2 and the 2nd run of the deployment seems to work, and there is no trouble until I shut down control1. -----Original Message----- From: Mark Goddard <mark(a)stackhpc.com> Sent: Wednesday, March 31, 2021 4:14 AM To: Braden, Albert <C-Albert.Braden(a)charter.com> Cc: openstack-discuss(a)lists.openstack.org Subject: [EXTERNAL] Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. On Tue, 30 Mar 2021 at 13:41, Braden, Albert <C-Albert.Braden(a)charter.com> wrote: > > I’ve created a heat stack and installed Openstack Train to test the Centos7->8 upgrade following the document here: > > > > https://docs.openstack.org/kolla-ansible/train/user/centos8.html#migrating-… > > > > I used the instructions here to successfully remove and replace control0 with a Centos8 box > > > > https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hos… > > > > After this my RMQ admin page shows all 3 nodes up, including the new control0. The name of the cluster is rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-2 /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-2 ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-0-replace',[]}, > > {'rabbit@chrnc-void-testupgrade-control-1',[]}, > > {'rabbit@chrnc-void-testupgrade-control-2',[]}]}] > > > > After that I create a new VM to verify that the cluster is still working, and then perform the same procedure on control1. When I shut down services on control1, the ansible playbook finishes successfully: > > > > kolla-ansible -i ../multinode stop --yes-i-really-really-mean-it --limit control1 > > … > > control1 : ok=45 changed=22 unreachable=0 failed=0 skipped=105 rescued=0 ignored=0 > > > > After this my RMQ admin page stops responding. When I check RMQ on the new control0 and the existing control2, the container is still up but RMQ is not running: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'. > > > > If I start it on control0 and control2, then the cluster seems normal and the admin page starts working again, and cluster status looks normal: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-0-replace ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-2', > > 'rabbit@chrnc-void-testupgrade-control-0-replace']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-2',[]}, > > {'rabbit@chrnc-void-testupgrade-control-0-replace',[]}]}] > > > > But my hypervisors are down: > > > > (openstack) [root@chrnc-void-testupgrade-build kolla-ansible]# ohll > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | 3 | chrnc-void-testupgrade-compute-2.dev.chtrse.com | QEMU | 172.16.2.106 | down | 5 | 8 | 2560 | 30719 | > > | 6 | chrnc-void-testupgrade-compute-0.dev.chtrse.com | QEMU | 172.16.2.31 | down | 5 | 8 | 2560 | 30719 | > > | 9 | chrnc-void-testupgrade-compute-1.dev.chtrse.com | QEMU | 172.16.0.30 | down | 5 | 8 | 2560 | 30719 | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > > > When I look at the nova-compute.log on a compute node, I see RMQ failures every 10 seconds: > > > > 172.16.2.31 compute0 > > 2021-03-30 03:07:54.893 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > 2021-03-30 03:07:55.905 7 INFO oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] Reconnected to AMQP server on 172.16.1.132:5672 via [amqp] client with port 56422. > > 2021-03-30 03:08:05.915 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > > > In the RMQ logs I see this every 10 seconds: > > > > 172.16.1.132 control2 > > [root@chrnc-void-testupgrade-control-2 ~]# tail -f /var/log/kolla/rabbitmq/rabbit\(a)chrnc-void-testupgrade-control-2.log |grep 172.16.2.31 > > 2021-03-30 03:07:54.895 [warning] <0.13247.35> closing AMQP connection <0.13247.35> (172.16.2.31:56420 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > client unexpectedly closed TCP connection > > 2021-03-30 03:07:55.901 [info] <0.15288.35> accepting AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) > > 2021-03-30 03:07:55.903 [info] <0.15288.35> Connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) has a client-provided name: nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e > > 2021-03-30 03:07:55.904 [info] <0.15288.35> connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e): user 'openstack' authenticated and granted access to vhost '/' > > 2021-03-30 03:08:05.916 [warning] <0.15288.35> closing AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > > > Why does RMQ fail when I shut down the 2nd controller after successfully replacing the first one? Hi Albert, Could you share the versions of RabbitMQ and erlang in both versions of the container? When initially testing this setup, I think we had 3.7.24 on both sides. Perhaps the CentOS 8 version has moved on sufficiently to become incompatible? Mark > > > > I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails. > > > > The contents of this e-mail message and > any attachments are intended solely for the > addressee(s) and may contain confidential > and/or legally privileged information. If you > are not the intended recipient of this message > or if this message has been addressed to you > in error, please immediately alert the sender > by reply e-mail and then delete this message > and any attachments. If you are not the > intended recipient, you are notified that > any use, dissemination, distribution, copying, > or storage of this message or any attachment > is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

1 0