December 2020 - openstack-discuss

[tc] weekly meeting skip
by Mohammed Naser 23 Dec '20

23 Dec '20

Hi there, Due to the upcoming holidays, we are skipping the next TC meetings on Dec 24th and Dec 31st. We will resume with our regular schedule on Thursday, January 7th. Thank you & happy holidays! Regards Mohammed -- Mohammed Naser VEXXHOST, Inc.

1 0

oslo message direct_mandatory_flag question
by Satish Patel 22 Dec '20

22 Dec '20

Folks, I am getting a very strange error from time to time on senlin logs and when it hits this error i can't do anything in senlin until i restart service. After googling i found this bug https://bugs.launchpad.net/oslo.messaging/+bug/1905965 It's pretty much related to my issue because I am running the latest victoria. Dec 22 13:56:44 os-lab-infra-1-senlin-container-16f24bbe senlin-wsgi-api[8188]: 2020-12-22 13:56:44.212 8188 ERROR oslo.messaging._drivers.impl_rabbit [-] [df314561-6415-4103-a1fd-14ab95182cfb] AMQP server on 10.65.6.176:5671 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error> Dec 22 13:56:44 os-lab-infra-1-senlin-container-16f24bbe senlin-wsgi-api[8188]: 2020-12-22 13:56:44.220 8188 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Dec 22 13:56:44 os-lab-infra-1-senlin-container-16f24bbe senlin-conductor[8250]: 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server [req-3c89475b-89fc-404b-8537-df7a587261d9 462618bed32745d2a9166bcc33fc117e f1502c79c70f4651be8ffc7b844b584f - - -] MessageUndeliverable error, source exception: Basic.return: (312) NO_ROUTE, routing_key: reply_54d93c43fe894ed18ce8092f4497306b, exchange: : : oslo_messaging.exceptions.MessageUndeliverable 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 184, in _process_incoming 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server message.reply(res) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 149, in reply 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server self._send_reply(conn, reply, failure) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 125, in _send_reply 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server conn.direct_send(self.reply_q, rpc_common.serialize_msg(msg)) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1320, in direct_send 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server self._ensure_publishing(self._publish_and_raises_on_missing_exchange, 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1202, in _ensure_publishing 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server self.ensure(method, retry=retry, error_callback=_error_callback) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 826, in ensure 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server ret, channel = autoretry_method() 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/kombu/connection.py", line 525, in _ensured 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server return fun(*args, **kwargs) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/kombu/connection.py", line 601, in __call__ 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server return fun(*args, channel=channels[0], **kwargs), channels[0] 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 815, in execute_method 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server method() 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1294, in _publish_and_raises_on_missing_exchange 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server self._publish(exchange, msg, routing_key=routing_key, 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1238, in _publish 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server self._producer.publish( 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/kombu/messaging.py", line 175, in publish 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server return _publish( 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/kombu/messaging.py", line 197, in _publish 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server return channel.basic_publish( 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/channel.py", line 1782, in basic_publish_confirm 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server self.wait([spec.Basic.Ack, spec.Basic.Nack], callback=confirm_handler) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/abstract_channel.py", line 86, in wait 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server self.connection.drain_events(timeout=timeout) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/connection.py", line 514, in drain_events 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server while not self.blocking_read(timeout): 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/connection.py", line 520, in blocking_read 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server return self.on_inbound_frame(frame) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/method_framing.py", line 77, in on_frame 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server callback(channel, msg.frame_method, msg.frame_args, msg) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/connection.py", line 526, in on_inbound_method 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server return self.channels[channel_id].dispatch_method( 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/abstract_channel.py", line 143, in dispatch_method 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server listener(*args) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/amqp/channel.py", line 2006, in _on_basic_return 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server callback(exc, exchange, routing_key, message) 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server File "/openstack/venvs/senlin-22.0.0.0b2.dev56/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 852, in on_return 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server raise exceptions.MessageUndeliverable(exception, exchange, routing_key, 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server oslo_messaging.exceptions.MessageUndeliverable 2020-12-22 13:56:44.461 8250 ERROR oslo_messaging.rpc.server Dec 22 13:56:45 os-lab-infra-1-senlin-container-16f24bbe senlin-wsgi-api[8188]: 2020-12-22 13:56:45.248 8188 INFO oslo.messaging._drivers.impl_rabbit [-] [df314561-6415-4103-a1fd-14ab95182cfb] Reconnected to AMQP server on 10.65.6.176:5671 via [amqp] client with port 44246. Dec 22 13:56:53 os-lab-infra-1-senlin-container-16f24bbe senlin-health-manager[8218]: 2020-12-22 13:56:53.954 8218 INFO senlin.engine.health_manager [req-c0118d24-9f32-47c8-926e-eb4b60f922cd 462618bed32745d2a9166bcc33fc117e f1502c79c70f4651be8ffc7b844b584f - - -] Health check passed for all nodes in cluster f51b3154-fb8e-43f2-b68b-03b091c1e0bf. Dec 22 13:57:23 os-lab-infra-1-senlin-container-16f24bbe senlin-health-manager[8218]: 2020-12-22 13:57:23.962 8218 INFO senlin.engine.health_manager [req-472f4403-fd66-422a-8a84-a0b0f51d94cc 462618bed32745d2a9166bcc33fc117e f1502c79c70f4651be8ffc7b844b584f - - -] Health check passed for all nodes in cluster f51b3154-fb8e-43f2-b68b-03b091c1e0bf. Dec 22 13:57:23 os-lab-infra-1-senlin-container-16f24bbe senlin-health-manager[8218]: 2020-12-22 13:57:23.963 8218 INFO senlin.engine.health_manager [req-a5ff1fe3-35e8-443b-b91a-aa1d78656569 462618bed32745d2a9166bcc33fc117e f1502c79c70f4651be8ffc7b844b584f - - -] Health check passed for all nodes in cluster f51b3154-fb8e-43f2-b68b-03b091c1e0bf. Dec 22 13:57:44 os-lab-infra-1-senlin-container-16f24bbe uwsgi[8188]: Tue Dec 22 13:57:44 2020 - uwsgi_response_writev_headers_and_body_do(): Connection reset by peer [core/writer.c line 306] during GET /v1/clusters?global_project=False (10.65.6.17) Dec 22 13:57:44 os-lab-infra-1-senlin-container-16f24bbe senlin-wsgi-api[8188]: 2020-12-22 13:57:44.448 8188 CRITICAL senlin-api [req-3c89475b-89fc-404b-8537-df7a587261d9 462618bed32745d2a9166bcc33fc117e f1502c79c70f4651be8ffc7b844b584f - - -] Unhandled error: OSError: write error 2020-12-22 13:57:44.448 8188 ERROR senlin-api OSError: write error 2020-12-22 13:57:44.448 8188 ERROR senlin-api Dec 22 13:57:53 os-lab-infra-1-senlin-container-16f24bbe senlin-health-manager[8218]: 2020-12-22 13:57:53.955 8218 INFO senlin.engine.health_manager [req-8a578e8e-438d-45c1-b81d-c6f3aed5fc81 462618bed32745d2a9166bcc33fc117e f1502c79c70f4651be8ffc7b844b584f - - -] Health check passed for all nodes in cluster f51b3154-fb8e-43f2-b68b-03b091c1e0bf. As per advice i have tired to add following snippet to fix this issue [DEFAULT] transport_url = rabbit://senlin:94d7aecb853145779db8f1dcb@10.65.6.176:5671//senlin?ssl=1 [oslo_messaging_rabbit] ssl = True direct_mandatory_flag = False But i got error when trying to set direct_mandatory_flag option ERROR oslo_service.service [-] Error starting t hread.: oslo_config.cfg.ConfigFileValueError: Value for option direct_mandatory_flag from LocationInfo(location=<Locations.user: (4, True)>, detail='/etc/senlin/senlin.conf') is not valid: invalid literal for int() with base 10: 'False' Any idea what is going on here?

2 2

[infra][tripleo] is it possible to control (tripleo) gate queue priority?
by Marios Andreou 22 Dec '20

22 Dec '20

Hello illustrious members of openstack infra tripleo-ci squad is wondering if it is possible for (some subset of) us to be able to set the priority of a particular patch/es in the tripleo queue. We've done this "manually" in the past, by abandoning all patches in the gate & then restoring in order and putting the priority patch at the top of the dependency queue. However abandoning all the things is completely disruptive for everyone else (sometimes that might be necessary if your queue is way too long but still...). So the question is, is there a better way to put a particular patch at the top of our queue when we need to do that? thanks for your thoughts, sorry if this has come up before I couldn't quickly find something in the list archives. regards, marios

4 5

[off-topic][tripleo] Merry Christmas!
by Carlos Camacho Gonzalez 22 Dec '20

22 Dec '20

Hi everyone! Its been a complex year for everyone, however, we managed to do amazing things, I hope we can get a boost of energy these holidays to do even better things next year, I wish you all the best and to stay healthy. https://imgur.com/a/5rSXMTe As usual, all the sources are in GH [1]. Merry Christmas! Carlos. [1]: https://github.com/ccamacho/tripleo-graphics/tree/master/x-mas/2020

1 0

[blazar] No IRC meetings during holiday season
by Pierre Riteau 21 Dec '20

21 Dec '20

Hello, I am on leave for the end of the year, so there will be no IRC meetings for the Blazar project until the week of January 4. I wish everyone a very happy holiday! Pierre Riteau (priteau)

1 0

[keystone] No weekly meeting Dec 22
by Nikolla, Kristi 21 Dec '20

21 Dec '20

Hi all, I won't be able to host the weekly IRC team meeting for keystone tomorrow. Best, Kristi Nikolla

1 0

Re: [placement] Train upgrade warning
by Seth Tunstall 21 Dec '20

21 Dec '20

Hello, As another person has mentioned (Chanu Romain), I am attempting an upgrade from Queens to Train. I have successfully upgraded Keystone and Glance, but while trying to separate Placement's databases out from Nova, I run into the the same issue with placement-status upgrade check: # placement-status upgrade check +-----------------------------------------------------------------+ | Upgrade Check Results | +-----------------------------------------------------------------+ | Check: Missing Root Provider IDs | | Result: Success | | Details: None | +-----------------------------------------------------------------+ | Check: Incomplete Consumers | | Result: Warning | | Details: There are -3511 incomplete consumers table records for | | existing allocations. Run the "placement-manage db | | online_data_migrations" command. | +-----------------------------------------------------------------+ Likewise, running placement-manage db online_data_migrations does nothing. I am running this placement migration on a dev cluster which has had the databases from a Prod cluster restored to it. This has worked well for me previously as a way to run through upgrades without touching the live system. Thanks very much, Seth

3 10

[tc] weekly update
by Mohammed Naser 21 Dec '20

21 Dec '20

Hi everyone, Here's an update for what happened in the OpenStack TC this week. You can get more information by checking for changes in openstack/governance repository. # Patches ## Open Reviews - Add glance-tempest-plugin to Glance https://review.opendev.org/c/openstack/governance/+/767666 - Remove Karbor project team https://review.opendev.org/c/openstack/governance/+/767056 - Add Resolution of TC stance on the OpenStackClient https://review.opendev.org/c/openstack/governance/+/759904 ## Project Updates - Deprecate openstack-ansible-galera_client role https://review.opendev.org/c/openstack/governance/+/765784 ## General Changes - Improve check-review-status https://review.opendev.org/c/openstack/governance/+/766249 - Clarify impact on releases for SIGs https://review.opendev.org/c/openstack/governance/+/752699 # Other Reminders - Due to the upcoming holidays we are skipping the next TC meetings on Dec 24th and Dec 31st. We will resume with our regular schedule on Thursday January 7th. Thanks for reading! Happy Holidays! Mohammed & Kendall -- Mohammed Naser VEXXHOST, Inc.

1 0

[all] Our next release name - OpenStack Xena!
by Sean McGinnis 21 Dec '20

21 Dec '20

Hello everyone! The naming poll is done [0], the legal vetting is complete, and we now have our X release name. All hail OpenStack Xena! Observant readers will note that Xena was not the 1st place name from the poll. After review for legal and copyright concerns, it was determined there would be conflicts with the top place Xanadu. Apologies to the Olivia Newton John fans out there. Xena was determined to be safe for our use, and was a close second. More details about the release naming process can be found on the governance page [1]. Please consider getting involved for the Y release naming. Thanks! Sean [0] https://civs.cs.cornell.edu/cgi-bin/results.pl?id=E_7e6e96070af39fe7 [1] https://governance.openstack.org/tc/reference/release-naming.html

1 0

New Openstack Deployment questions
by Thomas Wakefield 21 Dec '20

21 Dec '20

OpenStack deployment questions: If you were starting a new deployment of OpenStack today what OS would you use, and what tools would you use for deployment? We were thinking CentOS with Kayobe, but then CentOS changed their support plans, and I am hesitant to start a new project with CentOS. We do have access to RHEL licensing so that might be an option. We have also looked at OpenStack-Ansible for deployment. Thoughts? Thanks in advance. -Tom

9 13