June 2021 - openstack-discuss - lists.openstack.org

[all][tc] What's happening in Technical Committee: summary 4th June, 21: Reading: 5 min
by Ghanshyam Mann 05 Jun '21

05 Jun '21

Hello Everyone, Here is last week's summary of the Technical Committee activities. 1. What we completed this week: ========================= * Retired the sushy-cli[1]. * Replaced freenode ref with OFTC[2]. * Added TC resolution to move the IRC network from Freenode to OFTC[3] 2. TC Meetings: ============ * TC held this week meeting on Thursday; you can find the full meeting logs in the below link: - https://meetings.opendev.org/meetings/tc/2021/tc.2021-06-03-15.00.log.html * We will have next week's meeting on June 10th, Thursday 15:00 UTC[4]. 3. Activities In progress: ================== TC Tracker for Xena cycle ------------------------------ TC is using the etherpad[5] for Xena cycle working item. We will be checking and updating the status biweekly in the same etherpad. Open Reviews ----------------- * Two open reviews for ongoing activities[6]. MIgration from Freenode to OFTC ----------------------------------------- * All the required work for this migration is tracked in this etherpad[7] * TC resolution is merged[3]. * OFTC bot/logging is done. All project teams started their discussion/meetings to OFTC. * I have communicated the next steps on openstack-discuss ML[8] as well as to all PTLs individual email. * We are in 'Communicate with community' work where we need to update all contributor doc etc. Please finish this in your project and mark the progress in etherpad[7]. * This migration has been proposed to add in Open Infra newsletter OpenStack's news also[9]. * Topic change on Freenode channels will be done on June 11th. Nomination is open for the 'Y' release naming ------------------------------------------------------ * Y release naming process is started[10]. Nomination is open until June 10th feel free to propose names in below wiki ** https://wiki.openstack.org/wiki/Release_Naming/Y_Proposals Replacing ATC terminology with AC (Active Contributors) ------------------------------------------------------------------- * Governance charter change for ATC->AC has been merged [11]. * TC resolution to map the ATC with the new term AC from Bylaws' perspective is up[12]. 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[13]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [14] 3. Office hours: The Technical Committee offers a weekly office hour every Tuesday at 0100 UTC [15] 4. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://review.opendev.org/c/openstack/governance/+/792348 [2] https://review.opendev.org/c/openstack/governance/+/793864 [3] https://review.opendev.org/c/openstack/governance/+/793260 [4] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [5] https://etherpad.opendev.org/p/tc-xena-tracker [6] https://review.opendev.org/q/project:openstack/governance+status:open [7] https://etherpad.opendev.org/p/openstack-irc-migration-to-oftc [8] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022780.html [9] https://etherpad.opendev.org/p/newsletter-openstack-news [10] http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022383.html [11] https://review.opendev.org/c/openstack/governance/+/790092 [12] https://review.opendev.org/c/openstack/governance/+/794366 [13] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [14] http://eavesdrop.openstack.org/#Technical_Committee_Meeting [15] http://eavesdrop.openstack.org/#Technical_Committee_Office_hours -gmann

1 0

[Diversity] Diversity and Inclusion Meeting Reminder - OFTC
by Amy Marrich 04 Jun '21

04 Jun '21

The Diversity & Inclusion WG invites members of all OIF projects to attend our next meeting Monday June 7th, at 17:00 UTC in the #openinfra-diversity channel on OFTC. The agenda can be found at https://etherpad.openstack.org/p/ diversity-wg-agenda. Please feel free to add any topics you wish to discuss at the meeting. Thanks, Amy (spotz)

1 0

[novnc-console] Cannot connect to console
by Derek O keeffe 04 Jun '21

04 Jun '21

Hi all, This is my first post to this list so excuse me if I have not submitted correctly. I have installed openstack Victoria manually as a multi node setup. A controller & 3 computes. Everything works fine and the way it's expected. I have secured horizon with letsencrypt certs (for now) and again all is fine. When I did a test deploy I also used those certs to load the novnc console securely and it worked. My problem with my new deploy is that the console will not load no matter what I try. I get the following error when I enable debug mode in nova. 2021-06-04 15:54:11.004 356545 ERROR nova.console.websocketproxy Traceback (most recent call last):2021-06-04 15:54:11.004 356545 ERROR nova.console.websocketproxy File "/usr/lib/python3/dist-packages/websockify/websockifyserver.py", line 691, in top_new_client2021-06-04 15:54:11.004 356545 ERROR nova.console.websocketproxy client = self.do_handshake(startsock, address)2021-06-04 15:54:11.004 356545 ERROR nova.console.websocketproxy File "/usr/lib/python3/dist-packages/websockify/websockifyserver.py", line 578, in do_handshake2021-06-04 15:54:11.004 356545 ERROR nova.console.websocketproxy context.load_cert_chain(certfile=self.cert, keyfile=self.key, password=self.key_password)2021-06-04 15:54:11.004 356545 ERROR nova.console.websocketproxy PermissionError: [Errno 13] Permission denied If I don't have debug enabled I just get the permission denied error. I have switched to the nova user and confirmed I can access the certs directory and read the certs. All my nova services are running fine as well. My controller conf is the following:[default]ssl_only=true cert=/etc/letsencrypt/live/ <MY_CONTROLLER_FQDN>/fullchain.pemkey=/etc/letsencrypt/live/ <MY_CONTROLLER_FQDN>/privkey.pem [vnc]enabled = trueserver_listen = 0.0.0.0server_proxyclient_address = $my_ipnovncproxy_base_url = https://<MY_CONTROLLER_FQDN>:6080/vnc_auto.html My compute config is the following:[vnc]enabled = trueserver_listen = 0.0.0.0server_proxyclient_address = $my_ipnovncproxy_base_url = https://<MY_CONTROLLER_FQDN>:6080/vnc_auto.html If anyone could help that would be really appreciated or any advice to further troubleshoot!! I cannot see anything else in any logs but I might not be looking in the right place. Thank you in advance. Derek

2 2

[nova] stable branches nova-grenade-multinode job broken
by melanie witt 04 Jun '21

04 Jun '21

Hi all, FYI the nova-grenade-multinode CI job is known to be broken on stable branches at the moment due to a too new version of Ceph (Pacific) being installed that is incompatible with the older jobs. We have fixes proposed with the following patch (and its backports to victoria/ussuri/train) to convert the job to native Zuul v3: https://review.opendev.org/c/openstack/nova/+/794345 Once these patches merge, the CI should be passing again. Cheers, -melanie

1 0

[Interop] draft of presentation to the board
by Kanevsky, Arkady 04 Jun '21

04 Jun '21

https://docs.google.com/presentation/d/1-9H1cTXZxW0vCSTzfBe0aMKbd7nggd8SOHQ… Comments welcome. Arkady Kanevsky, Ph.D. SP Chief Technologist & DE Dell Technologies office of CTO Dell Inc. One Dell Way, MS PS2-91 Round Rock, TX 78682, USA Phone: 512 7204955

1 0

[kolla-ansible] kolla-ansible destroy
by at 04 Jun '21

04 Jun '21

Hi is kolla-ansible destroy "--tags" aware? What is the best way to remove all unwanted containers, configuration files, logs, etc. when you want to remove some service or move it to another node? Regards Adam Tomas

2 4

[TripleO] next irc meeting Tuesday 08 June @ 1400 UTC in OFTC #tripleo
by Marios Andreou 04 Jun '21

04 Jun '21

Reminder that the next TripleO irc meeting is: ** Tuesday 08 June 1400 UTC in OFTC irc channel: #tripleo ** ** https://wiki.openstack.org/wiki/Meetings/TripleO ** ** https://etherpad.opendev.org/p/tripleo-meeting-items ** Add anything you want to highlight at https://etherpad.opendev.org/p/tripleo-meeting-items This can be recently completed things, ongoing review requests, blocking issues, or anything else tripleo you want to share. Our last meeting was on May 25 - you can find logs there http://eavesdrop.openstack.org/meetings/tripleo/2021/tripleo.2021-05-25-14.… Hope you can make it on Tuesday, regards, marios

1 0

[Victoria][magnum][octavia]ingress-controller health degraded
by Luke Camilleri 04 Jun '21

04 Jun '21

Hi Everyone, we have the following problem that we are trying to identify the main cause: Basically we have deployed an ingress and an ingress-controller (using the following deployment file https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44…) the ingress controller deployment is successful with 1 replica of the ingress-controller pod, the Octavia LoadBalancer is successfully created and points to the NodePorts being published on each node. This was showing only 1 member in the LoadBalancers screen as healthy/online. I increased the replicas to 3, from the LoadBalancers screen in Horizon, I can see the service being reported as degraded and only the kubernetes-worker-nodes that have the ingress-controller pod/s deployed on are being reported as online, this behaviour is not the same as a standard deployment where the NodePort actually communicates with the ClusterIP:Port of the internal service and hence once there is a single pod UP the NodePorts are shown as up when queried: ingress-nginx-controller-74fd5565fb-d86h9 1/1 Running 0 14h 10.100.3.13 k8s-c1-prod-2-klctfd24lze6-node-1 <none> <none> ingress-nginx-controller-74fd5565fb-h9985 1/1 Running 0 15h 10.100.1.8 k8s-c1-prod-2-klctfd24lze6-node-0 <none> <none> ingress-nginx-controller-74fd5565fb-qkddq 1/1 Running 0 15h 10.100.1.7 k8s-c1-prod-2-klctfd24lze6-node-0 <none> <none> The below shows the status of the members in the pool as replica3: | 834750fe-e43e-408d-abc3-aad3dcde0fdb | member_0_node-0 | id | ACTIVE | 192.168.1.75 | 32054 | ONLINE | 1 | | 1ddffd80-acae-40b3-a2de-19be0a69a039 | member_0_node-2 | id | ACTIVE | 192.168.1.90 | 32054 | ERROR | 1 | | d4e4baa4-0a69-4775-8ea0-165a207f11ae | member_0_node-1| id | ACTIVE | 192.168.1.148 | 32054 | ONLINE | 1 | In fact to have the deployment spread across all 3 nodes, I had to increase the replicas until all 3 nodes had at least an instance of the ingress controller running on them (in this case it was replica 5). I do not believe this as being an Octavia issue as the health check is being done via TCP port number which is the NodePort exposed by Kubernetes and if the ingress-controller is not running on that node the port check fails, I added the label octavia mainly to get some input that may confirm the correct behavior of Octavia I am expecting to receive a healthy state when i check the members of the pool since I can query the ClusterIP from any worker node on ports 80 and 443 and the outcome is always successful but not when using the NodePort Thanks in advance

1 0

Re: [ops] rabbitmq queues for nova versioned notifications queues keep filling up
by Christopher McCarthy 04 Jun '21

04 Jun '21

Hi Ajay, We work around this by setting a TTL on our notifications queues via RabbitMQ policy definition. We include the following in our definitions.json for RabbitMQ: "policies":[ {"vhost": "/", "name": "notifications-ttl", "pattern": "^(notifications|versioned_notifications)\\.", "apply-to": "queues", "definition": {"message-ttl":600000}, "priority":0} ] This expires messages in the notifications and versioned_notifications queues after 10 minutes, which seems to work well for us. I believe we initially picked up this workaround from this[1] bug report. Hope this helps, - Chris -- Christopher McCarthy MathWorks cmccarth(a)mathworks.com [1] https://bugs.launchpad.net/charm-rabbitmq-server/+bug/1737170 Date: Wed, 2 Jun 2021 22:39:54 -0000 From: "Ajay Tikoo (BLOOMBERG/ 120 PARK)" <atikoo(a)bloomberg.net> To: openstack-discuss(a)lists.openstack.org Subject: [ops] rabbitmq queues for nova versioned notifications queues keep filling up Message-ID: <60B808BA00D0068401D80001_0_3025859@msclnypmsgsv04> Content-Type: text/plain; charset="utf-8" I am not sure if this is the right channel/format to post this question, so my apologies in advance if this is not the right place. We are using Openstack Rocky. Watcher needs versioned notifications to be enabled. However after enabling versioned notifications, the queues for versioned_notifications (info and error) keep filling up Based on the updates the the Watchers cluster data model, it appears that Watcher is consuming messages, but they still linger in these queues. So with nova versioned notifications disabled, Watcher is unable to update the cluster data model (between rebuild intervals), and with them enabled, it keeps filling up the MQ queues. What is the best way to resolve this? Thank you, Ajay Tikoo

1 0

[security-sig][cinder] propose vulnerability:managed tag for os-brick
by Brian Rosmaita 04 Jun '21

04 Jun '21

I've posted a patch to add the 'vulnerablity:managed' tag to the os-brick library: https://review.opendev.org/c/openstack/governance/+/794680 I just want to give a heads-up to the OpenStack Vulnerablity Management Team, since this will impact the VMT, though hopefully not very much. The Cinder team was under the impression that the VMT was already managing private security bugs for os-brick. The issue may not have come up before because usually there's a driver + connector involved and the bug gets filed under cinder (which is already tagged vulnerablity:managed). In any case, the cinder team discussed this at our recent midcycle meeting and decided that we appreciate the extra eyes and long-term perspective the VMT brings to the table, and we'd like to formalize a relation between the VMT and the os-brick library. cheers, brian

2 1