June 2022 - openstack-discuss - lists.openstack.org

[kolla-ansible][wallaby][nova][spice]
by Franck VEDEL 27 Jun '22

27 Jun '22

hello I have two problems and I ask you for a little help before doing stupid things. If I correctly diagnosed the first problem, which is to replace novnc by spice in my installation, it is not possible. I'm using Centos, I'm on the Wallaby version, and the last tag for the Docker image is Train. So my first question is: can I somehow replace novnc with something else, spice or another? Second question: I would like to switch from Wallaby to Xena (unless we can switch from Wallaby to Yoga... is it possible?) . After a year of operation, this is the first time I have had to do an update with an openstack in production (student instances have been deleted, end of the academic year) but not used for a few days. Is the following procedure correct: git clone --branch stable/xena https://opendev.org/openstack/kolla-ansible mv kolla-ansible kolla-ansible-xena (to keep multiple versions) pip install ./kolla-ansible-xena kolla-ansible -i multinode bootstrap-servers kolla-ansible -i multinode prechecks kolla-ansible -i multinode deploy What to do with globals.yml and the /etc/kolla/config overload? no changes ? Thank you in advance for your help, for your answers, and for an update procedure if you have one. Sincerely Franck

1 0

How is this overallocation possible?
by Russell Stather 27 Jun '22

27 Jun '22

See image below. How can I be using 38 vcpus on a compute with 24 total? Is this an error in horizon? [cid:988b50f9-b9c8-4ccc-9240-90f2d1ff3df6]

3 5

[Neutron] team meeting - 28.06.2022. - cancelled
by Lajos Katona 27 Jun '22

27 Jun '22

Hi, As I said in the last Team meeting (see [1]) I am OoO this Tuesday, so can't drive the coming meeting. [1]: https://meetings.opendev.org/meetings/networking/2022/networking.2022-06-21… Best wishes, and see you online. Lajos Katona (lajoskatona)

1 0

[neutron] Drivers meeting agenda - 24.06.2022.
by Lajos Katona 27 Jun '22

27 Jun '22

Hi Neutron Drivers, The agenda for tomorrow's drivers meeting is at [1]. * [RFE] Improve performance of bulk port create/delete with networking-generic-switch (#linkhttps:// bugs.launchpad.net/neutron/+bug/1976270) * Neutron RBAC not sharing subnet (#link https://bugs.launchpad.net/neutron/+bug/1975603) * [ovn]Floating IP adds distributed attributes (#link https://bugs.launchpad.net/neutron/+bug/1978039) * [RFE] Adding the "rekey" parameter to the api for strongswan like "dpd_action" (#link https://bugs.launchpad.net/neutron/+bug/1979044) [1] https://wiki.openstack.org/wiki/Meetings/NeutronDrivers#Agenda See you at the meeting tomorrow. Lajos Katona (lajoskatona)

2 2

[nova] NVLink passthrough
by Cory Hawkvelt 27 Jun '22

27 Jun '22

Hey team, Is anyone working with NVLink in their clouds? How are you handling passing through the right set of GPU's per NVLink 'group' I have servers with 4 sets of 2 way NVLinks(8 cards in pairs of 2) and I'm able to passthrough the PCI devices to the VM no problem but there is no guarantee that the NVLink pair get passed through together and if we end up with 1 GPU from one pair and 1 GPU from another pair then we run into all sorts of issues(As you'd expect) So I'm looking to understand how Nova can be NVLink aware I guess but I'm struggling to find any conversation ro material on the topic but I assume it's been done before? I did find this talk [1] which mentioned this problem but they write some sort of hack to sit in between nova and qemu, while quite a clever solution it seems like there must be a better way to do this in 2022? [1] - https://www.openstack.org/videos/summits/vancouver-2018/can-we-boost-more-h… Cheers, Cory

2 1

[cinder] Spec Freeze in effect
by Rajat Dhasmana 27 Jun '22

27 Jun '22

Hello Argonauts, As per the scheduled deadline[1], the cinder spec freeze is in effect. If you would like to continue working on your spec to get it merged and implemented in the Zed development cycle, please request a spec freeze exception as a new email to the Openstack Discuss mailing list[2] describing the reason for not making it into the deadline and hence it will be granted the spec freeze exception. The deadline for proposing the spec freeze exception is by 1st July, 2022 and the specs granted spec freeze exception should be merged by 8th July, 2022. Following are the specs that are targeted for the Zed cycle: 1) Spec to introduce additional task status field Patch: https://review.opendev.org/c/openstack/cinder-specs/+/818551 Needs an update from the author. 2) WIP: Add new volume transaction tracking Patch: https://review.opendev.org/c/openstack/cinder-specs/+/845176 Since this is a WIP, I'm not sure the intention is to get it into Zed specifically. 3) volume list query optimization Patch: https://review.opendev.org/c/openstack/cinder-specs/+/726070 Has a +2 but still lacks review from the main cinder core team. 4) Update original resource's az Patch: https://review.opendev.org/c/openstack/cinder-specs/+/778437 Hasn't been updated since 31st March so looks less likely to make it into Zed. 5) New Quota System Patch: https://review.opendev.org/c/openstack/cinder-specs/+/819693 Needs to be retargeted for Zed and updated as per PTG discussion. Also would like to request the cinder core team to take a look at the open specs. [1] https://releases.openstack.org/zed/schedule.html#z-cinder-spec-freeze [2] openstack-discuss(a)lists.openstack.org Thanks and regards Rajat Dhasmana

1 0

[neutron] bug deputy report for week of 2022-06-20
by Bence Romsics 27 Jun '22

27 Jun '22

Hi Neutron Team, Here comes the deputy report for last week: High: * https://bugs.launchpad.net/neutron/+bug/1979047 Interface attach fails with libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': File descriptor named '(null)' has not been found gate failure caused by libvirt bug in centos stream 9, affected tests blacklisted until centos fix arrives: https://review.opendev.org/c/openstack/neutron/+/846905 * https://bugs.launchpad.net/neutron/+bug/1979661 [stable/train] member_batch_update breaks contract with octavia-api interface fix merged: https://review.opendev.org/c/openstack/networking-ovn/+/847255 Medium: * https://bugs.launchpad.net/neutron/+bug/1979643 ``get_ports`` query done in ``check_baremetal_ports_dhcp_options`` cannot filter by VNIC type fix waiting for workflow+1: https://review.opendev.org/c/openstack/neutron/+/847346 Still in discussion with reporter: * https://bugs.launchpad.net/neutron/+bug/1979528 neutron dnsmasq keeps pushing the default gateway * https://bugs.launchpad.net/neutron/+bug/1979089 l3ha router delete race condition if state changes to master Cheers, Bence (rubasov)

1 0

Re: [Swift] [kolla] Swift issues in one cluster
by Albert Braden 27 Jun '22

27 Jun '22

We're running 2.23.3 but it appears that we are experiencing the bug (1). We tracked the problem down to a client who recently started using a java library to read large files from Swift. When he moves his activity to the other QA cluster, the problem follows. Am I guessing correctly that the bug was never fixed, and that (2) fixes a different problem? On Thursday, June 23, 2022, 10:43:44 AM EDT, Clay Gerrard <clay.gerrard(a)gmail.com> wrote: https://review.opendev.org/c/openstack/swift/+/575254 has been included in every released swift tag since 2.21.0 I believe Train included a swift version of at least 2.22 https://releases.openstack.org/train/#swift Nvidia doesn't use haproxy in front of our swift proxies, and we don't see BlockingIOError in tracebacks - the traceback might go away if you upgrade to the latest swift (2.29) and/or eventlet and/or python 3.8ish On Thu, Jun 23, 2022 at 8:49 AM Albert Braden <ozzzo(a)yahoo.com> wrote: Can anyone help with this Swift issue? It looks like we are being hit with this bug (1) but this bug doesn't seem to mention where/whether it was ever fixed. This (2) appears to be a fix, and it appears to have been merged, but it doesn't mention the bug, and it's not obvious to me what version it affects. Is anyone else encountering this problem? It appears that customers in this one cluster may be doing something to cause it; we're still trying to track down specifically what they are doing, that they aren't doing in the other clusters. We are running kolla-ansible Train on RHEL7. 1. https://bugs.launchpad.net/swift/+bug/1572719 2. https://opendev.org/openstack/swift/commit/0e81ffd1e1481a73146fce17f61f2ab9… On Wednesday, June 22, 2022, 05:10:10 PM EDT, Albert Braden <ozzzo(a)yahoo.com> wrote: Is this bug fixed in Train? https://bugs.launchpad.net/swift/+bug/1572719 On Tuesday, June 21, 2022, 09:26:47 AM EDT, Albert Braden <ozzzo(a)yahoo.com> wrote: All of our endpoints are https: +----------------------------------+--------+--------------+--------------+---------+-----------+---------------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+--------+--------------+--------------+---------+-----------+---------------------------------------------------------+ | <ID> | <region> | keystone | identity | True | internal | https://api-int.<region>.<domain>:5000 | | <ID> | <region>| swift | object-store | True | public | https://swift. <region>.<domain>:8080/v1/AUTH_%(tenant_id)s | | <ID> | <region>| swift | object-store | True | internal | https://swift.<region>.<domain>:8080/v1/AUTH_%(tenant_id)s | | <ID> | <region>| keystone | identity | True | admin | https://api-int. <region>.<domain>:35357 | | <ID> | <region>| keystone | identity | True | public | https://api-ext. <region>.<domain>:5000 | | <ID> | <region>| swift | object-store | True | admin | https://swift. <region>.<domain>:8080/v1 | +----------------------------------+--------+--------------+--------------+---------+-----------+---------------------------------------------------------+ I don't think this is causing the issue; all of our clusters are setup the same. We did think it was load at first, and got the 2 heaviest users to stop what they were doing, but that didn't make a difference. Our other QA cluster has similar load and identical hardware. When I look at the network graphs, I see traffic spiking up to 1G, but these are 25G interfaces, and none of the resources on the boxes are exhausted. CPU is 97% idle; memory is 30% used, disk is not full. It doesn't look like the problem is load-related. We see the haproxy connections stacking up even when load is low. What else could be causing this? On Friday, June 17, 2022, 11:12:36 PM EDT, Pete Zaitcev <zaitcev(a)redhat.com> wrote: On Fri, 17 Jun 2022 17:33:27 +0000 (UTC) Albert Braden <ozzzo(a)yahoo.com> wrote: > $ openstack container list > Unable to establish connection to https://swift.<region>.<domain>:8080/v1/AUTH_<project>: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer') Right away I have a question: why in the world are you connecting to 8080 with HTTPS? > (from Splunk): > Payload: swift-proxy-server: STDERR: File "/usr/lib64/python3.6/socket.py", line 604, in write#012 return self._sock.send(b) > Payload: swift-proxy-server: STDERR: BlockingIOError > Payload: swift-proxy-server: STDERR: os.read(self.rfd, 1) > Payload: swift-proxy-server: STDERR: File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 818, in process_request#012 proto.__init__(conn_state, self) This looks quite fishy to me, because the os.read is in swift/common/utils.py and it's responsible for the mutex. > When we look at network connections, we see haproxy stacking up (many lines of this): > > # netstat -ntup | sort -b -k2 -n -r | head -n +100 > tcp 5976932 0 127.0.0.1:60738 127.0.0.1:8080 ESTABLISHED 13045/haproxy > tcp 5976446 0 127.0.0.1:58480 127.0.0.1:8080 ESTABLISHED 13045/haproxy > tcp 5973217 0 127.0.0.1:33244 127.0.0.1:8080 ESTABLISHED 13045/haproxy > tcp 5973120 0 127.0.0.1:51836 127.0.0.1:8080 ESTABLISHED 13045/haproxy > tcp 5971968 0 127.0.0.1:58516 127.0.0.1:8080 ESTABLISHED 13045/haproxy > ... > > If we restart the swift_haproxy and swift_proxy_server containers then the problem goes away, and comes back over a few minutes. Where should we be looking for the root cause of this issue? Indeed if so many requests are established, you're in trouble. The best fix, I think, is to find the customer who's doing it and punish them. Otherwise, quotas and the ratelimiting middleware are your friends. There's also a possibility that your cluster is underperforming, although usually that results in 500 results first. But then again, at times users would "compensate" for issues by just launching way more requests, in effect DoS-ing the cluster even worse. -- Pete -- Clay Gerrard210 788 9431

2 5

[all] Failure to build (FTBFS) with Sphinx 5.0 and docutils 0.18
by Thomas Goirand 27 Jun '22

27 Jun '22

Hi, A number of bugs were filed against the Debian packages I maintain. Here's the list of bugs: https://bugs.debian.org/1013410 Nova https://bugs.debian.org/1013406 os-api-ref https://bugs.debian.org/1013402 openstacksdk https://bugs.debian.org/1013394 dogpile.cache https://bugs.debian.org/1013393 repoze.who https://bugs.debian.org/1013382 cloudkitty I'd appreciate a bit of help from the OpenStack project here, to fix these before Sphinx 5 and Docutils 0.18 are uploaded to Debian Unstable. If you do have a fix, a quick link in these bug reports would be lovely. Cheers, Thomas Goirand (zigo)

3 3

[all][horizon][django 4.0] Failure to build (FTBFS) with Django 4.0
by Thomas Goirand 27 Jun '22

27 Jun '22

Hi, A number of RC bugs were filed against my packages in Debian Unstable: https://bugs.debian.org/1013488 Horizon https://bugs.debian.org/1013475 django-pyscss https://bugs.debian.org/1013515 ironic-ui https://bugs.debian.org/1013605 django-formtools https://bugs.debian.org/1013603 enmerkar https://bugs.debian.org/1013506 ara If you have a fix, and can post the link to the patch in the Debian bug tracker, that'd be very much appreciated and help these packages not being auto-removed from Debian testing. Otherwise, you can always ping me on IRC. Cheers, Thomas Goirand (zigo)

2 2

[Ceph Rados Gateway] 403 when using S3 client
by Taltavull Jean-François 27 Jun '22

27 Jun '22

Hi All, I get an http 403 error code when I try to get the bucket list with Ubuntu (Focal) S3 client (awscli). S3 api has been activated in radosgw config file and EC2 credentials have been created and put in S3 client config file. Otherwise, everything is working fine with OpenStack client. My deployment: - OSA 23.2.0 - OpenStack Wallaby - Ceph and Rados GW Octopus Has any of you already experienced this kind of behaviour ? Many thanks, Jean-Francois

3 6

Re: [heat-translator][tacker] Propose Hiromu Asahina (h-asahina) for heat-translator core team
by Hiromu Asahina 27 Jun '22

27 Jun '22

Thank you for your warm feedback. I'll do my best :) By the way, me and Ueha-san who just joined the heat-translator core before me are not listed on heat-translator drivers [1]. I guess it means we don't have the permissions to control the heat-translator launchpad. Is it possible to add us to that list? Best Regards, Hiromu (h-asahina) [1] https://launchpad.net/~heat-translator-drivers/+members On 2022/06/27 13:42, Yasufumi Ogawa wrote: > > > > -------- Forwarded Message -------- > Subject: Re: [heat-translator][tacker] Propose Hiromu Asahina > (h-asahina) for heat-translator core team > Date: Fri, 24 Jun 2022 15:57:18 +0900 > From: Yasufumi Ogawa <yasufum.o(a)gmail.com> > To: Rico Lin <ricolin(a)ricolky.com> > CC: ueha.ayumu(a)fujitsu.com <ueha.ayumu(a)fujitsu.com>, > openstack-discuss(a)lists.openstack.org > <openstack-discuss(a)lists.openstack.org>, HADDLETON, Robert W (Bob) > <bob.haddleton(a)nokia.com>, Ghanshyam Mann <gmann(a)ghanshyammann.com> > > Thanks Rico and all! > > On 2022/06/24 5:03, Rico Lin wrote: >> Hi team >> >> Sorry for the late reply, just added Hiromu to the heat-translator-core. >> >> And welcome Hiromu. >> >> *Rico Lin* >> >> >> On Fri, Jun 24, 2022 at 2:30 AM Yasufumi Ogawa <yasufum.o(a)gmail.com >> <mailto:yasufum.o@gmail.com>> wrote: >> >> Hi heat-translator cores, >> >> We tacker team would like to keep contributing to maintain >> heat-translator, so appreciate if you agree to the proposal kindly. >> >> Cheers, >> Yasufumi >> >> On 2022/06/09 14:14, ueha.ayumu(a)fujitsu.com >> <mailto:ueha.ayumu@fujitsu.com> wrote: >> > Hi Rico, >> > >> > This is just a friendly reminder that we are waiting for your >> action. >> > Or please let us know if there are any other necessary processes. >> > Thanks. >> > >> > Best Regards, >> > Ueha >> > >> > -----Original Message----- >> > From: ueha.ayumu(a)fujitsu.com <mailto:ueha.ayumu@fujitsu.com> >> <ueha.ayumu(a)fujitsu.com <mailto:ueha.ayumu@fujitsu.com>> >> > Sent: Thursday, June 2, 2022 5:39 PM >> > To: openstack-discuss(a)lists.openstack.org >> <mailto:openstack-discuss@lists.openstack.org>; 'Rico Lin' >> <ricolin(a)ricolky.com <mailto:ricolin@ricolky.com>> >> > Subject: RE: [heat-translator][tacker] Propose Hiromu Asahina >> (h-asahina) for heat-translator core team >> > >> > Hi Rico, >> > >> > This proposal is great helpful for us! >> > There seems to be no particular objection, so could you add it to >> heat-translator-core if there is no problem? >> > Thanks. >> > >> > Best Regards, >> > Ueha >> > >> > -----Original Message----- >> > From: ueha.ayumu(a)fujitsu.com <mailto:ueha.ayumu@fujitsu.com> >> <ueha.ayumu(a)fujitsu.com <mailto:ueha.ayumu@fujitsu.com>> >> > Sent: Wednesday, May 25, 2022 1:40 PM >> > To: openstack-discuss(a)lists.openstack.org >> <mailto:openstack-discuss@lists.openstack.org> >> > Subject: RE: [heat-translator][tacker] Propose Hiromu Asahina >> (h-asahina) for heat-translator core team >> > >> > +1 >> > >> > Best Regards, >> > Ueha >> > >> > -----Original Message----- >> > From: Yoshito Ito <yoshito.itou.dr(a)hco.ntt.co.jp >> <mailto:yoshito.itou.dr@hco.ntt.co.jp>> >> > Sent: Wednesday, May 25, 2022 12:31 PM >> > To: openstack-discuss(a)lists.openstack.org >> <mailto:openstack-discuss@lists.openstack.org> >> > Subject: [heat-translator][tacker] Propose Hiromu Asahina >> (h-asahina) for heat-translator core team >> > >> > Hi heat-translator team! >> > >> > I would like to propose Hiromu Asahina (h-asahina) as a member of >> the heat-translator core team. I'm now not able to actively >> contribute to heat-translator, and he can lead the team instead >> of me. >> > >> > He is mainly contributing to Tacker by fixing issues, refactoring >> existing codes, and a lot of helpful reviews, with his experience as >> a research engineer in NTT. Tacker highly depends on heat-translator >> in its main functionality, so I believe he can manage it with his >> knowledge of Tacker. >> > >> > Please vote for his nomination by answering this mail till >> June 1st. >> > >> > Best Regards, >> > >> > Yoshito Ito >> > >> > >>

1 0

[magnum] [xena] [IPv6] Review Patch for dual-stack IPv4/6 work in Magnum-managed Kubernetes 1.21
by Namrata Sitlani 27 Jun '22

27 Jun '22

Hello All, We run release Xena, and we have deployed Kubernetes version 1.21.10 on Magnum with Fedora CoreOS version 34 and we are trying to have IPv6 support for that Kubernetes cluster, as Kubernetes 1.21 and later versions have dual-stack(IPv4/IPv6) support is enabled by default. To achieve that, we also need IPv6 support in magnum, which is not currently supported but there has been a patch for "Support IPv6 dual subnet" in a pending state for almost a year. Following is the patch link: https://review.opendev.org/c/openstack/magnum/+/802235 We are interested in the use case for dual-stack support in magnum. We would highly appreciate it if someone could help us land that patch. Kindly let us know if we can provide any additional information or help for the same. Thanks, Namrata Sitlani

1 0

[vmware-nsx] Missing releases and project status
by Takashi Kajinami 25 Jun '22

25 Jun '22

Hello, I recently noticed that the vmware-nsx repo doesn't have the stable/wallaby branch and the stable/yoga branch. Also, no release has been created since victoria release. There is a bug fix recently merged directly to stable/xena, and this was backported to ussuri and train with victoria skipped. https://review.opendev.org/c/x/vmware-nsx/+/836034 I'm quite confused with this because it looks like master and victoria are no longer maintained. May I ask for some clarification about the current status of the project, especially regarding the following points. - Is there any plan to create stable/wallaby and stable/yoga ? - Is there any plan to create a release for stable/wallaby and later ? - May I know the maintenance status of each branch ? - What is the future plan of that repo? Will you have stable/zed release ? I'm asking this because the plugin is supported by puppet-neutron but we are thinking of deprecating/removing the implementation if the plugin will be unmaintained. I'd appreciate your input on this. Thank you, Takashi

1 1

[all][tc] What's happening in Technical Committee: summary 24 June 2022: Reading: 5 min
by Ghanshyam Mann 25 Jun '22

25 Jun '22

Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * We had this week's meeting on 23 June. Most of the meeting discussions are summarized in this email. Meeting full logs are available @https://meetings.opendev.org/meetings/tc/2022/tc.2022-06-23-15.00.log.html * Next TC weekly meeting will be on 30 July Thursday at 15:00 UTC, feel free to add the topic on the agenda[1] by 29 July. 2. What we completed this week: ========================= * Defined "Emerging technology and inactive projects" framework[2]. Details can be found here[4]. Thanks to Slawek for working on this framework. A brief about this framework: 1. When a new project applies to be an official OpenStack project and they are not fully ready/doing things as OpenStack way. Still, they can be an official OpenStack with the agreement of filling the gap for OpenStack way. This framework will help such projects to have them early in the OpenStack community but also with the timeline of filling the gaps. (Skyline is a good example of such a project). 2. Inactive projects: If any project become inactive and no maintainer merges the things then this framework will help to detect such projects before the final release of work and give time to recover before we decide to retire them. This will avoid last-minute issues during the final release for example: what we had in the Yoga release[3] * Clarified "ATC" and "AC" terms in TC charter as per the bylaws perspective[5] * Added cinder ibm storwize charm repo[6] 3. Activities In progress: ================== TC Tracker for Zed cycle ------------------------------ * Zed tracker etherpad includes the TC working items[7], Two are completed and others items are in-progress. Open Reviews ----------------- * Four open reviews for ongoing activities[8]. Create the Environmental Sustainability SIG --------------------------------------------------- There is a proposal for 'Environmental Sustainability ' SIG[9]. Because of its wider scope at the OpenInfra projects level (not just OpenStack) we are discussing if it is a good idea to be an OpenStack SIG or something at the OpenInfra level like a working group under Board or so. New ELK service dashboard: e-r service ----------------------------------------------- Updates from dpawlik: "Changes in ci log processing: created account on docker hub for ci log processing project (credentials are in AWS secret service); enabled pushing logscraper and log gearman container images into the docker hub." Ananya working on integrating the Elastic recheck with Upstream opensearch[10] Dasm working on merging 'origin/rdo' into the merge branch[11] Consistent and Secure Default RBAC ------------------------------------------- We discussed the operator feedback in the policy popup meeting on Tuesday 21 June[12] where operators from KDDI joined and wrote their feedback in etherpad[13]. After considering the feedback and all the challenges we have with 'scope', we decided to drop the 'scope' implementation and work on implementing the 'project persona'. I have proposed the goal document updates[14] 2021 User Survey TC Question Analysis ----------------------------------------------- No update on this. The survey summary is up for review[15]. Feel free to check and provide feedback. Zed cycle Leaderless projects ---------------------------------- No updates on this. Only Adjutant project is leaderless/maintainer-less. We will check Adjutant's the situation again on ML and hope Braden will be ready with their company side permission[16]. Fixing Zuul config error ---------------------------- Requesting projects with zuul config error to look into those and fix them which should not take much time[17][18]. Project updates ------------------- * Add Cinder Dell EMC PowerStore charm[19] * Retire openstack-helm-deployments[20] 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[21]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [22] 3. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://review.opendev.org/c/openstack/governance/+/839880 [3] http://lists.openstack.org/pipermail/openstack-discuss/2022-March/027864.ht… [4] https://governance.openstack.org/tc/reference/emerging-technology-and-inact… [5] https://review.opendev.org/c/openstack/governance/+/844882 [6] https://review.opendev.org/c/openstack/governance/+/846900 [7] https://etherpad.opendev.org/p/tc-zed-tracker [8] https://review.opendev.org/q/projects:openstack/governance+status:open [9] https://review.opendev.org/c/openstack/governance-sigs/+/845336 [10] https://review.opendev.org/c/opendev/elastic-recheck/+/845777 [11] https://review.opendev.org/c/opendev/elastic-recheck/+/847405/ [12] https://etherpad.opendev.org/p/rbac-zed-ptg#L171 [13] https://etherpad.opendev.org/p/rbac-operator-feedback#L45 [14] https://review.opendev.org/c/openstack/governance/+/847418 [15] https://review.opendev.org/c/openstack/governance/+/836888 [16] http://lists.openstack.org/pipermail/openstack-discuss/2022-March/027626.ht… [17] https://etherpad.opendev.org/p/zuul-config-error-openstack [18] http://lists.openstack.org/pipermail/openstack-discuss/2022-May/028603.html [19] https://review.opendev.org/c/openstack/governance/+/846890 [20] https://review.opendev.org/c/openstack/governance/+/847413 [21] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [22] http://eavesdrop.openstack.org/#Technical_Committee_Meeting -gmann

1 0

[oslo] Propose to EOL stable/queens, stable/rocky on all the oslo scope
by Herve Beraud 24 Jun '22

24 Jun '22

Hi, On our last meeting of the oslo team we discussed the problem with broken stable branches (rocky and older) in oslo's projects [1]. Indeed, almost all these branches are broken. Előd Illés kindly generated a list of periodic-stable errors on Oslo's stable branches [2]. Given the lack of active maintainers on Oslo and given the current status of the CI in those branches, I propose to make them End Of Life. I will wait until the end of month for anyone who would like to maybe step up as maintainer of those branches and who would at least try to fix CI of them. If no one will volunteer for that, I'll EOLing those branches for all the projects under the oslo umbrella. Let us know your thoughts. Thank you for your attention. [1] https://meetings.opendev.org/meetings/oslo/2021/oslo.2021-10-04-15.00.log.t… [2] http://lists.openstack.org/pipermail/openstack-discuss/2021-July/023939.html -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

5 5

ovn-controller/OVS stranger behaviour
by Tiago Pires 24 Jun '22

24 Jun '22

Hi all, I'm trying to understand a stranger's behaviour regarding to ovn-controller/OVS. In my setup I have OVN 21.09/ OVS 2.16 and Ubuntu Xena and sometimes when a new VM is created, this VM can reach other VMs in east-west traffic (even in differents Chassis) but it can't reach an external network (e.g. Internet) through Chassi Gateway. I ran the following trace: # ovs-appctl ofproto/trace br-int in_port="93",icmp,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_ttl=64 And I got this output: Final flow: recirc_id=0xc157b1,eth,icmp,reg0=0x300,reg11=0xd,reg12=0x10,reg13=0xf,reg14=0x3,reg15=0x2,metadata=0x29,in_port=93,vlan_tci=0x0000,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0 Megaflow: recirc_id=0xc157b1,ct_state=+new-est-rel-rpl-inv+trk,ct_label=0/0x1,eth,icmp,in_port=93,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src= 192.168.40.128/26,nw_dst=8.0.0.0/7,nw_ttl=64,nw_frag=no Datapath actions: ct(commit,zone=15,label=0/0x1,nat(src)),set(eth(src=fa:16:3e:ec:7f:dd,dst=00:00:00:00:00:00)),set(ipv4(ttl=63)),userspace(pid=3451843211,controller(reason=1,dont_send=1,continuation=0,recirc_id=12670898,rule_cookie=0x3e26215e,controller_id=0,max_len=65535)) It seems the Datapath is querying the controller and I did not understand the reason. So, I did an ovn-controller recompute (ovn-appctl -t ovn-controller recompute) on the Chassi where the VM is placed to check if it could change the behaviour and I could trace the packet with success and the VM started to communicate with the Internet normally: Final flow: recirc_id=0x2,eth,icmp,reg0=0x300,reg11=0xd,reg12=0x10,reg13=0xf,reg14=0x3,reg15=0x2,metadata=0x29,in_port=93,vlan_tci=0x0000,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src=192.168.40.140,nw_dst=8.8.8.8,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=0,icmp_code=0 Megaflow: recirc_id=0x2,ct_state=+new-est-rel-rpl-inv+trk,ct_label=0/0x1,eth,icmp,tun_id=0/0xffffff,tun_metadata0=NP,in_port=93,dl_src=fa:16:3e:26:34:ef,dl_dst=fa:16:3e:65:68:6e,nw_src= 192.168.40.128/26,nw_dst=8.0.0.0/7,nw_ecn=0,nw_ttl=64,nw_frag=no Datapath actions: ct(commit,zone=15,label=0/0x1,nat(src)),set(tunnel(tun_id=0x2a,dst=10.X6.X3.133,ttl=64,tp_dst=6081,geneve({class=0x102,type=0x80,len=4,0x30002}),flags(df|csum|key))),set(eth(src=fa:16:3e:ec:7f:dd,dst=00:00:5e:00:04:00)),set(ipv4(ttl=63)),2 The Datapath action is using the tunnel with the Chassi Gateway. It happens always with new VMs but sometimes. After running the recompute on the Chassi, I created additional VMs and this issue did not happen. In my Chassi I have enable these parameters also: ovn-monitor-all="true" ovn-openflow-probe-interval="0" ovn-remote-probe-interval="180000" I did some troubleshooting and I'm seeing this error (ovs-vswitchd) always when a VM is created in a Chassi: 2022-06-23T11:47:08.385Z|07907|bridge|WARN|could not open network device tap8a43df0c-fd (No such device) 2022-06-23T11:47:09.282Z|07908|bridge|INFO|bridge br-int: added interface tap8a43df0c-fd on port 51 2022-06-23T11:47:09.645Z|07909|bridge|INFO|bridge br-int: added interface tap3200bf1c-20 on port 52 2022-06-23T11:47:19.329Z|07911|connmgr|INFO|br-int<->unix#1468: 430 flow_mods in the 7 s starting 10 s ago (410 adds, 20 deletes) On this commit http://patchwork.ozlabs.org/project/ovn/patch/1608197000-637-1-git-send-ema… it solved something similar to my issue. It seems the ovs-vswitchd is missing some flows and when I run the recompute it fixes it. So, in order to avoid this issue I'm testing at this moment to run the recompute through libvirt hook when a VM gets "started" status. Do you know this behaviour could be bug related? Regards, Tiago Pires Do you know this behaviour could be bug related?

2 2

[heat-translator][tacker] Propose Hiromu Asahina (h-asahina) for heat-translator core team
by Yoshito Ito 24 Jun '22

24 Jun '22

Hi heat-translator team! I would like to propose Hiromu Asahina (h-asahina) as a member of the heat-translator core team. I'm now not able to actively contribute to heat-translator, and he can lead the team instead of me. He is mainly contributing to Tacker by fixing issues, refactoring existing codes, and a lot of helpful reviews, with his experience as a research engineer in NTT. Tacker highly depends on heat-translator in its main functionality, so I believe he can manage it with his knowledge of Tacker. Please vote for his nomination by answering this mail till June 1st. Best Regards, Yoshito Ito

7 10

[ER] Revive elastic recheck
by Dariusz Smigiel 23 Jun '22

23 Jun '22

Hey Team! I started cherry picking changes from the rdo branch back to the master branch for Elastic Recheck. [1] I'm planning to continue doing so, to revive the master branch and continue development in the main branch. Some of the changes are incompatible between releases. We're gonna clear them case by case to enable it in the future. Thanks, Darius [1]: https://review.opendev.org/c/opendev/elastic-recheck/+/847259

1 1

[all][tc] Technical Committee next weekly meeting on 23 June 2022 at 1500 UTC
by Ghanshyam Mann 23 Jun '22

23 Jun '22

Hello Everyone, The technical Committee's next weekly meeting is scheduled for 23 June 2022, at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, 22 June at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

Swift issues in one cluster
by Albert Braden 23 Jun '22

23 Jun '22

We have multiple Swift clusters, all configured the same. One of them started failing this week. The symptom is that Swift commands take a long time to execute and sometimes they fail: $ openstack container list Unable to establish connection to https://swift.<region>.<domain>:8080/v1/AUTH_<project>: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer') When we look at the logs we see lots of swift-proxy-server errors: (from Splunk): Payload: swift-proxy-server: STDERR: File "/usr/lib64/python3.6/socket.py", line 604, in write#012 return self._sock.send(b) Payload: swift-proxy-server: STDERR: BlockingIOError Payload: swift-proxy-server: STDERR: os.read(self.rfd, 1) Payload: swift-proxy-server: STDERR: File "/usr/lib/python3.6/site-packages/eventlet/wsgi.py", line 818, in process_request#012 proto.__init__(conn_state, self) Payload: swift-proxy-server: STDERR: File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 397, in send Payload: swift-proxy-server: STDERR: return self._sock.send(b) When we look at network connections, we see haproxy stacking up (many lines of this): # netstat -ntup | sort -b -k2 -n -r | head -n +100 tcp 5976932 0 127.0.0.1:60738 127.0.0.1:8080 ESTABLISHED 13045/haproxy tcp 5976446 0 127.0.0.1:58480 127.0.0.1:8080 ESTABLISHED 13045/haproxy tcp 5973217 0 127.0.0.1:33244 127.0.0.1:8080 ESTABLISHED 13045/haproxy tcp 5973120 0 127.0.0.1:51836 127.0.0.1:8080 ESTABLISHED 13045/haproxy tcp 5971968 0 127.0.0.1:58516 127.0.0.1:8080 ESTABLISHED 13045/haproxy ... If we restart the swift_haproxy and swift_proxy_server containers then the problem goes away, and comes back over a few minutes. Where should we be looking for the root cause of this issue?

2 4

[TripleO] Proposing Brendan Shephard and Takashi Kajinami
by James Slagle 23 Jun '22

23 Jun '22

I'm formally proposing that we add Brendan Shephard and Takashi Kajinami to the core reviewers team for TripleO. Takashi is already considered core on puppet-tripleo, and I'm proposing we expand that to all of TripleO, and likewise for Brendan. They both have been contributing and reviewing for a little while now, and I feel it's time to add them to the team. Please +1 or let me know any concerns before Friday June 15th. Thanks! -- -- James Slagle --

15 15

[TripleO] Gate blocker - tempest test error - 'Failed to attach network adapter device'
by Ronelle Landy 23 Jun '22

23 Jun '22

Hello All, We have a new gate blocker on tripleo-ci-centos-9-standalone, scenario-010 and possibly other jobs. The jobs fail tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_hotplug_nic Details of the failure are in: https://bugs.launchpad.net/tripleo/+bug/1978969. We will reply to this thread as soon as we have more information and/or a fix. Please hold rechecks for now, Thank you, TripleO CI

1 2

Re: [kolla-ansible][xena] Facing network issue when providing SR-IOV support
by Rodolfo Alonso Hernandez 23 Jun '22

23 Jun '22

Hello Harshit: You need first to fix your underlying network configuration. If the interfaces created on "sriovnet1" can't talk to each other this is because your network is blocking this traffic. Check your TOR switches to enable/allow this VLAN range. Regards. On Thu, Jun 23, 2022 at 9:42 AM Harshit Mishra <harshit(a)voereir.com> wrote: > Hello Rodolfo, > > Thanks for your support. > > If the VMs are spawned without any issue, that could be a problem in the > underlying network deployment. When sending traffic from a VM with an > SR-IOV port, try to capture the traffic in the physical function. Please > check the physical network connected to the SR-IOV network cards allow VLAN > traffic in 1000:1009 (according to your configuration). > > Network I have configured of VLAN type in a given range. > > openstack network show 4cacd198-29c3-47b3-bf77-39f1d01c9a22 -c > provider:network_type -c provider:physical_network -c > provider:segmentation_id > +---------------------------+-----------+ > | Field | Value | > +---------------------------+-----------+ > | provider:network_type | vlan | > | provider:physical_network | sriovnet1 | > | provider:segmentation_id | 1002 | > +---------------------------+-----------+ > > > Note: I forget to add that Intra-communication between the VM's working > fine deployed on same HOST but inter-communication not working. > One more thing IP address is not assigning to interfaces of VM. We have to > configure static IP address manually. > > Please find below incoming traffic flow on PF. > > > # tcpdump -i enp8s0f1 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on enp8s0f1, link-type EN10MB (Ethernet), capture size 262144 > bytes > 12:39:28.584000 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, > Request from fa:16:3e:88:f5:3e (oui Unknown), length 285 > 12:39:39.183491 IP6 fe80::a236:9fff:fe24:803a > ip6-allrouters: ICMP6, > router solicitation, length 16 > 12:39:49.660269 IP6 fe80::a236:9fff:fe53:e598.5240 > ff02::15a.5240: UDP, > length 221 > 12:39:49.660897 IP6 fe80::a236:9fff:fe53:e59a.5240 > ff02::15a.5240: UDP, > length 221 > 12:39:54.105381 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, > Request from a0:36:9f:d8:0d:a8 (oui Unknown), length 293 > 12:39:54.897051 LLDP, length 46 > 12:39:56.281671 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, > Request from 3c:fd:fe:cd:06:71 (oui Unknown), length 293 > 12:40:06.286310 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, > Request from a0:36:9f:d8:0d:aa (oui Unknown), length 293 > 12:40:15.039416 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, > Request from 3c:fd:fe:cd:06:70 (oui Unknown), length 293 > 12:40:19.675902 IP6 fe80::a236:9fff:fe53:e598.5240 > ff02::15a.5240: UDP, > length 221 > 12:40:19.676390 IP6 fe80::a236:9fff:fe53:e59a.5240 > ff02::15a.5240: UDP, > length 221 > 12:40:25.708972 LLDP, length 46 > 12:40:29.288242 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, > Request from fa:16:3e:c4:3f:4f (oui Unknown), length 285 > 12:40:33.008653 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, > Request from fa:16:3e:88:f5:3e (oui Unknown), length 285 > > > > Another question: do the VMs with SR-IOV port receive the DHCP reply? If > so, at least you know the compute to controller communication is working > (assuming the DHCP agent is on the controller). > > No, VMs not receiving DHCP reply, you can refer above logs. Yes, DHCP > agent is on controller. > > Kindly let me know anything else is required. > > Thanks! > > Regards, > Harshit Mishra > > ------------------------------ > *From:* Rodolfo Alonso Hernandez <ralonsoh(a)redhat.com> > *Sent:* Wednesday, June 22, 2022 6:46 PM > *To:* Harshit Mishra <harshit(a)voereir.com> > *Subject:* Re: [kolla-ansible][xena] Facing network issue when providing > SR-IOV support > > Hello Harshit: > > If the VMs are spawned without any issue, that could be a problem in the > underlying network deployment. When sending traffic from a VM with an > SR-IOV port, try to capture the traffic in the physical function. Please > check the physical network connected to the SR-IOV network cards allow VLAN > traffic in 1000:1009 (according to your configuration). > > Another question: do the VMs with SR-IOV port receive the DHCP reply? If > so, at least you know the compute to controller communication is working > (assuming the DHCP agent is on the controller). > > Regards. > > On Wed, Jun 22, 2022 at 2:17 PM Harshit Mishra <harshit(a)voereir.com> > wrote: > > Hi! > > > I have deployed Openstack Xena using kolla-ansible (v13.0.1) on a > multi-node setup (1 controller+network, multiple computes). > > > > On this cluster I would like to have support for normal, as well as direct > (SR-IOV) vNIC types. > > > > I have done all the pre-requirement like configure VF on SR-IOV network > interface. > > > > In the current deployment, I have created two physnets, one for flat > network (called physnet1 on br-ex), and one for SR-IOV (called sriovnet1 on > Intel 10G 2P X520 card). I am creating one network of VXLAN type for normal > vNIC, and one of VLAN type on sriovnet1 (called sriov_network) for direct > vNIC. > > > > Mapping of PF with provider in sriov-agent.ini: > > > > # cat /etc/kolla/neutron-sriov-agent/sriov_agent.ini > > [sriov_nic] > > physical_device_mappings = sriovnet1:enp8s0f1 > > exclude_devices = > > > > [securitygroup] > > firewall_driver = neutron.agent.firewall.NoopFirewallDriver > > > > ---------------------------------------------- > > ml2_conf.ini on controller node: > > > > # cat /etc/kolla/neutron-server/ml2_conf.ini > > [ml2] > > type_drivers = flat,vlan,vxlan > > tenant_network_types = vxlan,vlan,flat > > mechanism_drivers = openvswitch,l2population,sriovnicswitch > > extension_drivers = port_security > > > > [ml2_type_vlan] > > network_vlan_ranges = physnet1,sriovnet1:1000:1009 > > > > [ml2_type_flat] > > flat_networks = physnet1,sriovnet1 > > > > [ml2_sriov] > > agent_required = False > > supported_pci_vendor_devs = 8086:10ed > > > > [ml2_type_vxlan] > > vni_ranges = 1:1000 > > > > ---------------------------------------------- > > > > # grep -nr enabled_filters /etc/kolla/nova-scheduler/ > > /etc/kolla/nova-scheduler/nova.conf:13:enabled_filters = > ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,SameHostFilter,DifferentHostFilter,AggregateInstanceExtraSpecsFilter,PciPassthroughFilter > > > > ---------------------------------------------- > > > > # grep passthrough /etc/kolla/nova-compute/nova.conf > > passthrough_whitelist = [{"physical_network": "sriovnet1", "devname": > "enp8s0f1"}] > > > > > > The interfaces on VMs from normal vNICs are able to communicate > inter-compute. The same is not working for the direct vNICs belonging to > the sriov_network. I have tried multiple config changes, but none seem to > work. > > > > Please help in solving this issue, or suggest a different way of achieving > this if this is wrong or not optimal. > > > > Thanks! > > Regards, > Harshit Mishra > >

1 0

Re: 504 Gateway time out error while loading OpenStack Projects
by Eugen Block 23 Jun '22

23 Jun '22

Please keep the responses in the list so others can see your responses, too, and chime in with their comments. How is your dashboard configured regarding cache? This is our config (/srv/www/openstack-dashboard/openstack_dashboard/local/local_settings.d/_100_local_settings.py): CACHES = { 'default': { 'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache', 'LOCATION': [ '<IP1>:11211', '<IP2>:11211', ] } } Our memcached config has both localhost and its management IP: MEMCACHED_PARAMS="-U 0 -m 64 -l 127.0.0.1,<IP1> -p 11211 -c 4096" The reason for that is we noticed performance improvements when we replaced the list of control nodes for all the services and point to localhost instead: # before #memcached_servers = controller01:11211,controller02:11211 # after memcached_servers = localhost:11211 Zitat von Adivya Singh <adivya1.singh(a)gmail.com>: > Hello, > > My setting in memcache looks like this, because it is default taken by > ansible > > -d > -v > -U 0 > -m 1024 > -p 11211 > -u memcache > -l 172.29.x.X > -c 4096 > -t 4 > > So should i change the time out perimeter to something else > > Regards > Adivya Singh > > On Wed, Jun 15, 2022 at 8:53 PM Maysa De Macedo Souza <mdemaced(a)redhat.com> > wrote: > >> Hello, >> >> Maybe increasing the value of a setting like >> memcache_pool_conn_get_timeout can help? >> >> Cheers, >> Maysa. >> >> On Wed, Jun 15, 2022 at 2:05 PM Eugen Block <eblock(a)nde.ag> wrote: >> >>> Hi, >>> >>> I remember having a similar issue a few years back in an older >>> openstack release, probably Ocata. We had difficulties loading *all* >>> instances in the admin tab in Horizon dashboard. It turned out to be a >>> memcached misconfiguration. I don't recall the details, unfortunately. >>> But I'd start with memcached. >>> >>> >>> Zitat von Adivya Singh <adivya1.singh(a)gmail.com>: >>> >>> > hi Team, >>> > >>> > We have Xena release installed on Ubuntu Openstack, Lately we are >>> getting >>> > "504 Gateway Timeout error ", As the we moved from 1 node to 3 node Open >>> > Stack >>> > i do not see any error in haproxy , neither there is a time out when i >>> ping >>> > external-VIP-IP configured. >>> > >>> > Apache 2 are also loaded, tried to delete the Horizon Container and >>> created >>> > it again, but no errors, Still the same errors. >>> > >>> > Any guesses on that >>> > >>> > Regards >>> > Adivya Singh >>> >>> >>> >>> >>>

1 0