February 2022 - openstack-discuss

[neutron][ci] Agenda for the CI meeting - Tuesday 15.02.2022
by Slawek Kaplonski 15 Feb '22

15 Feb '22

Hi, Agenda for today's CI meeting is at [1]. Please remember that our today's meeting will be video meeting on [2]. See You there at 1500 UTC today. [1] https://etherpad.opendev.org/p/neutron-ci-meetings [2] https://meetpad.opendev.org/neutron-ci-meetings -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[election][tacker][ptl] PTL candidacy for Z
by Yasufumi Ogawa 15 Feb '22

15 Feb '22

Hi, I'd like to propose my candidacy for Tacker PTL in Z cycle. I have been a PTL from Victoria and made an effort to implement the latest ETSI-NVF standard features to enable operators to deploy their services. In Yoga release, we are going to provide several container supports for deploying CNFs, multi-version APIs for mixed environment of legacy and cutting edge-products provided by several vendors and more. We also had several collaborative sessions with ETSI NFV for accelerating each activity of standardization and deployment of required features. In Z release, I would like to expand our target to VNFs for network acceleration other than proceed to implement VM/container management features which has been introduced in the previous releases. It has became one of the key requirements of the recent commercial use cases such as 5GC. Best regards, Yasufumi (yasufum)

1 0

[manila][ptl] Non-candidacy for the Z cycle
by Goutham Pacha Ravi 15 Feb '22

15 Feb '22

Hey Zorillas and other interested Stackers, I took over as Manila PTL prior to the Ussuri cycle and have had fun running the project for 5 cycles. The itch for change hasn't subsided. So, I'd love to hand off the PTL baton and go back to being a maintainer (or whatever else you'll let my pompous behind claim to be). Despair not though, there's someone that has bright ideas and energy to enhance this role that intends to nominate themselves to lead us through the Zorilla (Zed) cycle! I'm very proud that I had your encouragement and support for the past ~three years. We innovated and incorporated some fun practices that held the community together and grew it over time. We chipped away at tech debt in an organized fashion, created focus teams, added half a dozen core reviewers, mentored 20 (!) interns, squashed bugs, put together hackfests, collaborated on reviews and somehow made the pandemic-caused virtual project team gatherings fun as well. We will do so much more of this with the new PTL and the hyper-charged zorilla spirit :D So I'm very excited. Thank you for your support! -- Goutham Pacha Ravi

2 1

[ironic][election][ptl] PTL Candidacy for Zed Cycle
by Iury Gregory 15 Feb '22

15 Feb '22

Hello, Ironicers and stackers! I would like to submit my candidacy to continue serving as Ironic PTL for the Zed cycle. In the Yoga cycle, we are adding features to facilitate the cloud operator's life (e.g., Ironic now can calculate the default value for any enabled interface based on the `enabled_hardware_type`, verify steps). We fixed bugs related to SW Raid, virtual media deployments, and many others! Let's continue working together so we can take over the world! =) Best regards, Iury Gregory Melo Ferreira IRC: iurygregory -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Part of the ironic-core and puppet-manager-core team in OpenStack* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory(a)gmail.com <iurygregory(a)gmail.com>*

1 0

[elections][ptl][masakari]
by Sam Su (苏正伟) 15 Feb '22

15 Feb '22

Hi all, I write to submit my candidacy for the Masakari PTL position during the Z cycle. I took over as Masakari PTL during the Yoga cycle, and would like to serve the community for another cycle. Right now the biggest challenge of the Masakari project is lack of activity. We have been using Masakari in our cloud environment for years, so I want to continue to make some little improvement as feedback. Best regards for you. -suzhengwei

1 0

[election][Sahara]PTL candidacy for Z
by Juntingqiu Qiujunting (邱军婷) 15 Feb '22

15 Feb '22

Hi all: I write to submit my candidacy for the Sahara PTL position during the Z cycle.I have been PTL for the Yoga cycle, and I would like to serve the community for another cycle. In yoga cycle, we have discuss about the plugins are CDH, MapR, Storm, Spark and Vanilla will be removed from Sahara. I will continue this work on Z cycle. And assist the community to resolve some inappropriate issues in Yoga about Sahara project. At present, the Sahara project is not very active, and welcome my dears to actively participate in the construction of the Sahara project in Z cycle. Finally, expect the Sahara project to develop healthily and stably. Best regards for you. Fossen Qiu (fossenqiu) --------------------------------- Fossen Qiu | 邱军婷 CBRD | 云计算与大数据研发部 T: 18249256272 E: qiujunting(a)inspur.com<mailto:qiujunting@inspur.com> 浪潮云海

1 0

[election][puppet] PTL Candidacy for the "Z" release
by Takashi Kajinami 15 Feb '22

15 Feb '22

Hello, I'd like to announce my candidacy for the PTL role in Puppet OpenStack, for the Z release cycle. I've been involved in the project for a few years. I joined the core reviewers of the project in 2020. My contribution started with adding a few parameters but nowdays I'm covering literally all modules to keep these modules consistent with the latest service implementation, as well as helping release management. I'll list up some important items for the upcoming Z-release, which I'd like to prioritize. - Keep our implementation up to date and clean up old items to keep codes as simple as possible. - Extend support for Secure RBAC. We are adding partial support to enforce Secure RBAC in Keystone during Yoga. - Stabilize support for CentOS 9 Stream, which we are adding for Yoga. - Add support for the new Ubuntu LTS. - Improve scenario/component coverage by CI. Finally, as we are currently a small team, we always welcome new contributors, reviewers or whoever interested in helping the project ! Thank you for your consideration. Thank you, Takashi

1 0

[election][storlets] PTL Candidacy for the "Z" release
by Takashi Kajinami 15 Feb '22

15 Feb '22

Hi All, I'd like to announce my candidacy for PTL of Storlets for the z release cycle, continuing the role from past several cycles. The biggest change we're expecting for this cycle is removal of Python 2 support, following the same in Swift, which was recently announced[1]. We don't expect any challenges at this moment as Storlets has been fully complatble with Python 3 for several cycles, but the remaining work to clean up items left to keep Python 2 support (Python 2 jobs, for example) is our priority. [1] http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026925.… Thank you for your consideration. Thank you, Takashi

1 0

[all][tc] Relmgt team position on release cadence
by Thierry Carrez 15 Feb '22

15 Feb '22

Hi everyone, The (long) document below reflects the current position of the release management team on a popular question: should the OpenStack release cadence be changed? Please note that we only address the release management / stable branch management facet of the problem. There are other dimensions to take into account (governance, feature deprecation, supported distros...) to get a complete view of the debate. Introduction ------------ The subject of how often OpenStack should be released has been regularly debated in the OpenStack community. OpenStack started with a 3-month release cycle, then switched to 6-month release cycle starting with Diablo. It is often thought of a release management decision, but it is actually a much larger topic: a release cadence is a trade-off between pressure to release more often and pressure to release less often, coming in from a lot of different stakeholders. In OpenStack, it is ultimately a Technical Committee decision. But that decision is informed by the position of a number of stakeholders. This document gives historical context and describes the current release management team position. The current trade-off --------------------- The main pressure to release more often is to make features available to users faster. Developers get a faster feedback loop, hardware vendors ensure software is compatible with their latest products, and users get exciting new features. "Release early, release often" is a best practice in our industry -- we should generally aim at releasing as often as possible. But that is counterbalanced by pressure to release less often. From a development perspective, each release cycle comes with some process overhead. On the integrators side, a new release means packaging and validation work. On the users side, it means pressure to upgrade. To justify that cost, there needs to be enough user-visible benefit (like new features) in a given release. For the last 10 years for OpenStack, that balance has been around six months. Six months let us accumulate enough new development that it was worth upgrading to / integrating the new version, while giving enough time to actually do the work. It also aligned well with Foundation events cadence, allowing to synchronize in-person developer meetings date with start of cycles. What changed ------------ The major recent change affecting this trade-off is that the pace of new development in OpenStack slowed down. The rhythm of changes was divided by 3 between 2015 and 2021, reflecting that OpenStack is now a mature and stable solution, where accessing the latest features is no longer a major driver. That reduces some of the pressure for releasing more often. At the same time, we have more users every day, with larger and larger deployments, and keeping those clusters constantly up to date is an operational challenge. That increases the pressure to release less often. In essence, OpenStack is becoming much more like a LTS distribution than a web browser -- something users like moving slow. Over the past years, project teams also increasingly decoupled individual components from the "coordinated release". More and more components opted for an independent or intermediary-released model, where they can put out releases in the middle of a cycle, making new features available to their users. This increasingly opens up the possibility of a longer "coordinated release" which would still allow development teams to follow "release early, release often" best practices. All that recent evolution means it is (again) time to reconsider if the 6-month cadence is what serves our community best, and in particular if a longer release cadence would not suit us better. The release management team position on the debate -------------------------------------------------- While releasing less often would definitely reduce the load on the release management team, most of the team work being automated, we do not think it should be a major factor in motivating the decision. We should not adjust the cadence too often though, as there is a one-time cost in switching our processes. In terms of impact, we expect that a switch to a longer cycle will encourage more project teams to adopt a "with-intermediary" release model (rather than the traditional "with-rc" single release per cycle), which may lead to abandoning the latter, hence simplifying our processes. Longer cycles might also discourage people to commit to PTL or release liaison work. We'd probably need to manage expectations there, and encourage more frequent switches (or create alternate models). If the decision is made to switch to a longer cycle, the release management team recommends to switch to one year directly. That would avoid changing it again anytime soon, and synchronizing on a calendar year is much simpler to follow and communicate. We also recommend announcing the change well in advance. We currently have an opportunity of making the switch when we reach the end of the release naming alphabet, which would also greatly simplify the communications around the change. Finally, it is worth mentioning the impact on the stable branch work. Releasing less often would likely impact the number of stable branches that we keep on maintaining, so that we do not go too much in the past (and hit unmaintained distributions or long-gone dependencies). We currently maintain releases for 18 months before they switch to extended maintenance, which results in between 3 and 4 releases being maintained at the same time. We'd recommend switching to maintaining one-year releases for 24 months, which would result in between 2 and 3 releases being maintained at the same time. Such a change would lead to longer maintenance for our users while reducing backporting work for our developers. -- Thierry Carrez (ttx) On behalf of the OpenStack Release Management team

15 45

[election][tc] Candidacy for TC for Z
by Nikolla, Kristi 14 Feb '22

14 Feb '22

Hi all, I am announcing my candidacy for a position on the OpenStack Technical Committee. I'm a software engineer at the Mass Open Cloud[0], and the New England Research Cloud[1], two initiatives from major universities in the Boston area to create and operate public and open clouds that can federated and replicated. I've been involved with OpenStack for the past 7 years as a user, operator, and developer. This is not my first time running, I have previously served on the Technical Committee[2] and as Keystone PTL for 1 year during the Victoria and Wallaby cycles[3][4]. I am passionate about open source and governance and keeping the community healthy. My focus will be in facilitating the discussions that need to happen, whether they are internal to a team, between project teams, or between OpenStack and other projects. Please allow me once again the honor of representing you, and the opportunity to continue contributing to the community that I owe so much to and want to see flourish. I look forward to hopefully seeing you all very soon! Thank you for your time and consideration, Kristi Nikolla (he/him) [0]. https://massopen.cloud [1]. https://nerc.mghpcc.org [2]. https://opendev.org/openstack/election/raw/branch/master/candidates/victori… [3]. https://opendev.org/openstack/election/raw/branch/master/candidates/victori… [4]. https://opendev.org/openstack/election/raw/branch/master/candidates/wallaby…

1 0

[puppet][tc] Propose changing the release model to cycle-with-rc
by Takashi Kajinami 14 Feb '22

14 Feb '22

Hello, I already discussed this with a few cores last year but I'm raising this topic in ML to make an official decision. Currently Puppet OpenStack is following cycle-with-intermediary and creates a release every milestone. However our code is tightly related to the actual service implementations and having only puppet releases is not very useful. Considering the above point, and effort required to cut off releases per milestone, I'll propose changing our release model to cycle-with-rc , and creating a single release. Because we already created milestone releases for Yoga, I'm thinking of applying the change from next cycle(Z). Please let me know if you have any concerns. Thank you, Takashi

5 5

[blazar][election][ptl] PTL candidacy for Z
by Pierre Riteau 14 Feb '22

14 Feb '22

Hi, I would like to self-nominate for the role of PTL of Blazar for the Z release cycle. I have been PTL since the Stein cycle and I am willing to continue in this role. We have experienced an uptick in contributions during the Yoga cycle, merging some long-awaited features such as a calendar view in the dashboard or initial support for preemptible instances. I would like to keep this momentum going, improving Blazar for its existing users and plugging gaps preventing it from getting adopted by more deployments. Thank you for your support, Pierre Riteau (priteau)

1 0

[election][Designate][ptl] My candidacy for the "Z" release
by Michael Johnson 14 Feb '22

14 Feb '22

Hello OpenStack community, I would like to announce my candidacy for PTL of Designate for the "Z" cycle. I would like to continue to support the Designate team for the "Z" release. We have made good progress improving the Designate tempest tests and documentation, but we have more work to do. The team has also been focusing on cleaning up some of the accumulated technical debt, which I expect will continue into the "Z" cycle. Thank you for your support and your consideration for "Z", Michael Johnson (johnsom)

1 0

Outreachy OpenStack Coordinator change
by Mahati Chamarthy 14 Feb '22

14 Feb '22

Hello all, There has been a change in coordination responsibilities for Outreachy OpenStack <https://www.outreachy.org>. Samuel has decided to step down as a coordinator. I'd like to thank Samuel for his time and efforts to Outreachy. He has helped onboard many new contributors onto the Outreachy Openstack internship program. Thank you Samuel and hope our paths cross again! Good luck. I'd also like to welcome Dmitry Tantsur who kindly offered to coordinate along with me for Outreachy Openstack. Dmitry has been a long time OpenStack contributor and a core committer for the Ironic project. He has already been an Outreachy mentor for multiple rounds. For any queries regarding Outreachy, he can be reached at divius.inside(a)gmail.com (or) on IRC: dtantsur Welcome aboard Dmitry, thank you for stepping up! Best, Mahati

1 0

[neutron] lldpd Errors
by Ammad Syed 14 Feb '22

14 Feb '22

Hello, I am using ovn 21.09 with neutron 19.0. I am continuously seeing below lldpd errors in my journalctl and syslog of compute nodes. Is there anything to worry about? Feb 13 21:17:29 computehost02 lldpd[5025]: MSAP has changed for port tap30b18e08-00, sending a shutdown LLDPDU Feb 13 21:17:29 computehost02 lldpd[5025]: unable to send packet on real device for tap30b18e08-00: No such device or address Feb 13 21:17:29 computehost02 lldpd[5025]: unable to send packet on real device for tap48d8e15e-ee: No such device or address Feb 13 21:17:59 computehost02 lldpd[5025]: unable to send packet on real device for tap48d8e15e-ee: No such device or address Feb 13 21:17:59 computehost02 lldpd[5025]: MSAP has changed for port tap30b18e08-00, sending a shutdown LLDPDU Ammad

3 2

[election][Octavia][ptl] PTL candidacy for Z
by Gregory Thiemonge 14 Feb '22

14 Feb '22

Hi, I would like to nominate myself for Octavia PTL for the Z release cycle. I am the current PTL and I would like to continue to contribute to this role. For the Z release, my plan for Octavia is to enable persistence in taskflow in the amphorav2 driver, to start working on HTTP/3 support, to fix old known issues (handling multiple subnets on the VIP port, fixing multiple subnets issue on member ports). Thanks for your consideration, Gregory

1 0

Re: [kolla-ansible][nova] volume creation time
by Eugen Block 14 Feb '22

14 Feb '22

I haven't used iscsi as a backend yet, but for HDDs the speed looks relatable, on a system with HDD ceph backend the volume creation of a volume (image is 2 GB) takes about 40 seconds, as you yee the download is quite slow, the conversion is a little faster: Image download 541.00 MB at 28.14 MB/s Converted 2252.00 MB image at 172.08 MB/s With a factor of 10 (20 GB) I would probably end up with similar creation times. Just for comparison, this is almost the same image (also 2 GB) in a different ceph cluster where I mounted the cinder conversion path from cephfs, SSD pool: Image download 555.12 MB at 41.34 MB/s Converted 2252.00 MB image at 769.17 MB/s This volume was created within 20 seconds. You might also want to tweak these options: block_device_allocate_retries = 300 block_device_allocate_retries_interval = 10 These are the defaults: block_device_allocate_retries = 60 block_device_allocate_retries_interval = 3 This would fit your error message: > Volume be0f28eb-1045-4687-8bdb-5a6d385be6fa did not finish being > created even after we waited 187 seconds or 61 attempts. And its > status is downloading. It tried 60 times with a 3 second interval, apparently that's not enough. Can you see any bottlenecks in the network or disk utilization which would slow down the download? Zitat von Franck VEDEL <franck.vedel(a)univ-grenoble-alpes.fr>: > Hi Eugen, > thanks for your help > We have 3 servers (s1, s2 , s3) and an iscsi bay attached on s3. > Multinode: > [control] > s1 > s2 > > [compute] > s1 > s2 > s3 > > [storage] > s3 > > on s1: more /etc/kolla/globals.yml > ... > enable_cinder: "yes" > enable_cinder_backend_iscsi: "yes" > enable_cinder_backend_lvm: « yes" > enable_iscsid: « yes" > cinder_volume_group: "cinder-volumes » > ... > enable_glance_image_cache: "yes" > glance_cache_max_size: "21474836480" > glance_file_datadir_volume: « /images/« > ... > > on s3: /images is on the iscsi bay > mount |grep images > /dev/mapper/VG--IMAGES-LV--IMAGES on /images type xfs > (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=1024,swidth=1024,noquota) > > lsblk > sdf > 8:80 0 500G 0 disk > └─mpathc > 253:3 0 500G 0 mpath > └─mpathc1 > 253:4 0 500G 0 part > └─VG--IMAGES-LV--IMAGES > 253:5 0 500G 0 lvm /images > > > ls -l /images: > drwxr-x---. 5 42415 42415 4096 6 févr. 18:40 image-cache > drwxr-x---. 2 42415 42415 4096 4 févr. 15:16 images > drwxr-x---. 2 42415 42415 6 22 nov. 12:03 staging > drwxr-x---. 2 42415 42415 6 22 nov. 12:03 tasks_work_dir > > ls -l /images/image-cache > total 71646760 > -rw-r-----. 1 42415 42415 360841216 2 déc. 11:52 > 3e3aada8-7610-4c55-b116-a12db68f8ea4 > -rw-r-----. 1 42415 42415 237436928 28 nov. 16:56 > 6419642b-fcbd-4e5d-9c77-46a48d2af93f > -rw-r-----. 1 42415 42415 10975379456 26 nov. 14:59 > 7490e914-8001-4d56-baea-fabf80f425e1 > -rw-r-----. 1 42415 42415 21474836480 22 nov. 16:46 > 7fc7f9a6-ab0e-45cf-9c29-7e59f6aa68a5 > -rw-r-----. 1 42415 42415 2694512640 15 déc. 18:07 > 890fd2e8-2fac-42c6-956b-6b10f2253a56 > -rw-r-----. 1 42415 42415 12048400384 1 déc. 17:04 > 9a235763-ff0c-40fd-9a8d-7cdca3d3e9ce > -rw-r-----. 1 42415 42415 5949227008 15 déc. 20:41 > 9cbba37b-1de1-482a-87f2-631d2143cd46 > -rw-r-----. 1 42415 42415 566994944 6 déc. 12:32 > b6e29dd9-a66d-4569-a222-6fc0bd9b1b11 > -rw-r-----. 1 42415 42415 578748416 2 déc. 11:24 > c40953ee-4b39-43a5-8f6c-b48a046c38e9 > -rw-r-----. 1 42415 42415 16300544 27 janv. 12:19 > c88630c7-a7c6-44ff-bfa0-e5af4b1720e3 > -rw-r-----. 1 42415 42415 12288 6 févr. 18:40 cache.db > -rw-r-----. 1 42415 42415 12324503552 1 déc. 07:50 > e0d4fddd-5aa7-4177-a1d6-e6b4c56f12e8 > -rw-r-----. 1 42415 42415 6139084800 22 nov. 15:05 > eda93204-9846-4216-a6e8-c29977fdcf2f > -rw-r-----. 1 42415 42415 0 22 nov. 12:03 image_cache_db_init > drwxr-x---. 2 42415 42415 6 27 janv. 12:19 incomplete > drwxr-x---. 2 42415 42415 6 22 nov. 12:03 invalid > drwxr-x---. 2 42415 42415 6 22 nov. 12:03 queue > > on s1 > openstack image list > +--------------------------------------+-----------------------------+--------+ > | ID | Name > | Status | > +--------------------------------------+-----------------------------+--------+ > ….. > | 7fc7f9a6-ab0e-45cf-9c29-7e59f6aa68a5 | rocky8.4 > | active | > …. > | 7490e914-8001-4d56-baea-fabf80f425e1 | win10_2104 > | active | > …. > +———————————————————+-----------------------------+————+ > > > openstack image show 7fc7f9a6-ab0e-45cf-9c29-7e59f6aa68a5 > disk_format | raw > > when I try to add an instance from this image (2G RAM, 40G HDD): > [Error : Build of instance baa06bef-9628-407f-8bae-500ef7bce065 > aborted: Volume be0f28eb-1045-4687-8bdb-5a6d385be6fa did not finish > being created even after we waited 187 seconds or 61 attempts. And > its status is downloading. > > it’s impossible. I need to add the volume from image first, and > after add instance from volume. > > Is it normal ? > > > Franck > >> Le 7 févr. 2022 à 10:55, Eugen Block <eblock(a)nde.ag> a écrit : >> >> Hi Franck, >> >> although it's a different topic than your original question I >> wanted to comment on the volume creation time (maybe a new thread >> would make sense). What is your storage back end? If it is ceph, >> are your images in raw format? Otherwise cinder has to download the >> image from glance (to /var/lib/cinder/conversion) and convert it, >> then upload it back to ceph. It's similar with nova, nova stores >> base images in /var/lib/nova/instances/_base to prevent the compute >> nodes from downloading it every time. This may save some time for >> the download, but the upload has to happen anyway. And if you don't >> use shared storage for nova (e.g. for live-migration) you may >> encounter that some compute nodes are quicker creating an instance >> because they only have to upload, others will first have to >> download, convert and then upload it. >> >> You would see the conversion in the logs of cinder: >> >> INFO cinder.image.image_utils >> [req-f2062570-4006-464b-a1f5-d0d5ac34670d >> d71f59600f1c40c394022738d4864915 31b9b4900a4d4bdaabaf263d0b4021be - >> - -] Converted 2252.00 MB image at 757.52 MB/s >> >> Hope this helps. >> >> Eugen >> >> >> Zitat von Franck VEDEL <franck.vedel(a)univ-grenoble-alpes.fr>: >> >>> Sunday morning: my openstack works…. OUF. >>> the "kolla-ansible -i multimode mariadb_recovery" command (which >>> is magic anyway) fixed the problem and then the mariadb and nova >>> containers started. >>> Once solved the problems between my serv3 and the iscsi bay, >>> restart the container glance, everything seems to work. >>> >>>> 4 minutes to create a 20GB empty volume seems too long to me. For >>>> an actual 20GB image, it's going to depend on the speed of the >>>> backing storage tech. >>> 4 minutes is for a volume from an image. I will see this problem >>> next summer , I will retry to change the MTU value. >>> >>> Thanks a lot, really >>> >>> >>> Franck >>> >>>> Le 5 févr. 2022 à 17:08, Laurent Dumont >>>> <laurentfdumont(a)gmail.com> a écrit : >>>> >>>> Any chance to revert back switches + server? That would indicate >>>> that MTU was the issue. >>>> Dont ping the iscsi bay, ping between the controllers in >>>> Openstack, they are the ones running mariadb/galera. >>>> Since the icmp packets are small, it might not trigger the MTU >>>> issues. Can you try something like "ping -s 8972 -M do -c 4 >>>> $mariadb_host_2" from $ mariadb_host_1? >>>> What is your network setup on the servers? Two ports in a bond? >>>> Did you change both physical interface MTU + bond interface itself? >>>> >>>> 4 minutes to create a 20GB empty volume seems too long to me. For >>>> an actual 20GB image, it's going to depend on the speed of the >>>> backing storage tech. >>>> >>>> On Sat, Feb 5, 2022 at 1:51 AM Franck VEDEL >>>> <franck.vedel(a)univ-grenoble-alpes.fr >>>> <mailto:franck.vedel@univ-grenoble-alpes.fr>> wrote: >>>> Thanks for your help. >>>> >>>> >>>>> What was the starting value for MTU? >>>> 1500 >>>>> What was the starting value changed to for MTU? >>>> 9000 >>>>> Can ping between all your controllers? >>>> yes, all container starts except nova-conductor, nova-scheduler, maraidb >>>> >>>> >>>>> Do you just have two controllers running mariadb? >>>> yes >>>>> How did you change MTU? >>>> >>>> On the 3 servers: >>>> nmcli connection modify team0-port1 802-3-ethernet.mtu 9000 >>>> nmcli connection modify team1-port2 802-3-ethernet.mtu 9000 >>>> nmcli connection modify type team0 team.runner lack ethernet.mtu 9000 >>>> nmcli con down team0 >>>> nmcli con down team1 >>>> >>>> >>>>> Was the change reverted at the network level as well (switches >>>>> need to be configured higher or at the same MTU value then the >>>>> servers) >>>> I didn’t change Mtu on network (switches) , but ping -s >>>> 10.0.5.117 (iscsi bay) was working from serv3. >>>> >>>> I changed the value of the mtu because the creation of the >>>> volumes takes a lot of time I find (4 minutes for 20G, which is >>>> too long for what I want to do, the patience of the students >>>> decreases with the years) >>>> >>>> Franck >>>> >>>>> Le 4 févr. 2022 à 23:12, Laurent Dumont >>>>> <laurentfdumont(a)gmail.com <mailto:laurentfdumont@gmail.com>> a >>>>> écrit : >>>>> >>>>> What was the starting value for MTU? >>>>> What was the starting value changed to for MTU? >>>>> Can ping between all your controllers? >>>>> Do you just have two controllers running mariadb? >>>>> How did you change MTU? >>>>> Was the change reverted at the network level as well (switches >>>>> need to be configured higher or at the same MTU value then the >>>>> servers) >>>>> 4567 seems to be the port for galera (clustering for mariadb) <> >>>>> On Fri, Feb 4, 2022 at 11:52 AM Franck VEDEL >>>>> <franck.vedel(a)univ-grenoble-alpes.fr >>>>> <mailto:franck.vedel@univ-grenoble-alpes.fr>> wrote: >>>>> Hello, >>>>> I am in an emergency situation, quite catastrophic situation >>>>> because I do not know what to do. >>>>> >>>>> I have an Openstack cluster with 3 servers (serv1, serv2, >>>>> serv3). He was doing so well… >>>>> >>>>> >>>>> A network admin came to me and told me to change an MTU on the >>>>> cards. I knew it shouldn't be done...I shouldn't have done it. >>>>> I did it. >>>>> Of course, it didn't work as expected. I went back to my >>>>> starting configuration and there I have a big problem with >>>>> mariadb which is set up on serv1 and serv2. >>>>> >>>>> Here are my errors: >>>>> >>>>> >>>>> 2022-02-04 17:40:36 0 [ERROR] WSREP: failed to open gcomm >>>>> backend connection: 110: failed to reach primary view: 110 >>>>> (Connection timed out) >>>>> at gcomm/src/pc.cpp:connect():160 >>>>> 2022-02-04 17:40:36 0 [ERROR] WSREP: >>>>> gcs/src/gcs_core.cpp:gcs_core_open():209: Failed to open backend >>>>> connection: -110 (Connection timed out) >>>>> 2022-02-04 17:40:36 0 [ERROR] WSREP: >>>>> gcs/src/gcs.cpp:gcs_open():1475: Failed to open channel >>>>> 'openstack' at 'gcomm://10.0.5.109:4567,10.0.5.110:4567': <> >>>>> -110 (Connection timed out) >>>>> 2022-02-04 17:40:36 0 [ERROR] WSREP: gcs connect failed: >>>>> Connection timed out >>>>> 2022-02-04 17:40:36 0 [ERROR] WSREP: >>>>> wsrep::connect(gcomm://10.0.5.109:4567,10.0.5.110:4567 <>) >>>>> failed: 7 >>>>> 2022-02-04 17:40:36 0 [ERROR] Aborting >>>>> >>>>> >>>>> >>>>> >>>>> I do not know what to do. My installation is done with >>>>> kolla-ansible, mariadb docker restarts every 30 seconds. >>>>> >>>>> Can the "kolla-ansible reconfigure mariadb" command be a solution? >>>>> Could the command "kolla-ansible mariadb recovery" be a solution? >>>>> >>>>> Thanks in advance if you can help me. >>>>> >>>>> >>>>> >>>>> Franck >>>>> >>>>> >>>> >> >> >> >>

2 2

nova rabbitmq strange bug
by Satish Patel 14 Feb '22

14 Feb '22

Folks, I am running wallaby and Today i had rabbitMQ issue and i rebuild RabbitMQ. but now my all compute node throwing following error. not sure what is going on up they are not trying to come up and keep crashing with following error 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service [req-6c3cdaa3-ecf9-4bef-9def-6b6a69dfd30b - - - - -] Error starting thread.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 9510bd6c0d6a48e782ed7bc2887f32c8 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service Traceback (most recent call last): 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 433, in get 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service return self._queues[msg_id].get(block=True, timeout=timeout) 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/queue.py", line 322, in get 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service return waiter.wait() 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/queue.py", line 141, in wait 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service return get_hub().switch() 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service return self.greenlet.switch() 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service _queue.Empty 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service During handling of the above exception, another exception occurred: 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service Traceback (most recent call last): 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_service/service.py", line 807, in run_service 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service service.start() 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/service.py", line 159, in start 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service self.manager.init_host() 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/compute/manager.py", line 1413, in init_host 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service self.driver.init_host(host=self.host) 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 792, in init_host 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service self._register_instance_machine_type() 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 811, in _register_instance_machine_type 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service for instance in objects.InstanceList.get_by_host(context, hostname): 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_versionedobjects/base.py", line 175, in wrapper 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service result = cls.indirection_api.object_class_action_versions( 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service return cctxt.call(context, 'object_class_action_versions', 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/rpc/client.py", line 175, in call 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service self.transport._send(self.target, msg_ctxt, msg, 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/transport.py", line 123, in _send 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service return self._driver.send(target, ctxt, message, 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 680, in send 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service return self._send(target, ctxt, message, wait_for_reply, timeout, 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 669, in _send 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service result = self._waiter.wait(msg_id, timeout, 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in wait 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service message = self.waiters.get(msg_id, timeout=timeout) 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service File "/openstack/venvs/nova-23.1.1/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 435, in get 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service raise oslo_messaging.MessagingTimeout( 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 9510bd6c0d6a48e782ed7bc2887f32c8 2022-02-10 00:52:27.843 3660531 ERROR oslo_service.service

2 5

[largescale-sig] Next meeting: Feb 16th, 15utc
by Thierry Carrez 14 Feb '22

14 Feb '22

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 15UTC. You can doublecheck how that time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20220216T15 We'll look back at the new format for the Large Scale OpenStack show and discuss the next episode. Feel free to add other topics to the agenda: https://etherpad.openstack.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 0

[neutron] Bug deputy report (week starting on 2022-02-07)
by Bernard Cafarelli 14 Feb '22

14 Feb '22

Hi neutrinos, I was bug deputy last week, here is the list of reported bugs A quiet week overall, no high importance bugs without assignee High * [fullstack] Test "test_east_west_traffic" randomly failing - https://bugs.launchpad.net/neutron/+bug/1960338 Mostly affecting Wallaby, https://review.opendev.org/c/openstack/neutron/+/828231 backports seem to have helped * "ovn_agent._run_process" ovs-appctl fails setting the service log configuration - https://bugs.launchpad.net/neutron/+bug/1960514 devstack deployment issue for DPDK environments, OVN_BUILD_MODULES=True, patch by ralonsoh https://review.opendev.org/c/openstack/devstack/+/828877 Medium * snat is used instead of dnat_and_snat when L3GW resides on chassis - https://bugs.launchpad.net/neutron/+bug/1960405 elvira is checking this bug * [OVN] OVN_BUILD_MODULES/OVN_BUILD_FROM_SOURCE differences - https://bugs.launchpad.net/neutron/+bug/1960397 An ask-for-opinion bug on these variables, are they both needed? Low * Logging service writing errors in the logs when RPC required - https://bugs.launchpad.net/neutron/+bug/1960533 Service itself works fine, but it is better not to generate this kind of error message -- Bernard Cafarelli

1 0

[election][Cyborg]PTL candidacy for Z
by Brin Zhang(张百林) 14 Feb '22

14 Feb '22

Hi all, I write to submit my candidacy for the Cyborg PTL position during the Z cycle. I have been PTL for the Yoga cycle, and I would like to serve the community for another cycle. In yoga cycle, we have competed get device profile by name, clearly resource provide, updated devices, deployables' API docs, etc.. But we also have some new feature need to trace, e.g. vGPU support (splite a new spec wit usage of new trait of OWNER_NOVA, need to be down with Nova team), add xilinx FPGA driver, and add the UT to cover many scenarios. Another thing is to attract more contributors to the Cyborg contribution, to increase the activity of the Cyborg project, although I have started to do so. In summary, I want to keep the stability and diversity of accelerator devices in OpenStack. Best regards for you. Brin Zhang (brinzhang)

1 0

OpenStack Trove experimental datastores
by Ришат Азизов 12 Feb '22

12 Feb '22

Hello! Why were the experimental datastores removed in the Trove OpenStack Victoria release? https://review.opendev.org/c/openstack/trove/+/728419 <https://review.opendev.org/c/openstack/trove/+/728419> Thanks!

3 2

[elections][ptl][keystone] PTL Candidacy for Keystone
by Douglas Mendizabal 12 Feb '22

12 Feb '22

Hi All, I would like to announce my candidacy for Keystone PTL. Keystone is an important project for OpenStack and I would like to help ensure things keep moving forward. I'm very much interested in the Secure RBAC effort and I want to make sure that the Keystone project has everything it needs to support the greater community pursue the completion of this goal. Thanks, Douglas Mendizábal OFTC: dmendiza[m]

1 0

[elections][ptl][barbican]
by Douglas Mendizabal 12 Feb '22

12 Feb '22

Hi All, I would like to continue to serve as Barbican PTL for the Z cycle. Barbican is slowly moving forward, and I would like to see improvements continue for the next cycle. Thanks, Douglas Mendizábal OFTC: dmendiza[m]

1 0

[all][tc] What's happening in Technical Committee: summary 11th Feb, 21: Reading: 5 min
by Ghanshyam Mann 12 Feb '22

12 Feb '22

Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * We had this week meeting yesterday. Most of the meeting discussions are summarized below (Completed or in-progress activities section). Meeting full logs are available @https://meetings.opendev.org/meetings/tc/2022/tc.2022-02-10-15.00.log.html * Next week's meeting is on 17th Feb Thursday 15:00 UTC, feel free add the topic on the agenda[1] by 16th Feb. 2. What we completed this week: ========================= * Reverted the neutron-fwaas deprecation and it is maintained under neutron stadium now[2]. * Removed security-specs from security sig[3]. 3. Activities In progress: ================== TC Tracker for Yoga cycle ------------------------------ * This etherpad includes the Yoga cycle targets/working items for TC[4]. 2 out of 9 items are completed. Open Reviews ----------------- * Four open reviews for ongoing activities[5]. Dropping tags framework ------------------------------ yoctozepto has updated the part1 of dropping the tag framework[6] which is ready for review. He will continue working on part2 which is to update the release scripts and openstack navigation sites to remove the tag usage. Z release cycle name ------------------------- TC selection on proposed names is closed and selected names are under trademark checks by foundation, follow the ML[7] for updates. Z release cycle Technical Election --------------------------------------- Nomination for PTL and TC election are in-progress and will be open until 15th Feb[8]. I would like to encourage project leaders to stand for PTL and community members to run for TC. Please add your nomination before deadline[9]. TC position on release cadence ------------------------------------- We had separate call to discuss the next step and TC position on release cadence[10] and as discussed Dan has proposed the new release model as TC resolution[11]. Fixing Zuul config error ---------------------------- Requesting projects with zuul config error to look into those and fix them which should not take much time[12]. Project updates ------------------- * Retire js-openstack-lib (waiting on Adjutant to have new PTL/maintainer) [13] 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[14]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [15] 3. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://review.opendev.org/c/openstack/governance/+/828078 [3] https://review.opendev.org/c/openstack/governance/+/828018 [4] https://etherpad.opendev.org/p/tc-yoga-tracker [5] https://review.opendev.org/q/projects:openstack/governance+status:open [6] https://review.opendev.org/c/openstack/governance/+/822900 [7] http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026810.… [8] http://lists.openstack.org/pipermail/openstack-discuss/2022-February/027117… [9] https://governance.openstack.org/election/ [10] https://meetpad.opendev.org/OpenStackReleasecadence [11] https://review.opendev.org/c/openstack/governance/+/828777 [12] https://etherpad.opendev.org/p/zuul-config-error-openstack [13] https://review.opendev.org/c/openstack/governance/+/798540 [14] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [15] http://eavesdrop.openstack.org/#Technical_Committee_Meeting -gmann

1 0