January 2022 - openstack-discuss - lists.openstack.org

Live migration fails
by Szabo, Istvan (Agoda) 07 Aug '24

07 Aug '24

Hi, I have couple of compute nodes where the live migration fails with existing vms. When I quickly spawn a vm and try live migration it works so I assume shouldn't be a big problem with the compute node. However I have many existing vms where it fails with a servername not found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 ERROR nova.conductor.tasks.migrate [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Unable to find record for source node servername on servername: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Failed to compute_task_migrate_server: Compute host servername could not be found.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Setting instance to ACTIVE state.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.672 227612 ERROR oslo_messaging.rpc.server [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Exception during message handling: ComputeHostNotFound: Compute host am-osfecn-4025 Tried with this command: nova live-migration --block-migrate id. Any idea? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 16

[all] Dynamic Zuul results table in Gerrit 3
by Radosław Piliszek 04 Aug '22

04 Aug '22

Hello Fellow OpenStack and OpenDev Folks! TL;DR click on [3] and enjoy. I am starting this thread to not hijack the discussion happening on [1]. First of all, I would like to thank gibi (Balazs Gibizer) for hacking a way to get the place to render the table in the first place (pun intended). I have been a long-time-now user of [2]. I have improved and customised it for myself but never really got to share back the changes I made. The new Gerrit obviously broke the whole script so it was of no use to share at that particular state. However, inspired by gibi's work, I decided to finally sit down and fix it to work with Gerrit 3 and here it comes: [3]. Works well on Chrome with Tampermonkey. Not tested others. I hope you will enjoy this little helper (I do). I know the script looks super fugly but it generally boils down to a mix of styles of 3 people and Gerrit having funky UI rendering. Finally, I'd also like to thank hrw (Marcin Juszkiewicz) for linking me to the original Michel's script in 2019. [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/019051… [2] https://opendev.org/x/coats/src/commit/444c95738677593dcfed0cfd9667d4c4f0d5… [3] https://gist.github.com/yoctozepto/7ea1271c299d143388b7c1b1802ee75e Kind regards, -yoctozepto

5 7

[all] PyCharm Licenses Renewed till July 2021
by Swapnil Kulkarni 13 Jul '22

13 Jul '22

Hello, Happy to inform you the open source developer license for Pycharm has been renewed for 1 additional year till July 2021. Best Regards, Swapnil Kulkarni coolsvap at gmail dot com

8 13

Nova not updating to new size of an extended in-use / attached cinder volume (Ceph RBD) to guest
by Christian Rohmann 28 Jun '22

28 Jun '22

Hello Openstack-Discuss, after some digging I found out that cinder learned to resize attached (in-use) volumes quite a while ago with the introduction of the initial "extend 'in-use' volume" feature: https://review.opendev.org/c/openstack/cinder/+/454287/ The support was then extended to also cover Ceph RBD backed volumes with: https://review.opendev.org/c/openstack/nova/+/613039/ Since this is only about the cinder part, I was wondering if nova would ever find out and would actively rescan the device / volume and quintessentially present an increased new size to the guest. Apparently this is where a certain volume-extended event comes into play: https://review.opendev.org/c/openstack/nova/+/454322/ that is to be emitted by cinder when a volume had been extened. I then went head and tried this with the openstack cli (as Horizon does not seem to support / offer to resize in-use volumes. I run Openstack Train with Ceph RBD as storage. First I ran into an issue with the openstack cli (https://bugs.launchpad.net/cinder/+bug/1871759/comments/2 <https://bugs.launchpad.net/cinder/+bug/1871759/comments/2>) but using cinder extend $volumeid $newsize I was able to resize in-use volumes just fine. The only thing missing was the propagation to the guest. I played around with SCSI rescans, but that did not work, reboots also failed to do anyting in this regards. Stopping and starting the VM did work, but why having the capability to online-resize an attached volume when not being able to resize the guest volume as well? Apparently there seems to be an old and somewhat similar observation / bug with https://bugs.launchpad.net/nova/+bug/1369465 So I was simply wondering if this is expected to be working? Are there any special settings / options I need to set to enable this feature? Thanks and with kind regards Christian

4 9

[oslo] Propose to EOL stable/queens, stable/rocky on all the oslo scope
by Herve Beraud 24 Jun '22

24 Jun '22

Hi, On our last meeting of the oslo team we discussed the problem with broken stable branches (rocky and older) in oslo's projects [1]. Indeed, almost all these branches are broken. Előd Illés kindly generated a list of periodic-stable errors on Oslo's stable branches [2]. Given the lack of active maintainers on Oslo and given the current status of the CI in those branches, I propose to make them End Of Life. I will wait until the end of month for anyone who would like to maybe step up as maintainer of those branches and who would at least try to fix CI of them. If no one will volunteer for that, I'll EOLing those branches for all the projects under the oslo umbrella. Let us know your thoughts. Thank you for your attention. [1] https://meetings.opendev.org/meetings/oslo/2021/oslo.2021-10-04-15.00.log.t… [2] http://lists.openstack.org/pipermail/openstack-discuss/2021-July/023939.html -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

5 5

[neutron] Bug Deputy Report January 03 - 10
by Lucas Alvares Gomes 11 Apr '22

11 Apr '22

Hi, This is the Neutron bug report from January 3rd to 10th. Critical: * https://bugs.launchpad.net/neutron/+bug/1956344 - "Functional test test_gateway_chassis_rebalance is failing intermittently" - Assigned to: Rodolfo Alonso High: * https://bugs.launchpad.net/neutron/+bug/1956034 - "ovn load balancer health monitor cause mac address conflict" - Unassigned * https://bugs.launchpad.net/neutron/+bug/1956745 - " [ovn-octavia-provider] Load Balancer remained with ACTIVE state even with PENDING_UPDATE listener" - Assigned to: Luis Tomas Bolivar * https://bugs.launchpad.net/neutron/+bug/1956763 - "[OVN] Overlapping configuration for security group logging is not applied correctly" - Assigned to: Elvira Garcia Ruiz Medium: * https://bugs.launchpad.net/neutron/+bug/1956035 - "ovn load balancer member failover not working when accessed from floating ip" - Unassigned * https://bugs.launchpad.net/neutron/+bug/1956632 - "DNS integration documentation and scenario tests need update after dns_assignment calculation was changed" - Assigned to: Miguel Lavalle * https://bugs.launchpad.net/neutron/+bug/1956785 - "Duplicate validator TypeError for dict values Edit" - Assigned to: Harald Jensås Low: * https://bugs.launchpad.net/neutron/+bug/1956476 - "[OVN] Disallow multiple physnets per bridge" - Assigned to: Rodolfo Alonso * https://bugs.launchpad.net/neutron/+bug/1956770 - " keepalived no_track implemented in >= 2.0.3" - Assigned to: Rodolfo Alonso Needs further triage: * https://bugs.launchpad.net/neutron/+bug/1956435 - "OVS: support multiple segments per host" - Unassigned * https://bugs.launchpad.net/neutron/+bug/1956460 - "The VM can't get metadata from dhcp-agent" - Unassigned * https://bugs.launchpad.net/neutron/+bug/1956846 - "ha router duplicated routes" - Unassigned Cheers, Lucas

4 7

[TripleO] Core team cleanup
by James Slagle 30 Mar '22

30 Mar '22

Hello, I took a look at our core team, "tripleo-core" in gerrit. We have a few individuals who I feel have moved on from TripleO in their focus. I looked at the reviews from stackalytics.io for the last 180 days[1]. These individuals have less than 6 reviews, which is about 1 review a month: Bob Fournier Dan Sneddon Dmitry Tantsur Jiří Stránský Juan Antonio Osorio Robles Marius Cornea These individuals have publicly expressed that they are moving on from TripleO: Michele Baldessari wes hayutin I'd like to propose we remove these folks from our core team, while thanking them for their contributions. I'll also note that I'd still value +1/-1 from these folks with a lot of significance, and encourage them to review their areas of expertise! If anyone on the list plans to start reviewing in TripleO again, then I also think we can postpone the removal for the time being and re-evaluate later. Please let me know if that's the case. Please reply and let me know any agreements or concerns with this change. Thank you! [1] https://www.stackalytics.io/report/contribution?module=tripleo-group&projec… -- -- James Slagle --

5 6

[infra] gerrit is:mergeable predicate
by Brian Rosmaita 15 Mar '22

15 Mar '22

Hello Infra team, In Gerrit 3.4, the is:mergeable predicate is disabled by default [0]. It was kind of handy to be able to screen out patches that are in merge conflict when looking for reviews. Is there an alternative way of doing this, or would you be averse to restoring the previous behavior? (I promise not to complain about gerrit performance.) cheers, brian [0] https://www.gerritcodereview.com/3.4.html#ismergeable-predicate-is-disabled…

2 3

[ops][nova] How to update from N to N+3 (and manage nova services that don't start because they find too old compute nodes...)
by Massimo Sgaravatto 28 Feb '22

28 Feb '22

Dear all When we upgraded our Cloud from Rocky to Train we followed the following procedure: 1) Shutdown of all services on the controller and compute nodes 2) Update from Rocky to Stein of controller (just to do the dbsyncs) 3) Update from Stein to Train of controller 4) Update from Rocky to Train of compute nodes We are trying to do the same to update from Train to Xena, but now there is a problem because nova services on the controller node refuse to start since they find too old compute nodes (this is indeed a new feature, properly documented in the release notes). As a workaround we had to manually modify the "version" field of the compute nodes in the nova.services table. Is it ok, or is there a cleaner way to manage the issue ? Thanks, Massimo

5 10

[all][tc] Relmgt team position on release cadence
by Thierry Carrez 14 Feb '22

14 Feb '22

Hi everyone, The (long) document below reflects the current position of the release management team on a popular question: should the OpenStack release cadence be changed? Please note that we only address the release management / stable branch management facet of the problem. There are other dimensions to take into account (governance, feature deprecation, supported distros...) to get a complete view of the debate. Introduction ------------ The subject of how often OpenStack should be released has been regularly debated in the OpenStack community. OpenStack started with a 3-month release cycle, then switched to 6-month release cycle starting with Diablo. It is often thought of a release management decision, but it is actually a much larger topic: a release cadence is a trade-off between pressure to release more often and pressure to release less often, coming in from a lot of different stakeholders. In OpenStack, it is ultimately a Technical Committee decision. But that decision is informed by the position of a number of stakeholders. This document gives historical context and describes the current release management team position. The current trade-off --------------------- The main pressure to release more often is to make features available to users faster. Developers get a faster feedback loop, hardware vendors ensure software is compatible with their latest products, and users get exciting new features. "Release early, release often" is a best practice in our industry -- we should generally aim at releasing as often as possible. But that is counterbalanced by pressure to release less often. From a development perspective, each release cycle comes with some process overhead. On the integrators side, a new release means packaging and validation work. On the users side, it means pressure to upgrade. To justify that cost, there needs to be enough user-visible benefit (like new features) in a given release. For the last 10 years for OpenStack, that balance has been around six months. Six months let us accumulate enough new development that it was worth upgrading to / integrating the new version, while giving enough time to actually do the work. It also aligned well with Foundation events cadence, allowing to synchronize in-person developer meetings date with start of cycles. What changed ------------ The major recent change affecting this trade-off is that the pace of new development in OpenStack slowed down. The rhythm of changes was divided by 3 between 2015 and 2021, reflecting that OpenStack is now a mature and stable solution, where accessing the latest features is no longer a major driver. That reduces some of the pressure for releasing more often. At the same time, we have more users every day, with larger and larger deployments, and keeping those clusters constantly up to date is an operational challenge. That increases the pressure to release less often. In essence, OpenStack is becoming much more like a LTS distribution than a web browser -- something users like moving slow. Over the past years, project teams also increasingly decoupled individual components from the "coordinated release". More and more components opted for an independent or intermediary-released model, where they can put out releases in the middle of a cycle, making new features available to their users. This increasingly opens up the possibility of a longer "coordinated release" which would still allow development teams to follow "release early, release often" best practices. All that recent evolution means it is (again) time to reconsider if the 6-month cadence is what serves our community best, and in particular if a longer release cadence would not suit us better. The release management team position on the debate -------------------------------------------------- While releasing less often would definitely reduce the load on the release management team, most of the team work being automated, we do not think it should be a major factor in motivating the decision. We should not adjust the cadence too often though, as there is a one-time cost in switching our processes. In terms of impact, we expect that a switch to a longer cycle will encourage more project teams to adopt a "with-intermediary" release model (rather than the traditional "with-rc" single release per cycle), which may lead to abandoning the latter, hence simplifying our processes. Longer cycles might also discourage people to commit to PTL or release liaison work. We'd probably need to manage expectations there, and encourage more frequent switches (or create alternate models). If the decision is made to switch to a longer cycle, the release management team recommends to switch to one year directly. That would avoid changing it again anytime soon, and synchronizing on a calendar year is much simpler to follow and communicate. We also recommend announcing the change well in advance. We currently have an opportunity of making the switch when we reach the end of the release naming alphabet, which would also greatly simplify the communications around the change. Finally, it is worth mentioning the impact on the stable branch work. Releasing less often would likely impact the number of stable branches that we keep on maintaining, so that we do not go too much in the past (and hit unmaintained distributions or long-gone dependencies). We currently maintain releases for 18 months before they switch to extended maintenance, which results in between 3 and 4 releases being maintained at the same time. We'd recommend switching to maintaining one-year releases for 24 months, which would result in between 2 and 3 releases being maintained at the same time. Such a change would lead to longer maintenance for our users while reducing backporting work for our developers. -- Thierry Carrez (ttx) On behalf of the OpenStack Release Management team

15 45