December 2021 - openstack-discuss

Live migration fails
by Szabo, Istvan (Agoda) 07 Aug '24

07 Aug '24

Hi, I have couple of compute nodes where the live migration fails with existing vms. When I quickly spawn a vm and try live migration it works so I assume shouldn't be a big problem with the compute node. However I have many existing vms where it fails with a servername not found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 ERROR nova.conductor.tasks.migrate [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Unable to find record for source node servername on servername: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Failed to compute_task_migrate_server: Compute host servername could not be found.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Setting instance to ACTIVE state.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.672 227612 ERROR oslo_messaging.rpc.server [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Exception during message handling: ComputeHostNotFound: Compute host am-osfecn-4025 Tried with this command: nova live-migration --block-migrate id. Any idea? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 16

[all] Dynamic Zuul results table in Gerrit 3
by Radosław Piliszek 04 Aug '22

04 Aug '22

Hello Fellow OpenStack and OpenDev Folks! TL;DR click on [3] and enjoy. I am starting this thread to not hijack the discussion happening on [1]. First of all, I would like to thank gibi (Balazs Gibizer) for hacking a way to get the place to render the table in the first place (pun intended). I have been a long-time-now user of [2]. I have improved and customised it for myself but never really got to share back the changes I made. The new Gerrit obviously broke the whole script so it was of no use to share at that particular state. However, inspired by gibi's work, I decided to finally sit down and fix it to work with Gerrit 3 and here it comes: [3]. Works well on Chrome with Tampermonkey. Not tested others. I hope you will enjoy this little helper (I do). I know the script looks super fugly but it generally boils down to a mix of styles of 3 people and Gerrit having funky UI rendering. Finally, I'd also like to thank hrw (Marcin Juszkiewicz) for linking me to the original Michel's script in 2019. [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/019051… [2] https://opendev.org/x/coats/src/commit/444c95738677593dcfed0cfd9667d4c4f0d5… [3] https://gist.github.com/yoctozepto/7ea1271c299d143388b7c1b1802ee75e Kind regards, -yoctozepto

5 7

[all] PyCharm Licenses Renewed till July 2021
by Swapnil Kulkarni 13 Jul '22

13 Jul '22

Hello, Happy to inform you the open source developer license for Pycharm has been renewed for 1 additional year till July 2021. Best Regards, Swapnil Kulkarni coolsvap at gmail dot com

8 13

Nova not updating to new size of an extended in-use / attached cinder volume (Ceph RBD) to guest
by Christian Rohmann 28 Jun '22

28 Jun '22

Hello Openstack-Discuss, after some digging I found out that cinder learned to resize attached (in-use) volumes quite a while ago with the introduction of the initial "extend 'in-use' volume" feature: https://review.opendev.org/c/openstack/cinder/+/454287/ The support was then extended to also cover Ceph RBD backed volumes with: https://review.opendev.org/c/openstack/nova/+/613039/ Since this is only about the cinder part, I was wondering if nova would ever find out and would actively rescan the device / volume and quintessentially present an increased new size to the guest. Apparently this is where a certain volume-extended event comes into play: https://review.opendev.org/c/openstack/nova/+/454322/ that is to be emitted by cinder when a volume had been extened. I then went head and tried this with the openstack cli (as Horizon does not seem to support / offer to resize in-use volumes. I run Openstack Train with Ceph RBD as storage. First I ran into an issue with the openstack cli (https://bugs.launchpad.net/cinder/+bug/1871759/comments/2 <https://bugs.launchpad.net/cinder/+bug/1871759/comments/2>) but using cinder extend $volumeid $newsize I was able to resize in-use volumes just fine. The only thing missing was the propagation to the guest. I played around with SCSI rescans, but that did not work, reboots also failed to do anyting in this regards. Stopping and starting the VM did work, but why having the capability to online-resize an attached volume when not being able to resize the guest volume as well? Apparently there seems to be an old and somewhat similar observation / bug with https://bugs.launchpad.net/nova/+bug/1369465 So I was simply wondering if this is expected to be working? Are there any special settings / options I need to set to enable this feature? Thanks and with kind regards Christian

4 9

[oslo] Propose to EOL stable/queens, stable/rocky on all the oslo scope
by Herve Beraud 24 Jun '22

24 Jun '22

Hi, On our last meeting of the oslo team we discussed the problem with broken stable branches (rocky and older) in oslo's projects [1]. Indeed, almost all these branches are broken. Előd Illés kindly generated a list of periodic-stable errors on Oslo's stable branches [2]. Given the lack of active maintainers on Oslo and given the current status of the CI in those branches, I propose to make them End Of Life. I will wait until the end of month for anyone who would like to maybe step up as maintainer of those branches and who would at least try to fix CI of them. If no one will volunteer for that, I'll EOLing those branches for all the projects under the oslo umbrella. Let us know your thoughts. Thank you for your attention. [1] https://meetings.opendev.org/meetings/oslo/2021/oslo.2021-10-04-15.00.log.t… [2] http://lists.openstack.org/pipermail/openstack-discuss/2021-July/023939.html -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

5 5

[TripleO] Core team cleanup
by James Slagle 30 Mar '22

30 Mar '22

Hello, I took a look at our core team, "tripleo-core" in gerrit. We have a few individuals who I feel have moved on from TripleO in their focus. I looked at the reviews from stackalytics.io for the last 180 days[1]. These individuals have less than 6 reviews, which is about 1 review a month: Bob Fournier Dan Sneddon Dmitry Tantsur Jiří Stránský Juan Antonio Osorio Robles Marius Cornea These individuals have publicly expressed that they are moving on from TripleO: Michele Baldessari wes hayutin I'd like to propose we remove these folks from our core team, while thanking them for their contributions. I'll also note that I'd still value +1/-1 from these folks with a lot of significance, and encourage them to review their areas of expertise! If anyone on the list plans to start reviewing in TripleO again, then I also think we can postpone the removal for the time being and re-evaluate later. Please let me know if that's the case. Please reply and let me know any agreements or concerns with this change. Thank you! [1] https://www.stackalytics.io/report/contribution?module=tripleo-group&projec… -- -- James Slagle --

5 6

[all][tc] Relmgt team position on release cadence
by Thierry Carrez 14 Feb '22

14 Feb '22

Hi everyone, The (long) document below reflects the current position of the release management team on a popular question: should the OpenStack release cadence be changed? Please note that we only address the release management / stable branch management facet of the problem. There are other dimensions to take into account (governance, feature deprecation, supported distros...) to get a complete view of the debate. Introduction ------------ The subject of how often OpenStack should be released has been regularly debated in the OpenStack community. OpenStack started with a 3-month release cycle, then switched to 6-month release cycle starting with Diablo. It is often thought of a release management decision, but it is actually a much larger topic: a release cadence is a trade-off between pressure to release more often and pressure to release less often, coming in from a lot of different stakeholders. In OpenStack, it is ultimately a Technical Committee decision. But that decision is informed by the position of a number of stakeholders. This document gives historical context and describes the current release management team position. The current trade-off --------------------- The main pressure to release more often is to make features available to users faster. Developers get a faster feedback loop, hardware vendors ensure software is compatible with their latest products, and users get exciting new features. "Release early, release often" is a best practice in our industry -- we should generally aim at releasing as often as possible. But that is counterbalanced by pressure to release less often. From a development perspective, each release cycle comes with some process overhead. On the integrators side, a new release means packaging and validation work. On the users side, it means pressure to upgrade. To justify that cost, there needs to be enough user-visible benefit (like new features) in a given release. For the last 10 years for OpenStack, that balance has been around six months. Six months let us accumulate enough new development that it was worth upgrading to / integrating the new version, while giving enough time to actually do the work. It also aligned well with Foundation events cadence, allowing to synchronize in-person developer meetings date with start of cycles. What changed ------------ The major recent change affecting this trade-off is that the pace of new development in OpenStack slowed down. The rhythm of changes was divided by 3 between 2015 and 2021, reflecting that OpenStack is now a mature and stable solution, where accessing the latest features is no longer a major driver. That reduces some of the pressure for releasing more often. At the same time, we have more users every day, with larger and larger deployments, and keeping those clusters constantly up to date is an operational challenge. That increases the pressure to release less often. In essence, OpenStack is becoming much more like a LTS distribution than a web browser -- something users like moving slow. Over the past years, project teams also increasingly decoupled individual components from the "coordinated release". More and more components opted for an independent or intermediary-released model, where they can put out releases in the middle of a cycle, making new features available to their users. This increasingly opens up the possibility of a longer "coordinated release" which would still allow development teams to follow "release early, release often" best practices. All that recent evolution means it is (again) time to reconsider if the 6-month cadence is what serves our community best, and in particular if a longer release cadence would not suit us better. The release management team position on the debate -------------------------------------------------- While releasing less often would definitely reduce the load on the release management team, most of the team work being automated, we do not think it should be a major factor in motivating the decision. We should not adjust the cadence too often though, as there is a one-time cost in switching our processes. In terms of impact, we expect that a switch to a longer cycle will encourage more project teams to adopt a "with-intermediary" release model (rather than the traditional "with-rc" single release per cycle), which may lead to abandoning the latter, hence simplifying our processes. Longer cycles might also discourage people to commit to PTL or release liaison work. We'd probably need to manage expectations there, and encourage more frequent switches (or create alternate models). If the decision is made to switch to a longer cycle, the release management team recommends to switch to one year directly. That would avoid changing it again anytime soon, and synchronizing on a calendar year is much simpler to follow and communicate. We also recommend announcing the change well in advance. We currently have an opportunity of making the switch when we reach the end of the release naming alphabet, which would also greatly simplify the communications around the change. Finally, it is worth mentioning the impact on the stable branch work. Releasing less often would likely impact the number of stable branches that we keep on maintaining, so that we do not go too much in the past (and hit unmaintained distributions or long-gone dependencies). We currently maintain releases for 18 months before they switch to extended maintenance, which results in between 3 and 4 releases being maintained at the same time. We'd recommend switching to maintaining one-year releases for 24 months, which would result in between 2 and 3 releases being maintained at the same time. Such a change would lead to longer maintenance for our users while reducing backporting work for our developers. -- Thierry Carrez (ttx) On behalf of the OpenStack Release Management team

15 45

[magnum] dropping mesos code
by Mohammed Naser 01 Feb '22

01 Feb '22

Hi there, Does it make sense for us to drop the Mesos driver code from the code base? I don't know of anyone that actually uses it and I believe it's likely not been tested for years. Mohammed -- Mohammed Naser VEXXHOST, Inc.

4 5

[nova][powervm][zvm] Time to mark powervm and zvm drivers as deprecated for removal?
by Stephen Finucane 28 Jan '22

28 Jan '22

I've had a PR open against the pypowervm library for over a fortnight now [1] with no activity. This has prompted me to go looking into the state of the powervm driver and what I've found isn't great. It doesn't seem there has been any feature development on the driver for many years, and the last change I can see that wasn't simply of a case of "I need to do this to get tests to pass" was over three years ago [2]. The CI also doesn't appear to have been touched in years. The situation for zvm isn't any better. The last functional change to that driver was nearly 2 years ago and I wasn't able to spot any CI running (though it's possible I just missed this). This means the powervm and zvm drivers stand out from the other non-libvirt drivers in-tree, each of which are getting at least some activity (albeit out- of-tree for Hyper-V, from what I recall). It also begs the question: are these drivers something we want to keep around? If we do, how are we going to maintain them long term? If not, how aggressive should we be in removing that now dead code? I'll open patches to mark both powervm and zvm as deprecated shortly so that, assuming nothing changes in the interim, we can look to remove it early in the Z cycle. Please respond here or on the reviews if you have concerns. Stephen [1] https://github.com/powervm/pypowervm/pull/17 [2] Change ID I89ad36f19672368a1f795e1f29c5af6368ccfeec

1 1

Re: Re: [ops] [kolla] RabbitMQ High Availability
by Albert Braden 25 Jan '22

25 Jan '22

Replying from my home email because I've been asked to not email the list from my work email anymore, until I get permission from upper management. I'm not sure I follow. I was planning to add 2 lines to etc/kolla/config/global.conf: [oslo_messaging_rabbit] amqp_durable_queues = False Is that not sufficient? What is involved in configuring dedicated control exchanges for each service? What would that look like in the config? From: Herve Beraud <hberaud(a)redhat.com> Sent: Thursday, December 9, 2021 2:45 AM To: Bogdan Dobrelya <bdobreli(a)redhat.com> Cc: openstack-discuss(a)lists.openstack.org Subject: [EXTERNAL] Re: [ops] [kolla] RabbitMQ High Availability Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Le mer. 8 déc. 2021 à 11:48, Bogdan Dobrelya <bdobreli(a)redhat.com> a écrit : Please see inline >> I read this with great interest because we are seeing this issue. Questions: >> >> 1. We are running kola-ansible Train, and our RMQ version is 3.7.23. Should we be upgrading our Train clusters to use 3.8.x? >> 2. Document [2] recommends policy '^(?!(amq\.)|(.*_fanout_)|(reply_)).*'. I don't see this in our ansible playbooks, nor in any of the config files in the RMQ container. What would this look like in Ansible, and what should the resulting container config look like? >> 3. It appears that we are not setting "amqp_durable_queues = True". What does this setting look like in Ansible, and what file does it go into? > > Note that even having rabbit HA policies adjusted like that and its HA > replication factor [0] decreased (e.g. to a 2), there still might be > high churn caused by a large enough number of replicated durable RPC > topic queues. And that might cripple the cloud down with the incurred > I/O overhead because a durable queue requires all messages in it to be > persisted to a disk (for all the messaging cluster replicas) before they > are ack'ed by the broker. > > Given that said, Oslo messaging would likely require a more granular > control for topic exchanges and the durable queues flag - to tell it to > declare as durable only the most critical paths of a service. A single > config setting and a single control exchange per a service might be not > enough. Also note that therefore, amqp_durable_queue=True requires dedicated control exchanges configured for each service. Those that use 'openstack' as a default cannot turn the feature ON. Changing it to a service specific might also cause upgrade impact, as described in the topic [3]. The same is true for `amqp_auto_delete=True`. That requires dedicated control exchanges else it won't work if each service defines its own policy on a shared control exchange (e.g `openstack`) and if policies differ from each other. [3] https://review.opendev.org/q/topic:scope-config-opts > > There are also race conditions with durable queues enabled, like [1]. A > solution could be where each service declare its own dedicated control > exchange with its own configuration. > > Finally, openstack components should add perhaps a *.next CI job to test > it with durable queues, like [2] > > [0] https://www.rabbitmq.com/ha.html#replication-factor > > [1] > https://zuul.opendev.org/t/openstack/build/aa514dd788f34cc1be3800e6d7dba0e8… > > [2] https://review.opendev.org/c/openstack/nova/+/820523 > >> >> Does anyone have a sample set of RMQ config files that they can share? >> >> It looks like my Outlook has ruined the link; reposting: >> [2] https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit > > > -- > Best regards, > Bogdan Dobrelya, > Irc #bogdando -- Best regards, Bogdan Dobrelya, Irc #bogdando -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

6 16