April 2021 - openstack-discuss - lists.openstack.org

Live migration fails
by Szabo, Istvan (Agoda) 07 Aug '24

07 Aug '24

Hi, I have couple of compute nodes where the live migration fails with existing vms. When I quickly spawn a vm and try live migration it works so I assume shouldn't be a big problem with the compute node. However I have many existing vms where it fails with a servername not found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 ERROR nova.conductor.tasks.migrate [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Unable to find record for source node servername on servername: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Failed to compute_task_migrate_server: Compute host servername could not be found.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Setting instance to ACTIVE state.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.672 227612 ERROR oslo_messaging.rpc.server [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Exception during message handling: ComputeHostNotFound: Compute host am-osfecn-4025 Tried with this command: nova live-migration --block-migrate id. Any idea? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 16

[all] Dynamic Zuul results table in Gerrit 3
by Radosław Piliszek 05 Aug '22

05 Aug '22

Hello Fellow OpenStack and OpenDev Folks! TL;DR click on [3] and enjoy. I am starting this thread to not hijack the discussion happening on [1]. First of all, I would like to thank gibi (Balazs Gibizer) for hacking a way to get the place to render the table in the first place (pun intended). I have been a long-time-now user of [2]. I have improved and customised it for myself but never really got to share back the changes I made. The new Gerrit obviously broke the whole script so it was of no use to share at that particular state. However, inspired by gibi's work, I decided to finally sit down and fix it to work with Gerrit 3 and here it comes: [3]. Works well on Chrome with Tampermonkey. Not tested others. I hope you will enjoy this little helper (I do). I know the script looks super fugly but it generally boils down to a mix of styles of 3 people and Gerrit having funky UI rendering. Finally, I'd also like to thank hrw (Marcin Juszkiewicz) for linking me to the original Michel's script in 2019. [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/019051… [2] https://opendev.org/x/coats/src/commit/444c95738677593dcfed0cfd9667d4c4f0d5… [3] https://gist.github.com/yoctozepto/7ea1271c299d143388b7c1b1802ee75e Kind regards, -yoctozepto

5 7

Nova not updating to new size of an extended in-use / attached cinder volume (Ceph RBD) to guest
by Christian Rohmann 28 Jun '22

28 Jun '22

Hello Openstack-Discuss, after some digging I found out that cinder learned to resize attached (in-use) volumes quite a while ago with the introduction of the initial "extend 'in-use' volume" feature: https://review.opendev.org/c/openstack/cinder/+/454287/ The support was then extended to also cover Ceph RBD backed volumes with: https://review.opendev.org/c/openstack/nova/+/613039/ Since this is only about the cinder part, I was wondering if nova would ever find out and would actively rescan the device / volume and quintessentially present an increased new size to the guest. Apparently this is where a certain volume-extended event comes into play: https://review.opendev.org/c/openstack/nova/+/454322/ that is to be emitted by cinder when a volume had been extened. I then went head and tried this with the openstack cli (as Horizon does not seem to support / offer to resize in-use volumes. I run Openstack Train with Ceph RBD as storage. First I ran into an issue with the openstack cli (https://bugs.launchpad.net/cinder/+bug/1871759/comments/2 <https://bugs.launchpad.net/cinder/+bug/1871759/comments/2>) but using cinder extend $volumeid $newsize I was able to resize in-use volumes just fine. The only thing missing was the propagation to the guest. I played around with SCSI rescans, but that did not work, reboots also failed to do anyting in this regards. Stopping and starting the VM did work, but why having the capability to online-resize an attached volume when not being able to resize the guest volume as well? Apparently there seems to be an old and somewhat similar observation / bug with https://bugs.launchpad.net/nova/+bug/1369465 So I was simply wondering if this is expected to be working? Are there any special settings / options I need to set to enable this feature? Thanks and with kind regards Christian

4 9

[nova][dev] Revisiting qemu emulation where guest arch != host arch
by Apsey, Christopher 06 Jan '22

06 Jan '22

All, A few years ago I asked a question[1] about why nova, when given a hw_architecture property from glance for an image, would not end up using the correct qemu-system-xx binary when starting the guest process on a compute node if that compute nodes architecture did not match the proposed guest architecture. As an example, if we had all x86 hosts, but wanted to run an emulated ppc guest, we should be able to do that given that at least one compute node had qemu-system-ppc already installed and libvirt was successfully reporting that as a supported architecture to nova. It seemed like a heavy lift at the time, so it was put on the back burner. I am now in a position to fund a contract developer to make this happen, so the question is: would this be a useful blueprint that would potentially be accepted? Most of the time when people want to run an emulated guest they would just nest it inside of an already running guest of the native architecture, but that severely limits observability and the task of managing any more than a handful of instances in this manner quickly becomes a tangled nightmare of networking, etc. I see real benefit in allowing this scenario to run natively so all of the tooling that exists for fleet management 'just works'. This would also be a significant differentiator for OpenStack as a whole. Thoughts? [1] http://lists.openstack.org/pipermail/openstack-operators/2018-August/015653… Chris Apsey Director | Georgia Cyber Range GEORGIA CYBER CENTER 100 Grace Hopper Lane | Augusta, Georgia | 30901 https://www.gacybercenter.org

2 2

Help with eventlet 0.26.1 and dnspython >= 2
by Thomas Goirand 05 Oct '21

05 Oct '21

Hi, As you may know, eventlet is incompatible with dnspython >= 2.0.0.0rc1. See [1] for the details. However, Debian unstable has 2.0.0. Would there be some good soul willing to help me fix this situation? I would need a patch to fix this, but I'm really not sure how to start. Cheers, Thomas Goirand (zigo)

8 15

[all]Introduction to venus which is the project of log management and has been contributed to the OpenStack community
by Liye Pang(逄立业) 10 Sep '21

10 Sep '21

Hello everyone, after feedback from a large number of operations and maintenance personnel in InCloud OpenStack, we developed the log management project “Venus” for the OpenStack projects and that has contributed to the OpenStack community. The following is an introduction to “Venus”. If there is interest in the community, we are interested in proposing it to become an official OpenStack project in the future. Background In the day-to-day operation and maintenance of large-scale cloud platform, the following problems are encountered： l Time-consuming for log querying while the server increasing to thousands. l Difficult to retrieve logs, since there are many modules in the platform, e.g. systems service, compute, storage, network and other platform services. l The large amount and dispersion of log make faults are difficult to be discovered. l Because of distributed and interaction between components of the cloud platform, and scattered logs between components, it will take more time to locate problems. About Venus According to the key requirements of OpenStack in log storage, retrieval, analysis and so on, we introduced Venus project, a unified log management module. This module can provide a one-stop solution to log collection, cleaning, indexing, analysis, alarm, visualization, report generation and other needs, which involves helping operator or maintainer to quickly solve retrieve problems, grasp the operational health of the platform, and improve the management capabilities of the cloud platform. Additionally, this module plans to use machine learning algorithms to quickly locate IT failures and root causes, and improve operation and maintenance efficiency. Application scenario Venus played a key role in the following scenarios： l Retrieval: Provide a simple and easy-to-use way to retrieve all log and the context. l Analysis: Realize log association, field value statistics, and provide multi-scene and multi-dimensional visual analysis reports. l Alerts：Convert retrieval into active alerts to realize the error finding in massive logs. l Issue location: Establish a chain relationship and knowledge graphs to quickly locate problems. Overall structure The architecture of log management system based on Venus and elastic search is as follows: Diagram 0: Architecture of Venus venus_api: API module，provide API、rest-api service. venus_manager: Internal timing task module to realize the core functions of the log system. Current progress The current progress of the Venus project is as follows: l Collection：Develop fluentd collection tasks based on collectd to read, filter, format and send plug-ins for OpenStack, operating systems, and platform services, etc. l Index：Dealing with multi-dimensional index data in elasticsearch, and provide more concise and comprehensive authentication interface to return query results. l Analysis：Analyzing and display the related module errors, Mariadb connection errors, and Rabbitmq connection errors. l Alerts：Develop alarm task code to set threshold for the number of error logs of different modules at different times, and provides alarm services and notification services. l Location：Develop the call chain analysis function based on global_requested series, which can show the execution sequence, time and error information, etc., and provide the export operation. l Management：Develop configuration management functions in the log system, such as alarm threshold setting, timing task management, and log saving time setting, etc. Application examples Two examples of Venus application scenarios are as follows. 1. The virtual machine creation operation was performed on the cloud platform and it was found that the virtual machine was not created successfully. First, we can find the request id of the operation and jump to the virtual machine creation call chain page. Then, we can query the calling process, view and download the details of the log of the call. 2. In the cloud platform, the error log of each module can be converted into alarms to remind the users. Further, we can retrieve the details of the error log and error log statistics. Next step The next step of the Venus project is as follows: * Collection：In addition to fluent, other collection plugins such as logstash will be integrated. * Analysis: Explore more operation and maintenance scenarios, and conduct statistical analysis and alarm on key data. * display: The configuration, analysis and alarm of Venus will be integrated into horizon in the form of plugin. * location: Form clustering log and construct knowledge map, and integrate algorithm class library to locate the root cause of the fault. Venus Project Registry Venus library: https://opendev.org/inspur/venus You can grab the source code using the following git command: git clone https://opendev.org/inspur/venus.git Venus Demo Youtu.be: <https://youtu.be/mE2MoEx3awM> https://youtu.be/mE2MoEx3awM

7 15

placement service issue (placement-status upgrade check failure)
by Deepak Mohanty 13 Jul '21

13 Jul '21

I need your help with installation of placement service on Ubuntu 20.04. I have followed the steps in: https://docs.openstack.org/placement/wallaby/install/install-ubuntu.html I am facing the following issue when I try to verify the installation: $ placement-status upgrade check Error: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/oslo_upgradecheck/upgradecheck.py", line 196, in run return conf.command.action_fn() File "/usr/lib/python3/dist-packages/oslo_upgradecheck/upgradecheck.py", line 104, in check result = func_name(self, **kwargs) File "/usr/lib/python3/dist-packages/oslo_upgradecheck/common_checks.py", line 41, in check_policy_json policy_path = conf.find_file(conf.oslo_policy.policy_file) File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 2543, in find_file raise NotInitializedError() oslo_config.cfg.NotInitializedError: call expression on parser has not been invoked I did a bit of debugging and found that the policy file is: policy.json. I could not find any policy.json on my machine. I created an empty policy.json at: /etc/placement/policy.json (user and group = placement). That too did not work. I have followed the steps in the install guide twice, deleting the placement database and other resources and recreating them. I continue to see the same issue. Thank you for your time.

5 6

[all][stable] Ocata - End of Life
by Előd Illés 10 Jun '21

10 Jun '21

Hi, Sorry, this will be long :) as there are 3 topics around old stable branches and 'End of Life'. 1. Deletion of ocata-eol tagged branches With the introduction of Extended Maintenance process [1][2] some cycles ago, the 'End of Life' (EOL) process also changed: * branches were no longer EOL tagged and "mass-deleted" at the end of maintenance phase * EOL'ing became a project decision * if a project decides to cease maintenance of a branch that is in Extended Maintenance, then they can tag their branch with $series-eol However, the EOL-tagging process was not automated or redefined process-wise, so that meant the branches that were tagged as EOL were not deleted. Now (after some changing in tooling) Release Management team finally will start to delete EOL-tagged branches. In this mail I'm sending a *WARNING* to consumers of old stable branches, especially *ocata*, as we will start deleting the *ocata-eol* tagged branches in a *week*. (And also newer *-eol branches later on) 2. Ocata branch Beyond the 1st topic we must clarify the future of Ocata stable branch in general: tempest jobs became broken about ~ a year ago. That means that projects had two ways forward: a. drop tempest testing to unblock gate b. simply don't support ocata branch anymore As far as I see the latter one happened and stable/ocata became unmaintained probably for every projects. So my questions are regarding this: * Is any project still using/maintaining their stable/ocata branch? * If not: can Release Team initiate a mass-EOL-tagging of stable/ocata? 3. The 'next' old stable branches Some projects still support their Pike, Queens and Rocky branches. These branches use Xenial and py2.7 and both are out of support. This results broken gates time to time. Especially nowadays. These issues suggest that these branches are closer and closer to being unmaintained. So I call the attention of interested parties, who are for example still consuming these stable branches and using them downstream to put effort on maintaining the branches and their CI/gates. It is a good practice for stable maintainers to check if there are failures in their projects' periodic-stable jobs [3], as those are good indicators of the health of their stable branches. And if there are, then try to fix it as soon as possible. [1] https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.… [2] https://docs.openstack.org/project-team-guide/stable-branches.html#maintena… [3] http://lists.openstack.org/pipermail/openstack-stable-maint/2021-April/date… Thanks, Előd

1 2

[neutron][interop][refstack] New tests and capabilities to track in interop
by Martin Kopec 07 Jun '21

07 Jun '21

Hi everyone, I would like to further discuss the topics we covered with the neutron team during the PTG [1]. * adding address_group API capability It's tested by tests in neutron-tempest-plugin. First question is if tests which are not directly in tempest can be a part of a non-add-on marketing program? It's possible to move them to tempest though, by the time we do so, could they be marked as advisory? * Shall we include QoS tempest tests since we don't know what share of vendors enable QoS? Could it be an add-on? These tests are also in neutron-tempest-plugin, I assume we're talking about neutron_tempest_plugin.api.test_qos tests. If we want to include these tests, which program should they belong to? Do we wanna create a new one? [1] https://etherpad.opendev.org/p/neutron-xena-ptg Thanks, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA

4 8

Question regarding the production status of DB2 in Trove
by Bekir Fajkovic 01 Jun '21

01 Jun '21

Hello! One question regarding the development status of DB2 inside Trove. I see that DB2 is still in experimental phase, when could it be expected for that datastore type to get production ready status? Best Regards. Bekir Fajkovic Senior DBA Mobile: +46 70 019 48 47 www.citynetwork.eu | www.citycloud.com INNOVATION THROUGH OPEN IT INFRASTRUCTURE ISO 9001, 14001, 27001, 27015 & 27018 CERTIFIED

3 8