December 2020 - openstack-discuss

[magnum][api] Error system library fopen too many open files with magnum-auto-healer
by Ionut Biru 14 Jan '21

14 Jan '21

Hi guys, I have an issue with magnum api returning an error after a while: Server-side error: "[('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]" Log file: https://paste.xinu.at/6djE/ This started to appear after I enabled the template auto_healing_controller = magnum-auto-healer, magnum_auto_healer_tag = v1.19.0. Currently, I only have 4 clusters. After that the API is in error state and doesn't work unless I restart it. -- Ionut Biru - https://fleio.com

4 23

[oslo][TC] Dropping lower-constraints testing
by hberaud 12 Jan '21

12 Jan '21

Hello, As you already surely know, we (the openstack project) currently face some issues with our lower-constraints jobs due to pip's latest resolver feature. By discussing this topic with Thierry Carrez (ttx) from an oslo point of view, we reached the same conclusion that it is more appropriate to drop this kind of tests because the complexity and recurring pain needed to maintain them now exceeds the benefits provided by this mechanismes. Also we should notice that the number of active maintainers is declining, so we think that this is the shortest path to solve this problem on oslo for now and for the future too. In a first time I tried to fix our gates by fixing our lower-constraints project by project but with around ~36 projects to maintain this is a painful task, especially due to nested oslo layers inside oslo himself... I saw the face of the hell of dependencies. So, in a second time I submitted a series of patches to drop these tests [1]. But before moving further with that we would appreciate discussing this with the TC. For now the patches are ready and we just have to push the good button accordingly to our choices (+W or abandon). Normally all the oslo projects that need to be fixed are covered by [1]. Thoughts? Thanks for reading. [1] https://review.opendev.org/q/topic:%22oslo_lc_drop%22+(status:open%20OR%20s… -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

9 13

[tripleO] Customised Cinder-Volume fails at 'Paunch 5' during overcloud deployment
by Igal Katzir 07 Jan '21

07 Jan '21

Hello all, I am trying to deploy RHOSP16.1 (based on ‘train’ distribution) for Certification purposes. I have build a container for our cinder driver and trying to deploy it. Deployment runs almost till the end and fails at stage when it tries to configure Pacemaker; Here is the last message: "Info: Applying configuration version '1609231063'", "Notice: /Stage[main]/Pacemaker::Corosync/File_line[pcsd_bind_addr]/ensure: created", "Info: /Stage[main]/Pacemaker::Corosync/File_line[pcsd_bind_addr]: Scheduling refresh of Service[pcsd]", "Info: /Stage[main]/Pacemaker::Service/Service[pcsd]: Unscheduling all events on Service[pcsd]", "Info: Class[Pacemaker::Corosync]: Unscheduling all events on Class[Pacemaker::Corosync]", "Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Cinder::Volume_bundle/Pacemaker::Resource::Bundle[openstack-cinder-volume]/Pcmk_bundle[openstack-cinder-volume]: Dependency Pcmk_property[property-overcloud-controller-0-cinder-volume-role] has failures: true", "Info: Creating state file /var/lib/puppet/state/state.yaml", "Notice: Applied catalog in 382.92 seconds", "Changes:", " Total: 1", "Events:", " Success: 1", " Failure: 2", " Total: 3", I have verified that all packages on my container-image (Pacemaker,Corosync, libqb,and pcs) are installed with same versions as the overcloud-controller. But seems that something is still missing, because deployment with the default openstack-cinder-volume image completes successfully. Can anyone help with debugging this? Let me know if more info needed. Thanks in advance, Igal

2 4

[stable][grenade][qa][nova][swift] s-proxy unable to start due to missing runtime deps
by Lee Yarwood 07 Jan '21

07 Jan '21

Hello all, I wanted to raise awareness of the following issue and to seek some feedback on my approach to workaround it: ImportError: No module named keystonemiddleware.auth_token https://bugs.launchpad.net/swift/+bug/1909018 This was introduced after I landed the following devstack backport stopping projects from installing their test-requirements.txt deps: Stop installing test-requirements with projects https://review.opendev.org/q/I8f24b839bf42e2fb9803dc7df3a30ae20cf264eb For the time being to workaround this in various other gates I've suggested that we disable Swift in Grenade on stable/train: zuul: Disable swift services until bug #1909018 is resolved https://review.opendev.org/c/openstack/grenade/+/768224 This finally allowed openstack/nova to pass on stable/train with the following changes to lower-constraints.txt and test-requirements.txt: [stable-only] Cap bandit to 1.6.2 and raise hacking, flake8 and stestr https://review.opendev.org/c/openstack/nova/+/766171/ Are there any objections to disabling Swift in Grenade for the time being on stable/train? Would anyone have any objections to also disabling it on stable/stein via devstack-gate? Many thanks in advance, -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76

2 4

[neutron] SR-IOV mechanism driver configuration, plugin.ini
by Gabriel Omar Gamero Montenegro 07 Jan '21

07 Jan '21

Dear all, I'm following the OpenStack guide for the implementation of SR-IOV mechanism driver. I'm planning to incorporate this driver to my current OpenStack deployment (Queens). Config SR-IOV Guide: https://docs.openstack.org/neutron/queens/admin/config-sriov.html At point 2, section "Configure neutron-server (Controller)" they said that I have to add the 'plugin.ini' file as a parameter to the neutron-server service. To do this they require to <<edit the appropriate initialization script to configure the neutron-server service to load the plugin configuration file>>: --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini I'd like to know a few things: (1) Which plugin.ini file are talking about? (2) How to set up the neutron-server initialization script to add the plugin.ini file? I understand that this varies between OS distro (I'm currently using Ubuntu 16.04 LTS server) Here are some things I tried... I got the following results executing this command: systemctl status neutron-server.service ● neutron-server.service - OpenStack Neutron Server Loaded: loaded (/lib/systemd/system/neutron-server.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2020-12-29 18:13:50 -05 Main PID: 38590 (neutron-server) Tasks: 44 Memory: 738.8M CPU: 29.322s CGroup: /system.slice/neutron-server.service ├─38590 /usr/bin/python2 /usr/bin/neutron-server --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --log-file=/var/log/neutron/neutron-server.log ... I see 2 things: (i) When neutron-server is exectured, the following parameters are passed: --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --log-file=/var/log/neutron/neutron-server.log (ii) The file '/lib/systemd/system/neutron-server.service' is loaded and it has the following content: ... ExecStart=/etc/init.d/neutron-server systemd-start ... This indicates me that it's executing '/etc/init.d/neutron-server' script. So I suppose this is the file indicated to add the parameters of the SR-IOV OpenStack documentation, but I have no idea where to put them. For Red-Hat distros I found this documentation with the following configuration: https://access.redhat.com/documentation/en-us/ red_hat_enterprise_linux_openstack_platform/7/html/networking_guide /sr-iov-support-for-virtual-networking vi /usr/lib/systemd/system/neutron-server.service ... ExecStart=/usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini --log-file /var/log/neutron/server.log Thanks in advance, Gabriel Gamero

3 2

[qa][tempest] Update language in tempest code base
by Arx Cruz 05 Jan '21

05 Jan '21

Hello, I would like to start a discussion regarding the topic. At this moment in time we have an opportunity to be a more open and inclusive project by eliminating outdated naming conventions from tempest codebase, such as blacklist, whitelist. We should take the opportunity and do our best to replace outdated terms with their more inclusive alternatives. As you can see in [1] the TripleO project is already working on this initiative, and I would like to work on this as well on the tempest side. Any thoughts? Shall I start with a sepc, adding deprecation warnings? [1] https://review.opendev.org/#/c/740013/1/specs/victoria/renaming_rules.rst Kind regards, -- Arx Cruz Software Engineer Red Hat EMEA <https://www.redhat.com> arxcruz(a)redhat.com @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

6 9

[infra][magnum][ci] Issues installing bashate and coverage
by Spyros Trigazis 05 Jan '21

05 Jan '21

Hello infra, openstack-tox-lower-constraints fails for bashate and coverage. (Maybe more, I bumped bashate and it failed for coverage. I don;t want to waste more resources on our CI) eg https://review.opendev.org/c/openstack/magnum/+/765881 https://review.opendev.org/c/openstack/magnum/+/765979 Do we miss something? Thanks, Spyros

4 6

[Ussuri] Auto shutdown VM
by Deepa KR 05 Jan '21

05 Jan '21

Hi All We have a Openstack setup with the Ussuri Version and I am regularly facing auto shutdown of a few VMs (ubuntu16.04) randomly . If I restart then the instance is back . From logs I was able to see the messages below . WARNING nova.compute.manager [req-2a21d455-ac04-44aa-b248-4776e5109013 813f3fb52c434e38991bb90aa4771541 10b5279cb6f64ca19871f132a2cee1a3 - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Received unexpected event network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for instance with vm_state active and task_state None. INFO nova.compute.manager [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] VM Stopped (Lifecycle Event) INFO nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. syslog:Nov 13 07:01:07 fgshwbucehyp04 nova-compute[2680204]: 2020-11-13 07:01:07.684 2680204 WARNING nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4 nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance is already powered off in the hypervisor when stop is called. nova.virt.libvirt.driver [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance already shutdown. nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully. nova.compute.manager [req-7a0a0d03-e286-42f0-9e36-38a432f236f3 d9ca03b9d0884d51a26a39b6c82f02eb 304d859c43df4de4944ca5623f7f455c - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Get console output nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully. I searched a few blogs and forums but couldn't find a solution to it . Few mentioned to add sync_power_state_interval=-1 in /etc/nova/nova.conf .But understood that this will help only when nova stops vm. But in this case vm itself is shutting down (Instance shutdown by itself. Calling the stop API) Also no memory issue in VM nor the hypervisor. Also did apt-get upgrade . It would be great if anyone can shed light to this issue. Regards, Deepa K R Sent from my iPhone

7 15

some error abount ironic-python-agent-builder
by Ankele zhang 04 Jan '21

04 Jan '21

Hi~ I have an OpenStack platform in Rocky version. I use ironic-python-agent-builder to build a tinyipa image to customing HardwareManager for 'RAID configuration' in Ironic cleaning steps. While I follow the steps ' https://docs.openstack.org/ironic-python-agent-builder/latest/admin/tinyipa…' to build tinyipa image, it occurs error as following: [image: image.png] [image: image.png] so, what the IPA_SOURCE_DIR means? Do I need to download the source code of the ironic-python-agent and copy it to /opt/stack/ before this? Looking forward to your reply. Ankele

2 1

[Ceilometer][all]Visualising Ceilometer
by Adekunbi Adewojo 31 Dec '20

31 Dec '20

Hi there, I want to get the results in ceilometer into Grafana dashboard so that I can visualise them. Is there a manual on this or can someone point me in the right direction please? Currently, I interact with ceilometer using it's API from my code to get some metrics result but would love to visualise them. Also, I came across Sensu and thinks this could help. Can it be used to visualise metrics gotten from ceilometer? I am interested in getting VM metrics in my code and at the same time be able to visualise them when applications on the VM are stress tested. Thanks.

3 3