December 2020 - openstack-discuss

[TripleO] moving stable/rocky for tripleo repos to unmaintained (+ then EOL) OK?
by Marios Andreou 20 Jan '21

20 Jan '21

Hello TripleO I would like to propose that we move all tripleo stable/rocky repos [1] to "unmaintained", with a view to tagging as end-of-life in due course. This will allow us to focus our efforts on keeping the check and gate queues green and continue to deliver weekly promotions for the more recent and active stable/* branches train ussuri victoria and master. The stable/rocky repos have not had much action in the last few months - I collected some info at [2] about the most recent stable/rocky commits for each of the tripleo repos. For many of those there are no commits in the last 6 months and for some even longer. The tripleo stable/rocky repos were tagged as "extended maintenance" (rocky-em) [2] in April 2020 with [3]. We have already reduced our CI commitment for rocky - these [4] are the current check/gate jobs and these [5] are the jobs that run for promotion to current-tripleo. However maintaining this doesn’t make sense if we are not even using it e.g. merging things into tripleo-* stable/rocky. Please raise your objections or any other comments or thoughts about this. Unless there are any blockers raised here, the plan is to put this into motion early in January. One still unanswered question I have is that since there is no ‘unmaintained’ tag, in the same way as we have the <release>-em or <release-eol> for extended maintenance and end-of-life, do we simply _declare_ that the repos are unmaintained? Then after a period of “0 to 6 months” per [6] we can tag the tripleo repos with rocky-eol. If any one reading this knows please tell us! Thanks for reading! regards, marios [1] https://releases.openstack.org/teams/tripleo.html#rocky [2] http://paste.openstack.org/raw/800464/ [3] https://review.opendev.org/c/openstack/releases/+/709912 [4] http://dashboard-ci.tripleo.org/d/3-DYSmOGk/jobs-exploration?orgId=1&var-in… [5] http://dashboard-ci.tripleo.org/d/3-DYSmOGk/jobs-exploration?orgId=1&fullsc… [6] https://docs.openstack.org/project-team-guide/stable-branches.html#maintena…

3 5

[ops][nova][designate] Does anyone rely on fully-qualified instance names?
by Stephen Finucane 14 Jan '21

14 Jan '21

When attaching a port to an instance, nova will check for DNS support in neutron and set a 'dns_name' attribute if found. To populate this attribute, nova uses a sanitised version of the instance name, stored in the instance.hostname attribute. This sanitisation simply strips out any unicode characters and replaces underscores and spaces with dashes, before truncating to 63 characters. It does not currently replace periods and this is the cause of bug 1581977 [1], where an instance name such as 'ubuntu20.04' will fail to schedule since neutron identifies '04' as an invalid TLD. The question now is what to do to resolve this. There are two obvious paths available to us. The first is to simply catch these invalid hostnames and replace them with an arbitrary hostname of format 'Server-{serverUUID}'. This is what we currently do for purely unicode instance names and is what I've proposed at [2]. The other option is to strip all periods, or rather replace them with hyphens, when sanitizing the instance name. This is more predictable but breaks the ability to use the instance name as a FQDN. Such usage is something I'm told we've never supported, but I'm concerned that there are users out there who are relying on this all the same and I'd like to get a feel for whether this is the case first. So, the question: does anyone currently rely on this inadvertent "feature"? Cheers, Stephen [1] https://launchpad.net/bugs/1581977 [2] https://review.opendev.org/c/openstack/nova/+/764482

9 23

[magnum][api] Error system library fopen too many open files with magnum-auto-healer
by Ionut Biru 14 Jan '21

14 Jan '21

Hi guys, I have an issue with magnum api returning an error after a while: Server-side error: "[('system library', 'fopen', 'Too many open files'), ('BIO routines', 'BIO_new_file', 'system lib'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')]" Log file: https://paste.xinu.at/6djE/ This started to appear after I enabled the template auto_healing_controller = magnum-auto-healer, magnum_auto_healer_tag = v1.19.0. Currently, I only have 4 clusters. After that the API is in error state and doesn't work unless I restart it. -- Ionut Biru - https://fleio.com

4 23

[oslo][TC] Dropping lower-constraints testing
by hberaud 12 Jan '21

12 Jan '21

Hello, As you already surely know, we (the openstack project) currently face some issues with our lower-constraints jobs due to pip's latest resolver feature. By discussing this topic with Thierry Carrez (ttx) from an oslo point of view, we reached the same conclusion that it is more appropriate to drop this kind of tests because the complexity and recurring pain needed to maintain them now exceeds the benefits provided by this mechanismes. Also we should notice that the number of active maintainers is declining, so we think that this is the shortest path to solve this problem on oslo for now and for the future too. In a first time I tried to fix our gates by fixing our lower-constraints project by project but with around ~36 projects to maintain this is a painful task, especially due to nested oslo layers inside oslo himself... I saw the face of the hell of dependencies. So, in a second time I submitted a series of patches to drop these tests [1]. But before moving further with that we would appreciate discussing this with the TC. For now the patches are ready and we just have to push the good button accordingly to our choices (+W or abandon). Normally all the oslo projects that need to be fixed are covered by [1]. Thoughts? Thanks for reading. [1] https://review.opendev.org/q/topic:%22oslo_lc_drop%22+(status:open%20OR%20s… -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

9 13

[tripleO] Customised Cinder-Volume fails at 'Paunch 5' during overcloud deployment
by Igal Katzir 08 Jan '21

08 Jan '21

Hello all, I am trying to deploy RHOSP16.1 (based on ‘train’ distribution) for Certification purposes. I have build a container for our cinder driver and trying to deploy it. Deployment runs almost till the end and fails at stage when it tries to configure Pacemaker; Here is the last message: "Info: Applying configuration version '1609231063'", "Notice: /Stage[main]/Pacemaker::Corosync/File_line[pcsd_bind_addr]/ensure: created", "Info: /Stage[main]/Pacemaker::Corosync/File_line[pcsd_bind_addr]: Scheduling refresh of Service[pcsd]", "Info: /Stage[main]/Pacemaker::Service/Service[pcsd]: Unscheduling all events on Service[pcsd]", "Info: Class[Pacemaker::Corosync]: Unscheduling all events on Class[Pacemaker::Corosync]", "Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Cinder::Volume_bundle/Pacemaker::Resource::Bundle[openstack-cinder-volume]/Pcmk_bundle[openstack-cinder-volume]: Dependency Pcmk_property[property-overcloud-controller-0-cinder-volume-role] has failures: true", "Info: Creating state file /var/lib/puppet/state/state.yaml", "Notice: Applied catalog in 382.92 seconds", "Changes:", " Total: 1", "Events:", " Success: 1", " Failure: 2", " Total: 3", I have verified that all packages on my container-image (Pacemaker,Corosync, libqb,and pcs) are installed with same versions as the overcloud-controller. But seems that something is still missing, because deployment with the default openstack-cinder-volume image completes successfully. Can anyone help with debugging this? Let me know if more info needed. Thanks in advance, Igal

2 4

[stable][grenade][qa][nova][swift] s-proxy unable to start due to missing runtime deps
by Lee Yarwood 08 Jan '21

08 Jan '21

Hello all, I wanted to raise awareness of the following issue and to seek some feedback on my approach to workaround it: ImportError: No module named keystonemiddleware.auth_token https://bugs.launchpad.net/swift/+bug/1909018 This was introduced after I landed the following devstack backport stopping projects from installing their test-requirements.txt deps: Stop installing test-requirements with projects https://review.opendev.org/q/I8f24b839bf42e2fb9803dc7df3a30ae20cf264eb For the time being to workaround this in various other gates I've suggested that we disable Swift in Grenade on stable/train: zuul: Disable swift services until bug #1909018 is resolved https://review.opendev.org/c/openstack/grenade/+/768224 This finally allowed openstack/nova to pass on stable/train with the following changes to lower-constraints.txt and test-requirements.txt: [stable-only] Cap bandit to 1.6.2 and raise hacking, flake8 and stestr https://review.opendev.org/c/openstack/nova/+/766171/ Are there any objections to disabling Swift in Grenade for the time being on stable/train? Would anyone have any objections to also disabling it on stable/stein via devstack-gate? Many thanks in advance, -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76

2 4

[neutron] SR-IOV mechanism driver configuration, plugin.ini
by Gabriel Omar Gamero Montenegro 07 Jan '21

07 Jan '21

Dear all, I'm following the OpenStack guide for the implementation of SR-IOV mechanism driver. I'm planning to incorporate this driver to my current OpenStack deployment (Queens). Config SR-IOV Guide: https://docs.openstack.org/neutron/queens/admin/config-sriov.html At point 2, section "Configure neutron-server (Controller)" they said that I have to add the 'plugin.ini' file as a parameter to the neutron-server service. To do this they require to <<edit the appropriate initialization script to configure the neutron-server service to load the plugin configuration file>>: --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini I'd like to know a few things: (1) Which plugin.ini file are talking about? (2) How to set up the neutron-server initialization script to add the plugin.ini file? I understand that this varies between OS distro (I'm currently using Ubuntu 16.04 LTS server) Here are some things I tried... I got the following results executing this command: systemctl status neutron-server.service ● neutron-server.service - OpenStack Neutron Server Loaded: loaded (/lib/systemd/system/neutron-server.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2020-12-29 18:13:50 -05 Main PID: 38590 (neutron-server) Tasks: 44 Memory: 738.8M CPU: 29.322s CGroup: /system.slice/neutron-server.service ├─38590 /usr/bin/python2 /usr/bin/neutron-server --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --log-file=/var/log/neutron/neutron-server.log ... I see 2 things: (i) When neutron-server is exectured, the following parameters are passed: --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --log-file=/var/log/neutron/neutron-server.log (ii) The file '/lib/systemd/system/neutron-server.service' is loaded and it has the following content: ... ExecStart=/etc/init.d/neutron-server systemd-start ... This indicates me that it's executing '/etc/init.d/neutron-server' script. So I suppose this is the file indicated to add the parameters of the SR-IOV OpenStack documentation, but I have no idea where to put them. For Red-Hat distros I found this documentation with the following configuration: https://access.redhat.com/documentation/en-us/ red_hat_enterprise_linux_openstack_platform/7/html/networking_guide /sr-iov-support-for-virtual-networking vi /usr/lib/systemd/system/neutron-server.service ... ExecStart=/usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini --log-file /var/log/neutron/server.log Thanks in advance, Gabriel Gamero

3 2

[qa][tempest] Update language in tempest code base
by Arx Cruz 05 Jan '21

05 Jan '21

Hello, I would like to start a discussion regarding the topic. At this moment in time we have an opportunity to be a more open and inclusive project by eliminating outdated naming conventions from tempest codebase, such as blacklist, whitelist. We should take the opportunity and do our best to replace outdated terms with their more inclusive alternatives. As you can see in [1] the TripleO project is already working on this initiative, and I would like to work on this as well on the tempest side. Any thoughts? Shall I start with a sepc, adding deprecation warnings? [1] https://review.opendev.org/#/c/740013/1/specs/victoria/renaming_rules.rst Kind regards, -- Arx Cruz Software Engineer Red Hat EMEA <https://www.redhat.com> arxcruz(a)redhat.com @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

6 9

[infra][magnum][ci] Issues installing bashate and coverage
by Spyros Trigazis 05 Jan '21

05 Jan '21

Hello infra, openstack-tox-lower-constraints fails for bashate and coverage. (Maybe more, I bumped bashate and it failed for coverage. I don;t want to waste more resources on our CI) eg https://review.opendev.org/c/openstack/magnum/+/765881 https://review.opendev.org/c/openstack/magnum/+/765979 Do we miss something? Thanks, Spyros

4 6

[Ussuri] Auto shutdown VM
by Deepa KR 05 Jan '21

05 Jan '21

Hi All We have a Openstack setup with the Ussuri Version and I am regularly facing auto shutdown of a few VMs (ubuntu16.04) randomly . If I restart then the instance is back . From logs I was able to see the messages below . WARNING nova.compute.manager [req-2a21d455-ac04-44aa-b248-4776e5109013 813f3fb52c434e38991bb90aa4771541 10b5279cb6f64ca19871f132a2cee1a3 - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Received unexpected event network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for instance with vm_state active and task_state None. INFO nova.compute.manager [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] VM Stopped (Lifecycle Event) INFO nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. syslog:Nov 13 07:01:07 fgshwbucehyp04 nova-compute[2680204]: 2020-11-13 07:01:07.684 2680204 WARNING nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4 nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance is already powered off in the hypervisor when stop is called. nova.virt.libvirt.driver [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance already shutdown. nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully. nova.compute.manager [req-7a0a0d03-e286-42f0-9e36-38a432f236f3 d9ca03b9d0884d51a26a39b6c82f02eb 304d859c43df4de4944ca5623f7f455c - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Get console output nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully. I searched a few blogs and forums but couldn't find a solution to it . Few mentioned to add sync_power_state_interval=-1 in /etc/nova/nova.conf .But understood that this will help only when nova stops vm. But in this case vm itself is shutting down (Instance shutdown by itself. Calling the stop API) Also no memory issue in VM nor the hypervisor. Also did apt-get upgrade . It would be great if anyone can shed light to this issue. Regards, Deepa K R Sent from my iPhone

7 15