November 2020 - openstack-discuss

[largescale-sig] OpenStack DB Archiver
by Arnaud Morin 10 Feb '21

10 Feb '21

Hello large-scalers! TLDR: we opensource a tool to help reducing size of databases. See https://github.com/ovh/osarchiver/ Few months ago, we released a tool, name osarchiver, which we are using on our production environment (at OVH) to help reduce the size of our tables in mariadb (or mysql) In fact, some tables are well know to grow very quickly. We use it, for example, to clean the OpenStack mistral database from old tasks, actions and executions which are older than a year. Another use case could be to archive some data in another table (e.g. with _archived as suffix) if they are 6 months old, and delete this data after 1 year. The source code of this tool is available here: https://github.com/ovh/osarchiver/ We were wondering if some other users would be interested in using the tool, and maybe move it under the opendev governance? Feel free to contact us and/or answer this thread. Cheers, -- Arnaud, Pierre-Samuel and OVH team

12 18

[all] CI test result table in the new gerrit review UI
by Balázs Gibizer 26 Jan '21

26 Jan '21

Hi, I understand that adapting the old CI test result table to the new gerrit review UI is not a simple task. As a temporary measure I hacked together a tampermonkey user script[1] that does a limited version of the CI table injection from the browser side. I tested it with recent chrome and firefox. Let me know if you have problems with it and I'll try to help. Also feel free to adapt it to your own needs. Cheers, gibi [1] https://gist.github.com/gibizer/717e0cb5b0da0d60d5ecf1e18c0c171a

15 34

[ops][nova][designate] Does anyone rely on fully-qualified instance names?
by Stephen Finucane 14 Jan '21

14 Jan '21

When attaching a port to an instance, nova will check for DNS support in neutron and set a 'dns_name' attribute if found. To populate this attribute, nova uses a sanitised version of the instance name, stored in the instance.hostname attribute. This sanitisation simply strips out any unicode characters and replaces underscores and spaces with dashes, before truncating to 63 characters. It does not currently replace periods and this is the cause of bug 1581977 [1], where an instance name such as 'ubuntu20.04' will fail to schedule since neutron identifies '04' as an invalid TLD. The question now is what to do to resolve this. There are two obvious paths available to us. The first is to simply catch these invalid hostnames and replace them with an arbitrary hostname of format 'Server-{serverUUID}'. This is what we currently do for purely unicode instance names and is what I've proposed at [2]. The other option is to strip all periods, or rather replace them with hyphens, when sanitizing the instance name. This is more predictable but breaks the ability to use the instance name as a FQDN. Such usage is something I'm told we've never supported, but I'm concerned that there are users out there who are relying on this all the same and I'd like to get a feel for whether this is the case first. So, the question: does anyone currently rely on this inadvertent "feature"? Cheers, Stephen [1] https://launchpad.net/bugs/1581977 [2] https://review.opendev.org/c/openstack/nova/+/764482

9 23

[qa][tempest] Update language in tempest code base
by Arx Cruz 05 Jan '21

05 Jan '21

Hello, I would like to start a discussion regarding the topic. At this moment in time we have an opportunity to be a more open and inclusive project by eliminating outdated naming conventions from tempest codebase, such as blacklist, whitelist. We should take the opportunity and do our best to replace outdated terms with their more inclusive alternatives. As you can see in [1] the TripleO project is already working on this initiative, and I would like to work on this as well on the tempest side. Any thoughts? Shall I start with a sepc, adding deprecation warnings? [1] https://review.opendev.org/#/c/740013/1/specs/victoria/renaming_rules.rst Kind regards, -- Arx Cruz Software Engineer Red Hat EMEA <https://www.redhat.com> arxcruz(a)redhat.com @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

6 9

[Ussuri] Auto shutdown VM
by Deepa KR 04 Jan '21

04 Jan '21

Hi All We have a Openstack setup with the Ussuri Version and I am regularly facing auto shutdown of a few VMs (ubuntu16.04) randomly . If I restart then the instance is back . From logs I was able to see the messages below . WARNING nova.compute.manager [req-2a21d455-ac04-44aa-b248-4776e5109013 813f3fb52c434e38991bb90aa4771541 10b5279cb6f64ca19871f132a2cee1a3 - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Received unexpected event network-vif-unplugged-e97839a1-bbc4-4d26-af30-768ca3630ce9 for instance with vm_state active and task_state None. INFO nova.compute.manager [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] VM Stopped (Lifecycle Event) INFO nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. syslog:Nov 13 07:01:07 fgshwbucehyp04 nova-compute[2680204]: 2020-11-13 07:01:07.684 2680204 WARNING nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4 nova.compute.manager [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance is already powered off in the hypervisor when stop is called. nova.virt.libvirt.driver [req-8261f607-4f1e-459d-85d4-e269694dd477 - - - - -] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance already shutdown. nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully. nova.compute.manager [req-7a0a0d03-e286-42f0-9e36-38a432f236f3 d9ca03b9d0884d51a26a39b6c82f02eb 304d859c43df4de4944ca5623f7f455c - default default] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Get console output nova.virt.libvirt.driver [-] [instance: 28cd861c-ef15-444a-a902-9cac643c72b5] Instance destroyed successfully. I searched a few blogs and forums but couldn't find a solution to it . Few mentioned to add sync_power_state_interval=-1 in /etc/nova/nova.conf .But understood that this will help only when nova stops vm. But in this case vm itself is shutting down (Instance shutdown by itself. Calling the stop API) Also no memory issue in VM nor the hypervisor. Also did apt-get upgrade . It would be great if anyone can shed light to this issue. Regards, Deepa K R Sent from my iPhone

7 15

Re: [placement] Train upgrade warning
by Seth Tunstall 21 Dec '20

21 Dec '20

Hello, As another person has mentioned (Chanu Romain), I am attempting an upgrade from Queens to Train. I have successfully upgraded Keystone and Glance, but while trying to separate Placement's databases out from Nova, I run into the the same issue with placement-status upgrade check: # placement-status upgrade check +-----------------------------------------------------------------+ | Upgrade Check Results | +-----------------------------------------------------------------+ | Check: Missing Root Provider IDs | | Result: Success | | Details: None | +-----------------------------------------------------------------+ | Check: Incomplete Consumers | | Result: Warning | | Details: There are -3511 incomplete consumers table records for | | existing allocations. Run the "placement-manage db | | online_data_migrations" command. | +-----------------------------------------------------------------+ Likewise, running placement-manage db online_data_migrations does nothing. I am running this placement migration on a dev cluster which has had the databases from a Prod cluster restored to it. This has worked well for me previously as a way to run through upgrades without touching the live system. Thanks very much, Seth

3 10

[tc][all][searchlight ] Retiring the Searchlight project
by Ghanshyam Mann 07 Dec '20

07 Dec '20

Hello Everyone, As you know, Searchlight is a leaderless project for the Wallaby cycle, which means there is no PTL candidate to lead it in the Wallaby cycle. 'No PTL' and no liaisons for DPL model is one of the criteria which triggers TC to start checking the health, maintainers of the project for dropping the project from OpenStack Governance[1]. TC discussed the leaderless project in PTG[2] and checked if the project has maintainers and what activities are done in the Victoria development cycle. It seems no functional changes in Searchlight repos except few gate fixes or community goal commits[3]. Based on all these checks and no maintainer for Searchlight, TC decided to drop this project from OpenStack governance in the Wallaby cycle. Ref: Mandatory Repository Retirement resolution [4] and the detailed process is in the project guide docs [5]. If your organization product/customer use/rely on this project then this is the right time to step forward to maintain it otherwise from the Wallaby cycle, Searchlight will move out of OpenStack governance by keeping their repo under OpenStack namespace with an empty master branch with 'Not Maintained' message in README. If someone from old or new maintainers shows interest to continue its development then it can be re-added to OpenStack governance. With that thanks to Searchlight contributors and PTLs for maintaining this project. [1] https://governance.openstack.org/tc/reference/dropping-projects.html [2] https://etherpad.opendev.org/p/tc-wallaby-ptg [3] https://www.stackalytics.com/?release=victoria&module=searchlight-group&met… [4] https://governance.openstack.org/tc/resolutions/20190711-mandatory-reposito… [5] https://docs.openstack.org/project-team-guide/repository.html#retiring-a-re… -gmann

5 8

[tc][all][qinling] Retiring the Qinling project
by Ghanshyam Mann 07 Dec '20

07 Dec '20

Hello Everyone, As you know, Qinling is a leaderless project for the Wallaby cycle, which means there is no PTL candidate to lead it in the Wallaby cycle. 'No PTL' and no liaisons for DPL model is one of the criteria which triggers TC to start checking the health, maintainers of the project for dropping the project from OpenStack Governance[1]. TC discussed the leaderless project in PTG[2] and checked if the project has maintainers and what activities are done in the Victoria development cycle. It seems no functional changes in Qinling repos except few gate fixes or community goal commits[3]. Based on all these checks and no maintainer for Qinling, TC decided to drop this project from OpenStack governance in the Wallaby cycle. Ref: Mandatory Repository Retirement resolution [4] and the detailed process is in the project guide docs [5]. If your organization product/customer use/rely on this project then this is the right time to step forward to maintain it otherwise from the Wallaby cycle, Qinling will move out of OpenStack governance by keeping their repo under OpenStack namespace with an empty master branch with 'Not Maintained' message in README. If someone from old or new maintainers shows interest to continue its development then it can be re-added to OpenStack governance. With that thanks to Qinling contributors and PTLs (especially lxkong ) for maintaining this project. [1] https://governance.openstack.org/tc/reference/dropping-projects.html [2] https://etherpad.opendev.org/p/tc-wallaby-ptg [3] https://www.stackalytics.com/?release=victoria&module=qinling-group&metric=… [4] https://governance.openstack.org/tc/resolutions/20190711-mandatory-reposito… [5] https://docs.openstack.org/project-team-guide/repository.html#retiring-a-re… -gmann

4 6

[largescale-sig] Next meeting: December 2, 15utc
by Thierry Carrez 02 Dec '20

02 Dec '20

Hi everyone, We'll have a Large Scale SIG meeting this Wednesday in #openstack-meeting-3 on IRC, at 15UTC. You can doublecheck how it translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20201202T15 Our main topic will be to discuss how to best collect operator scaling experiences. Feel free to add other topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting Talk to you all later, -- Thierry Carrez

1 1

[neutron] bug deputy report Nov 23-29
by Oleg Bondarev 01 Dec '20

01 Dec '20

Please find new bugs summary for the week Nov 23-27. Nothing critical. 2 RFEs for neutron drivers team to discuss. High: - https://bugs.launchpad.net/neutron/+bug/1905700 - [OVN] ovn-metadata-agent: "RowNotFound: Cannot find Chassis with name..." when starting the agent - Gate failure - fix approved: https://review.opendev.org/c/openstack/neutron/+/764318 - https://bugs.launchpad.net/neutron/+bug/1905726 - Qos plugin performs too many queries - fixes on review: - https://review.opendev.org/c/openstack/neutron/+/764433 - https://review.opendev.org/c/openstack/neutron/+/764454 Medium: - https://bugs.launchpad.net/neutron/+bug/1905271 - [OVS] Polling cycle continuously interrupted by L2 population (when enabled) - fix on review: https://review.opendev.org/c/openstack/neutron/+/755313 - https://bugs.launchpad.net/neutron/+bug/1905551 - functional: test_gateway_chassis_rebalance fails - Unassigned - https://bugs.launchpad.net/neutron/+bug/1905568 - Sanity checks missing port_name while adding tunnel port - fix approved: https://review.opendev.org/c/openstack/neutron/+/764171 - https://bugs.launchpad.net/neutron/+bug/1905611 - OVN.ovsdb_probe_interval takes effect only after initial database dump - Unassigned Low: - https://bugs.launchpad.net/neutron/+bug/1905538 - Some OVS bridges may lack OpenFlow10 protocol - fix on review: https://review.opendev.org/c/openstack/neutron/+/764150 Wishlist: - https://bugs.launchpad.net/neutron/+bug/1905276 - Overriding hypervisor name for resource provider always requires a complete list of interfaces/bridges - fix on review - https://review.opendev.org/c/openstack/neutron/+/763563 - https://bugs.launchpad.net/neutron/+bug/1905268 - port list performance for trunks can be optimized - fix on review - https://review.opendev.org/c/openstack/neutron/+/763777 RFEs: - https://bugs.launchpad.net/neutron/+bug/1905295 - [RFE] Allow multiple external gateways on a router - https://bugs.launchpad.net/neutron/+bug/1905391 - [RFE] VPNaaS support for OVN Won't fix: - https://bugs.launchpad.net/neutron/+bug/1905552 - neutron-fwaas netlink conntrack driver would catch error while conntrack rules protocol is 'unknown' - fwaas is not supported Expired Bugs: - https://bugs.launchpad.net/bugs/1896592 - [neutron-tempest-plugin] test_dhcpv6_stateless_* clashing when creating a IPv6 subnet - https://bugs.launchpad.net/bugs/1853632 - designate dns driver does not use domain settings for auth Thanks, Oleg --- Advanced Software Technology Lab Huawei

2 1