May 2021 - openstack-discuss - lists.openstack.org

[all][infra][qa] Retiring Logstash, Elasticsearch, subunit2sql, and Health
by Clark Boylan 09 Jul '21

09 Jul '21

Hello everyone, Xenial has recently reached the end of its life. Our logstash+kibana+elasticsearch and subunit2sql+health data crunching services all run on Xenial. Even without the distro platform EOL concerns these services are growing old and haven't received the care they need to keep running reliably. Additionally these services represent a large portion of our resource consumption: * 6 x 16 vcpu + 60GB RAM + 1TB disk Elasticsearch servers * 20 x 4 vcpu + 4GB RAM logstash-worker servers * 1 x 2 vcpu + 2GB RAM logstash/kibana central server * 2 x 8 vcpu + 8GB RAM subunit-worker servers * 64GB RAM + 500GB disk subunit2sql trove db server * 1 x 4 vcpu + 4GB RAM health server To put things in perspective, they account for more than a quarter of our control plane servers, occupying over a third of our block storage and in excess of half the total memory footprint. The OpenDev/OpenStack Infra team(s) don't seem to have the time available currently to do the major lifting required to bring these services up to date. I would like to propose that we simply turn them off. All of these services operate off of public data that will not be going away (specifically job log content). If others are interested in taking this on they can hook into this data and run their own processing pipelines. I am sure not everyone will be happy with this proposal. I get it. I came up with the idea for the elasticsearch job log processing way back at the San Diego summit. I spent many many many hours since working to get it up and running and to keep it running. But pragmatism means that my efforts and the team's efforts are better spent elsewhere. I am happy to hear feedback on this. Thank you for your time. Clark

10 13

[Octavia dashboard] Impossible to fill the subnet field when creating a load balancer
by Taltavull Jean-Francois 09 Jul '21

09 Jul '21

Hi All, On my OSA Victoria deployment (on Ubuntu 20.04), the "Create Load Balancer" form does not allow me to fill the mandatory "subnet" field and therefore I can't create a load balancer with Horizon. On the other hand, Octavia works fine and I can create load balancers with the CLI, Terraform, etc. Has one of you already faced such a situation ? Best regards, Jean-François

3 6

[nova] spec review day
by Balazs Gibizer 05 Jul '21

05 Jul '21

Hi, During the today's meeting it came up that we should have a spec review day around M1 (and later one more before M2). As M1 is 27th of May, I propose to do a spec review day on 25th of May, which is a Tuesday. Let me know if the timing does not good for you. The rules are the usual. Let's use this day to focus on open specs, trying to reach agreement on as many thing as possible with close cooperation during the day. Cheers, gibi

1 3

[Swift] Object replication failures on newly upgraded servers
by Mark Kirkwood 01 Jul '21

01 Jul '21

HI, I'm in the process of upgrading a Swift cluster from 2.7/Mitaka to 2.23/Train. While in general it seems to be going well, I'm noticing non-zero object replication failures on the upgraded nodes only, e.g: $ curl http://localhost:6000/recon/replication/object {"replication_last": 1622156911.019487, "replication_stats": {"rsync": 40580, "success": 4141229, "attempted": 2081856, "remove": 4083, "suffix_count": 14960481, "failure": 26550, "hashmatch": 4127197, "failure_nodes": {"10.11.18.67": {"obj08": 2348, "obj09": 60, "obj10": 3030, "obj02": 34, "obj03": 25, "obj01": 44, "obj06": 1498, "obj07": 28, "obj04": 69, "obj05": 36}, "10.11.18.68": {"obj03": 6901, "obj01": 293, "obj06": 1901, "obj04": 10281, "obj10": 1}, "10.12.18.76": {"obj10": 1}}, "suffix_sync": 1785, "suffix_hash": 2778}, "object_replication_last": 1622156911.019487, "replication_time": 1094.7836411476135, "object_replication_time": 1094.7836411476135} Examining the logs (/var/log/swift/object.log and /var/log/syslog) these are not throwing up any red flags (i.e no failing rsyncs noted). Any suggesting about how to get more information about what went wrong e.g: "10.11.18.67": {"obj08": 2348}, how to find what those 2348 errors were? regards Mark P.s: basic sanity checking is ok - uploaded objects go where they should and can be retrieved for 2.7 or 2.23 servers ok (the old and new version servers agree about object placement)

2 4

[tc][all] What should we have as community goals for Y series?( Starting community-wide goals ideas)
by Rico Lin 30 Jun '21

30 Jun '21

Dear all, We're now in R-22 week for Xena cycle which sounds like a perfect time to start calling for community-wide goals ideas for Y-series. According to the goal process schedule [1], we need to find potential goals, and champions before Xena milestone-1 and provide proper discussion in the community right after that to give a clear view and detail on each goal. And if we would like to keep up with the schedule, we should start right away to identify potential goals. So please help to provide ideas for Y series community-wide goals in [2]. Community-wide goals are important in terms of solving and improving a technical area across OpenStack as a whole. It has a lot more benefits to be considered from users as well from a developer's perspective. See [3] for more details about community-wide goals and processes. Also, you can refer to the backlogs of community-wide goals from this[4] and victoria cycle goals[5] (also ussuri[6]). We took cool-down cycle goal step for Xena cycle [7], so no selected goals for Xena. [1] https://governance.openstack.org/tc/goals/#goal-selection-schedule [2] https://etherpad.opendev.org/p/y-series-goals [3] https://governance.openstack.org/tc/goals/index.html [4] https://etherpad.openstack.org/p/community-goals [5] https://etherpad.openstack.org/p/YVR-v-series-goals [6] https://etherpad.openstack.org/p/PVG-u-series-goals [7] https://review.opendev.org/c/openstack/governance/+/770616 *Rico Lin* OIF Board director, OpenStack TC, Multi-arch SIG chair, Heat PTL, Senior Software Engineer@EasyStack

10 13

Customization of nova-scheduler
by Levon Melikbekjan 16 Jun '21

16 Jun '21

Hello Openstack team, is it possible to customize the nova-scheduler via Python? If yes, how? Best regards Levon

8 16

[all] CRITICAL: Upcoming changes to the OpenStack Community IRC this weekend
by Ghanshyam Mann 16 Jun '21

16 Jun '21

Greetings contributors & community members! With recent events, the Technical Committee held an emergency meeting today (Wednesday, May 26th, 2021) regarding Freenode IRC and what our decision would be [1]. Earlier in the week, the consensus amongst the TC was to gather more information from the individual projects, and make a decision from there[2]. With #rdo, #ubuntu, and #wikipedia having been hijacked, the consensus amongst the TC and the community members who were able to attend the meeting was to move away from Freenode as soon as possible. The TC agreed that this move away from Freenode needs to be a community-wide move to the same, new IRC network for all projects to avoid splintering of the community. As has been long-planned in the event of a contingency, we will be moving to OFTC. We recognize this is a contentious topic, and ultimately we seek to ensure community continuity before evolution to something beyond IRC, as many have expressed interest in doing via Mailing List discussions. At this point, we had to make a decision to solve the immediate problem in the simplest and most expedient way possible, so this is that announcement. We welcome continued discussion about future alternatives on the other threads. With this in mind, we suggest the following steps. Everyone: ======= 1. Do NOT change any channel topics to represent this change. This is likely to result in the channel being taken over by Freenode and will disrupt communications within our community. 2. Register your nicknames on OFTC [3][4] 3. Be *prepared* to join your channels on OFTC[4]. The OpenStack community channels have already been registered on OFTC and await you. 4. Continue to use Freenode for OpenStack discussions until the bots have been moved and the official cut-over takes place this coming weekend. We anticipate using OFTC starting Monday, May 31st. Projects/Project Leaders: ==================== 1. Projects should work to get a few volunteers to staff their project channels on Freenode, for the near future to help redirect people to OFTC. This should occur via private messages to avoid a ban. 2. Continue to hold project meetings on Freenode until the bots are enabled on OFTC. 3. Update project wikis/documentation with the new IRC network information. We ask that you consider referring to the central contributor guide[5]. 4. The TC is asking that projects take advantage of this time of change to consider moving project meetings from the #openstack-meeting* channels to their project channel. 5. Please avoid discussing the move to OFTC in Freenode channels as this may also trigger a takeover of the channel. We are working on getting our bots over to OFTC, and they will be moved over the weekend. Starting Monday May 31, the bots will be on OFTC. Communication regarding this migration will take place on OFTC[4] in #openstack-dev, and we're working on updating the contributor guide[5] to reflect this migration. Sincerely, The OpenStack TC and community leaders who came together to agree on a path forward. [1]: https://etherpad.opendev.org/p/openstack-irc [2]: https://etherpad.opendev.org/p/feedback-on-freenode [3]: https://www.oftc.net/Services/#register-your-account [4]: https://www.oftc.net/ [5]: https://docs.openstack.org/contributors/common/irc.html

3 6

[all][stable] Ocata - End of Life
by Előd Illés 09 Jun '21

09 Jun '21

Hi, Sorry, this will be long :) as there are 3 topics around old stable branches and 'End of Life'. 1. Deletion of ocata-eol tagged branches With the introduction of Extended Maintenance process [1][2] some cycles ago, the 'End of Life' (EOL) process also changed: * branches were no longer EOL tagged and "mass-deleted" at the end of maintenance phase * EOL'ing became a project decision * if a project decides to cease maintenance of a branch that is in Extended Maintenance, then they can tag their branch with $series-eol However, the EOL-tagging process was not automated or redefined process-wise, so that meant the branches that were tagged as EOL were not deleted. Now (after some changing in tooling) Release Management team finally will start to delete EOL-tagged branches. In this mail I'm sending a *WARNING* to consumers of old stable branches, especially *ocata*, as we will start deleting the *ocata-eol* tagged branches in a *week*. (And also newer *-eol branches later on) 2. Ocata branch Beyond the 1st topic we must clarify the future of Ocata stable branch in general: tempest jobs became broken about ~ a year ago. That means that projects had two ways forward: a. drop tempest testing to unblock gate b. simply don't support ocata branch anymore As far as I see the latter one happened and stable/ocata became unmaintained probably for every projects. So my questions are regarding this: * Is any project still using/maintaining their stable/ocata branch? * If not: can Release Team initiate a mass-EOL-tagging of stable/ocata? 3. The 'next' old stable branches Some projects still support their Pike, Queens and Rocky branches. These branches use Xenial and py2.7 and both are out of support. This results broken gates time to time. Especially nowadays. These issues suggest that these branches are closer and closer to being unmaintained. So I call the attention of interested parties, who are for example still consuming these stable branches and using them downstream to put effort on maintaining the branches and their CI/gates. It is a good practice for stable maintainers to check if there are failures in their projects' periodic-stable jobs [3], as those are good indicators of the health of their stable branches. And if there are, then try to fix it as soon as possible. [1] https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.… [2] https://docs.openstack.org/project-team-guide/stable-branches.html#maintena… [3] http://lists.openstack.org/pipermail/openstack-stable-maint/2021-April/date… Thanks, Előd

1 2

[Neutron] sriov network setup for victoria - clarification needed
by forums＠mossakowski.ch 09 Jun '21

09 Jun '21

Hello, I have two victoria environments: 1) a working one, standard setup with separate dedicated interface for sriov (pt0 and pt1) 2) a broken one, where I'm trying to reuse one of already used interfaces (ens2f0 or ens2f1) for sriov. ens2f0 is used for several VLANs (mgmt and storage) and ens2f1 is a neutron external interface which I bridged for VLAN tenant networks. On both I have enabled 63 VFs, it's a standard intetl 10Gb x540 adapter. On broken environment, when I'm trying to boot a VM with sriov port that I created before, I see this error shown on below gist: https://gist.github.com/moss2k13/8e6272cbe7748b2c5210fab291360e0b I'm investigating this for couple days now but I'm out of ideas so I'd like to ask for your support. Is this possible to achieve what I'm trying to do on 2nd environment? To use PF as normal interface and use its VFs for sriov-agent at the same time? Regards, Piotr Mossakowski

3 5

[neutron][interop][refstack] New tests and capabilities to track in interop
by Martin Kopec 07 Jun '21

07 Jun '21

Hi everyone, I would like to further discuss the topics we covered with the neutron team during the PTG [1]. * adding address_group API capability It's tested by tests in neutron-tempest-plugin. First question is if tests which are not directly in tempest can be a part of a non-add-on marketing program? It's possible to move them to tempest though, by the time we do so, could they be marked as advisory? * Shall we include QoS tempest tests since we don't know what share of vendors enable QoS? Could it be an add-on? These tests are also in neutron-tempest-plugin, I assume we're talking about neutron_tempest_plugin.api.test_qos tests. If we want to include these tests, which program should they belong to? Do we wanna create a new one? [1] https://etherpad.opendev.org/p/neutron-xena-ptg Thanks, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA

4 8