Open Stack

Hello Everyone,

I am writing the Technical Committee discussion details from the Zed cycle PTG this week. We had a lot of
discussions and that is why it is a long email but I tried to categorize the details per topic so that you can easily spot
the discussion you are interested in.

TC + Community leader's interaction
============================
We continue the interaction session in this PTG also. The main idea here is to interact with community leaders
and ask for their feedback on TC. I am happy to see more attendance this time. Below are the topics we discussed
in this feedback session.

* Updates from TC:

** Status on previous PTG interaction sessions' Action items/feedback:
*** TC to do more technical guides for the project: DONE
We have a unified limit as the first technical guide to start in this[1].
*** TC to define a concrete checklist to check goal readiness before we select a goal: DONE
We decoupled the community-wide goal from the release cycle[2] and also added the goal readiness checklist[3].
*** Spread the word: DONE
We have updated a new page in project-team-guide where projects can spread the word/story[4]

** Do not do ignorant Recheck:

Recheck without seeing the failure even those are not related to the proposed change is very
costly and end up consuming a lot of infra resources. Checking the failure and adding comments in why
you are doing a recheck will give related projects a chance to know the frequent failures happening in
upstream CI and so does we can spend more time fixing them.

Dan Smith has added detailed documentation on "How to Handle Test Failures"[5], this can be used
to educate the contributors on recheck and how to debug the failure reason.

*** Action Item:
**** PTL to start spreading the word and monitor ignorant recheck in their weekly meeting etc.
**** On ignorant recheck, Zuul to add a comment with the link to recheck practice documentation.

** Spread the word/project outreach:

This was brought up again in this PTG but I forget to update that we did work on this in the Yoga cycle and
updated the project-team-guide to list the places where the project can spread the word/story[4]. It is
ready for projects to refer it and know where they can post their stories/updates.

* Recognition to the new contributors and encourage/help them to continue contributing to OpenStack:

When there is a new contributor who contributes a significant amount of work to any project then we should
have some way to appreciate them. It can be an appreciation certificate/badge from the foundation or TC,
appreciation on social media, especially on linked-in. In TC, we will check what is the best possible and less
costly way to do so. We checked in cross-community session with k8s and they seem to have coupon/badge
distribution from CNCF foundation. 

2021 user survey analysis
===================

Jay has prepared the summary of the 2021 TC survey[6] and we covered it at a high level, especially the big changes
from the previous survey.  As the next step, we will review the proposed summary and merge it. Also, we talked about
updating the upgrade question when we start the tick/tock release.

Prepare to migrate for SQLAlchemy 2.0
==============================

sqlalchemy 2.0 is under development and might have many incompatible changes. To prepare OpenStack in advance,
Stephen sent the mail on what are changes and project needs to do[7], also gave the project status. oslo.db is all good
to use the sqlalchemy  2.0 and the required neutron change is also merged. Thanks to Stephen for driving it and taking
care of various project work.

Elections Analysis:
==============

Project elections are not going well and we end up with a lot of missing nominations. This cycle, we had 17 projects on
the leaderless list and out of it, 16 missed the nomination. There are various factors that lead to missing nomination,
short notice from the election (1-2 weeks), PTLs are not active on ML, Language, Time Zone etc. By seeing the
number of leaderless projects end up having the new PTLs (this cycle it was just 1 project) and having such projects repeating
the nomination miss, we thought election in those projects are not actually required. TC agrees to change the leader assignment
for such a project and instead of finding PTL, we will propose to move these projects to the DPL model. WIth the DPL model, we
do not need to repeat the PTL assignment in every cycle. 

Please note, moving them to DPL model will not be automatically done instead that will be done when there are required liaisons
to take care of DPL responsibility.

Community-wide goals Check In & discussion:
===================================

* FIPS goal
ade_lee gave the current status of this FIPS work progress in various projects[8]. Current FIPS jobs are based on the centos9 stream
which is not so stable but we will keep monitoring it. Canonical has suggested a way to set up FIPS jobs on ubuntu and will give the
foundation keys to do FIPS testing and periodically rotate them.  One challenge is the horizon which has the dependencies on Django
fix and that will be present in Django higher version (4.0 or later) and it is difficult for the horizon to upgrade to that, especially the current
bandwidth team has. He also proposed the different milestones to finish the FIPS work[9].

The Technical Committee agreed to have this as a 'selected goal' and as the next step ade_lee will propose it in governance with defined
milestone and we will review it. Thanks, ade_lee for all your work and energy in this.
 
* RBAC goal
I have summarised it in a separate email, please refer to it
- http://lists.openstack.org/pipermail/openstack-discuss/2022-April/028103.html

Release Cadence Adjustment:
=======================

In TC and leaders' interaction sessions, Dan presented an overview of the new release cadence and explain the things which will be
changed. After that project followed the discussion in their project PTG sessions and then brought the question in the Thursday TC
session. There was a question if we can remove the things (which are deprecated in tick release) in tock release and the answer is yes.

We discussed the automation of deprecation/upgrade release notes from tock release to tick release and it is fine but we need to make
sure we do not make tick release notes so big that people start ignoring those. We will see how it goes in a few of the initial tick release
notes. Below are the various other things which need changes: 

* ACTION: 
** Deprecation document in project-team-guide[10]:
*** Add the master deployment case as a recommendation
*** Intermediate release upgrade case
*** Update 4-a paragraph to clarify the 12 months case and things deprecated in tick can be removed in the next tock release.

** Testing runtime
*** In tock release, it is ok to add/test the new python version but do not upgrade the distro version.
*** In tick release: no change and keep doing the same what we do currently 

** Stable branch 
*** Update release and stable team with what is proposed in 
https://governance.openstack.org/tc/resolutions/20220210-release-cadence-adjustment.html#example-sequence

** tick-tock name:
***gmann to check foundation about trademark check on "tick", "tock" name

Remove release naming instead just use the number
=======================================

In yoga cycle, we changed the release naming convention to add the number also in <year>.<release count in that year>
format[11]. But the release name is less fun nowadays and more of a conflict or objection from the community. Starting
from Ussuri, I have seen objections on many release names and the most recent is 'Zed'.  Being one of the release name
process coordinators for the last couple of releases, I feel that I am doing a good job and have to accept the blame/objection. 
Also, the process of selecting the release name has its own cost in terms of TC, community time as well as the cost of the
legal check by the foundation.  Having only the release number will help to understand how old that release is and also in
understanding the tick-tock releases.

The Drawback of removing the release name is that it requires changes in release tooling and will be less helpful in marketing.
The former one is a one-time change and later once can be adjusted with having some tag lines.
For example, "'OpenStack 2022.2':  The integration of your effort"

Considering all the facts, TC agreed to start the release name drop process in Gerrit and the community can add their feedback
on that.

Gate Health and Checks
===================

This is something TC started monitoring and helping on gate health and it is going in a good direction. TC will continue doing it
in every weekly meeting. We encourage every project to keep monitoring their rechecks and TC also will keep eye on those
when we notice a large number of rechecks without comment. 

* Infra resource checks:
The next thing we checked was if there are enough infra resources we have to have the smooth development of OpenStack in
the Zed cycle. Clark and fungi gave updates on those and we are all in a good position at least not in a panic situation. But to
monitor the required infra resources correctly, TC will work on listing the required and good-to-have services we need for
OpenStack development. That way we will be able to judge the infra resource situation in a better way.

* ELK service:
The new dashboard for ELK service is ready to use, you can find the dashboard and credential information in this review[12]. We
encourage community members to start using it and provide feedback. Accordingly, Clark will plan to stop the older ELK servers
in a month or so. Thanks dpawlik for working on it.

* nested-virt capable instances for testing
We discussed it and they are available and can be requested from the job definition but there is always a risk of a slow job and
timing failure and it is preferred to use those in the project gate only and not in the integrated/cross-project gate,

Testing strategy:
=============

* Direction on the lower constraints
As lower constraints are not well tested and always causing issues in upstream CI, we are discussing it again to find some permanent
solution. We discussed both cases 1. keeping lower bound in requirements.txt files and 2. lower-constraints.txt file and its testing. We
agree to keep the 1st one but drop the 2nd one completely. Below is the agreement and action items:

** AGREE/ACTION ITEM:
*** Write up how downstream distro can test their constraint file with OpenStack upstream.
*** Keep the lower bound in requirements.txt but add a line saying that they are the best effort but not tested. If they are wrong
then file a bug or fix it. 
*** Drop the lower-constraints.txt file, l-c tox env and its testing on master as well as on all stable branches.

TC will add the resolution and communicate that on ML too. 

* Release specific job template for testing runtime:
As brought up in ML[13], release-specific template does not actually work for independent releases model repo where the OpenStack
bot does not update the template. Stephen added a new generic template to use in such repo. But if we see these release-specific
templates are adding an extra step to update in every repo on a new release. Having a generic template and handling the python version
jobs with branch variant[14] will work fine. But the generic template will move all the projects to new testing runtime at the same time from
a central place. We agree to do it but with proper communication to all the projects and give some window to test/fix them before we switch
to new testing versions.

* Performance testing/monitoring
Performance testing is good to do but as per our current CI resources, it is very difficult. We discussed few ideas where we can keep eyes
on the performance aspect:
** At the start, we can monitor the memory footprint via performance stats in CI jobs
** Before and after DB query counting (how does this patch affect the number of DB queries)
** For API performance, we can monitor all the API requests via a single behind tls proxy gateway and collect the stats.

* When to remove the deprecated devstack-gate?
We agree to remove it once stable/wallaby is EM and will update the same in README file also with the expected date of stable/wallaby to
be EM (2022-10-14).

Improvement in project governance (continue from Yoga PTG....)
================================================

This is regarding how we can better keep eyes on the less or no active projects and detect them earlier in the cycle to same the release,
infra resources. In the Yoga release, we faced the issue of a few projects being broken during the last date of the final release. We discussed
it in the last PTG and agreed to start the 'tech-preview' idea. To define what can be entry and exit criteria for any project to be in 'tech-preview',
we need input from the release team on stopping the auto release of projects or so. The release team was busy in their PTG at the same time
we had this discussion so I will continue the discussion in TC weekly meetings.  

Current state of translations in OpenStack 
================================

Ianychoi and Seongsoocho attended this session and gave us a detailed picture of the translation work, what is the current state and what
all work is required to do including migration from Zanata to weblate.  The main issue here is we need more people helping in this work.
We also asked who uses those translations and they should come up to help.  As the next step, we agreed to add a new question to the user
survey ("Do you use i18n translations in your deployments? If yes please tell us which languages you use") and get some stats about translation
usage. Meanwhile, Ian and seongsoocho team will continue maintaining. Thanks to Ian and Seongsoocho for their best effort to maintain it.

Cross community sessions with the k8s steering committee team:
=================================================

k8s steering committee members (Paris, Dims, Bob, and Tim) joined the OpenStack Technical Committee in PTG. We discussed the various
topics about Legal support/coverage for contributors, especially in the security disclosure process and export control. 
We asked if k8s also have the same level of language translation support in code/doc as that we have in OpenStack and they have only for the
documentation which is also if anyone proposes to do.  Then we discussed the long-term sustainability efforts, especially for the experience
contributors who can take higher responsibility as well as train the new contributors. This is an issue in both communities and none of us
have any solution to this.

In the last, we discussed how k8s recognize the contributor's good work. In k8s along with appreciating on slack, ML, they issue the coupon and
badges from the CNCF foundation.

TC Zed Activities checks and planning
=============================

This is the last hour of the PTG and we started with the yoga cycle retrospective.

* Yoga Retrospective
 We are doing good in gate health monitoring as well doing more technical work. On the improvement side, we need to be faster at making
the decision on things that are in discussion for long period instead of keeping them open. 

* Pop Up Team Check In
After checking with the status and need of both active popup teams, we decided to continue with both in Zed cycle.

* TC weekly Meeting time check
We will keep the current time until daylight saving time changes again. Also, we will continue the video call once a month. 

* TC liaison continue or drop?
As we have improved the interaction with the project with the weekly meetings as well as in PTG sessions (TC+Leaders interaction session), we
agreed to drop the TC liaisons completely. 

* I will prepare the Zed cycle TC Tracker (activities we will do in Zed cycle)

* Next week's TC meeting is cancelled and we will resume meeting from 21st April onwards.

That is all from PTG, thanks for reading it and stay safe!

[1] https://docs.openstack.org/project-team-guide/technical-guides/unified-limits.html
[2] https://review.opendev.org/c/openstack/governance/+/816387
[3] https://review.opendev.org/c/openstack/governance/+/835102
[4] https://docs.openstack.org/project-team-guide/spread-the-word.html
[5] https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures
[6] https://review.opendev.org/c/openstack/governance/+/836888
[7] http://lists.openstack.org/pipermail/openstack-discuss/2021-August/024122.html
[8] https://etherpad.opendev.org/p/qa-zed-ptg-fips
[9] https://etherpad.opendev.org/p/zed-ptg-fips-goal
[10] https://docs.openstack.org/project-team-guide/deprecation.html
[11] https://review.opendev.org/c/openstack/governance/+/829563
[12] https://review.opendev.org/c/openstack/governance-sigs/+/835838
[13] http://lists.openstack.org/pipermail/openstack-discuss/2022-March/027676.html
[14] https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/833892