March 2024 - openstack-discuss - lists.openstack.org

[all][packaging][release][qa] pep-517 and pip 23
by smooney＠redhat.com 06 May '24

06 May '24

hi folks, TL;DR ----- this is a long email so the tl;dr is we should add pyproject.toml ``` [build-system] requires = ["pbr>=5.7.0", "setuptools>=36.6.0", "wheel"] build-backend = "pbr.build" ``` to all openstack projects to supprot pip 23.1+ Long version ------------ As some of you may know the ptg happend last week an done of hte topic raised there was a know limitation in devstack where GLOBAL_VENV=True does not work on centos/fedora derived distros. while i generally dont use centos/fedora derived distros for my daily work i have been using devstack for over 10 years at this point so when im told it broken i ocationlly will go fix it. In this case the issue is not with devstack (it often isnt) and is actully in our collective reliance on a deprecated functionality and how we package openstack today. Back story, for better or worse when openstack was created the landscape of python packaging was significantly different then it is today. To help corodinate our dependcies the openstack comunity went all in on requirements.txt and developed upper-constraits to manage our requirements and developed PBR (python build Reasonableness). both of those approches have served us well and likely will continue to do so for some time. Since then pyproject.toml has started to gain popularity and pep-517 (A build-system independent format for source trees) has been adotpted but so far openstack as a comunity has not embraced those changes. (although pbr already supprot pep-517 :) ). Until now this has not been a problem however with recent release of pip things are now breaking. Problem Context: - openstack does not provide pyporject.toml files - openstack project configure setuptools to use pbr via setup.py - setuptools support generating scripts form entry points(console_scripts) which we use for project binaries( nova- compute, neutron-server, ...) - openstack api services use a pbr extension know as wsgi_scripts which adds a similar functionality for generating wsgi application mains file - devstack due to coming changes in distro to block "sudo pip install" now installs openstack in a single by default on debian distros. on centos/fedora distros it currently almost functional (but not). - devstack installs in editable mode because that is how most developers use devstack for developemnt What changed: - pip 20.2 deprecated support for falling back to calling setup.py when pyproject.toml is not present - https://pip.pypa.io/en/stable/news/#v20-2 - https://github.com/pypa/pip/issues/8368 - at this point when pyproject.yaml is not called pip calls setup.py install [-e] <path to repo or package name> - setup.py invokes pbr to generate wsgi_scripts and other files - in pip 23.1 the fallback was removed https://pip.pypa.io/en/stable/news/#v23-1 - pip now builds a temp local wheel instead of calling setup.py install when pyproject.toml is not found - in editable mode (-e) the default way the wheel is built does not invoke pbr properly to generate the wsgi files - as a result the wsgi files are not installed to the venv bin dir and this causes devstack to fail to start keystone (or any other wsgi api) The fix, there are a few ways to approch this: - clamp pip (https://review.opendev.org/c/openstack/devstack/+/898845) this is a terible idea and it breaks debian - clamping pip below 23.1 works on centos for now however OVS on debian already requires pep 517 for installing - as such when that version of ovs (or other projects start enforcing that) hits centos it will break - add a minimal pyproject.toml to all openstack project - example for keystone https://review.opendev.org/c/openstack/keystone/+/899519 - this example jsut adds the minium pyproject.toml to enable pbr to be used as the build system - it does not remove supprot for setup.py but it will not be used by default going forward whit new version of pip - all config is still in setup.cfg - add pyproject.toml support but remove setup.py support https://review.opendev.org/c/opendev/bindep/+/816741 - this goes one step futrue and actuly nulls out setup.py - we could take this approach if we want to but its not required. The reason for this email is to get agreement on how to support pip 23.1+ My proposal would be to add pyproject.toml with the minimal config required to enable pip 23.2 but not drop the setup.py support in this cycle. i would suggest that we can deprecate installing via setup.py in caracal if that is desirable but not make that change until the next slurp to give time to projects and packages to catch up. if there are no objections to that i would like to ask what is the preferd way to create and submit the patches for the minimal pyproject.toml ``` [build-system] requires = ["pbr>=5.7.0", "setuptools>=36.6.0", "wheel"] build-backend = "pbr.build" ``` i can propose patches to the main project deployed by devstack by default. i.e. nova/neutron ectra but it might be better if someone form the core teams of each project could take on that task and chasing the reviews. if someone has a script to generate a patch for a set of repos that just add the above pyporject.toml then we can use that to keep a consistent topic/commit message across all patches. my proposal for testing this is two fold. in ci we can test the patches are functional by using a DNM patch to devstack to force enable the gloal venv on centos and depend on the set of patches, i will modify https://review.opendev.org/c/openstack/devstack/+/898845 to do that locally we can test that the patch is functional by doing the following substituting keystone for the relevant project. ``` git clone https://opendev.org/openstack/keystone cd keystone python3 -m venv .venv . .venv/bin/activate python3 -m pip install -U pip python3 -m pip install -e . ls .venv/bin/keystone* ``` the ls will not contain the wsgi script with the addition of the pyproject.toml it should. on debian sid i can reproduce the same issue the problem we are seeing on centos 9 now will start impacting other distos eventually so its good to fix this now. if i start proposing this to glance/nova/neutron/cinder/swift/horizon/placement (to get a default devstack working) can i get cores to review? can cores form each project nominate peopel to review or volenteer to do this work. i have created https://etherpad.opendev.org/p/pep-517-and-pip-23 to track this work i have some other work that i think we can do in parallel like addign pyporject.toml support to the pti requirements if we go this way but that i think we can discuss on a governacne patch to updated it. anyway this is a lot to digest so feedback is welcome. if someone else would like to drive this that is also welcome (i.e. tc/release team) regards sean.

10 27

[largescale-sig] Next meeting: March 27, 9utc
by Thierry Carrez 24 Apr '24

24 Apr '24

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 9UTC. Arnaud Morin will be chairing. You can doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20240327T09 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

2 3

[oslo][release][requirements] broken watch_log_file depends on unmaintained pyinotify
by Takashi Kajinami 19 Apr '24

19 Apr '24

Hello, # This is mainly a oslo topic but I'm adding release team and requirements # team because we likely need new stable/2024.1 release of oslo.log. I looked into an old oslo.log bug[1] during the weekend and noticed that the watch_log_file option in oslo.log has been broken for some time. Looking into the problem further, I leaned that pyinotify[1] library, which is used in this feature, hasn't been updated for 9 years (!!). What is worse, pyinotify dependes on asynccore which has been removed from Python 3.12 and I'm skeptical that we can fix pyinotify to make it work with Python 3.12. There was an old discussion to replace pyinotify by inorify[3] but itnofiy hasn't been updated for 4 years and looks also unmaintained. I think the only option we have here is deprecating the functionality in 2024.1 so that we can remove it in early phase of 2024.2. Because we are quite close to 2024.1 GA, we have to make the decision early. I've already proposed the change for deprecation[4]. If you have any concerns then please reply to this email or leave your comments in the review. Thank you, Takashi [1] https://bugs.launchpad.net/oslo.log/+bug/1740111 [2] https://github.com/seb-m/pyinotify [3] https://github.com/dsoprea/PyInotify [4] https://review.opendev.org/c/openstack/oslo.log/+/914788 -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit

3 5

[designate] Support for DNSSEC // Bump in the wire signer via BIND9 or Knot
by Christian Rohmann 18 Apr '24

18 Apr '24

Hello, I was wondering is anybody knew about the current state of DNSSEC signing for Designate-managed zones. Since Designate MDNS serves as primary they actually should do the signing / provide already signed zones via zone transfers. Adding support for DNSSEC this was last discussed for Kilo [1], but that spec was never finished, DNSSEC support never implemented. One approach to do this is using a bump in the wire signer [2][3][4] and have an intermediate BIND9 or Knot server doing the signing. Has anybody implemented something of this kind? If so, how do your users receive their initial DS / DNSKEY for the parent zone? Regards Christian [1] https://review.opendev.org/c/openstack/designate-specs/+/132571 [2] https://bind9.readthedocs.io/en/latest/dnssec-guide.html#bump-in-the-wire-s… [3] https://jpmens.net/2023/07/22/adieu-opendnssec-bienvenido-knot-dns/ [4] https://labs.ripe.net/author/anandb/dnssec-signer-migration/

4 7

[heat] Parametrization of Neutron trunk subports scaling down
by Lajos Katona 17 Apr '24

17 Apr '24

Hi, With Hot templates we run into a scaling issue with Neutron trunk subports. The task is to create Neutron trunks with "vlan_count" number of subports, and with stack update be able to add or remove subports of the created trunk. We tried with resourceGroup, and that works well if we increase the number of subports. The problem comes with decrease, as with resourceGroups I can't (my lack of knowledge) tell heat_engine that first update the trunk and remove the subport and after that delete the port. And Neutron API gives back conflict as the port which heat_engine tries to delete is still used by the trunk. Do you know a way to accomplish such an operation with Hot templates? Thanks in advance for the help. Best wishes Lajos Katona (lajoskatona)

2 4

[kolla-ansible][Antelope][Heat] Heat UpdateWaitConditionHandle Issues
by Devin Patterson 16 Apr '24

16 Apr '24

Hello, We are creating a workflow to use Heat auto-scaling groups with wait conditions. The ideal goal would be to have the wait condition clear and require another signal when the Heat stack is updated. Trying to use the resource OS::Heat::UpdateWaitConditionHandle instead of OS::Heat::WaitConditionHandle. This works during initial creation, but when I udpdate the stack with a new configuration, I get the following error. UpdateWaitConditionHandle.update() got an unexpected keyword argument 'new_template_id' Any ideas on how to use wait conditions during stack updates or to fix the above error. Thanks in advance. Devin

2 2

[all][tc] Modernize Python Stack
by Niklas Schwarz 10 Apr '24

10 Apr '24

Hello, I have just ran over the mailing listing post OpenStack 2024.2 testing runtime by Ganshyam Mann [1] on dropping support for python3.8 for the upcoming Openstack 2024.2 release. This post and some small contributions to some openstack components have lead me to write this post. What I would like to suggest to modernize the Python-Stack used for OpenStack. Such a modernization includes: - A current python version - A modern and up to date static code analysis - Introduce type annotations Let me explain what I mean: A current python version: The current supported Python versions are Python 3.8 to Python 3.11 [2]. As of writing, OpenStack Yoga was just moved to unmaintained. To install this release, e.g. on an Ubuntu Jammy operating system the packages are included in the Ubuntu-Packages. The default Python version on an Ubuntu 22.04 operating system is Python3.10. And looking at the support of the Python versions [3] Python 3.8 to 3.10 only receive security updates but the end is near for Python 3.8 to be out of support. The policy defined which Python version should be used for testing [4] will result that Python 3.8 is still used for the OpenStack 2024.1 release [5]. And this will also result in required support for Python3.8 in the upcoming releases of OpenStack. I my opinion the policy should be reworked and the OpenStack releases should be more aggressive on dropping support for older Python versions. Modern features of the language cannot be otherwise be used, therefor improvements in speed, security and general toolchain/ecosystem cannot be fully leveraged. By defining the lower bound of the Python Version we are limiting ourself to this version but we might also reduce the features we can use because of deprecations of the newer versions . A modern and up to date static code analysis: Currently flake8 and pylint are used both of which are modern analysis tools. But their configuration is in my opinion not up to date. Lets have a look at the configuration of the neutron-lib [6]. As seen a lot of rules are disabled by the configuration and are needed at the current moment at the state of the project. In addition some rules need to be adjusted such as the line length (80 characters is a very aggressive limit and on most modern displays a line length of 100 to 120 can fit without any problems, personal opinion). The adjustment of the static code analysis besides opinionated rules will lead me to my third point to modernize the Python stack used Introduce type annotations Python type annotations are well supported in the current Python versions and through the typing package additional types are added. The introduction of type annotations will improve the developer experience. This is also motivated by the Python foundation in pep-484 [7]. Because of the type annotation all modern IDEs can provide (more precise) type hints. The type hints can and in in my experience will reduce the introduction of bugs (accessing members or calling methods, no logical bugs) and will shorten the time to develop a new feature or fix a logical bug and do not run in runtime exceptions. The datatypes passed to a method are clear and we do not need to dig down a rabbithole to find out what is passed to a method. The name of a variable does not always tell the truth. As an example I want to look at a patch [8] for the neutron-vpnaas component. Christian Rohmann and I worked on this patch for the component together (this has also lead me to write this email) to improve the reconciling for VPNaaS. It took as along time to find out that a RouterInformation object is passed in to a method instead of a router but the variable was named router. We both agreed on that in the first place the naming of the variable was wrong but type annotations would have saved us a good amount of time. This does not mean that the naming is not important and that type annotations will save the world, the context which includes the name of variables is important and type annotations will add additional information to the context. And I have just touched the surface on what is possible with the introduction of type annotations, thinking of automatic document generation, test generation, even code generation,... As a reference to my point modernizing the static analysis this can then also include the introduction of mypy to check the types used. Of course such a change cannot be done in one big commit and needs time but I suggest to start adding type annotations and improve the overall code quality of the OpenStack services, starting with the oslo packages as they are used by mostly all OpenStack components. Of course such changes cannot be done for such a huge project in one day. It took a long time to remove support for Python 2. So I do not expect such changes to happen immediately. But it would be nice to have a discussion about it and maybe create a road map and adjust some policy on supported Python versions and create new policies how to write Python code and what additional tools should be used. I can also understand if not everything is possible but the last two points need to be addressed in my opinion. These changes will lead overall to better code quality but most important the introduction to the big code base will be easier for new developers and reviewers will also have a better time. Best regards Niklas [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [2] https://docs.openstack.org/tempest/latest/supported_version.html [3] https://devguide.python.org/versions/ [4] https://governance.openstack.org/tc/resolutions/20181024-python-update-proc… [5] https://governance.openstack.org/tc/reference/runtimes/2024.1.html [6] https://github.com/openstack/neutron-lib/blob/master/.pylintrc [7] https://peps.python.org/pep-0484/#rationale-and-goals [8] https://review.opendev.org/c/openstack/neutron-vpnaas/+/875745

15 27

[kolla-ansible][horizon]Create instance default parameters
by Franck VEDEL 10 Apr '24

10 Apr '24

Hi, With horizon, is it possible de have this default parameter when you create an instance: in source, create a new volume: « No ». And.. if it’s possible, how configure this with kolla-ansible ? Thank’s for your help. Franck

2 4

Nova filter based on VM metadata?
by Ken D'Ambrosio 08 Apr '24

08 Apr '24

Hey, all. I'm looking for a way to to filter to a particular aggregate based on arbitrary VM metadata. (E.g., maybe FACTER facts from Puppet that we pass to the VM as metadata.) Could someone point me toward documentation that might outline how to accomplish that? As I see it, there's really one thing I need to understand -- the rest I can probably glean from extant filters: how (or if) a filter could access a given VM's metadata from the API creation request (if that's even possible). Thanks! -Ken

3 3

[openstack-ansible] : Ussuri minor version update.
by keshav bareja 04 Apr '24

04 Apr '24

Hi Team I am planning to update my environment from the current ussuri minor version to the latest minor version available for the Ussuri. I found the procedure below. https://docs.openstack.org/openstack-ansible/ussuri/admin/upgrades/minor-up… I have some queries on this: 1. How to find the current minor release version of the Ussuri deployed on my environment? Is there any command or way that can help? 2. What is the latest minor version available for Ussuri? In the Ussuri release notes page I could see 2.10, can someone please confirm if 2.10 is the latest version? 3. Also I need to know the bug fixes between my current release to the latest minor ussuri release. 4. Can someone confirm what all pre-requisite I need to take care of before going with the update. 5. Anyone having experience with such minor updates , Please share precaution and recommendations, Ussuri Series Release Notes <https://docs.openstack.org/releasenotes/openstack-ansible/victoria.html> <https://docs.openstack.org/releasenotes/openstack-ansible/train.html> <https://bugs.launchpad.net/openstack-ansible/+filebug?field.title=Ussuri%20…> 21.2.10 <https://docs.openstack.org/releasenotes/openstack-ansible/ussuri.html#relno…>Security Issues <https://docs.openstack.org/releasenotes/openstack-ansible/ussuri.html#secur…> - This release eliminates following security issues: - OSSA-2021-005: https://security.openstack.org/ossa/OSSA-2021-005.html - OSSA-2021-006: https://security.openstack.org/ossa/OSSA-2021-006.html

2 5