January 2024 - openstack-discuss - lists.openstack.org

[all][packaging][release][qa] pep-517 and pip 23
by smooney＠redhat.com 06 May '24

06 May '24

hi folks, TL;DR ----- this is a long email so the tl;dr is we should add pyproject.toml ``` [build-system] requires = ["pbr>=5.7.0", "setuptools>=36.6.0", "wheel"] build-backend = "pbr.build" ``` to all openstack projects to supprot pip 23.1+ Long version ------------ As some of you may know the ptg happend last week an done of hte topic raised there was a know limitation in devstack where GLOBAL_VENV=True does not work on centos/fedora derived distros. while i generally dont use centos/fedora derived distros for my daily work i have been using devstack for over 10 years at this point so when im told it broken i ocationlly will go fix it. In this case the issue is not with devstack (it often isnt) and is actully in our collective reliance on a deprecated functionality and how we package openstack today. Back story, for better or worse when openstack was created the landscape of python packaging was significantly different then it is today. To help corodinate our dependcies the openstack comunity went all in on requirements.txt and developed upper-constraits to manage our requirements and developed PBR (python build Reasonableness). both of those approches have served us well and likely will continue to do so for some time. Since then pyproject.toml has started to gain popularity and pep-517 (A build-system independent format for source trees) has been adotpted but so far openstack as a comunity has not embraced those changes. (although pbr already supprot pep-517 :) ). Until now this has not been a problem however with recent release of pip things are now breaking. Problem Context: - openstack does not provide pyporject.toml files - openstack project configure setuptools to use pbr via setup.py - setuptools support generating scripts form entry points(console_scripts) which we use for project binaries( nova- compute, neutron-server, ...) - openstack api services use a pbr extension know as wsgi_scripts which adds a similar functionality for generating wsgi application mains file - devstack due to coming changes in distro to block "sudo pip install" now installs openstack in a single by default on debian distros. on centos/fedora distros it currently almost functional (but not). - devstack installs in editable mode because that is how most developers use devstack for developemnt What changed: - pip 20.2 deprecated support for falling back to calling setup.py when pyproject.toml is not present - https://pip.pypa.io/en/stable/news/#v20-2 - https://github.com/pypa/pip/issues/8368 - at this point when pyproject.yaml is not called pip calls setup.py install [-e] <path to repo or package name> - setup.py invokes pbr to generate wsgi_scripts and other files - in pip 23.1 the fallback was removed https://pip.pypa.io/en/stable/news/#v23-1 - pip now builds a temp local wheel instead of calling setup.py install when pyproject.toml is not found - in editable mode (-e) the default way the wheel is built does not invoke pbr properly to generate the wsgi files - as a result the wsgi files are not installed to the venv bin dir and this causes devstack to fail to start keystone (or any other wsgi api) The fix, there are a few ways to approch this: - clamp pip (https://review.opendev.org/c/openstack/devstack/+/898845) this is a terible idea and it breaks debian - clamping pip below 23.1 works on centos for now however OVS on debian already requires pep 517 for installing - as such when that version of ovs (or other projects start enforcing that) hits centos it will break - add a minimal pyproject.toml to all openstack project - example for keystone https://review.opendev.org/c/openstack/keystone/+/899519 - this example jsut adds the minium pyproject.toml to enable pbr to be used as the build system - it does not remove supprot for setup.py but it will not be used by default going forward whit new version of pip - all config is still in setup.cfg - add pyproject.toml support but remove setup.py support https://review.opendev.org/c/opendev/bindep/+/816741 - this goes one step futrue and actuly nulls out setup.py - we could take this approach if we want to but its not required. The reason for this email is to get agreement on how to support pip 23.1+ My proposal would be to add pyproject.toml with the minimal config required to enable pip 23.2 but not drop the setup.py support in this cycle. i would suggest that we can deprecate installing via setup.py in caracal if that is desirable but not make that change until the next slurp to give time to projects and packages to catch up. if there are no objections to that i would like to ask what is the preferd way to create and submit the patches for the minimal pyproject.toml ``` [build-system] requires = ["pbr>=5.7.0", "setuptools>=36.6.0", "wheel"] build-backend = "pbr.build" ``` i can propose patches to the main project deployed by devstack by default. i.e. nova/neutron ectra but it might be better if someone form the core teams of each project could take on that task and chasing the reviews. if someone has a script to generate a patch for a set of repos that just add the above pyporject.toml then we can use that to keep a consistent topic/commit message across all patches. my proposal for testing this is two fold. in ci we can test the patches are functional by using a DNM patch to devstack to force enable the gloal venv on centos and depend on the set of patches, i will modify https://review.opendev.org/c/openstack/devstack/+/898845 to do that locally we can test that the patch is functional by doing the following substituting keystone for the relevant project. ``` git clone https://opendev.org/openstack/keystone cd keystone python3 -m venv .venv . .venv/bin/activate python3 -m pip install -U pip python3 -m pip install -e . ls .venv/bin/keystone* ``` the ls will not contain the wsgi script with the addition of the pyproject.toml it should. on debian sid i can reproduce the same issue the problem we are seeing on centos 9 now will start impacting other distos eventually so its good to fix this now. if i start proposing this to glance/nova/neutron/cinder/swift/horizon/placement (to get a default devstack working) can i get cores to review? can cores form each project nominate peopel to review or volenteer to do this work. i have created https://etherpad.opendev.org/p/pep-517-and-pip-23 to track this work i have some other work that i think we can do in parallel like addign pyporject.toml support to the pti requirements if we go this way but that i think we can discuss on a governacne patch to updated it. anyway this is a lot to digest so feedback is welcome. if someone else would like to drive this that is also welcome (i.e. tc/release team) regards sean.

10 27

Accessing virsh in openstack wallaby | tripleo
by Swogat Pradhan 24 Mar '24

24 Mar '24

Hi, Can someone please tell me how can i access the virsh utility on the compute node on openstack wallaby? Previously we used nova_libvirt but in wallaby there are multiple containers and i am unable to use virsh utility to edit my domain.xml file. List of containers: [root@dcn01-hci-2 ~]# podman ps | grep nova 10d981158842 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo kolla_start 2 months ago Up 2 months ago nova_virtlogd_wrapper e91667516afc 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo kolla_start 2 months ago Up 2 months ago nova_virtsecretd a4b07fe0e833 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo kolla_start 2 months ago Up 2 months ago nova_virtnodedevd d7e1db393e9c 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo kolla_start 2 months ago Up 2 months ago nova_virtstoraged 5308a171793e 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo kolla_start 2 months ago Up 2 months ago nova_virtqemud f490a6249ba1 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo kolla_start 2 months ago Up 6 weeks ago nova_virtproxyd ab0441e56957 172.25.201.68:8787/tripleomaster/openstack-nova-compute:current-tripleo kolla_start 2 months ago Up 2 months ago (healthy) nova_migration_target f35fbba5b690 172.25.201.68:8787/tripleomaster/openstack-nova-compute:current-tripleo kolla_start 2 months ago Up 6 weeks ago (unhealthy) nova_compute 82bc01993106 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo /usr/sbin/virtlog... 2 months ago Up 2 months ago Can someone please suggest how I can i edit an instance xml file in this setup? With regards, Swogat Pradhan

2 5

[Freezer] Is Freezer dead?
by pi3.14＠tuta.io 18 Mar '24

18 Mar '24

Hi folks, is Freezer still supported? I don't see any real development going on with it. Is it recommended to use in a production Cloud environment? Are there other free and open source alternatives to Freezer? Thanks Roby

8 19

[swift] Increase part power on account/container ring
by Thiago De Moraes Teixeira 13 Mar '24

13 Mar '24

Morning, Here we have a cluster with Accounts/Containers ring with part power 8 and our Objects ring with part power 20 (builted later). Now we want to migrate the 'old' Accounts/Containers ring to a new one, with higher part power, like 20. Is there a better way to do this? I'm doing some crazy tests with SAIO and multiples storage nodes, based in build a new ring with part power 20 and just swap the old files (account/container.ring.gz) with the new ones and let the replicators do their jobs, moving *.db files to their new home partition. Until today these tests worked fine, but I don't know how to do it on a large scale, with some terabytes of data. Just wondering about some ways to do that :) And before/after this change, is there any command in swift that i can audit this entire process, ensuring that cluster data remains the same? Thanks! -- _‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’._ * **‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*

3 8

[heat][monasca][sahara][senlin] Features dependent on inactive projects
by Takashi Kajinami 08 Mar '24

08 Mar '24

Hello, As you know, some of the OpenStack projects were marked inactive during this cycle. There are still on-going discussions but from my view these are very likely excluded from the upcoming 2024.1 release and it's unclear yet if any of these projects will be kept. Currently heat provides features integrated with some of the inactive projects (Monasca, Sahara and Senlin), and we have to decide how we deal with these features[1]. The main concern is that we can't maintain the features for the inactive projects, and IMO we should retire these features if these inactive projects are abandoned or are no longer maintained in the consistent way with the other active projects. Based on the current situation, I'd propose the following steps for upcoming cycles. - Deprecate the features dependent on inactive projects in 2023.2 release - Note that we have to remove these features directly during this cycle in case any problems related to these project block the upcoming heat release. - Remove these features in 2024.1 release unless we are sure that these projects are kept maintained as active projects in the openstack namespace, with following the global release cycle. Please let me know if you have any concern about the proposed plan. If I hear no objections then I'll propose patches for deprecation. Thank you, Takashi Kajinami irc: tkajinam [1] I've submitted a series of patches to remove these features and these would give a clear view about the features affected. https://review.opendev.org/q/topic:%22inactive-project-removal%22+project:o…

3 6

[cinder][glance][nova][telemetry][monasca][oslo][vmware] Can we deprecate(and remove) VMWare support ?
by Takashi Kajinami 07 Mar '24

07 Mar '24

Hi all, As you might know, some of the OpenStack projects currently support usage of VMWare component. However recently the VMware virt driver in nova was marked experimental in 2023.2 release [1] and it has been discussed that the driver will be removed in 2023.2 release[2], after deprecation is backported to 2023.1. (Please someone from nova team correct me if I'm wrong) So my current question is; Does it make sense to deprecate all of the features to support VMWare component ? My understanding is that these features were added to be used along with VMWare virt driver in nova, but once the virt driver is removed this use case is voided. I'll list up the features within the discussion scope below. nova: VMWare virt driver (this is already experimental by [1]) cinder: VMware Storage Driver (vmdk) glance: VMWare Datastore (I proposed deprecating this in [3][4]) ceilometer: VMWare virt driver (this was deprecated by [5]) os-brick: VmdkConnector monasca-agent: vcenter collector (*1) oslo.vmware: the whole project NOTE: These features are found based on dependency on oslo.vmware but there can be a few not listed here. Outside of OpenStack's governance, there are two projects dependent on oslo.vmware, networking-vsphere and vmware-nsx but both of these look inactive. Because of SLURP, we have to complete deprecation during in 2024.1 release so that we can drop the feature right after 2024.1. If we fail then we have to maintain these features until 2025.1 (the features already deprecated can be removed earlier, though). So I'd like to get feedback and decision before deadline for 2024.1 Thank you, Takashi (*1) Monasca is now inactive so I doubt anyone would work on this topic before it is retired. [1] https://review.opendev.org/c/openstack/nova/+/863911 [2] https://review.opendev.org/c/openstack/nova/+/897017 [3] https://review.opendev.org/c/openstack/glance_store/+/893696 [4] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [5] https://review.opendev.org/c/openstack/ceilometer/+/894540

6 21

[Watcher] How use Watcher to scheduler Openstack resource without cpu_usage
by Nguyễn Hữu Khôi 06 Mar '24

06 Mar '24

Hello guys. I want to use watcher to scheduler openstack resources but some openstack versions are missing cpu_usage metric. How do we use Watcher without it? Thank you. Nguyen Huu Khoi

3 5

[openstack][skyline] Some works with skyline
by Nguyễn Hữu Khôi 25 Feb '24

25 Feb '24

Hello guys. I am working on Skyline. I can integrate my monitor tab on it. I will try to do something about it. If it works properly, I can contribute. It is a very nice project. I hope it will grow and grow in the future. Thanks Skyline Team. [image: image.png] Nguyen Huu Khoi

3 7

[tc][murano] Status of the murano project
by Takashi Kajinami 20 Feb '24

20 Feb '24

Hello, As I brought in another thread about yaql library maintenance[1], It seems murano project is not active now, based on the following points. * The changes merged during recent two years but all of these are gate fix * Its CI is currently broken and no one is trying to fix it * I tried reaching the current PTL by email or gerrit review but didn't get any feedback even after waiting for a few weeks I'm wondering if murano project should be added to the inactive projects, or at least its project health and plan needs to be reviewed by TC, before we include this to Caracal deliverables. # Though it might be too later to drop the deliverables now Thank you Takashi Kajinami irc: tkajinam [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack…

4 15

Encountering a live migration issue on Kolla Ansible Deployment
by khurshidsx＠gmail.com 15 Feb '24

15 Feb '24

Hello all, I recently set up a cluster using Kolla Ansible, but I'm encountering a live migration issue. I would appreciate any advice or insights you can provide. Error Message is rule:create_port_binding is disallowed by policy Neutron server returns request_ids: ['req-dce2d22c-349b-4759-a0f2-8bebbfbb1aee'] Here are some logs: 2024-01-25 04:55:51.599 23 INFO neutron.api.v2.resource [req-a380e1bc-a94b-4935-be0d-76fcc94becc4 req-dce2d22c-349b-4759-a0f2-8bebbfbb1aee 636854b888d7441eafcd21255e1378f1 b07c4b6d7b2241c187ed84f30b163817 - - default default] create failed (client error): Access was denied to this resource. 2024-01-25 04:55:51.601 23 INFO neutron.wsgi [req-a380e1bc-a94b-4935-be0d-76fcc94becc4 req-dce2d22c-349b-4759-a0f2-8bebbfbb1aee 636854b888d7441eafcd21255e1378f1 b07c4b6d7b2241c187ed84f30b163817 - - default default] 192.168.252.254,192.168.252.12 "POST /v2.0/ports/e70fab29-d0cd-4db4-8fbc-a572da1bff64/bindings HTTP/1.1" status: 403 len: 308 time: 0.0207033 Neutron server returns request_ids: ['req-dce2d22c-349b-4759-a0f2-8bebbfbb1aee'] 2024-01-25 04:55:51.815 19 ERROR nova.conductor.manager Neutron server returns request_ids: ['req-dce2d22c-349b-4759-a0f2-8bebbfbb1aee'] 2024-01-25 04:55:51.815 19 ERROR nova.conductor.manager Neutron server returns request_ids: ['req-dce2d22c-349b-4759-a0f2-8bebbfbb1aee'] Neutron server returns request_ids: ['req-dce2d22c-349b-4759-a0f2-8bebbfbb1aee']: nova.exception.Forbidden: rule:create_port_binding is disallowed by policy

3 5