May 2023 - openstack-discuss - lists.openstack.org

[Octavia] importlib fails to read first 5 chars from file
by Mihai Gheorghe 09 May '23

09 May '23

Hello, We are experiencing a strange problem with Octavia API in an Openstack deployment. A bit of context: we have an Openstack Ussuri deployment (HA) on Ubuntu Focal. Most Openstack services are running on LXD containers on 3 baremetals. There are also two OVN-dedicated-chassis which act as gateways and nova-compute nodes. A couple of days ago we updated all the packages in the environment to the latest versions (baremetals and lxc containers). After the updates, we started seeing problems with the Octavia API. The API would randomly give an internal server error. Octavia API is running under Apache2 WSGI on 3 LXD containers on 3 separate nodes. Looking into one of the Octavia container under /var/log/apache2/octavia_error.log, we found the following error: [Fri Apr 28 10:11:07.509268 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] mod_wsgi (pid=1312420): Failed to exec Python script file '/usr/bin/octavia-wsgi'. [Fri Apr 28 10:11:07.509455 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] mod_wsgi (pid=1312420): Exception occurred processing WSGI script '/usr/bin/octavia-wsgi'. [Fri Apr 28 10:11:07.511688 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] Traceback (most recent call last): [Fri Apr 28 10:11:07.511884 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "/usr/bin/octavia-wsgi", line 52, in <module> [Fri Apr 28 10:11:07.512260 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] application = setup_app() [Fri Apr 28 10:11:07.512343 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "/usr/lib/python3/dist-packages/octavia/api/app.py", line 59, in setup_app [Fri Apr 28 10:11:07.512435 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] pecan_config = get_pecan_config() [Fri Apr 28 10:11:07.512513 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "/usr/lib/python3/dist-packages/octavia/api/app.py", line 40, in get_pecan_config [Fri Apr 28 10:11:07.512529 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] return pecan_configuration.conf_from_file(filename) [Fri Apr 28 10:11:07.512543 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "/usr/lib/python3/dist-packages/pecan/configuration.py", line 176, in conf_from_file [Fri Apr 28 10:11:07.512553 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] SourceFileLoader(module_name, abspath).load_module(module_name) [Fri Apr 28 10:11:07.512567 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap_external>", line 522, in _check_name_wrapper [Fri Apr 28 10:11:07.512705 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap_external>", line 1027, in load_module [Fri Apr 28 10:11:07.512725 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap_external>", line 852, in load_module [Fri Apr 28 10:11:07.512737 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap>", line 265, in _load_module_shim [Fri Apr 28 10:11:07.512750 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap>", line 702, in _load [Fri Apr 28 10:11:07.512763 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap>", line 671, in _load_unlocked [Fri Apr 28 10:11:07.512775 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap_external>", line 844, in exec_module [Fri Apr 28 10:11:07.512788 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap_external>", line 981, in get_code [Fri Apr 28 10:11:07.512800 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap_external>", line 911, in source_to_code [Fri Apr 28 10:11:07.512839 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed [Fri Apr 28 10:11:07.512869 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "/usr/lib/python3/dist-packages/octavia/api/config.py", line 1 [Fri Apr 28 10:11:07.512893 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] Copyright 2014 Rackspace [Fri Apr 28 10:11:07.512921 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] ^ [Fri Apr 28 10:11:07.512938 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] IndentationError: unexpected indent The original /usr/lib/python3/dist-packages/octavia/api/config.py file is added to the bug report. As seen in the file, the first line starts with a comment, but python is trying to run it as code. Removing the first 5 charactersin the file so that string above starts right after the comment yelds the following traceback: [Wed Apr 26 17:08:22.609980 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] mod_wsgi (pid=330647): Failed to exec Python script file '/usr/bin/octavia-wsgi'. [Wed Apr 26 17:08:22.610114 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] mod_wsgi (pid=330647): Exception occurred processing WSGI script '/usr/bin/octavia-wsgi'. [Wed Apr 26 17:08:22.611039 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] Traceback (most recent call last): [Wed Apr 26 17:08:22.611097 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] File "/usr/bin/octavia-wsgi", line 52, in <module> [Wed Apr 26 17:08:22.611106 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] application = setup_app() [Wed Apr 26 17:08:22.611125 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] File "/usr/lib/python3/dist-packages/octavia/api/app.py", line 59, in setup_app [Wed Apr 26 17:08:22.611135 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] pecan_config = get_pecan_config() [Wed Apr 26 17:08:22.611152 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] File "/usr/lib/python3/dist-packages/octavia/api/app.py", line 40, in get_pecan_config [Wed Apr 26 17:08:22.611160 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] return pecan_configuration.conf_from_file(filename) [Wed Apr 26 17:08:22.611178 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] File "/usr/lib/python3/dist-packages/pecan/configuration.py", line 168, in conf_from_file [Wed Apr 26 17:08:22.611327 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] compiled = compile(f.read(), abspath, 'exec') [Wed Apr 26 17:08:22.611355 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] File "/usr/lib/python3/dist-packages/octavia/api/config.py", line 1 [Wed Apr 26 17:08:22.611381 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] right 2014 Rackspace [Wed Apr 26 17:08:22.611419 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] ^ [Wed Apr 26 17:08:22.611437 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] SyntaxError: invalid syntax It seems that when the source file is read by python, the first 5 characters are skipped for some reason. Digging deeper, I see no reason for importlib to trim those characters. Internally it uses _io.open_code().read() which should return the entire file without trimming anything If adding the comment after the 5th char on the first line, the file is read as is should but the error moves to a different file: [Fri Apr 28 10:20:24.730194 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] mod_wsgi (pid=1316806): Failed to exec Python script file '/usr/bin/octavia-wsgi'. [Fri Apr 28 10:20:24.730379 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] mod_wsgi (pid=1316806): Exception occurred processing WSGI script '/usr/bin/octavia-wsgi'. [Fri Apr 28 10:20:24.732137 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] Traceback (most recent call last): [Fri Apr 28 10:20:24.732294 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] File "/usr/bin/octavia-wsgi", line 52, in <module> [Fri Apr 28 10:20:24.732316 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] application = setup_app() [Fri Apr 28 10:20:24.732340 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] File "/usr/lib/python3/dist-packages/octavia/api/app.py", line 62, in setup_app [Fri Apr 28 10:20:24.732353 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] return pecan_make_app( [Fri Apr 28 10:20:24.732381 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] File "/usr/lib/python3/dist-packages/pecan/__init__.py", line 86, in make_app [Fri Apr 28 10:20:24.732400 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] app = Pecan(root, **kw) [Fri Apr 28 10:20:24.732453 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] File "/usr/lib/python3/dist-packages/pecan/core.py", line 832, in __init__ [Fri Apr 28 10:20:24.732475 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] super(Pecan, self).__init__(*args, **kw) [Fri Apr 28 10:20:24.732516 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] File "/usr/lib/python3/dist-packages/pecan/core.py", line 240, in __init__ [Fri Apr 28 10:20:24.732536 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] root = self.__translate_root__(root) [Fri Apr 28 10:20:24.732582 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] File "/usr/lib/python3/dist-packages/pecan/core.py", line 275, in __translate_root__ [Fri Apr 28 10:20:24.732618 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] module = __import__(name, fromlist=fromlist) [Fri Apr 28 10:20:24.732645 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] File "/usr/lib/python3/dist-packages/octavia/api/root_controller.py", line 24, in <module> [Fri Apr 28 10:20:24.732716 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] from octavia.api.v2 import controllers as v2_controller [Fri Apr 28 10:20:24.732760 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] File "/usr/lib/python3/dist-packages/octavia/api/v2/controllers/__init__.py", line 19, in <module> [Fri Apr 28 10:20:24.732821 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] from octavia.api.v2.controllers import availability_zone_profiles [Fri Apr 28 10:20:24.732908 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] File "/usr/lib/python3/dist-packages/octavia/api/v2/controllers/availability_zone_profiles.py", line 1 [Fri Apr 28 10:20:24.732952 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] Copyright 2019 Verizon Media [Fri Apr 28 10:20:24.733024 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] ^ [Fri Apr 28 10:20:24.733107 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] SyntaxError: invalid syntax As far as we can tell, this problem appears when importlib reads the file and hands it over incomplete (missing the first 5 chars). This issue only occurs on the Octavia containers and not on the other Openstack services. The other thing is that the fault is not consistent. It happens at Apache2 reload or restart but not everytime. Once in a few restarts. When the problem occurs, Octavia API is down. Env details: OS: Ubuntu 20.04 Kernel: 5.4.0-147-generic Openstack: Ussuri Octavia: ii octavia-api 6.2.2-0ubuntu1 all OpenStack Load Balancer as a Service - API frontend ii octavia-common 6.2.2-0ubuntu1 all OpenStack Load Balancer as a Service - Common files ii octavia-driver-agent 6.2.2-0ubuntu1 all OpenStack Load Balancer Service - Driver Agent ii octavia-health-manager 6.2.2-0ubuntu1 all OpenStack Load Balancer Service - Health manager ii octavia-housekeeping 6.2.2-0ubuntu1 all OpenStack Load Balancer Service - Housekeeping manager ii octavia-worker 6.2.2-0ubuntu1 all OpenStack Load Balancer Service - Worker ii python3-octavia 6.2.2-0ubuntu1 all OpenStack Load Balancer as a Service - Python libraries ii python3-octavia-lib 2.0.0-0ubuntu1 all Library to support Octavia provider drivers ii python3-ovn-octavia-provider 0.1.0-0ubuntu1 all OpenStack Octavia Integration with OVN - Python 3 library Python: Python 3.8.10 LXD: lxd 5.13-cea5ee2 24758 latest/stable canonical✓ - A bug was filled on launchpad for this issue: https://bugs.launchpad.net/ubuntu/+source/importlib/+bug/2018011 Thanks! Bug #2018011 “python3 - importlib fails to read first 5 chars fr...” : Bugs : importlib package : Ubuntu<https://bugs.launchpad.net/ubuntu/+source/importlib/+bug/2018011> A bit of context: we have an Openstack Ussuri deployment (HA) on Ubuntu Focal. Most Openstack services are running on LXD containers on 3 baremetals. There are also two OVN-dedicated-chassis which act as gateways and nova-compute nodes. A couple of days ago we updated all the packages in the environment to the latest versions (baremetals and lxc containers). After the updates, we started seeing problems with the Octavia API. The API would randomly give an internal server error. Octavia API is... bugs.launchpad.net

1 0

[kolla-ansible][xena][vmware][neutron]
by Andrej Moravčík 09 May '23

09 May '23

Hello community, I am evaluating OpenStack release Xena with VMware vSphere deployed using Kolla-ansible. I would really appreciate if someone with similar environment can confirm my findings or share his/her experience. I did some research and choose Xena, because it is still maintained release and vmware-nsx latest stable branch is xena. I have built customised Docker images with additional Neutron plugins from opendev.org/x/vmware-nsx<http://opendev.org/x/vmware-nsx>, then I deployed single controller using all-in-one Ansible inventory with customised globals.yml according https://docs.openstack.org/kolla-ansible/xena/reference/compute/vmware-guid… 1) Neutron creates flat or VLAN type provider networks and add related Distributed Virtual Port Group (PG) to vSphere Distributed Switch (vDS or DVS). PG is automatically created with naming convention “<network name>”-“<network ID suffix>", which may not be suitable for same use cases, eg. PG was created by Cisco ACI ahead of Neutron and is part of ACI EPG. Question: is this PG naming convention fixed by design or is it configurable? If I want to use existing vDS PG with Neutron how this can be achieved? 2) Neutron metadata and DHCP agents are not accessible to Nova instance attached to DVS PG. I can deploy simple instance from Cirros image, but I cannot provision DHCP or fixed IP address. Provisioning via config drive doesn’t work either, I filled bug report to Nova … https://bugs.launchpad.net/nova/+bug/2018973 The documentation says … "For VMware DVS, the Neutron DHCP agent does not attaches to Open vSwitch inside VMware environment, but attach to the Open vSwitch bridge called br-dvs on the OpenStack side and replies to/receives DHCP packets through VLAN. Similar to what the DHCP agent does, Neutron metadata agent attaches to br-dvs bridge and works through VLAN.” Question: Is it expected that Kolla-ansible creates working configurations for Nova and Neutron, if all related options in globals.yml are set up? Or further adjustment is needed on Openstack or vCenter side? Thank you. — Andrej Moravcik ________________________________ Táto správa a jej prílohy sú dôverné. Prosím, narábajte s týmito informáciami ako s dôvernými a nezverejňujte, nekopírujte ani neposielajte tieto informácie bez predchádzajúceho súhlasu spoločnosti Aliter Technologies, a.s. Ak nie ste adresátom indikovaným v záhlaví tejto správy alebo želaný príjemca, upovedomte prosím spoločnosť Aliter Technologies, a.s. odpoveďou na tento e-mail a pôvodnú správu vymažte zo svojho systému. Aliter Technologies - registrovaná ochranná známka pre EU, USA This message and any attachments are confidential. Please treat the information as confidential, and do not disclose, copy or deliver this message to anyone without Aliter Technologies' approval. If you are not the addressee indicated in this message or an intended recipient please notify Aliter Technologies a.s. by return e-mail and delete the message from your system. Aliter Technologies - Registered in WIPO & U.S. Patent and Trademark Office

1 0

[neutron][Secure RBAC] New policies enabled by default
by Slawek Kaplonski 09 May '23

09 May '23

Hi, It's just a heads up that [1] was merged recently and Neutron is using new, secure RBAC policies by default now. If You would see any issues with that, please report bug(s) and let me know. [1] https://review.opendev.org/c/openstack/neutron/+/879827 -- Slawek Kaplonski Principal Software Engineer Red Hat

3 4

[neutron] Bug deputy May 01 - May 07
by Rodolfo Alonso Hernandez 09 May '23

09 May '23

Hello Neutrinos: This is the bug report from last week. High: * https://bugs.launchpad.net/neutron/+bug/2018727: [SRBAC] API policies for get_policy_*_rule are wrong. Assigned to Slawek. Medium: * https://bugs.launchpad.net/neutron/+bug/2018009: ML2 context not considering tags at network creation and update. Assigned to Miro. * https://bugs.launchpad.net/neutron/+bug/2018289: OVN trunk subport missing bounding information. Assigned to Arnau. * https://bugs.launchpad.net/neutron/+bug/2018529: [OVN] Virtual ports cannot be used as VM ports. Assigned to Rodolfo. * https://bugs.launchpad.net/neutron/+bug/2018585: [SRBAC]New policies change the behavior for check rule type. Assigned to Rodolfo. * https://bugs.launchpad.net/neutron/+bug/2018599: Disable config option use_random_fully does not work. *Not assigned*. * https://bugs.launchpad.net/neutron/+bug/2018737: neutron-dynamic-routing announces routes for disabled routers. Assigned to Felix. Low: * https://bugs.launchpad.net/neutron/+bug/2018474: Unnecessary agent resource version. *Not assigned*. Regards.

1 0

Stepping down as Ironic core
by Arne Wiebalck 09 May '23

09 May '23

Dear all, With a change in my role at work about a year ago, the time I have had available for upstream technical work has dropped significantly. I would therefore like to step down from my role as one of the Ironic cores. I am really grateful to have worked closely with this team of incredibly talented, helpful and kind people, and would like to use this occasion to thank them for all the support they have provided during these past years ... and of course for all the fun we had! Cheers, Arne -- Arne Wiebalck CERN IT

4 3

Using Veeam
by Robert Winbladh 09 May '23

09 May '23

Hi, Not sure I am in the right place but thought I would give it a go. We have a s3 service using IBM Spectrum Scale that builds on/uses Openstack Swift. The problem that we are having is that if the client/user is using Veeam the bucket gets locked. IBM explains that it has something to do with outstanding requests/timeouts. And the bucket gets locked, and the client/user/Veeam cannot access it any longer. This happens all the time. So my questions is: * What can I do about it? Anything that can be tuned in Swift? * IBM tells me to spread the load across multiple buckets - but Veeam does not have support for that (as far as I understand) * Any kind of tuning that needs to be done in Veeam (if anyone would know) * We have tried lowering "concurrent tasks" to 1, but still the bucket gets locked. [cid:image001.png@01D98259.E2C0B2C0] Robert Winbladh Seniorkonsult Technical Operations Manager Incident Manager Problem Manager Change Control Board member Asigra Certified Engineer IBM Spectrum Protect

1 0

[gate][tripleo] gate blocker
by Ananya Banerjee 09 May '23

09 May '23

Hello, All Centos 8 jobs which deploy standalone are failing at the moment. Please hold rechecks if you hit standalone deploy failure on Centos 8 jobs. We are working on the bug: https://bugs.launchpad.net/tripleo/+bug/2018588 Thanks, Ananya -- Ananya Banerjee, RHCSA, RHCE-OSP Software Engineer Red Hat EMEA <https://www.redhat.com/> anbanerj(a)redhat.com M: +491784949931 IM: frenzy_friday @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com/>

1 1

[all][tc][release] Hold the Python3.8 dropping
by Ghanshyam Mann 09 May '23

09 May '23

Hello Everyone, As you know the gate is breaking due to Python3.8 drop[1][2], which we discussed in the TC channel and meeting[3]. TC is preparing the guidelines on not dropping Python3.8 in 2023.2 pti. Meanwhile, governance guidelines are prepared, this email is a heads-up to hold the dropping of Python3.8 from your project/repo. Do not bump the min py version in setup.cfg (do not bump version in python_requires ). @Release team, It will be helpful if you can check/hold the release of any project/lib dropping the Python3.8. [1] https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033450.h… [2] https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033454.h… [3] https://meetings.opendev.org/meetings/tc/2023/tc.2023-04-25-18.00.log.html#… -gmann

1 1

[tc] Technical Committee next weekly meeting on May 9, 2023
by Nikolla, Kristi 09 May '23

09 May '23

Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held on Tuesday, May 9, 2023 at 1800 UTC on #openstack-tc on OFTC IRC Please propose items to the agenda by editing the wiki page at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting At the end of today I will send out an email with the finalized agenda. Thank you, Kristi Nikolla

1 1

[Diversity] Diversity and Inclusion WG Meeting reminder
by Amy Marrich 08 May '23

08 May '23

This is a reminder that the Diversity and Inclusion WG will be meeting tomorrow at 14:00 UTC in the #openinfra-diversity channel on OFTC. We hope members of all OpenInfra projects join us as we review the Foundation-wide diversity survey questions for release at Summit. Thanks, Amy (spotz)

1 0

Fwd: (Openstack-Cinder) Error
by Adivya Singh 08 May '23

08 May '23

HI, Can you please guide me, There is this error which i am getting "fatal: [t1w_cinder_volumes_container-29d71abf]: FAILED! => {"msg": "The conditional check 'cinder_backend_lvm_inuse | bool' failed. The error was: error while evaluating conditional (cinder_backend_lvm_inuse | bool): {{ (cinder_backends|default(\"\")|to_json).find(\"cinder.volume.drivers.lvm.LVMVolumeDriver\") != -1 }}: {'lvm': {'iscsi_ip_address': '{{ 172.29.244.50 }}', 'volume_backend_name': 'LVM_iSCSI', 'volume_driver': 'cinder.volume.drivers.lvm.LVMVolumeDriver', 'volume_group': 'cloud'}}: float object has no element 244\n\nThe error appears to be in '/opt/openstack-ansible/playbooks/common-playbooks/cinder.yml': line 67, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n - name: Configure container (cinder-volume) when lvm is in-use\n ^ here\n"} I understand this error to some extent, but want to confirm with you if any body have seen similiar kind of error storage_hosts: s1w: ip: 172.29.236.50 container_vars: #cinder_storage_availability_zone: nova #cinder_default_availability_zone: nova cinder_backends: lvm: volume_backend_name: LVM_iSCSI volume_driver: cinder.volume.drivers.lvm.LVMDriver volume_group: cloud-volume iscsi_ip_address: "{{ cinder_storage_address }}" limit_container_types: cinder_volume Regards Adivya Singh

1 0

[tc][all] Technical Committee Weekly Summary -- May 5, 2023
by Nikolla, Kristi 08 May '23

08 May '23

Hi all, Here’s another edition of “What’s happening on the Technical Committee.” Meeting ======= The Technical Committee met on May 2, 2023 on the Zoom. A recording of the meeting can be found on Youtube at https://youtu.be/e-IUZ1ymi_A The next meeting will be held on May 9, 2023 at 18:00UTC. For more information visit https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting Happenings ========== Adding Python 3.8 to PTI and expectation of Python support ---------------------------------------------------------- Based on the TC discussion during the last weekly meeting, the TC proposed a change adding Python 3.8 as a PTI requirement for libraries[0] and a proposal attempting to clarify which Python versions we should aim to support in general[1]. [0]. https://review.opendev.org/c/openstack/governance/+/882165 [1]. https://review.opendev.org/c/openstack/governance/+/882154 Changes ======= • Merged • Align release naming terminology (documentation-change) | https://review.opendev.org/c/openstack/governance/+/881706 • Appoint Jerry Zhou as Sahara PTL (formal-vote) | https://review.opendev.org/c/openstack/governance/+/881186 • Retire puppet-rally - Step 3: Retire Repository (project-update) | https://review.opendev.org/c/openstack/governance/+/880018 • New Open • Appoint Dmitriy Rabotyagov as Vitrage PTL (formal-vote) | https://review.opendev.org/c/openstack/governance/+/882139 • Appoint Hasan Acar as Monasca PTL (formal-vote) | https://review.opendev.org/c/openstack/governance/+/882374 • Add py38 as a PTI requirement for libraries (formal-vote) | https://review.opendev.org/c/openstack/governance/+/882165 • Clarify expectations on keeping Python versions (formal-vote) | https://review.opendev.org/c/openstack/governance/+/882154 • Abandoned • All Open • https://review.opendev.org/q/project:openstack/governance+status:open How to contact the TC ===================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send an email with the tag [tc] on the openstack-discuss mailing list. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 16:00 UTC 3. IRC: Ping us using the 'tc-members' keyword on the #openstack-tc IRC channel on OFTC.

1 0

[Nova] Clean up dead vms entries from DB
by Satish Patel 07 May '23

07 May '23

Folks, I have a small environment where controllers and computers run on the same nodes and yes that is a bad idea. But now what happened the machine got OOM out and crashed and stuck somewhere in Zombie stat. I have deleted all vms but it didn't work. Later I used the "virsh destroy" command to delete those vms and everything recovered but my openstack hypervisor state show" command still says you have 91 VMs hanging in DB. How do i clean up vms entered from nova DB which doesn't exist at all. I have tried the command "nova-manage db archive_deleted_rows" but it didn't help. How does nova sync DB with current stat? # openstack hypervisor stats show This command is deprecated. +----------------------+--------+ | Field | Value | +----------------------+--------+ | count | 3 | | current_workload | 11 | | disk_available_least | 15452 | | free_disk_gb | 48375 | | free_ram_mb | 192464 | | local_gb | 50286 | | local_gb_used | 1911 | | memory_mb | 386570 | | memory_mb_used | 194106 | | running_vms | 91 | | vcpus | 144 | | vcpus_used | 184 | +----------------------+--------+ Technically I have only single VM # openstack server list --all +--------------------------------------+---------+--------+---------------------------------+------------+----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+---------+--------+---------------------------------+------------+----------+ | b33fd79d-2b90-41dd-a070-29f92ce205e7 | foo1 | ACTIVE | 1=100.100.75.22, 192.168.1.139 | ubuntu2204 | m1.small | +--------------------------------------+---------+--------+---------------------------------+------------+----------+

2 1

Re: openstack-discuss Digest, Vol 55, Issue 17
by BEDDA Fadhel 07 May '23

07 May '23

Good morning, I set up openstack, on an Ubuntu 22.04 VM. Installation went well: I created an account and a project, as well as all the steps for creating an Ubuntu 18 instance. But when launching the instance, I only have error messages. My question: Is there someone who can help me and provide me with a procedure complete and tested. With my sincere thanks Le sam. 6 mai 2023 à 10:21, <openstack-discuss-request(a)lists.openstack.org> a écrit : > Send openstack-discuss mailing list submissions to > openstack-discuss(a)lists.openstack.org > > To subscribe or unsubscribe via the World Wide Web, visit > > https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss > > or, via email, send a message with subject or body 'help' to > openstack-discuss-request(a)lists.openstack.org > > You can reach the person managing the list at > openstack-discuss-owner(a)lists.openstack.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of openstack-discuss digest..." > > > Today's Topics: > > 1. [Nova] Clean up dead vms entries from DB (Satish Patel) > 2. [OPENSTACK][rabbitmq] using quorum queues (Nguy?n H?u Kh?i) > 3. Re: openstack-discuss Digest, Vol 55, Issue 16 (BEDDA Fadhel) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 5 May 2023 15:14:18 -0400 > From: Satish Patel <satish.txt(a)gmail.com> > To: OpenStack Discuss <openstack-discuss(a)lists.openstack.org> > Subject: [Nova] Clean up dead vms entries from DB > Message-ID: > <CAPgF-fqVSj3YKx7cYSF3gzAbL54veXM= > S18K5-sX6XMCbjTWKA(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Folks, > > I have a small environment where controllers and computers run on the same > nodes and yes that is a bad idea. But now what happened the machine got OOM > out and crashed and stuck somewhere in Zombie stat. I have deleted all vms > but it didn't work. Later I used the "virsh destroy" command to delete > those vms and everything recovered but my openstack hypervisor state show" > command still says you have 91 VMs hanging in DB. > > How do i clean up vms entered from nova DB which doesn't exist at all. > > I have tried the command "nova-manage db archive_deleted_rows" but it > didn't help. How does nova sync DB with current stat? > > # openstack hypervisor stats show > This command is deprecated. > +----------------------+--------+ > | Field | Value | > +----------------------+--------+ > | count | 3 | > | current_workload | 11 | > | disk_available_least | 15452 | > | free_disk_gb | 48375 | > | free_ram_mb | 192464 | > | local_gb | 50286 | > | local_gb_used | 1911 | > | memory_mb | 386570 | > | memory_mb_used | 194106 | > | running_vms | 91 | > | vcpus | 144 | > | vcpus_used | 184 | > +----------------------+--------+ > > > Technically I have only single VM > > # openstack server list --all > > +--------------------------------------+---------+--------+---------------------------------+------------+----------+ > | ID | Name | Status | Networks > | Image | Flavor | > > +--------------------------------------+---------+--------+---------------------------------+------------+----------+ > | b33fd79d-2b90-41dd-a070-29f92ce205e7 | foo1 | ACTIVE | 1=100.100.75.22, > 192.168.1.139 | ubuntu2204 | m1.small | > > +--------------------------------------+---------+--------+---------------------------------+------------+----------+ >

2 1

Re: openstack-discuss Digest, Vol 55, Issue 16
by BEDDA Fadhel 06 May '23

06 May '23

Bonjour, J’ai mis en place openstack, sur une VM Ubuntu 22.04. L’installation s’est bien passé: J’ai créé un compte et un projet, ainsi que toute les étapes pour la création d’une instance Ubuntu 18. Mais lors de lancement de l’instance, je n’ai que des messages d’erreur. Ma question: Est ce qu’il y’a quelqu’un qui peut m’aider et me fournir une procédure complète et testée. Avec mes vifs remerciements Le ven. 5 mai 2023 à 20:09, <openstack-discuss-request(a)lists.openstack.org> a écrit : > Send openstack-discuss mailing list submissions to > openstack-discuss(a)lists.openstack.org > > To subscribe or unsubscribe via the World Wide Web, visit > > https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss > > or, via email, send a message with subject or body 'help' to > openstack-discuss-request(a)lists.openstack.org > > You can reach the person managing the list at > openstack-discuss-owner(a)lists.openstack.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of openstack-discuss digest..." > > > Today's Topics: > > 1. [release] Release countdown for week R-21, May 08-12 (El?d Ill?s) > 2. Re: Kubernetes Conformance 1.24 + 1.25 (Guilherme Steinm?ller) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 5 May 2023 14:30:58 +0000 > From: El?d Ill?s <elod.illes(a)est.tech> > To: "openstack-discuss(a)lists.openstack.org" > <openstack-discuss(a)lists.openstack.org> > Subject: [release] Release countdown for week R-21, May 08-12 > Message-ID: > < > VI1P18901MB07511B14C9655567C4A5EC88FF729(a)VI1P18901MB0751.EURP189.PROD.OUTLOOK.COM > > > > Content-Type: text/plain; charset="utf-8" > > Development Focus > ----------------- > > The Bobcat-1 milestone is next week, on May 11th, 2023! Project team > plans for the 2023.2 Bobcat cycle should now be solidified. > > General Information > ------------------- > > Libraries need to be released at least once per milestone period. Next > week, the release team will propose releases for any library which had > changes but has not been otherwise released since the 2023.1 Antelope > release. > PTLs or release liaisons, please watch for these and give a +1 to > acknowledge them. If there is some reason to hold off on a release, let > us know that as well, by posting a -1. If we do not hear anything at all > by the end of the week, we will assume things are OK to proceed. > > NB: If one of your libraries is still releasing 0.x versions, start > thinking about when it will be appropriate to do a 1.0 version. The > version number does signal the state, real or perceived, of the library, > so we strongly encourage going to a full major version once things are > in a good and usable state. > > Upcoming Deadlines & Dates > -------------------------- > > Bobcat-1 milestone: May 11th, 2023 > OpenInfra Summit Vancouver (including PTG): June 13-15, 2023 > Final 2023.2 Bobcat release: October 4th, 2023 > > > El?d Ill?s > irc: elodilles @ #openstack-release >

1 0

[release] Release countdown for week R-21, May 08-12
by Elõd Illés 05 May '23

05 May '23

Development Focus ----------------- The Bobcat-1 milestone is next week, on May 11th, 2023! Project team plans for the 2023.2 Bobcat cycle should now be solidified. General Information ------------------- Libraries need to be released at least once per milestone period. Next week, the release team will propose releases for any library which had changes but has not been otherwise released since the 2023.1 Antelope release. PTLs or release liaisons, please watch for these and give a +1 to acknowledge them. If there is some reason to hold off on a release, let us know that as well, by posting a -1. If we do not hear anything at all by the end of the week, we will assume things are OK to proceed. NB: If one of your libraries is still releasing 0.x versions, start thinking about when it will be appropriate to do a 1.0 version. The version number does signal the state, real or perceived, of the library, so we strongly encourage going to a full major version once things are in a good and usable state. Upcoming Deadlines & Dates -------------------------- Bobcat-1 milestone: May 11th, 2023 OpenInfra Summit Vancouver (including PTG): June 13-15, 2023 Final 2023.2 Bobcat release: October 4th, 2023 Előd Illés irc: elodilles @ #openstack-release

1 0

[neutron] Drivers meeting cancelled today
by Rodolfo Alonso Hernandez 05 May '23

05 May '23

Hello Neutrinos: Due to the lack of agenda [1], today's drivers meeting is cancelled. Have a nice weekend. [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers

1 0

[dev][ceilometer] Add custom uptime metric to Ceilometer
by Angel Santana 04 May '23

04 May '23

Guys, I deployed Ceilometer into my OpenStack deployment using the Juju charm. I’m trying to add a custom metric that records the uptime of instances every hour. I’m totally lost on how to archive this or if it’s possible. Any help would be appreciate. So far I tried playing around with the meters.yaml and polling.yaml to see if I could pull it off with no luck. Is this possible? how should I do it? Thanks in advance. Regards, Ángel

3 2

Certbot auto renew
by Derek O keeffe 04 May '23

04 May '23

Hi all, We're having a problem with renewing letsencrypt certs via certbot in an external Neutron network where a security group is locking down HTTP+HTTPS access to select IP ranges. As far as we know the IP address for the Certbot ACME challenge server is always changing and therefore a static security group can't be set up to allow in traffic from that server. We have experimented with using UFW rules instead thinking we may be able to write a script to open port 80 periodically to allow the ACME challenge through, then close it back up, but it hasn't worked as we'd hoped either (either all traffic is blocked or the security group immediately takes precedence). Is there any way to programmatically enable + disable a security group as needed using something like OpenstackSDK to achieve the same thing? Thanks in advance. Regards,Derek

3 3

[openstack-ansible] Proposing Neil Hanlon for core reviewer
by Dmitriy Rabotyagov 04 May '23

04 May '23

Hi everyone, I'm pleased to propose Neil Hanlon as OpenStack-Ansible Core Reviewer. Neil has helped us a lot lately with maintenance of RHEL-like distros - both CentOS Stream and Rocky Linux, and basically has brought in support for the last one. Neil is present at meetings and always is responsive in IRC. At the same time they were providing useful reviews lately [1] If there are no objections, I will add Neil to the team on 25th of April 2023. Until then, please feel free to provide your feedback/opinions on the matter. [1] https://www.stackalytics.io/report/contribution?module=openstackansible-gro…

6 7

[openstack-helm] Nominate Karl Kloppenborg for openstack-helm-core
by Vladimir Kozhukalov 04 May '23

04 May '23

Dear Openstack-helmers, I would like to suggest Karl Kloppenborg as a member of the core review team. He's been actively doing code review recently and also contributed as a developer. He has been using Openstack-Helm for a few years now, so his experience as a user is also to our advantage. I hope for your support. -- Best regards, Kozhukalov Vladimir

2 2

[all][tc][vitrage] Final call for Vitrage project maintainers
by Ghanshyam Mann 03 May '23

03 May '23

Hello Everyone, You might have noticed that the Vitrage project is leaderless in this cycle[1]. Eyal, who was the previous PTL and the only maintainer of this project for more than two years[2], will not be able to continue in this project[3]. We thank and appreciate all previous maintainers' work on this project. This is the final call for maintainers. If anyone is using or interested in this project, this is the right time to step up and let us know in this email or IRC(OFTC) #openstack-tc channel. If there is no response by the end of April 2023, we will discuss the retirement of this project. [1] https://etherpad.opendev.org/p/2023.2-leaderless [2] https://www.stackalytics.io/?module=vitrage-group&release=wallaby [3] https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033333.h… -gmann

2 2

OpenInfra Live - May 4, 2023 at 9 a.m. CT / 14:00 UTC
by Kristin Barrientos 03 May '23

03 May '23

Hi everyone, This week’s OpenInfra Live episode is brought to you by the OpenInfra Foundation Staff. Episode: May the OpenInfra Force Be With You: Preview of the OpenInfra Summit! Join us for an insightful and informative discussion as we give you the inside scoop on what’s in store at the OpenInfra Summit, Vancouver, happening, June 13-15! Don’t miss your chance to learn how you can get involved! Speakers: Wes Wilson, Allison Price, Jimmy McArthur, Kendall Nelson Date and time: May 4, 2023, at 9 a.m. CT. (14:00 UTC) You can watch us live on: YouTube: https://www.youtube.com/live/g_nTOpLGftw?feature=share LinkedIn: https://www.linkedin.com/events/7056677974057648128/comments/ WeChat: recording will be posted on OpenStack WeChat after the live stream Have an idea for a future episode? Share it now at ideas.openinfra.live. Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

[dev][infra][qa][tact-sig] Future of Fedora and CentOS Stream Test Images
by Jeremy Stanley 03 May '23

03 May '23

tl;dr is that the OpenDev Collaboratory is looking at potentially scaling back on Fedora-based test platform support. Several options are presented in this service-discuss post: https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/t… If you have an opinion on using Fedora in our CI jobs, please follow up to the discussion there. -- Jeremy Stanley

1 0

[largescale-sig] Next meeting: May 3, 8utc
by Thierry Carrez 03 May '23

03 May '23

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 8UTC, our EU+APAC-friendly time. You can doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20230503T08 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1