May 2023 - openstack-discuss - lists.openstack.org

[kolla] [train] [neutron]
by Albert Braden 11 May '23

11 May '23

1 0

Extending the Forum
by Kendall Nelson 11 May '23

11 May '23

Hello Everyone, We received feedback that critical Forum conversations were not able to make the schedule due to the intentional work on our part to avoid PTG / Forum overlap which ended up restricting the number of Forum discussion proposals we were able to accept. To mitigate this, we have opened up the Forum room for a third day. We have created a form to collect critical Forum proposals. We will leave the form open for one week to give all timezones fair representation. The deadline to submit a proposal for a forum discussion is Thursday May, 18th at 7:00 UTC. Shortly after, the OpenInfra Foundation will make final decisions and program the Forum on Thursday. -Kendall Nelson [1] https://openinfrafoundation.formstack.com/forms/forum_expansion

1 0

[publiccloud-sig] Bi-weekly meeting times adjusted to 0700 UTC
by Felix Kronlage-Dammers 11 May '23

11 May '23

Hi, since some of us (to be honest, probably mostly me ;) have a conflicting meeting during summer time with the slot of the publiccloud-sig we discussed moving it to 0700 UTC during summer time. Since nobody in the past two meetings spoke against that, this was concluded yesterday. During summer time the publiccloud SIG will meet at 0700 UTC, not at 0800 UTC. The change is also reflected in the wiki page: https://wiki.openstack.org/wiki/PublicCloudSIG See you next time (and I'll set myself reminder to send out the reminder ;) felix -- GPG: 824CE0F0 / 2082 651E 5104 F989 4D18 BB2E 0B26 6738 824C E0F0 https://hazardous.org/ - fkr(a)hazardous.org - fkr@irc - @felixkronlage

1 0

[OSSA-2023-003] cinder, glance_store, nova, os-brick: Unauthorized volume access through deleted volume attachments (CVE-2023-2088)
by Jeremy Stanley 10 May '23

10 May '23

============================================================================ OSSA-2023-003: Unauthorized volume access through deleted volume attachments ============================================================================ :Date: May 10, 2023 :CVE: CVE-2023-2088 Affects ~~~~~~~ - Cinder: <20.2.1, >=21.0.0 <21.2.1, ==22.0.0 - Glance_store: <3.0.1, >=4.0.0 <4.1.1, >=4.2.0 <4.3.1 - Nova: <25.1.2, >=26.0.0 <26.1.2, ==27.0.0 - Os-brick: <5.2.3, >=6.0.0 <6.1.1, >=6.2.0 <6.2.2 Description ~~~~~~~~~~~ An unauthorized access to a volume could occur when an iSCSI or FC connection from a host is severed due to a volume being unmapped on the storage system and the device is later reused for another volume on the same host. **Scope:** Only deployments with iSCSI or FC volumes are affected. However, the fix for this issue includes a configuration change in Nova and Cinder that may impact you on your next upgrade regardless of what backend storage technology you use. See the *Configuration change* section below, and item 4(B) in the *Patches and Associated Deployment Changes* for details. This data leak can be triggered by two different situations. **Accidental case:** If there is a problem with network connectivity during a normal detach operation, OpenStack may fail to clean the situation up properly. Instead of force-detaching the compute node device, Nova ignores the error, assuming the instance has already been deleted. Due to this incomplete operation OpenStack may end up selecting the wrong multipath device when connecting another volume to an instance. **Intentional case:** A regular user can create an instance with a volume, and then delete the volume attachment directly in Cinder, which neglects to notify Nova. The compute node SCSI plumbing (over iSCSI/FC) will continue trying to connect to the original host/port/LUN, not knowing the attachment has been deleted. If a subsequent volume attachment re-uses the host/port/LUN for a different instance and volume, the original instance will gain access to it once the SCSI plumbing reconnects. Configuration Change -------------------- To prevent the intentional case, the Block Storage API provided by Cinder must only accept attachment delete requests from Nova for instance-attached volumes. A complicating factor is that Nova deletes an attachment by making a call to the Block Storage API on behalf of the user (that is, by passing the user's token), which makes the request indistinguishable from the user making this request directly. The solution is to have Nova include a service token along with the user's token so that Cinder can determine that the detach request is coming from Nova. The ability for Nova to pass a service token has been supported since Ocata, but has not been required until now. Thus, deployments that are not currently sending service user credentials from Nova will need to apply the relevant code changes and also make configuration changes to solve the problem. Patches and Associated Deployment Changes ----------------------------------------- Given the above analysis, a thorough fix must include the following elements: 1. The os-brick library must implement the ``force`` option for fibre channel, which which has only been available for iSCSI until now (covered by the linked patches). 2. Nova must call os-brick with the ``force`` option when disconnecting volumes from deleted instances (covered by the linked patches). 3. In deployments where Glance uses the cinder glance_store driver, glance must call os-brick with the ``force`` option when disconnecting volumes (covered by the linked patches). 4. Cinder must distinguish between safe and unsafe attachment delete requests and reject the unsafe ones. This part of the fix has two components: a. The Block Storage API will return a 409 (Conflict) for a request to delete an attachment if there is an instance currently using the attachment, **unless** the request is being made by a service (for example, Nova) on behalf of a user (covered by the linked patches). b. In order to recognize that a request is being made by a service on behalf of a user, Nova must be configured to send a service token along with the user token. If this configuration change is not made, the cinder change will reject **any** request to delete an attachment associated with a volume that is attached to an instance. Nova must be configured to send a service token to Cinder, and Cinder must be configured to accept service tokens. This is described in the following document and **IS NOT AUTOMATICALLY APPLIED BY THE LINKED PATCHES:** (Using service tokens to prevent long-running job failures) https://docs.openstack.org/cinder/latest/configuration/block-storage/servic… The Nova patch mentioned in step 2 includes a similar document more focused on Nova: doc/source/admin/configuration/service-user-token.rst 5. The cinder glance_store driver does not attach volumes to instances; instead, it attaches volumes directly to the Glance node. Thus, the Cinder change in step 4 will recognize an attachment-delete request coming from Glance as safe and allow it. (Of course, we expect that you will have applied the patches in steps 1 and 3 to your Glance nodes.) Errata ~~~~~~ An additional nova patch is required to fix a minor regression in periodic tasks and some nova-manage actions (errata 1). Also a patch to tempest is needed to account for behavior changes with fixes in place (errata 2). Patches ~~~~~~~ - https://review.opendev.org/882836 (2023.1/antelope cinder) - https://review.opendev.org/882851 (2023.1/antelope glance_store) - https://review.opendev.org/882858 (2023.1/antelope nova) - https://review.opendev.org/882859 (2023.1/antelope nova errata 1) - https://review.opendev.org/882843 (2023.1/antelope os-brick) - https://review.opendev.org/882835 (2023.2/bobcat cinder) - https://review.opendev.org/882834 (2023.2/bobcat glance_store) - https://review.opendev.org/882847 (2023.2/bobcat nova) - https://review.opendev.org/882852 (2023.2/bobcat nova errata 1) - https://review.opendev.org/882840 (2023.2/bobcat os-brick) - https://review.opendev.org/882876 (2023.2/bobcat tempest errata 2) - https://review.opendev.org/882869 (Wallaby nova) - https://review.opendev.org/882870 (Wallaby nova errata 1) - https://review.opendev.org/882839 (Xena cinder) - https://review.opendev.org/882855 (Xena glance_store) - https://review.opendev.org/882867 (Xena nova) - https://review.opendev.org/882868 (Xena nova errata 1) - https://review.opendev.org/882848 (Xena os-brick) - https://review.opendev.org/882838 (Yoga cinder) - https://review.opendev.org/882854 (Yoga glance_store) - https://review.opendev.org/882863 (Yoga nova) - https://review.opendev.org/882864 (Yoga nova errata 1) - https://review.opendev.org/882846 (Yoga os-brick) - https://review.opendev.org/882837 (Zed cinder) - https://review.opendev.org/882853 (Zed glance_store) - https://review.opendev.org/882860 (Zed nova) - https://review.opendev.org/882861 (Zed nova errata 1) - https://review.opendev.org/882844 (Zed os-brick) Credits ~~~~~~~ - Jan Wasilewski from Atman (CVE-2023-2088) - Gorka Eguileor from Red Hat (CVE-2023-2088) References ~~~~~~~~~~ - https://launchpad.net/bugs/2004555 - http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2088 Notes ~~~~~ - Limited Protection Against Accidents... If you are only concerned with protecting against the accidental case described earlier in this document, steps 1-3 above should be sufficient. Note, however, that only applying steps 1-3 leaves your cloud wide open to the intentional exploitation of this vulnerability. Therefore, we recommend that the full fix be applied to all deployments. - Using Configuration as a Short-Term Mitigation... An alternative approach to mitigation can be found in OSSN-0092 https://wiki.openstack.org/wiki/OSSN/OSSN-0092 - The stable/xena and stable/wallaby branches are under extended maintenance and will receive no new point releases, but patches for them are provided as a courtesy where available. OSSA History ~~~~~~~~~~~~ - 2023-05-10 - Errata 2 - 2023-05-10 - Errata 1 - 2023-05-10 - Original Version -- Jeremy Stanley OpenStack Vulnerability Management Team

1 0

Cinder Bug Report 2023-05-10
by Sofia Enriquez 10 May '23

10 May '23

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> *Medium* - [rbac] Volume type's extra_specs dict is empty. <https://bugs.launchpad.net/cinder/+bug/2018467> - *Status:* Unassigned. - cinder services report_state hangs for a longer period than report_interval when losing database connection. <https://bugs.launchpad.net/cinder/+bug/2018595> - *Status:* Unassigned. *Low* - HPE 3par - Unable to use small QoS Latency value (less than 1) <https://bugs.launchpad.net/cinder/+bug/2018994> - *Status:* Fix proposed to master <https://review.opendev.org/c/openstack/cinder/+/882266>. Cheers -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[cinder] create a multiattach volume in two ways?
by Tony Liu 10 May '23

10 May '23

Hi, [1] says "Currently you can create a multiattach volume in two ways." I see one way is with volume type. What's the another way? [1] https://docs.openstack.org/cinder/xena/admin/blockstorage-volume-multiattac… Thanks! Tony

2 3

[heat][magnum][tacker] Future of SoftwareDeployment support
by Takashi Kajinami 10 May '23

10 May '23

Hello, We discussed this briefly in the past thread where we discussed maintenance of os-*-agent repos, and also talked about this topic during Heat PTG, but I'd like to formalize the discussion to get a clear agreement. Heat has been supporting SoftwareDeployment resources to configure software in instances using some agents such as os-collect-config[1]. [1] https://docs.openstack.org/heat/latest/template_guide/software_deployment.h… This feature was initially developed to be used by TripleO (IIUC), but TripleO is retired now and we are losing the first motivation to maintain the feature. # Even TripleO replaced most of its usage of softwaredeployment by config-download lately. Because the heat project team has drunk dramatically recently, we'd like to put more focus on core features. For that aim we are now wondering if we can deprecate and remove this feature, and would like to hear from anyone who has any concerns about this. Quickly looking through the repos, it seems currently Magnum and Tacker are using SoftwareDeployment, and it'd be nice especially if we can understand their current requirements. 1. Magnum It seems SoftwareDeployment is used by k8s_fedora_atomic_v1 driver but I'm not too sure whether this driver is still supported, because Fedora Atomic was EOLed a while ago, right ? 2. Tacker SoftwareDeployment can be found in only test code in the tacker repo. We have some references kept in heat-translator which look related to TOSCA templates. Thank you, Takashi Kajinami

4 8

Unable to provision HCI node | Wallaby
by Swogat Pradhan 09 May '23

09 May '23

Hi, When i am running node provision command i am getting the following error: The full traceback is: File "/tmp/ansible_metalsmith_instances_payload_tmufs89r/ansible_metalsmith_instances_payload.zip/ansible/modules/metalsmith_instances.py", line 427, in main File "/tmp/ansible_metalsmith_instances_payload_tmufs89r/ansible_metalsmith_instances_payload.zip/ansible/modules/metalsmith_instances.py", line 340, in provision File "/usr/lib64/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/tmp/ansible_metalsmith_instances_payload_tmufs89r/ansible_metalsmith_instances_payload.zip/ansible/modules/metalsmith_instances.py", line 372, in _provision_instance File "/usr/lib/python3.6/site-packages/metalsmith/_provisioner.py", line 488, in wait_for_provisioning raise exceptions.DeploymentFailed(str(exc)) "networks": [ { "network": "ctlplane", "vif": true }, { "network": "internal_api", "subnet": "internal_apis2_subnet" }, { "network": "tenant", "subnet": "tenants2_subnet" }, { "network": "storage", "subnet": "storages2_subnet" }, { "network": "storage_mgmt", "subnet": "storage_mgmts2_subnet" } ], "nics": [ { "network": "ctlplane" } ], "ssh_public_keys": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCb/KTQTouURPWFO0R1zGEIKXYUDQb4+pIgNuTJ7zA43jz9nyGt/pH1pnZAq71NmfL+sICFPf4uvhqFkMU7T0eC3TVuP87kZ8pAMh0D+NPqtRQJxK0WZO2h67dUKBAtW5WamxABhDh+XqmRPXE8Fl1VvzlKO/KZGUClk24BNGjt/nqho9FGskNY/vZLQV/gZzrKfHUYpcDxIg6NpnNE6bGdy4tyL4JwYcdTP8ovU1JhKMXSTjk8WrGz/OQB8a4Pgq0WWgIRWvwZDuUwFWiK6aV7gkjhMXwwtZ7jq9fQOs/hcleATX7Cq77ayJW8DusxQLynbMrh/dBs3Smgp1Ncd3rthpJz2ujBg7ymScCq4ya0W1RkbzZ6H8kVT0hppO93Ip8VFTMnPDoWONQFdAbAVhaamycVMxxbGatHVYGGXIMMBEL1MO4ncIW4f46vNffIhmoTvG20ncy2zba7hk9D4NMbqEpyR3NG1BnMHW0h9bYp2K9+jOWS9MUfakD8kNi8Ff0= stack(a)hkg2director.bdxworld.com ", 2023-05-09 22:18:08.565576 | 48d539a1-1679-0477-63a8-000000000018 | FATAL | Provision instances | localhost | error={ "changed": false, "invocation": { "module_args": { "api_timeout": null, "auth": null, "auth_type": null, "availability_zone": null, "ca_cert": null, "clean_up": false, "client_cert": null, "client_key": null, "concurrency": 1, "instances": [ { "config_drive": { "meta_data": { "instance-type": "DistributedComputeHCI" } }, "hostname": "dcn01-hci-2", "image": { "href": "file:///var/lib/ironic/images/overcloud-full.raw", "kernel": "file:///var/lib/ironic/images/overcloud-full.vmlinuz", "ramdisk": "file:///var/lib/ironic/images/overcloud-full.initrd" }, "name": "17107583-48da-43eb-9e38-63e250848d05", "network_config": { "template": "/home/stack/dcn01/hci_network_bond.j2" }, "networks": [ { "network": "ctlplane", "vif": true }, { "network": "internal_api", "subnet": "internal_apis2_subnet" }, { "network": "tenant", "subnet": "tenants2_subnet" }, { "network": "storage", "subnet": "storages2_subnet" }, { "network": "storage_mgmt", "subnet": "storage_mgmts2_subnet" } ], "nics": [ { "network": "ctlplane" } ], "ssh_public_keys": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCb/KTQTouURPWFO0R1zGEIKXYUDQb4+pIgNuTJ7zA43jz9nyGt/pH1pnZAq71NmfL+sICFPf4uvhqFkMU7T0eC3TVuP87kZ8pAMh0D+NPqtRQJxK0WZO2h67dUKBAtW5WamxABhDh+XqmRPXE8Fl1VvzlKO/KZGUClk24BNGjt/nqho9FGskNY/vZLQV/gZzrKfHUYpcDxIg6NpnNE6bGdy4tyL4JwYcdTP8ovU1JhKMXSTjk8WrGz/OQB8a4Pgq0WWgIRWvwZDuUwFWiK6aV7gkjhMXwwtZ7jq9fQOs/hcleATX7Cq77ayJW8DusxQLynbMrh/dBs3Smgp1Ncd3rthpJz2ujBg7ymScCq4ya0W1RkbzZ6H8kVT0hppO93Ip8VFTMnPDoWONQFdAbAVhaamycVMxxbGatHVYGGXIMMBEL1MO4ncIW4f46vNffIhmoTvG20ncy2zba7hk9D4NMbqEpyR3NG1BnMHW0h9bYp2K9+jOWS9MUfakD8kNi8Ff0= stack(a)hkg2director.bdxworld.com ", "user_name": "heat-admin" } ], "interface": "public", "log_level": "info", "region_name": null, "state": "present", "timeout": 3600, "validate_certs": null, "wait": true } }, "logging": "Created port dcn01-hci-2-ctlplane (UUID ce763b38-b047-40bc-be44-9c9da8bd2e30) for node singapore-HCI3 (UUID 17107583-48da-43eb-9e38-63e250848d05) with {'network_id': '1fad76a3-aa2a-4213-8c37-eb89629da523', 'name': 'dcn01-hci-2-ctlplane'}\nAttached port dcn01-hci-2-ctlplane (UUID ce763b38-b047-40bc-be44-9c9da8bd2e30) to node singapore-HCI3 (UUID 17107583-48da-43eb-9e38-63e250848d05)\nProvisioning started on node singapore-HCI3 (UUID 17107583-48da-43eb-9e38-63e250848d05)\n", "msg": "Node 17107583-48da-43eb-9e38-63e250848d05 reached failure state \"deploy failed\"; the last error is Deploy step deploy.prepare_instance_boot failed: Failed to install a bootloader when deploying node 17107583-48da-43eb-9e38-63e250848d05. Error: Installing GRUB2 boot loader to device /dev/sda failed with Unexpected error while running command.\nCommand: chroot /tmp/tmp8_wa6g6q /bin/sh -c \"grub2-install /dev/sda\"\nExit code: 1\nStdout: ''\nStderr: 'grub2-install: error: this utility cannot be used for EFI platforms because it does not support UEFI Secure Boot.\\n'." I have already provisioned 2 other nodes using the same hardware, but the node provisioning is failing in this particular node. With regards, Swogat Pradhan

2 1

[neutron] When num of port mappings is substantial, the response time of List API so slow
by 韩光宇 09 May '23

09 May '23

Hello， I have a Victoria cluster with 1 controller node(also have nova-compute service) and two compute node. ALL node boasts 128 CPU cores and 512GB of memory. I established 1500 port mappings for a single floating IP A. At this point, the response time for the "List Floating IP Port Forwardings" API[1] becomes remarkably slow, approaching a duration of 9 minutes. Does anyone comprehend the cause of this phenomenon or how should I investigate it? I am immensely grateful for any assistance provided. Best regards, Han Guangyu [1] https://docs.openstack.org/api-ref/network/v2/index.html#list-floating-ip-p…

2 2

[Octavia] importlib fails to read first 5 chars from file
by Mihai Gheorghe 09 May '23

09 May '23

Hello, We are experiencing a strange problem with Octavia API in an Openstack deployment. A bit of context: we have an Openstack Ussuri deployment (HA) on Ubuntu Focal. Most Openstack services are running on LXD containers on 3 baremetals. There are also two OVN-dedicated-chassis which act as gateways and nova-compute nodes. A couple of days ago we updated all the packages in the environment to the latest versions (baremetals and lxc containers). After the updates, we started seeing problems with the Octavia API. The API would randomly give an internal server error. Octavia API is running under Apache2 WSGI on 3 LXD containers on 3 separate nodes. Looking into one of the Octavia container under /var/log/apache2/octavia_error.log, we found the following error: [Fri Apr 28 10:11:07.509268 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] mod_wsgi (pid=1312420): Failed to exec Python script file '/usr/bin/octavia-wsgi'. [Fri Apr 28 10:11:07.509455 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] mod_wsgi (pid=1312420): Exception occurred processing WSGI script '/usr/bin/octavia-wsgi'. [Fri Apr 28 10:11:07.511688 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] Traceback (most recent call last): [Fri Apr 28 10:11:07.511884 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "/usr/bin/octavia-wsgi", line 52, in <module> [Fri Apr 28 10:11:07.512260 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] application = setup_app() [Fri Apr 28 10:11:07.512343 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "/usr/lib/python3/dist-packages/octavia/api/app.py", line 59, in setup_app [Fri Apr 28 10:11:07.512435 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] pecan_config = get_pecan_config() [Fri Apr 28 10:11:07.512513 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "/usr/lib/python3/dist-packages/octavia/api/app.py", line 40, in get_pecan_config [Fri Apr 28 10:11:07.512529 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] return pecan_configuration.conf_from_file(filename) [Fri Apr 28 10:11:07.512543 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "/usr/lib/python3/dist-packages/pecan/configuration.py", line 176, in conf_from_file [Fri Apr 28 10:11:07.512553 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] SourceFileLoader(module_name, abspath).load_module(module_name) [Fri Apr 28 10:11:07.512567 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap_external>", line 522, in _check_name_wrapper [Fri Apr 28 10:11:07.512705 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap_external>", line 1027, in load_module [Fri Apr 28 10:11:07.512725 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap_external>", line 852, in load_module [Fri Apr 28 10:11:07.512737 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap>", line 265, in _load_module_shim [Fri Apr 28 10:11:07.512750 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap>", line 702, in _load [Fri Apr 28 10:11:07.512763 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap>", line 671, in _load_unlocked [Fri Apr 28 10:11:07.512775 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap_external>", line 844, in exec_module [Fri Apr 28 10:11:07.512788 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap_external>", line 981, in get_code [Fri Apr 28 10:11:07.512800 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap_external>", line 911, in source_to_code [Fri Apr 28 10:11:07.512839 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed [Fri Apr 28 10:11:07.512869 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] File "/usr/lib/python3/dist-packages/octavia/api/config.py", line 1 [Fri Apr 28 10:11:07.512893 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] Copyright 2014 Rackspace [Fri Apr 28 10:11:07.512921 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] ^ [Fri Apr 28 10:11:07.512938 2023] [wsgi:error] [pid 1312420:tid 140012169602816] [remote 127.0.0.1:59914] IndentationError: unexpected indent The original /usr/lib/python3/dist-packages/octavia/api/config.py file is added to the bug report. As seen in the file, the first line starts with a comment, but python is trying to run it as code. Removing the first 5 charactersin the file so that string above starts right after the comment yelds the following traceback: [Wed Apr 26 17:08:22.609980 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] mod_wsgi (pid=330647): Failed to exec Python script file '/usr/bin/octavia-wsgi'. [Wed Apr 26 17:08:22.610114 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] mod_wsgi (pid=330647): Exception occurred processing WSGI script '/usr/bin/octavia-wsgi'. [Wed Apr 26 17:08:22.611039 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] Traceback (most recent call last): [Wed Apr 26 17:08:22.611097 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] File "/usr/bin/octavia-wsgi", line 52, in <module> [Wed Apr 26 17:08:22.611106 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] application = setup_app() [Wed Apr 26 17:08:22.611125 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] File "/usr/lib/python3/dist-packages/octavia/api/app.py", line 59, in setup_app [Wed Apr 26 17:08:22.611135 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] pecan_config = get_pecan_config() [Wed Apr 26 17:08:22.611152 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] File "/usr/lib/python3/dist-packages/octavia/api/app.py", line 40, in get_pecan_config [Wed Apr 26 17:08:22.611160 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] return pecan_configuration.conf_from_file(filename) [Wed Apr 26 17:08:22.611178 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] File "/usr/lib/python3/dist-packages/pecan/configuration.py", line 168, in conf_from_file [Wed Apr 26 17:08:22.611327 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] compiled = compile(f.read(), abspath, 'exec') [Wed Apr 26 17:08:22.611355 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] File "/usr/lib/python3/dist-packages/octavia/api/config.py", line 1 [Wed Apr 26 17:08:22.611381 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] right 2014 Rackspace [Wed Apr 26 17:08:22.611419 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] ^ [Wed Apr 26 17:08:22.611437 2023] [wsgi:error] [pid 330647:tid 140629468657408] [remote 127.0.0.1:55142] SyntaxError: invalid syntax It seems that when the source file is read by python, the first 5 characters are skipped for some reason. Digging deeper, I see no reason for importlib to trim those characters. Internally it uses _io.open_code().read() which should return the entire file without trimming anything If adding the comment after the 5th char on the first line, the file is read as is should but the error moves to a different file: [Fri Apr 28 10:20:24.730194 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] mod_wsgi (pid=1316806): Failed to exec Python script file '/usr/bin/octavia-wsgi'. [Fri Apr 28 10:20:24.730379 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] mod_wsgi (pid=1316806): Exception occurred processing WSGI script '/usr/bin/octavia-wsgi'. [Fri Apr 28 10:20:24.732137 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] Traceback (most recent call last): [Fri Apr 28 10:20:24.732294 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] File "/usr/bin/octavia-wsgi", line 52, in <module> [Fri Apr 28 10:20:24.732316 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] application = setup_app() [Fri Apr 28 10:20:24.732340 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] File "/usr/lib/python3/dist-packages/octavia/api/app.py", line 62, in setup_app [Fri Apr 28 10:20:24.732353 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] return pecan_make_app( [Fri Apr 28 10:20:24.732381 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] File "/usr/lib/python3/dist-packages/pecan/__init__.py", line 86, in make_app [Fri Apr 28 10:20:24.732400 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] app = Pecan(root, **kw) [Fri Apr 28 10:20:24.732453 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] File "/usr/lib/python3/dist-packages/pecan/core.py", line 832, in __init__ [Fri Apr 28 10:20:24.732475 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] super(Pecan, self).__init__(*args, **kw) [Fri Apr 28 10:20:24.732516 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] File "/usr/lib/python3/dist-packages/pecan/core.py", line 240, in __init__ [Fri Apr 28 10:20:24.732536 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] root = self.__translate_root__(root) [Fri Apr 28 10:20:24.732582 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] File "/usr/lib/python3/dist-packages/pecan/core.py", line 275, in __translate_root__ [Fri Apr 28 10:20:24.732618 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] module = __import__(name, fromlist=fromlist) [Fri Apr 28 10:20:24.732645 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] File "/usr/lib/python3/dist-packages/octavia/api/root_controller.py", line 24, in <module> [Fri Apr 28 10:20:24.732716 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] from octavia.api.v2 import controllers as v2_controller [Fri Apr 28 10:20:24.732760 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] File "/usr/lib/python3/dist-packages/octavia/api/v2/controllers/__init__.py", line 19, in <module> [Fri Apr 28 10:20:24.732821 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] from octavia.api.v2.controllers import availability_zone_profiles [Fri Apr 28 10:20:24.732908 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] File "/usr/lib/python3/dist-packages/octavia/api/v2/controllers/availability_zone_profiles.py", line 1 [Fri Apr 28 10:20:24.732952 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] Copyright 2019 Verizon Media [Fri Apr 28 10:20:24.733024 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] ^ [Fri Apr 28 10:20:24.733107 2023] [wsgi:error] [pid 1316806:tid 140384699016960] [remote 127.0.0.1:41802] SyntaxError: invalid syntax As far as we can tell, this problem appears when importlib reads the file and hands it over incomplete (missing the first 5 chars). This issue only occurs on the Octavia containers and not on the other Openstack services. The other thing is that the fault is not consistent. It happens at Apache2 reload or restart but not everytime. Once in a few restarts. When the problem occurs, Octavia API is down. Env details: OS: Ubuntu 20.04 Kernel: 5.4.0-147-generic Openstack: Ussuri Octavia: ii octavia-api 6.2.2-0ubuntu1 all OpenStack Load Balancer as a Service - API frontend ii octavia-common 6.2.2-0ubuntu1 all OpenStack Load Balancer as a Service - Common files ii octavia-driver-agent 6.2.2-0ubuntu1 all OpenStack Load Balancer Service - Driver Agent ii octavia-health-manager 6.2.2-0ubuntu1 all OpenStack Load Balancer Service - Health manager ii octavia-housekeeping 6.2.2-0ubuntu1 all OpenStack Load Balancer Service - Housekeeping manager ii octavia-worker 6.2.2-0ubuntu1 all OpenStack Load Balancer Service - Worker ii python3-octavia 6.2.2-0ubuntu1 all OpenStack Load Balancer as a Service - Python libraries ii python3-octavia-lib 2.0.0-0ubuntu1 all Library to support Octavia provider drivers ii python3-ovn-octavia-provider 0.1.0-0ubuntu1 all OpenStack Octavia Integration with OVN - Python 3 library Python: Python 3.8.10 LXD: lxd 5.13-cea5ee2 24758 latest/stable canonical✓ - A bug was filled on launchpad for this issue: https://bugs.launchpad.net/ubuntu/+source/importlib/+bug/2018011 Thanks! Bug #2018011 “python3 - importlib fails to read first 5 chars fr...” : Bugs : importlib package : Ubuntu<https://bugs.launchpad.net/ubuntu/+source/importlib/+bug/2018011> A bit of context: we have an Openstack Ussuri deployment (HA) on Ubuntu Focal. Most Openstack services are running on LXD containers on 3 baremetals. There are also two OVN-dedicated-chassis which act as gateways and nova-compute nodes. A couple of days ago we updated all the packages in the environment to the latest versions (baremetals and lxc containers). After the updates, we started seeing problems with the Octavia API. The API would randomly give an internal server error. Octavia API is... bugs.launchpad.net

1 0