June 2024 - openstack-discuss - lists.openstack.org

Kolla Ansible on Ubuntu 20.04 - cloud-init & other network issues
by Tobias McNulty 10 Oct '24

10 Oct '24

Hi, I'm attempting to use Kolla Ansible 14.6.0 to deploy OpenStack Yoga on a small 3-node Ubuntu 20.04 cluster. The nodes have 128 GB RAM each, dual Xeon processors, and dual 10G Intel NICs. The NICs are connected to access ports on a 10G switch with separate VLANs for the local and external networks. All the playbooks run cleanly, but cloud-init is failing in the Ubuntu 20.04 and 22.04 VMs I attempt to boot. The VM images are unmodified from https://cloud-images.ubuntu.com/, and cloud-init works fine if I mount a second volume with user-data. The error is a timeout attempting to reach 169.254.169.254. This occurs both when booting a VM in an internal routed network and directly in an external network. I tried various neutron plugin agents (ovn, linuxbridge, and openvswitch both with and without firewall_driver = openvswitch <https://docs.openstack.org/kolla-ansible/latest/reference/networking/neutro…>) first with a clean install of the entire OS each time, all with the same result. Running tcpdump looking for 169.254.169.254 shows nothing. As a possible clue, the virtual NICs are unable to pass any traffic (e.g., to reach an external DHCP server) unless I completely disable port security on the interface (even if the associated security group is wide open). But disabling port security does not fix cloud-init (not to mention I don't really want to disable port security). Are there any additional requirements related to deploying OpenStack with Kolla on Ubuntu 20.04? This is a fairly vanilla configuration using the multinode inventory as a starting point. I tried to follow the Quick Start <https://docs.openstack.org/kolla-ansible/yoga/user/quickstart.html> as closely as possible; the only material difference I see is that I'm using the same 3 nodes for control + compute. I am using MAAS so it's easy to get a clean OS install on all three nodes ahead of each attempt. I plan to try again with the standard (non-HWE) kernel just in case, but otherwise I am running out of ideas. In case of any additional clues, here are my globals.yml and inventory file, along with the playbook I'm using to configure the network, images, VMs, etc., after bootstrapping the cluster: https://gist.github.com/tobiasmcnulty/7dbbdbc67abc08cbb013bf5983852ed6 Thank you in advance for any advice! Cheers, Tobias

5 9

[all][tc][ptls] Eventlet: Python 3.13 - Attach your seatbelt
by Herve Beraud 06 Sep '24

06 Sep '24

Hi all, As Python 3.13 is close to become a reality, new concerning incompatibilities between this new CPython version and Eventlet also become a reality: - https://github.com/eventlet/eventlet/issues/964 Fedora Python maintainers proposed a patch, but for now the problem remains unsolved. Also, many people reported bugs related to incompatibility between python3.12/dnspython/eventlet/pip-system-cert. There is an example who summarize well the problem https://github.com/eventlet/eventlet/issues/957 Do you have some visibility concerning the adoption of Python 3.13 into Openstack? Turbulences ahead... -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

4 5

PTG October 2024 Team Signup
by Kendall Nelson 29 Aug '24

29 Aug '24

Hello Everyone, Hopefully you have seen by now, the next OpenInfra PTG[1] which will take place October 21-25, 2024 virtually! Registration is now open[2]! To sign up your team, you must complete the survey[3] by August 30th at 7:00 UTC. We NEED accurate contact information (email and IRC nickname ideally) for the moderator of your team’s sessions. This is because the survey information will be used to organize the schedule signups which will be done via the PTGBot. If you are not on IRC, please get setup[4] on the OFTC network and join #openinfra-events. You are also encouraged to familiarize yourself with the PTGBot documentation[5] as well. If you have any questions, please reach out to ptg(a)openinfra.dev! Information about signing up for time slots will be sent to moderators after the team signup deadline. Continue to visit openinfra.dev/ptg for updates. -Kendall (diablo_rojo) [1] PTG Site: https://openinfra.dev/ptg/ [2] PTG Registration: http://ptg.openinfra.dev/ [3] Team Survey: https://openinfrafoundation.formstack.com/forms/oct2024_ptg_survey [4] Setup IRC: https://docs.openstack.org/contributors/common/irc.html [5] PTGBot README: https://opendev.org/openstack/ptgbot/src/branch/master/README.rst

1 2

networking-onos plugin
by Ganesan, Sathya 14 Aug '24

14 Aug '24

Good Day, Is there an alternative integration available for ONOS and Openstack since networking-onos is archived? Sathya Ganesan Geico Tech | Data Center Core Network Engineering - SDN | sganesan(a)geico.com<mailto:sganesan@geico.com> | 240.478.2121 | M-F 9a-5:30p EST Sensitivity: Company-Internal ==================== This email/fax message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution of this email/fax is prohibited. If you are not the intended recipient, please destroy all paper and electronic copies of the original message.

2 2

Live migration fails
by Szabo, Istvan (Agoda) 07 Aug '24

07 Aug '24

Hi, I have couple of compute nodes where the live migration fails with existing vms. When I quickly spawn a vm and try live migration it works so I assume shouldn't be a big problem with the compute node. However I have many existing vms where it fails with a servername not found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 ERROR nova.conductor.tasks.migrate [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Unable to find record for source node servername on servername: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Failed to compute_task_migrate_server: Compute host servername could not be found.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Setting instance to ACTIVE state.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.672 227612 ERROR oslo_messaging.rpc.server [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Exception during message handling: ComputeHostNotFound: Compute host am-osfecn-4025 Tried with this command: nova live-migration --block-migrate id. Any idea? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 16

[openstack][kolla-ansible]about upgrading openstack
by Nguyễn Hữu Khôi 04 Aug '24

04 Aug '24

Hello. I read this doc about upgrading openstack https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html But I would like to know if We install a new kolla-ansible(version to upgrade) on the new deployment host, apply changes with password.yml, global.yml and multinode and keep old /etc/kolla/config. Finally, we use kolla-ansible deploy instead of kolla-ansible upgrade. Are these steps ok? Pls correct me if I am wrong. Thank you. Regards Nguyen Huu Khoi

3 8

[RALLY 4.1.0] Verifier reconfigure fails when in "no admin" mode
by Jean-François TALTAVULL 16 Jul '24

16 Jul '24

Hi All, I'm using a Rally (4.1.0) deployment with existing users and no admin credentials. So the command "rally env show" outputs "admin": null in the json and it's ok. But the problem is that the command "rally verify configure-verifier -reconfigure" fails with: "rally_openstack.common.credential.OpenStackCredential() argument after ** must be a mapping, not NoneType" I've been struggling with this error for a number of hours with no results :-( Any idea ? I looked for a known bug in launchpad but found nothing. Thanks, Jean-Francois

2 4

Re: [neutron] ovs: multiple rows in Manager patch
by Brian Haley 16 Jul '24

16 Jul '24

Just sending response to the list in case anyone else has the same issue. On 6/17/24 1:09 PM, stream_unpicked390(a)simplelogin.com wrote: > Hi, > > On 6/13/24 21:38, Brian Haley - haleyb.dev(a)gmail.com wrote: >> Hi, >> >> On 6/13/24 4:53 AM, stream_unpicked390(a)simplelogin.com wrote: >>> Hi Brian, >>> On 6/12/24 23:04, Brian Haley - haleyb.dev(a)gmail.com wrote: >>>> Hi, >>>> >>>> On 6/12/24 12:43 PM, stream_unpicked390(a)simplelogin.com wrote: >>>>> Hi everyone! >>>>> >>>>> this is my first time on a mailing list, so please excuse any >>>>> "obvious" mistakes and feel free to correct me :) >>>>> >>>>> I'm currently setting up a minimal OpenStack system manually on one >>>>> server. The setup is nearly complete but I'm running into issues >>>>> with neutron. After fully completing the neutron setup as mentioned >>>>> in the docs, the neutron-ovs-service refuses to start, showing the >>>>> error message >>>> >>>> Sorry, but I have to ask which "docs" are you using? >>> >>> thanks for your quick answer! I followed the OpenStack install guide >>> under https://docs.openstack.org/2024.1/install/. For neutron >>> specifically, I used the document linked under >>> https://docs.openstack.org/neutron/2024.1/install/install-ubuntu.html. >>> >>> As I mentioned, I'm currently setting up everything on just one >>> server, the documentations scenario describes two servers. This is >>> the only intentional diversion I made from the docs. I've been trying >>> to setup a self-service network. >>> >>>> I don't see ptcp used in any of the examples here for ovsdb_connection: >>>> >>>> https://docs.openstack.org/neutron/latest/install/ >>> >>> I couldn't find any example of the ovsdb_connection and I don't >>> remember modifying any value like that either. The problem existed >>> "right out of the box". Can you point me to the part of the linked >>> documentation where the ovsdb_connection config is discussed? >> >> Instead of trying to go through the details of anything that might be >> wrong, if you are just trying to get a single-node Openstack running >> have you tried using devstack [0]? It would be much simpler if this is >> just for testing purposes. > This is a production setup so devstack was out of the question. Because > of our very tight constraints, we first have to set up a single node, > before being able to add additional servers. Definitely not ideal.. Your > hint about ovsdb_connection helped me out! While I couldn't find any > setting with the name ovsdb_connection already set to an, apparently > wrong, string, I set ovsdb_connection to the value mentioned by you. > This probably overwrote something I couldn't find, because after > restarting neutron, my issues were gone! Thanks again for looking into > my issue and guiding me on the right path! Glad it worked out, thanks for responding. -Brian >> [0] https://github.com/openstack-dev/devstack.git >> >>>>> multiple rows in Manager patch "ptcp:6640:127.0.0.0" >>>> >>>> Two things look wrong here, both the IP address and port. Typically >>>> this string would be "ptcp:127.0.0.1:6640" or "ptcp:0.0.0.0:6640" in >>>> ml2_conf.ini. >>> >>> Thanks! I will try to take a look at the aforementioned config as >>> soon as possible and report back! >>> >>>> >>>> -Brian >>>> >>>>> I dumped the ovs database which only shows one row in the Manager >>>>> table. After many days of debugging and googleing I'm out of ideas >>>>> how this could be solved.. >>>>> I already posted this issue on serverfault >>>>> (https://serverfault.com/questions/1160418/openstack-ovs-multiple-rows-in-ma…) only after that post finding this mailing list! >>>>> >>>>> Best regards >>>>> Talooola >>>>> >>>> >>

2 1

[swift] Weird log corruption or something more serious?
by Sam Morrison 16 Jul '24

16 Jul '24

Hi, We are debugging a weird issue with our swift cluster (currently running 2.29.2) We got a report from a user that they were getting upload failures and then sent us this from their logs: 2024-06-25 23:30:06,829 | swiftclient.service | ERROR | Object PUT failed: https://swift-host/v1/PROJECT_ID_A/prod_result/195e5e8e-334a-11ef-ba70-0a58… 401 Unauthorized [first 60 chars of response] b'<html><h1>Unauthorized</h1><p>This server could not verify t' (txn: txab25fcfcf3354bbbaf105-00667b52fe) Searching our logs for that transaction ID I found: 203.101.227.241 127.0.0.1 25/Jun/2024/23/30/06 PUT /v1/AUTH_PROJECT_ID_B/nexus-migration/nexus-data/blobs/default/content/vol-27/chap-25/6bc30bae-4f5e-40a8-bc63-096679af1a58.bytes HTTP/1.0 401 - rclone/v1.59.0%2Cpython-swiftclient-4.2.0 gAAAAABme1L7tlfE... - 131 - txab25fcfcf3354bbbaf105-00667b52fe - 0.0440 - - 1719358206.765954018 1719358206.809994936 0 Note this is a completely different project ID from our side, timestamp is correct too. We also have this: ERROR WSGI: code 400, message Bad request syntax ('\x89PNG') (txn: txab25fcfcf3354bbbaf105-00667b52fe) (client_ip: XX.XX.XX.XX) Upon further analysis we see this quite a lot, mostly from attempted hackers but lots from real users and lots seem to be related to rclone. Eg. Jun 27 08:24:46 swift-host proxy-server[70043]: ERROR WSGI: code 400, message Bad request syntax ('PUT /v1/AUTH_PROJECT_ID_B/nexus-migration/nexus-data/elasticsearch/nexus/nodes/0/indices/42b6a6a76dae683a647546ac02a6883108ac7df5/0/index/segments_4PUT /v1/AUTH_PROJECT_ID_B/nexus-migration/nexus-data/elasticsearch/nexus/nodes/0/indices/4db62ff1327dfe236d85b1dd1a6a9e8139e7e4cf/0/index/_67t.cfs HTTP/1.1') (txn: txd57a3d00bb1942c6a390e-00667c952a) (client_ip: XX.XX.XX.XX) Anyone know what is going on here? Thanks, Sam

1 1

[skyline][tc] Graduating skyline out of the "Emerging Projects" list
by Goutham Pacha Ravi 09 Jul '24

09 Jul '24

Hi, We were discussing in a recent TC meeting that Skyline seems to be ready to lose the "Emerging project" tag [1]. The arguments supporting it's graduation are: - The project is actively maintained, and has had consistent releases since the Zed release [2][3][4][5] - The project is being used in production cloud deployments [6] - Operators have requested skyline's inclusion in deployment and configuration management projects [7][8] Wu Wenxiang / Skyline team - is this assessment correct? If yes, are there any objections to removing Skyline from [1]? If it helps, the project team can re-assess the OpenStack project requirements we have [9] and work on any deficiencies. Thanks, On behalf of the OpenStack TC, Goutham Pacha Ravi OpenStack TC Chair [1] https://governance.openstack.org/tc/reference/emerging-technology-and-inact… [2] https://releases.openstack.org/zed/#skyline-apiserver [3] https://releases.openstack.org/antelope/#skyline-apiserver [4] https://releases.openstack.org/bobcat/#skyline-apiserver [5] https://releases.openstack.org/caracal/#skyline-apiserver [6] https://governance.openstack.org/tc/user_survey/analysis-2023.html#how-do-y… [7] https://review.opendev.org/q/topic:%22osa/skyline%22 [8] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack…

2 2