From mnasiadka at gmail.com Sat Jul 1 09:15:36 2023 From: mnasiadka at gmail.com (=?UTF-8?Q?Micha=C5=82_Nasiadka?=) Date: Sat, 1 Jul 2023 11:15:36 +0200 Subject: [kolla] kolla-toolbox Ubuntu 22 Erlang RMQ dependencies In-Reply-To: <73508178567141aab7bfd076400780a7@nl.team.blue> References: <73508178567141aab7bfd076400780a7@nl.team.blue> Message-ID: Hi Bram, This has been fixed in stable git repos, but not yet released in pypi - please install kolla directly from git. W dniu pt., 30.06.2023 o 12:52 Bram Kranendonk napisa?(a): > Hi all, > > > I'm trying to build the kolla-toolbox image for Ubuntu 22 source > for 2023.1 using kolla v16.0.0 > > > This is however not possible due to broken packages for Erlang/RabbitMQ > (tried without package cache): > > > INFO:kolla.common.utils.kolla-toolbox: rabbitmq-server : Depends: > erlang-base (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be > installed or > INFO:kolla.common.utils.kolla-toolbox: > erlang-base-hipe (< 1:26.0) but it is not installable or > INFO:kolla.common.utils.kolla-toolbox: > esl-erlang (< 1:26.0) but it is not installable > INFO:kolla.common.utils.kolla-toolbox: Depends: > erlang-crypto (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be > installed or > INFO:kolla.common.utils.kolla-toolbox: > esl-erlang (< 1:26.0) but it is not installable > INFO:kolla.common.utils.kolla-toolbox: Depends: > erlang-eldap (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be > installed or > INFO:kolla.common.utils.kolla-toolbox: > esl-erlang (< 1:26.0) but it is not installable > INFO:kolla.common.utils.kolla-toolbox: Depends: > erlang-inets (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be > installed or > INFO:kolla.common.utils.kolla-toolbox: > esl-erlang (< 1:26.0) but it is not installable > INFO:kolla.common.utils.kolla-toolbox: Depends: > erlang-mnesia (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be > installed or > INFO:kolla.common.utils.kolla-toolbox: > esl-erlang (< 1:26.0) but it is not installable > INFO:kolla.common.utils.kolla-toolbox: Depends: > erlang-os-mon (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be > installed or > INFO:kolla.common.utils.kolla-toolbox: > esl-erlang (< 1:26.0) but it is not installable > INFO:kolla.common.utils.kolla-toolbox: Depends: > erlang-parsetools (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be > installed or > INFO:kolla.common.utils.kolla-toolbox: > esl-erlang (< 1:26.0) but it is not installable > INFO:kolla.common.utils.kolla-toolbox: Depends: > erlang-public-key (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be > installed or > INFO:kolla.common.utils.kolla-toolbox: > esl-erlang (< 1:26.0) but it is not installable > INFO:kolla.common.utils.kolla-toolbox: Depends: > erlang-runtime-tools (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to > be installed or > INFO:kolla.common.utils.kolla-toolbox: > esl-erlang (< 1:26.0) but it is not installable > INFO:kolla.common.utils.kolla-toolbox: Depends: > erlang-ssl (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be > installed or > INFO:kolla.common.utils.kolla-toolbox: > esl-erlang (< 1:26.0) but it is not installable > INFO:kolla.common.utils.kolla-toolbox: Depends: > erlang-syntax-tools (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to > be installed or > INFO:kolla.common.utils.kolla-toolbox: > esl-erlang (< 1:26.0) but it is not installable > INFO:kolla.common.utils.kolla-toolbox: Depends: > erlang-tools (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be > installed or > INFO:kolla.common.utils.kolla-toolbox: > esl-erlang (< 1:26.0) but it is not installable > INFO:kolla.common.utils.kolla-toolbox: Depends: > erlang-xmerl (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be > installed or > INFO:kolla.common.utils.kolla-toolbox: > esl-erlang (< 1:26.0) but it is not installable > INFO:kolla.common.utils.kolla-toolbox:E: Unable to correct problems, you > have held broken packages. > > > I saw a recent commit that pins the RabbitMQ version, but this commit was > reverted later on. > > Does anyone has a idea how I can fix this issue? > > Thanks, > > > > . Bram Kranendonk > System Engineer > > Oostmaaslaan 71 > > (15e etage) > 3063 AN Rotterdam > The Netherlands > > [image: team.blue logo] > > > -- Micha? Nasiadka mnasiadka at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From haleyb.dev at gmail.com Sat Jul 1 22:54:55 2023 From: haleyb.dev at gmail.com (Brian Haley) Date: Sat, 1 Jul 2023 18:54:55 -0400 Subject: [neutron] Bug deputy report for week of June 26th Message-ID: <8b23be0a-35ec-b52c-2d0f-f4a14f295b31@gmail.com> Hi, I was Neutron bug deputy last week. Below is a short summary about the reported bugs. All but one bug has a patch proposed, so great job with that :) -Brian Critical bugs ------------- * https://bugs.launchpad.net/neutron/+bug/2025126 - Functional MySQL sync test failing with oslo/sqlalchemy master branch - https://review.opendev.org/c/openstack/neutron/+/886961 * https://bugs.launchpad.net/neutron/+bug/2025486 - [stable/wallaby] neutron-tempest-plugin-scenario-ovn-wallaby fails on ovn git clone - https://review.opendev.org/c/openstack/devstack/+/887184 High bugs --------- * https://bugs.launchpad.net/neutron/+bug/2025129 - DvrLocalRouter init references namespace before it is created - https://review.opendev.org/c/openstack/neutron/+/887036 * https://bugs.launchpad.net/neutron/+bug/2025202 - Execute neutron-ovn-db-sync-util report TypeError - https://review.opendev.org/c/openstack/neutron/+/881771 * https://bugs.launchpad.net/neutron/+bug/2025341 - flows lost with noop firewall driver at ovs-agent restart while the db is down - https://review.opendev.org/c/openstack/neutron/+/887257 * https://bugs.launchpad.net/neutron/+bug/2025368 - [sRBAC] Subnets permissions now inherit from the parent resource Network - https://review.opendev.org/c/openstack/neutron/+/887287 Medium bugs ----------- * https://bugs.launchpad.net/neutron/+bug/2025056 - Router ports without IP addresses shouldn't be allowed to deletion using port's API directly - https://review.opendev.org/c/openstack/neutron/+/886983 * https://bugs.launchpad.net/neutron/+bug/2025246 - [OVN] Improve log for the exception handling of ovn_l3/plugin.py - https://review.opendev.org/c/openstack/neutron/+/887153 * https://bugs.launchpad.net/neutron/+bug/2025264 - [ovn][DVR]FIP traffic centralized in DVR environments - https://review.opendev.org/c/openstack/neutron/+/886626 * https://bugs.launchpad.net/neutron/+bug/2025466 - [OVN] OVN agent should register the correct config options - https://review.opendev.org/c/openstack/neutron/+/887363 Low bugs -------- * https://bugs.launchpad.net/neutron/+bug/2025144 - [OVN] ``update_floatingip`` should handle the case when only the QoS policy is updated - https://review.opendev.org/c/openstack/neutron/+/886974 * https://bugs.launchpad.net/neutron/+bug/2025410 - After `openstack port set no-fixed-ip` command, the port's ip and mac's arp are still in qrouter without deletion - Could not reproduce with ML2/OVS, added questions Misc bugs --------- * https://bugs.launchpad.net/neutron/+bug/2024976 - iptable rules restoring error in l3-agent and openvswitch-agent - Not a bug, system issue Wishlist bugs ------------- * https://bugs.launchpad.net/neutron/+bug/2025055 - [rfe][ml2] Add a new API that supports cloning a specified security group - Asked about applicability, mentioned default SG RFE, https://bugs.launchpad.net/neutron/+bug/1983053 From satish.txt at gmail.com Sun Jul 2 02:53:44 2023 From: satish.txt at gmail.com (Satish Patel) Date: Sat, 1 Jul 2023 22:53:44 -0400 Subject: [OPENSTACK][rabbitmq] using quorum queues In-Reply-To: References: Message-ID: Thank you! I will deploy it with Quorum Queue and give you my feedback. On Sun, Jun 18, 2023 at 8:02?PM Nguy?n H?u Kh?i wrote: > Helo, > With kolla ansible > > nano /etc/kolla/config/global.conf > > [oslo_messaging_rabbit] > rabbit_quorum_queue = true > > but you need destroy (rm rabbitmq container and its volume ) and redeploy > new one to make quorum queues work. > > Nguyen Huu Khoi > > > On Sun, Jun 18, 2023 at 9:04?AM Satish Patel wrote: > >> Great! This is good to know that Quorum is a good solution. >> >> Do you have a config to enable in kolla-ansible deployment? >> >> On Thu, Jun 15, 2023 at 5:43?AM Arnaud Morin >> wrote: >> >>> Hey, >>> >>> We are also using quorum in some regions and plan to enable quorum >>> everwhere. >>> >>> Note that we also manage to enable quorum for transient queues (using a >>> custom patch as it's not doable with current oslo.messaging, see my >>> request in [1]). >>> We also introduced some custom changes in py-amqp to handle correctly >>> the rabbit disconnections (see [2] and [3]). >>> >>> So far, the real improvment is achieved thanks to the combination of all >>> of these changes, enabling quorum queue only was not enough for us to >>> notice any improvment. >>> >>> The downside of quorum queues is that it consume more power on the >>> rabbit cluster: you need more IO, CPU, RAM and network bandwith for the >>> same number of queues (see [4]). >>> It has to be taken into account. >>> >>> Cheers, >>> Arnaud. >>> >>> [1] >>> https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033343.html >>> [2] https://github.com/celery/py-amqp/pull/410 >>> [3] https://github.com/celery/py-amqp/pull/405 >>> [4] >>> https://plik.ovh/file/nHCny7psDCTrEm76/Pq9ASO9wUd8HRk4C/s_1686817000.png >>> >>> >>> >>> On 13.06.23 - 09:14, Sa Pham wrote: >>> > Dear Kh?i, >>> > >>> > Thanks for your reply. >>> > >>> > >>> > >>> > On Tue, Jun 13, 2023 at 9:05?AM Nguy?n H?u Kh?i < >>> nguyenhuukhoinw at gmail.com> >>> > wrote: >>> > >>> > > Hello. >>> > > Firstly, when I used the classic queue and sometimes, my rabbitmq >>> cluster >>> > > was broken, the computers showed state down and I needed to restart >>> the >>> > > computer service to make it up. Secondly, 1 of 3 controller is down >>> but my >>> > > system still works although it is not very first as fully >>> controller. I ran >>> > > it for about 3 months compared with classic. My openstack is Yoga >>> and use >>> > > Kolla-Ansible as a deployment tool, >>> > > Nguyen Huu Khoi >>> > > >>> > > >>> > > On Tue, Jun 13, 2023 at 8:43?AM Sa Pham wrote: >>> > > >>> > >> Hi Kh?i, >>> > >> >>> > >> Why do you say using the quorum queue is more stable than the >>> classic >>> > >> queue ? >>> > >> >>> > >> Thanks, >>> > >> >>> > >> >>> > >> >>> > >> On Tue, Jun 13, 2023 at 7:26?AM Nguy?n H?u Kh?i < >>> > >> nguyenhuukhoinw at gmail.com> wrote: >>> > >> >>> > >>> Hello Huettner, >>> > >>> I have used the quorum queue since March and it is ok until now. It >>> > >>> looks more stable than the classic queue. Some feedback to you. >>> > >>> Thank you. >>> > >>> Nguyen Huu Khoi. >>> > >>> >>> > >>> >>> > >>> >>> > >>> On Mon, May 8, 2023 at 1:14?PM Felix H?ttner >>> >>> > >>> wrote: >>> > >>> >>> > >>>> Hi Nguyen, >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> we are using quorum queues for one of our deployments. So fare we >>> did >>> > >>>> not have any issue with them. They also seem to survive restarts >>> without >>> > >>>> issues (however reply queues are still broken afterwards in a >>> small amount >>> > >>>> of cases, but they are no quorum/mirrored queues anyway). >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> So I would recommend them for everyone that creates a new cluster. >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> -- >>> > >>>> >>> > >>>> Felix Huettner >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> *From:* Nguy?n H?u Kh?i >>> > >>>> *Sent:* Saturday, May 6, 2023 4:29 AM >>> > >>>> *To:* OpenStack Discuss >>> > >>>> *Subject:* [OPENSTACK][rabbitmq] using quorum queues >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> Hello guys. >>> > >>>> >>> > >>>> IS there any guy who uses the quorum queue for openstack? Could >>> you >>> > >>>> give some feedback to compare with classic queue? >>> > >>>> >>> > >>>> Thank you. >>> > >>>> >>> > >>>> Nguyen Huu Khoi >>> > >>>> >>> > >>>> Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist >>> nur >>> > >>>> f?r die Verwertung durch den vorgesehenen Empf?nger bestimmt. >>> > >>>> Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den >>> > >>>> Absender bitte unverz?glich in Kenntnis und l?schen diese E Mail. >>> > >>>> >>> > >>>> Hinweise zum Datenschutz finden Sie hier >>> > >>>> . >>> > >>>> >>> > >>>> >>> > >>>> This e-mail may contain confidential content and is intended only >>> for >>> > >>>> the specified recipient/s. >>> > >>>> If you are not the intended recipient, please inform the sender >>> > >>>> immediately and delete this e-mail. >>> > >>>> >>> > >>>> Information on data protection can be found here >>> > >>>> . >>> > >>>> >>> > >>> >>> > >> >>> > >> -- >>> > >> Sa Pham Dang >>> > >> Skype: great_bn >>> > >> Phone/Telegram: 0986.849.582 >>> > >> >>> > >> >>> > >> >>> > >>> > -- >>> > Sa Pham Dang >>> > Skype: great_bn >>> > Phone/Telegram: 0986.849.582 >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.acosta at luizalabs.com Mon Jul 3 12:09:03 2023 From: roberto.acosta at luizalabs.com (Roberto Bartzen Acosta) Date: Mon, 3 Jul 2023 09:09:03 -0300 Subject: [nova] - libvirt - nl_recv returned with error: No buffer space available Message-ID: Hi OpenStack folks, We are seeing the following error when creating VMs: Jun 28 19:33:29 SR-0001 libvirtd[5289]: nl_recv returned with error: No buffer space available This problem occurs in deployments with different OS versions, for example: Ubuntu 20.04.4 LTS ii libnl-3-200:amd64 3.4.0-1 ii libvirt-daemon 6.0.0-0ubuntu8.16 Ubuntu 22.04 LTS ii libnl-3-200:amd64 3.5.0-0.1 ii libvirt-daemon 8.0.0-1ubuntu7.5 The hypervisor has around 40 native interfaces (bounds, vlans, tunneling bridges, mgmt bridges, physical NICs, etc), and plus 40 or more tap interfaces (depending on the number of VMs allocated), but we observed errors with more than 40 tap interfaces added to the host. Is anyone else noticing this? more details about the problem [1] Regards, Roberto [1] - https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/2025647 -- _?Esta mensagem ? direcionada apenas para os endere?os constantes no cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o imediatamente anuladas e proibidas?._ *?**?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o poder? aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos?.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From bram.kranendonk at nl.team.blue Mon Jul 3 12:13:28 2023 From: bram.kranendonk at nl.team.blue (Bram Kranendonk) Date: Mon, 3 Jul 2023 12:13:28 +0000 Subject: [kolla] kolla-toolbox Ubuntu 22 Erlang RMQ dependencies In-Reply-To: References: <73508178567141aab7bfd076400780a7@nl.team.blue>, Message-ID: <6e43acc2941649b3b263e919008096e3@nl.team.blue> Works like a charm. Thanks MIchal. . Bram Kranendonk System Engineer Oostmaaslaan 71 (15e etage) 3063 AN Rotterdam The Netherlands [team.blue logo] ________________________________ Van: Micha? Nasiadka Verzonden: zaterdag 1 juli 2023 11:15:36 Aan: Bram Kranendonk CC: openstack-discuss at lists.openstack.org Onderwerp: Re: [kolla] kolla-toolbox Ubuntu 22 Erlang RMQ dependencies Hi Bram, This has been fixed in stable git repos, but not yet released in pypi - please install kolla directly from git. W dniu pt., 30.06.2023 o 12:52 Bram Kranendonk napisa?(a): Hi all, I'm trying to build the kolla-toolbox image for Ubuntu 22 source for 2023.1 using kolla v16.0.0 This is however not possible due to broken packages for Erlang/RabbitMQ (tried without package cache): INFO:kolla.common.utils.kolla-toolbox: rabbitmq-server : Depends: erlang-base (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: erlang-base-hipe (< 1:26.0) but it is not installable or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-crypto (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-eldap (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-inets (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-mnesia (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-os-mon (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-parsetools (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-public-key (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-runtime-tools (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-ssl (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-syntax-tools (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-tools (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox: Depends: erlang-xmerl (< 1:26.0) but 1:26.0.2-1rmq1ppa1~ubuntu22.04.1 is to be installed or INFO:kolla.common.utils.kolla-toolbox: esl-erlang (< 1:26.0) but it is not installable INFO:kolla.common.utils.kolla-toolbox:E: Unable to correct problems, you have held broken packages. I saw a recent commit that pins the RabbitMQ version, but this commit was reverted later on. Does anyone has a idea how I can fix this issue? Thanks, . Bram Kranendonk System Engineer Oostmaaslaan 71 (15e etage) 3063 AN Rotterdam The Netherlands [team.blue logo] -- Micha? Nasiadka mnasiadka at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From katonalala at gmail.com Mon Jul 3 14:39:55 2023 From: katonalala at gmail.com (Lajos Katona) Date: Mon, 3 Jul 2023 16:39:55 +0200 Subject: [Neutron] Tap as a service In-Reply-To: References: Message-ID: Hi, I suppose you would like to use taas with ovs, am I right? For that be sure that agent_extension taa is loaded, in the ovs-agent log: INFO neutron.agent.agent_extensions_manager [-] Loaded agent extensions: ['taas'] If that is ok, when you create tap-services or tap-mirrors (and for the start of the agent) you can check the ovs flows if everything is as expected. For some help please check my doc under review: https://review.opendev.org/c/openstack/tap-as-a-service/+/828382 Hope you can move forward. Lajos altidor JB ezt ?rta (id?pont: 2023. j?n. 30., P, 14:10): > Thanks! > Basically I need to port mirror traffic to an IDS instance. TaaaS seemed > the most promising option. > I couldn't find a documentated way of installing it on openstack with juju > and maas and tried to do something similar than the devstack multi-node > (link). But my tap flows and ports stay down. > do you know of a better option to port mirror? > My logic was : install taas on the neutron lxd and all the nodes + > necessary configurations. Is there something I'm missing? > JB > > On Fri, Jun 30, 2023, 03:28 Lajos Katona wrote: > >> Hi, >> If you have technical questions I can help (I hope), but sadly I don't >> know if deployment tool integration is ready for tap-as-a-service. >> >> Lajos Katona (lajoskatona) >> >> altidor JB ezt ?rta (id?pont: 2023. j?n. 29., Cs, >> 19:31): >> >>> Hello, >>> Can anyone help me with the setup of Tap as a Service on Openstack with >>> Juju and Maas? >>> I've seen tutorials for installing it on DevStack but nothing for large >>> scale setups. >>> So far I've tried doing a similar process to this: >>> https://zhuanlan.zhihu.com/p/101599786 >>> basically install the apt on the neutron_api lxd and on the compute >>> nodes, with configurations that are done in the above tutorial. >>> Sounds simple enough. >>> But the tap_services and tap_flows stay in Down state. >>> Any ideas??? Also I'm open to another way to do port mirroring if this >>> doesn't work. >>> >>> JB >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From radoslaw.piliszek at gmail.com Mon Jul 3 15:20:19 2023 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Mon, 3 Jul 2023 17:20:19 +0200 Subject: [ironic] Capping off storyboard migration In-Reply-To: References: Message-ID: Hi Jay, I was thinking very-long-term if Storyboard was to go away and Ironic wanted a self-sovereign storage of that data. ;-) And I agree wholeheartedly with the last sentence of yours - about taking an action even if not ideal. I believe it's a common approach in Fortune 500 companies and many aspiring to be. Things rarely solve themselves by themselves. Radek -yoctozepto On Fri, 30 Jun 2023 at 19:04, Jay Faulkner wrote: > > Hey Radek, > > One of the nice things is that the stories will all still be in storyboard, along with their tasks and all associated information. Automatically closing the issues does not remove the history, but instead changes the status quo from "users/contributors who submitted those are being ignored" to "users/contributors who submitted those now understand where to go to get attention". > > Given there's also a giant backlog of older launchpad bugs to triage before we are "caught up" as a project, I intend on placing extra effort there. > > Ironic's bugtracking situation is an unfortunate reality, and I think taking action -- even if it's not the ideal action -- is better than maintaining the status quo. > > -JayF From fungi at yuggoth.org Mon Jul 3 15:43:49 2023 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 3 Jul 2023 15:43:49 +0000 Subject: [ironic] Capping off storyboard migration In-Reply-To: References: Message-ID: <20230703154349.nncnxn6wiwp274ns@yuggoth.org> On 2023-07-03 17:20:19 +0200 (+0200), Rados?aw Piliszek wrote: > I was thinking very-long-term if Storyboard was to go away and Ironic > wanted a self-sovereign storage of that data. ;-) > > And I agree wholeheartedly with the last sentence of yours - about > taking an action even if not ideal. I believe it's a common > approach in Fortune 500 companies and many aspiring to be. Things > rarely solve themselves by themselves. [...] From an OpenDev standpoint, we don't currently have any plans to take down the StoryBoard service. Should we decide to, there would almost certainly be ample warning and I at least would want to make sure we have some archival copy of all the stories it contained for posterity, even if that means asking for help from others with a vested interest in its preservation (much like we did with keeping ask.openstack.org online in a read-only state and then later switching to a link to the Internet Archive). -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From caiql at toyou.com.cn Mon Jul 3 16:46:08 2023 From: caiql at toyou.com.cn (=?UTF-8?B?6JSh5ZCv6b6Z?=) Date: Tue, 4 Jul 2023 00:46:08 +0800 (CST) Subject: =?UTF-8?B?W2NpbmRlcl1bZHJpdmVyc10gbmV3IGRyaXZlciBtZXJnZSBmcmVlemUgcXVlc3Rpb24=?= Message-ID: <997373634.584897.1688402768093.JavaMail.xmail@bj-wm-cp-12> An HTML attachment was scrubbed... URL: From ddanelski at cloudferro.com Mon Jul 3 10:22:02 2023 From: ddanelski at cloudferro.com (Dominik Danelski) Date: Mon, 3 Jul 2023 10:22:02 +0000 Subject: [nova] Scheduler Optimiser Message-ID: <09981137-ce97-df6b-55f9-038df7f10498@cloudferro.com> Hello, I would like to introduce you to the tool developed under the working title "Scheduler Optimiser". It is meant to test the effectiveness of different Scheduler configurations, both weights and filters on a given list of VM orders and in a semi-realistic infrastructure. My company - CloudFerro - has been preparing in-house for the last few months and foresees publishing the project as FOSS once it reaches the MVP stage. To make the final result more useful to the community and speed up the development (and release), I humbly ask for your expertise: Are you aware of previous similar efforts? Do you notice some flaws in the current approach? What, in your opinion, are more important aspects of the infrastructure behaviour, and what can be relatively safely ignored in terms of the effect on Scheduler results/allocation? Project objectives: * Use Devstack (or another OpenStack deployer) with a real Scheduler to replay a list of compute VM orders, either real from one's infrastructure or artificially created. * Assess the effectiveness of the scheduling in various terms like: "How many machines of a given type can still be allocated at the moment?" using plug-in "success meters". In a strict sense, the project does not simulate THE Scheduler but interacts with it. * Use fake-virt to emulate huge architectures on a relatively tiny test bench. * Have as little as possible, and ideally no changes to the Devstack's code that could not be included in the upstream repository. The usage should be as simple as: 1. Install Devstack. 2. Configure Devstack's cluster with its infrastructure information like flavours and hosts. 3. Configure Scheduler for a new test case. 4. Replay VM orders. 5. Repeat steps 3 and 4 to find better Scheduler settings. * Facilitate creating a minimal required setup of the test bench. Not by replacing standard Devstack scripts, but mainly through tooling for quick rebuilding data like flavours, infrastructure state, and other factors relevant to the simulation. Outside of the scope: * Running continuous analysis on the production environment, even if some plug-ins could be extracted for this purpose. * Retaining information about users and projects when replaying orders. * (Probably / low priority) replaying actions other than VM creation/deletion as they form a minority of operations and ignoring them should not have a distinct effect on the comparison experiments. Current state: ? ?Implemented: * Recreating flavours from JSON file exported via OpenStack CLI. * Replaying a list of orders in the form of (creation_date, termination_date, resource_id (optional), flavor_id) with basic flavour properties like VCPU, RAM, and DISK GB. The orders are replayed consecutively. * Plug-in success-rater mechanism which runs rater classes (returning quantified success measure) after each VM add/delete action, retains their intermediate history and "total success" - how it is defined is implementation dependent. First classes interacting with Placement like: "How many VMs of flavours x (with basic parameters for now) can fit in the cluster?" or "How many hosts are empty?". Missing: * Recreating hosts, note the fake-virt remark from "Risks and Challenges". * Tools facilitating Scheduler configuration. * Creating VMs with more parameters like VGPU, traits, and aggregates. * (Lower priority) saving the intermediate state of the cluster during simulation i.e. allocations to analyse it without rerunning the experiment. Currently, only the quantified meters are saved. * Gently failing and saving all information in case of resource depletion: close to completion, handling one exception type in upper layers is needed. * More success meters. Risks and Challenges: * Currently, the tool replays actions one by one, it waits for each creation and deletion to be complete before running success raters and taking another order. Thus, the order of actions is important, but not their absolute time and temporal density. This might skip some side-effects of a realistic execution. * Similarly, to the above, fake-virt provides simple classes that will not reproduce some behaviours of real-world hypervisors. An explicit Scheduler avoids hosts that had recently failed to allocate a VM, but most likely fake-virt will not mock such behaviour. * Fake-virt should reproduce a real diverse infrastructure instead of x copies of the same flavour. This might be the only, but very important change to the OpenStack codebase. If successful, it could benefit other projects and tests as well. Even though the list of missing features is seemingly larger, the most important parts of the program are already there, so we hope to finish the MVP development in a relatively short amount of time. We are going to publish it as FOSS in either case, but as mentioned your observations would be very much welcome at this stage. I am also open to answering more questions about the project. Kind regards Dominik Danelski From alsotoes at gmail.com Mon Jul 3 20:08:27 2023 From: alsotoes at gmail.com (Alvaro Soto) Date: Mon, 3 Jul 2023 14:08:27 -0600 Subject: [nova] Scheduler Optimiser In-Reply-To: <09981137-ce97-df6b-55f9-038df7f10498@cloudferro.com> References: <09981137-ce97-df6b-55f9-038df7f10498@cloudferro.com> Message-ID: Nice idea, but my main question is, how do you plan to beat the ones implemented currently? I'm working a little researching a little with some techniques to try to beat the random resource allocation schedulers. Can you share more about your research and/or implementation idea? Cheers. --- Alvaro Soto. Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you. ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people. On Mon, Jul 3, 2023, 2:00 PM Dominik Danelski wrote: > > Hello, > > > I would like to introduce you to the tool developed under the working > title "Scheduler Optimiser". It is meant to test the effectiveness of > different Scheduler configurations, both weights and filters on a given > list of VM orders and in a semi-realistic infrastructure. > > My company - CloudFerro - has been preparing in-house for the last few > months and foresees publishing the project as FOSS once it reaches the > MVP stage. To make the final result more useful to the community and > speed up the development (and release), I humbly ask for your expertise: > Are you aware of previous similar efforts? Do you notice some flaws in > the current approach? What, in your opinion, are more important aspects > of the infrastructure behaviour, and what can be relatively safely > ignored in terms of the effect on Scheduler results/allocation? > > > Project objectives: > > * Use Devstack (or another OpenStack deployer) with a real Scheduler > to replay a list of compute VM orders, either real from one's > infrastructure or artificially created. > * Assess the effectiveness of the scheduling in various terms like: > "How many machines of a given type can still be allocated at the > moment?" using plug-in "success meters". In a strict sense, the > project does not simulate THE Scheduler but interacts with it. > * Use fake-virt to emulate huge architectures on a relatively tiny > test bench. > * Have as little as possible, and ideally no changes to the Devstack's > code that could not be included in the upstream repository. The > usage should be as simple as: 1. Install Devstack. 2. Configure > Devstack's cluster with its infrastructure information like flavours > and hosts. 3. Configure Scheduler for a new test case. 4. Replay VM > orders. 5. Repeat steps 3 and 4 to find better Scheduler settings. > * Facilitate creating a minimal required setup of the test bench. Not > by replacing standard Devstack scripts, but mainly through tooling > for quick rebuilding data like flavours, infrastructure state, and > other factors relevant to the simulation. > > > Outside of the scope: > > * Running continuous analysis on the production environment, even if > some plug-ins could be extracted for this purpose. > * Retaining information about users and projects when replaying orders. > * (Probably / low priority) replaying actions other than VM > creation/deletion as they form a minority of operations and ignoring > them should not have a distinct effect on the comparison experiments. > > > Current state: > > Implemented: > > * Recreating flavours from JSON file exported via OpenStack CLI. > * Replaying a list of orders in the form of (creation_date, > termination_date, resource_id (optional), flavor_id) with basic > flavour properties like VCPU, RAM, and DISK GB. The orders are > replayed consecutively. > * Plug-in success-rater mechanism which runs rater classes (returning > quantified success measure) after each VM add/delete action, retains > their intermediate history and "total success" - how it is defined > is implementation dependent. First classes interacting with > Placement like: "How many VMs of flavours x (with basic parameters > for now) can fit in the cluster?" or "How many hosts are empty?". > > > Missing: > > * Recreating hosts, note the fake-virt remark from "Risks and > Challenges". > * Tools facilitating Scheduler configuration. > * Creating VMs with more parameters like VGPU, traits, and aggregates. > * (Lower priority) saving the intermediate state of the cluster during > simulation i.e. allocations to analyse it without rerunning the > experiment. Currently, only the quantified meters are saved. > * Gently failing and saving all information in case of resource > depletion: close to completion, handling one exception type in upper > layers is needed. > * More success meters. > > > Risks and Challenges: > > * Currently, the tool replays actions one by one, it waits for each > creation and deletion to be complete before running success raters > and taking another order. Thus, the order of actions is important, > but not their absolute time and temporal density. This might skip > some side-effects of a realistic execution. > * Similarly, to the above, fake-virt provides simple classes that will > not reproduce some behaviours of real-world hypervisors. An explicit > Scheduler avoids hosts that had recently failed to allocate a VM, > but most likely fake-virt will not mock such behaviour. > * Fake-virt should reproduce a real diverse infrastructure instead of > x copies of the same flavour. This might be the only, but very > important change to the OpenStack codebase. If successful, it could > benefit other projects and tests as well. > > > Even though the list of missing features is seemingly larger, the most > important parts of the program are already there, so we hope to finish > the MVP development in a relatively short amount of time. We are going > to publish it as FOSS in either case, but as mentioned your observations > would be very much welcome at this stage. I am also open to answering > more questions about the project. > > > Kind regards > > Dominik Danelski > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdhasman at redhat.com Mon Jul 3 20:35:22 2023 From: rdhasman at redhat.com (Rajat Dhasmana) Date: Tue, 4 Jul 2023 02:05:22 +0530 Subject: [cinder][drivers] new driver merge freeze question In-Reply-To: <997373634.584897.1688402768093.JavaMail.xmail@bj-wm-cp-12> References: <997373634.584897.1688402768093.JavaMail.xmail@bj-wm-cp-12> Message-ID: Hi, Thank you for proposing the driver. I've added it to the list of current drivers[1] for 2023.2 Bobcat cycle. Currently I can see that the patch is failing on multiple jobs i.e. pep8, unit, docs, grenade etc which needs to be fixed. Also the third party CI is failing as you have already mentioned. We can push for reviews but ideally the gate should be passing and the CI should be ready which is a bare minimum for the driver to be reviewed. I will mention this driver in our weekly cinder meeting on Wednesday, 1400 UTC on #openstack-meeting-alt channel, feel free to attend the meeting on IRC. Regarding the deadline, it can be extended given the driver is almost ready and might only require minor/cosmetic changes. Also the extension provided is also for a limited time frame (had been 1 week in the past) so the driver should be ready in 1 extra week given the exception is provided. Currently I would recommend fixing the gate jobs and getting the third party CI working so it is easier to push for reviews. [1] https://etherpad.opendev.org/p/cinder-2023-2-bobcat-drivers Thanks Rajat Dhasmana On Mon, Jul 3, 2023 at 10:22?PM ??? wrote: > Hello. I am working on the TOYOU NetStor cinder volume driver, > https://review.opendev.org/c/openstack/cinder/+/886942. I am working > hard to address some test cases that failed with our third-party-CI. In > the meantime, I would appreciate reviews so that I will be in a strong > position to ask for a new driver merge deadline exception if I'm only > "mostly done" when we hit the deadline this Friday, 7 July. My question is > whether it will be possible to ask for a new driver merge deadline > exception this cycle, or if the deadline is immovable? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ueha.ayumu at fujitsu.com Tue Jul 4 07:53:34 2023 From: ueha.ayumu at fujitsu.com (Ayumu Ueha (Fujitsu)) Date: Tue, 4 Jul 2023 07:53:34 +0000 Subject: [kuryr][tacker] fails to bump the k8s version to v1.26 Message-ID: Hi kuryr-kubetenetes team, Tacker uses the kuryr-kubernetes as the setup for the kubernetes Environment. In the Bobcat version, we will bump the supported version of kubenetes to 1.26.6, and when I tried to bump the version in devstack's local.conf by the patch [1], the following error occurred. (please refer the details to Zuul log [2]) ============================== [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [kubelet-check] Initial timeout of 40s passed. Unfortunately, an error has occurred: timed out waiting for the condition This error is likely caused by: - The kubelet is not running - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled) If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands: - 'systemctl status kubelet' - 'journalctl -xeu kubelet' Additionally, a control plane component may have crashed or exited when started by the container runtime. To troubleshoot, list all containers using your preferred container runtimes CLI. Here is one example how you may list all running Kubernetes containers by using crictl: - 'crictl --runtime-endpoint unix:///var/run/crio/crio.sock ps -a | grep kube | grep -v pause' Once you have found the failing container, you can inspect its logs with: - 'crictl --runtime-endpoint unix:///var/run/crio/crio.sock logs CONTAINERID' error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster ============================== I know it is not yet supporting version 1.26 of the K8S at the kuryr-kubernetes as for now, but do you know any way to avoid the above error? I suspect this is due to a change from version 1.25, but I'm not sure which one is affecting... Also, when will you support the K8S v 1.26 at the kuryr-kubernetes? Will the Bobcat release support it? Please kindly let me know if you know anything. Thank you. [1] https://review.opendev.org/c/openstack/tacker/+/886935 Change the parameters: KURYR_KUBERNETES_VERSION: 1.25.6 to 1.26.6 CRIO_VERSION: 1.25 to 1.26 [2] https://zuul.opendev.org/t/openstack/build/1a4061da74e640368da133ba219b54d9/log/controller-k8s/logs/devstacklog.txt#9761-9816 Best Regards, Ueha -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Tue Jul 4 11:08:39 2023 From: smooney at redhat.com (smooney at redhat.com) Date: Tue, 04 Jul 2023 12:08:39 +0100 Subject: [nova] Scheduler Optimiser In-Reply-To: References: <09981137-ce97-df6b-55f9-038df7f10498@cloudferro.com> Message-ID: <4ca79d6072090cf0aec8ebc9145e5875a0fae699.camel@redhat.com> not to bias either approach but just to ensure you both understand the current recommendation. when configuring the scheduler there are some rules fo tumb that shoudl be followed. first if it can be done by both a scheduler filter and placement then placement should be faster. when specifing the filters operators a encouraged to order them following a psudo id3 algorithm approch in other words always put the filters that elimiate the most host first i.e aggreate* filters then simple host specific filters (num instances) then complex host spefific filters liek the numa toplogy filter. finally if a filter is not useful for your cloud(i.e. you have not pci devices for passthough) you can remove it from the enabled filters list for the weighers all weigher are enabled by default and all enabled filters run on each host that passed the filtes. so there is no ordering to consider with weighers but if you dont need them for your cloud just like the filters you can remove them form the enabled set. for placement there are only 2 things you can really tweak, the max number of results and randomisation of allcoation candiates. https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.max_placement_results https://docs.openstack.org/placement/latest/configuration/config.html#placement.randomize_allocation_candidates i know cern set the max_placement_results very low. i.e <20 the default is 1000 but most dont modify this. i generally think settign randomize_allocation_candidates is good espcially if you reduce the max_placement_results as noted there are quite a few flags to delegate filters to placemetn lik https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.query_placement_for_routed_network_aggregates https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.limit_tenants_to_placement_aggregate https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.placement_aggregate_required_for_tenants https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.query_placement_for_availability_zone https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.query_placement_for_image_type_support https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.image_metadata_prefilter these often replace filteres like the az filter with a more effeicnt placement query so they should generally be prefer however they dotn always have exactly the same behavior so sometimes you cant just swap form one to anohter if you rely on the semantics of the filter that are changed. there are some options like https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.host_subset_size https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.shuffle_best_same_weighed_hosts that allow you to add a little bit of additional randomness to the host selection which may help with spreading/packing or overall better utilisation of host but it should not really affect performance of the schduelr. i generally recommend setting shuffle_best_same_weighed_hosts=true but leaving host_subset_size at its default of 1 that way you only get randomness if there are multiple equally good options. by default the scheduler should be determinist today if you do not adjust either host_subset_size or shuffle_best_same_weighed_hosts i don't know if that will help either of your research but those are the current best pratices for cofniguring the scheduler. regards sean. On Mon, 2023-07-03 at 14:08 -0600, Alvaro Soto wrote: > Nice idea, but my main question is, how do you plan to beat the ones > implemented currently? > > I'm working a little researching a little with some techniques to try to > beat the random resource allocation schedulers. > > Can you share more about your research and/or implementation idea? > > Cheers. > > --- > Alvaro Soto. > > Note: My work hours may not be your work hours. Please do not feel the need > to respond during a time that is not convenient for you. > ---------------------------------------------------------- > Great people talk about ideas, > ordinary people talk about things, > small people talk... about other people. > > On Mon, Jul 3, 2023, 2:00 PM Dominik Danelski > wrote: > > > > > Hello, > > > > > > I would like to introduce you to the tool developed under the working > > title "Scheduler Optimiser". It is meant to test the effectiveness of > > different Scheduler configurations, both weights and filters on a given > > list of VM orders and in a semi-realistic infrastructure. > > > > My company - CloudFerro - has been preparing in-house for the last few > > months and foresees publishing the project as FOSS once it reaches the > > MVP stage. To make the final result more useful to the community and > > speed up the development (and release), I humbly ask for your expertise: > > Are you aware of previous similar efforts? Do you notice some flaws in > > the current approach? What, in your opinion, are more important aspects > > of the infrastructure behaviour, and what can be relatively safely > > ignored in terms of the effect on Scheduler results/allocation? > > > > > > Project objectives: > > > > ? * Use Devstack (or another OpenStack deployer) with a real Scheduler > > ??? to replay a list of compute VM orders, either real from one's > > ??? infrastructure or artificially created. > > ? * Assess the effectiveness of the scheduling in various terms like: > > ??? "How many machines of a given type can still be allocated at the > > ??? moment?" using plug-in "success meters". In a strict sense, the > > ??? project does not simulate THE Scheduler but interacts with it. > > ? * Use fake-virt to emulate huge architectures on a relatively tiny > > ??? test bench. > > ? * Have as little as possible, and ideally no changes to the Devstack's > > ??? code that could not be included in the upstream repository. The > > ??? usage should be as simple as: 1. Install Devstack. 2. Configure > > ??? Devstack's cluster with its infrastructure information like flavours > > ??? and hosts. 3. Configure Scheduler for a new test case. 4. Replay VM > > ??? orders. 5. Repeat steps 3 and 4 to find better Scheduler settings. > > ? * Facilitate creating a minimal required setup of the test bench. Not > > ??? by replacing standard Devstack scripts, but mainly through tooling > > ??? for quick rebuilding data like flavours, infrastructure state, and > > ??? other factors relevant to the simulation. > > > > > > Outside of the scope: > > > > ? * Running continuous analysis on the production environment, even if > > ??? some plug-ins could be extracted for this purpose. > > ? * Retaining information about users and projects when replaying orders. > > ? * (Probably / low priority) replaying actions other than VM > > ??? creation/deletion as they form a minority of operations and ignoring > > ??? them should not have a distinct effect on the comparison experiments. > > > > > > Current state: > > > > ??? Implemented: > > > > ? * Recreating flavours from JSON file exported via OpenStack CLI. > > ? * Replaying a list of orders in the form of (creation_date, > > ??? termination_date, resource_id (optional), flavor_id) with basic > > ??? flavour properties like VCPU, RAM, and DISK GB. The orders are > > ??? replayed consecutively. > > ? * Plug-in success-rater mechanism which runs rater classes (returning > > ??? quantified success measure) after each VM add/delete action, retains > > ??? their intermediate history and "total success" - how it is defined > > ??? is implementation dependent. First classes interacting with > > ??? Placement like: "How many VMs of flavours x (with basic parameters > > ??? for now) can fit in the cluster?" or "How many hosts are empty?". > > > > > > Missing: > > > > ? * Recreating hosts, note the fake-virt remark from "Risks and > > Challenges". > > ? * Tools facilitating Scheduler configuration. > > ? * Creating VMs with more parameters like VGPU, traits, and aggregates. > > ? * (Lower priority) saving the intermediate state of the cluster during > > ??? simulation i.e. allocations to analyse it without rerunning the > > ??? experiment. Currently, only the quantified meters are saved. > > ? * Gently failing and saving all information in case of resource > > ??? depletion: close to completion, handling one exception type in upper > > ??? layers is needed. > > ? * More success meters. > > > > > > Risks and Challenges: > > > > ? * Currently, the tool replays actions one by one, it waits for each > > ??? creation and deletion to be complete before running success raters > > ??? and taking another order. Thus, the order of actions is important, > > ??? but not their absolute time and temporal density. This might skip > > ??? some side-effects of a realistic execution. > > ? * Similarly, to the above, fake-virt provides simple classes that will > > ??? not reproduce some behaviours of real-world hypervisors. An explicit > > ??? Scheduler avoids hosts that had recently failed to allocate a VM, > > ??? but most likely fake-virt will not mock such behaviour. > > ? * Fake-virt should reproduce a real diverse infrastructure instead of > > ??? x copies of the same flavour. This might be the only, but very > > ??? important change to the OpenStack codebase. If successful, it could > > ??? benefit other projects and tests as well. > > > > > > Even though the list of missing features is seemingly larger, the most > > important parts of the program are already there, so we hope to finish > > the MVP development in a relatively short amount of time. We are going > > to publish it as FOSS in either case, but as mentioned your observations > > would be very much welcome at this stage. I am also open to answering > > more questions about the project. > > > > > > Kind regards > > > > Dominik Danelski > > > > From dwilde at redhat.com Tue Jul 4 13:23:06 2023 From: dwilde at redhat.com (Dave Wilde) Date: Tue, 4 Jul 2023 08:23:06 -0500 Subject: [keystone] Weekly Meeting Cancelled for 04-Jul-2023 In-Reply-To: <1e51854a-2016-4d0c-8176-6201d142fe82@Spark> References: <1e51854a-2016-4d0c-8176-6201d142fe82@Spark> Message-ID: <64719fd8-1b55-4c7d-9966-a32f71dae3d3@Spark> Sorry for the late notice but I?m cancelling the weekly meeting for today as it?s a public holiday here in the U.S. Please reach out if you need anything. Thanks, /Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralonsoh at redhat.com Tue Jul 4 15:30:57 2023 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Tue, 4 Jul 2023 17:30:57 +0200 Subject: [neutron] os-ken migration to Launchpad Message-ID: Hello Neutrinos: It was decided today in the Neutron meeting that we are moving from storyboard [1] to the Neutron launchpad site. If a new feature or bug is needed in the "os-ken" project, a new bug in [3] should be reported with the tag *[os-ken]* in the title. Regards. [1]https://storyboard.openstack.org/#!/project/1079 [2]https://launchpad.net/neutron [3]https://bugs.launchpad.net/neutron -------------- next part -------------- An HTML attachment was scrubbed... URL: From jamesleong123098 at gmail.com Tue Jul 4 21:42:01 2023 From: jamesleong123098 at gmail.com (James Leong) Date: Tue, 4 Jul 2023 16:42:01 -0500 Subject: [horizon][keystone][kolla-ansible] Authentication failure Message-ID: Hi All, I am using the yoga version of OpenStack with the deployment tool of kolla-ansible. I am currently facing the below error when logging in via federated login using Globus Auth. " Login failed: An error occurred authenticating. Please try again later." When attempting to login, we are able to redirect the page to globus and process the request. However, when it comes back to the horizon login page, I am getting an authentication error. I have set up my keystone identity provider in globals.yml as below. keystone_identity_providers: - name: "globus" openstack_domain: "Default" protocol: "openid" identifier: "https://auth.globus.org" public_name: "Authenticate via Globus Auth" attribute_mapping: "globus" metadata_folder: "/home/user/osmetadata" keystone_federation_oidc_jwks_uri: "https://auth.globus.org/jwk.json" keystone_identity_mappings: - name: "globus" file: "/home/user/globus.json" Apart from specifying the identity provider and mapping, below are the other configurations we have set up when deploying. kolla_enable_tls_internal: "no" kolla_enable_tls_external: "yes" kolla_enable_tls_backend: "no" kolla_verify_tls_backend: "yes" Thanks for the help, James -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Tue Jul 4 23:08:24 2023 From: fungi at yuggoth.org (Jeremy Stanley) Date: Tue, 4 Jul 2023 23:08:24 +0000 Subject: [neutron] os-ken migration to Launchpad In-Reply-To: References: Message-ID: <20230704230824.4xv7ikoi2n5uvzag@yuggoth.org> On 2023-07-04 17:30:57 +0200 (+0200), Rodolfo Alonso Hernandez wrote: > It was decided today in the Neutron meeting that we are moving from > storyboard [1] to the Neutron launchpad site. [...] When you're ready, propose a change to openstack/project-config editing the gerrit/projects.yaml file to remove the use-storyboard flag from the openstack/os-ken project. Once that merges I'll update the project description in StoryBoard to link to the neutron project in Launchpad and flag the project inactive so that it shouldn't continue to show up in autocompletion results for new tasks. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From nguyenhuukhoinw at gmail.com Wed Jul 5 02:07:34 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Wed, 5 Jul 2023 09:07:34 +0700 Subject: [horizon][keystone][kolla-ansible] Authentication failure In-Reply-To: References: Message-ID: Hello, You need enable keystone debug log to find exact what is wrong, Nguyen Huu Khoi On Wed, Jul 5, 2023 at 4:48?AM James Leong wrote: > Hi All, > > I am using the yoga version of OpenStack with the deployment tool of > kolla-ansible. I am currently facing the below error when logging in via > federated login using Globus Auth. > > " Login failed: An error occurred authenticating. Please try again later." > > When attempting to login, we are able to redirect the page to globus and > process the request. However, when it comes back to the horizon login page, > I am getting an authentication error. I have set up my keystone identity > provider in globals.yml as below. > > keystone_identity_providers: > - name: "globus" > openstack_domain: "Default" > protocol: "openid" > identifier: "https://auth.globus.org" > public_name: "Authenticate via Globus Auth" > attribute_mapping: "globus" > metadata_folder: "/home/user/osmetadata" > keystone_federation_oidc_jwks_uri: "https://auth.globus.org/jwk.json" > > keystone_identity_mappings: > - name: "globus" > file: "/home/user/globus.json" > > Apart from specifying the identity provider and mapping, below are the > other configurations we have set up when deploying. > > kolla_enable_tls_internal: "no" > kolla_enable_tls_external: "yes" > kolla_enable_tls_backend: "no" > kolla_verify_tls_backend: "yes" > > Thanks for the help, > James > -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Wed Jul 5 06:55:25 2023 From: skaplons at redhat.com (Slawek Kaplonski) Date: Wed, 05 Jul 2023 08:55:25 +0200 Subject: [neutron] os-ken migration to Launchpad In-Reply-To: References: Message-ID: <5414604.KvVS0tvQ03@p1> Hi, Dnia wtorek, 4 lipca 2023 17:30:57 CEST Rodolfo Alonso Hernandez pisze: > Hello Neutrinos: > > It was decided today in the Neutron meeting that we are moving from > storyboard [1] to the Neutron launchpad site. If a new feature or bug is > needed in the "os-ken" project, a new bug in [3] should be reported with > the tag *[os-ken]* in the title. Additionally to the "tag" in the title, I think it would be also good to have such tag added in the "tags" section. It will allow easier filtering for bugs related to os-ken. > > Regards. > > [1]https://storyboard.openstack.org/#!/project/1079 > [2]https://launchpad.net/neutron > [3]https://bugs.launchpad.net/neutron > -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From skaplons at redhat.com Wed Jul 5 07:36:46 2023 From: skaplons at redhat.com (Slawek Kaplonski) Date: Wed, 05 Jul 2023 09:36:46 +0200 Subject: [neutron] CI meeting on 11.07.2023 cancelled Message-ID: <9093109.ugfZM50Qif@p1> Hi, I will be on PTO next week as well as some other members of the Neutron team. So we decided yesterday to cancel next week's CI meeting. See You on meeting on Tuesday 18th of July. -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From rdhasman at redhat.com Wed Jul 5 13:17:18 2023 From: rdhasman at redhat.com (Rajat Dhasmana) Date: Wed, 5 Jul 2023 18:47:18 +0530 Subject: [cinder][all] Cinder to EOL all EM branches Message-ID: Hi All, This is a followup to my last email[1] where a lot of discussion happened around the nature of EM branches and was/is it a successful model or not. This was also discussed during the Vancouver summit + PTG event and also recently in the TC meeting concluding that we will start a discussion again on ML regarding the EM model. However, my original intent of the mail was not to start a wider discussion about the EM model (which is a good discussion but not my intent) but a way to take feedback on Cinder team's decision to EOL the current EM branches and if it affects any operators, vendors etc. This mail is another reminder and a way to acquire more feedback on the above decision so the Cinder team can plan the process accordingly. We already have patches proposed to EOL all EM branches[2]. [1] https://lists.openstack.org/pipermail/openstack-discuss/2023-June/033980.html [2] https://review.opendev.org/q/topic:cinder-eol-june2023 Thanks Rajat Dhasmana -------------- next part -------------- An HTML attachment was scrubbed... URL: From anyrude10 at gmail.com Tue Jul 4 09:29:49 2023 From: anyrude10 at gmail.com (Anirudh Gupta) Date: Tue, 4 Jul 2023 14:59:49 +0530 Subject: [TripleO] - Issue Faced in Building Overcloud Images Message-ID: Hi Team, I am trying to build openstack wallaby images following the below link: https://docs.openstack.org/project-deploy-guide/tripleo-docs/wallaby/deployment/install_overcloud.html#deploy-the-overcloud But following the below issue: 2023-07-04 05:20:25.560 | --> Starting dependency resolution 2023-07-04 05:20:25.632 | --> Finished dependency resolution 2023-07-04 05:20:25.636 | Error: 2023-07-04 05:20:25.636 | Problem: cannot install the best candidate for the job *2023-07-04 05:20:25.636 | - nothing provides pacemaker-libs(x86-64) = 2.1.6-2.el8 needed by pacemaker-remote-2.1.6-2.el8.x86_642023-07-04 05:20:25.636 | (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate*2023-07-04 05:20:26.084 | Unmount /home/stack/image/dib_build.9WnO74VD/mnt/tmp/yum 2023-07-04 05:20:26.111 | Unmount /home/stack/image/dib_build.9WnO74VD/mnt/tmp/in_target.d 2023-07-04 05:20:26.136 | Unmount /home/stack/image/dib_build.9WnO74VD/mnt/sys 2023-07-04 05:20:26.161 | Unmount /home/stack/image/dib_build.9WnO74VD/mnt/proc 2023-07-04 05:20:26.191 | Unmount /home/stack/image/dib_build.9WnO74VD/mnt/dev/pts 2023-07-04 05:20:26.221 | Unmount /home/stack/image/dib_build.9WnO74VD/mnt/dev 2023-07-04 05:20:26.843 | INFO diskimage_builder.block_device.blockdevice [-] State already cleaned - no way to do anything here Exception occured while running the command Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run return super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_image.py", line 120, in take_action manager.build() File "/usr/lib/python3.6/site-packages/tripleo_common/image/build.py", line 85, in build elements, options, packages, extra_options) File "/usr/lib/python3.6/site-packages/tripleo_common/image/image_builder.py", line 140, in build_image raise subprocess.CalledProcessError(process.returncode, cmd) subprocess.CalledProcessError: Command '['disk-image-create', '-a', 'amd64', '-o', './overcloud-full', '-t', 'qcow2', '-p', 'pythoollector,sos,device-mapper-multipath,openstack-heat-agents,os-net-config,jq,python3-dbus', '--min-tmpfs=7', '--mkfs-options', '-s baremetal', 'openvswitch', 'overcloud-agent', 'overcloud-base', 'overcloud-controller', 'overcloud-compute', 'overcloud-ceph-stora'stable-interface-names', 'grub2', 'element-manifest', 'dynamic-login', 'iptables', 'enable-packages-install', 'override-pip-and-venerate', 'remove-machine-id', 'remove-resolvconf', 'openssh', 'disable-nouveau', 'selinux-permissive', 'interface-names']' return 1. I can see the same issue reported in bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=2017520 https://www.mail-archive.com/users at lists.rdoproject.org/msg01009.html Steps to Reproduce: 1. export OS_YAML="/usr/share/openstack-tripleo-common/image-yaml/overcloud-images-centos8.yaml" 2. export DIB_YUM_REPO_CONF="/etc/yum.repos.d/delorean* /etc/yum.repos.d/tripleo-centos-*" 3. export STABLE_RELEASE="wallaby" 4. openstack overcloud image build --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-images-python3.yaml --config-file $OS_YAML The same issue is being observed in "XENA" release as well. Can someone please help Regards Anirudh Gupta -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Wed Jul 5 18:14:21 2023 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Wed, 05 Jul 2023 11:14:21 -0700 Subject: [cinder][all] Cinder to EOL all EM branches In-Reply-To: References: Message-ID: <18927432cab.b35b5b43911163.6905573119740994317@ghanshyammann.com> ---- On Wed, 05 Jul 2023 06:17:18 -0700 Rajat Dhasmana wrote --- > Hi All, > > This is a followup to my last email[1] where a lot of discussion happenedaround the nature of EM branches and was/is it a successful model or not. > This was also discussed during the Vancouver summit?+ PTG event and alsorecently in the TC meeting concluding that we will start a discussion again onML regarding the EM model. > However, my original intent of the mail was not to start a wider discussion aboutthe EM model (which is a good discussion but not my intent) but a way to takefeedback on Cinder team's decision to EOL the current EM branches and if itaffects any operators, vendors etc. > This mail is another reminder and a way to acquire more feedback on the abovedecision so the Cinder team can plan the process accordingly. We already havepatches proposed to EOL all EM branches[2]. Thanks Rajat for pushing the EM discussion. But the consensus on EM future as community and Cinder moving all EM to EOL are closely related. Even though we were not able to get any solution to existing EM issue and TC is working on that and hopefully we will come to the agreement soon (even it might be killing the EM process). If Cinder make all their EM to EOL, it will end up other integrated projects to do the same because of how complex the testing will be and if any CVE need fix in project A also need fix in Cinder repo. IMO, we should wait for overall decision on EM so that we can keep consistency to our operator. Whatever the agreement will be, it will be same for all projects and give a clear consistency to Operators. -gmann > [1]?https://lists.openstack.org/pipermail/openstack-discuss/2023-June/033980.html[2]?https://review.opendev.org/q/topic:cinder-eol-june2023 > ThanksRajat Dhasmana From fungi at yuggoth.org Wed Jul 5 20:18:55 2023 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 5 Jul 2023 20:18:55 +0000 Subject: [cinder][all][tc] Cinder to EOL all EM branches In-Reply-To: <18927432cab.b35b5b43911163.6905573119740994317@ghanshyammann.com> References: <18927432cab.b35b5b43911163.6905573119740994317@ghanshyammann.com> Message-ID: <20230705201854.mcr67unio2s3uzly@yuggoth.org> On 2023-07-05 11:14:21 -0700 (-0700), Ghanshyam Mann wrote: [...] > Even though we were not able to get any solution to existing EM > issue and TC is working on that and hopefully we will come to the > agreement soon (even it might be killing the EM process). [...] > IMO, we should wait for overall decision on EM so that we can keep > consistency to our operator. Whatever the agreement will be, it > will be same for all projects and give a clear consistency to > Operators. [...] While I appreciate that this decision on the part of the Cinder contributors puts the overall OpenStack community in a bit of a quandary, it would be good to let them know how long they're being asked to wait for official feedback from the TC, so that they can proceed with their original plan (backed by existing stable maintenance policy) if the debate continues to drag out. Can the TC please commit to reaching a conclusion by some specific deadline? -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From swogatpradhan22 at gmail.com Thu Jul 6 06:30:39 2023 From: swogatpradhan22 at gmail.com (Swogat Pradhan) Date: Thu, 6 Jul 2023 12:00:39 +0530 Subject: Node provision | stuck in black screen after kernel selection window | Wallaby Message-ID: Hi, I am trying to provision a new compute node in my central site using the director node that I have already used to deploy multiple sites compute, ceph along with central compute, ceph and controller nodes. For adding the new node, I introspected the node, setup capabilities then ran the node provision command. But it is failing at growfs step as the node is not reachable after booting. ERROR: [WARNING]: Unhandled error in Python interpreter discovery for host overcloud- novacompute-4: Failed to connect to the host via ssh: ssh: connect to host 172.25.201.93 port 22: No route to host 2023-07-06 01:28:38.370427 | 48d539a1-1679-0502-e4b9-000000000017 | UNREACHABLE | Find the growvols utility | overcloud-novacompute-4 After logging into the console of the bare metal server, I found that the OS is not coming UP. I see the bootloader screen, it selects the centos 1st kernel and then goes into black screen. I don't know why it is happening or how to fix it. I thought maybe there was some hardware issue, but i tried installing centos manually and i am getting the console output, without any issues. But for some reason the os is not booting up after the bootloader kernel window and the console is getting stuck in black screen. Can someone please help? With regards, Swogat Pradhan -------------- next part -------------- An HTML attachment was scrubbed... URL: From zigo at debian.org Thu Jul 6 08:48:43 2023 From: zigo at debian.org (Thomas Goirand) Date: Thu, 6 Jul 2023 10:48:43 +0200 Subject: [horizon] The state of the gate for the last 1 month at least Message-ID: <632bba9f-3743-bc3d-33af-e66f21ec134b@debian.org> Hi team! It's been more than a month that it's impossible to merge anything in Horizon, due to the gate being broken because of one single functional test being broken: FAILED openstack_dashboard/test/integration_tests/tests/test_floatingips.py::TestFloatingipAssociateDisassociate::test_floatingip_associate_disassociate I do understand that you guys want to fix this, and I do understand how hard this puzzle is, with selenium, the chrome driver, jquery and all. It's far from simple, and I very much welcome the team effort. However, could we just disable that unit test until this is fixed, so that simple patches like this one can be merged? https://review.opendev.org/c/openstack/horizon/+/885577 Or more complex one from my colleague? https://review.opendev.org/c/openstack/horizon/+/885570 More than a month without merging any patch is a long time... Cheers, Thomas Goirand (zigo) From zigo at debian.org Thu Jul 6 09:26:09 2023 From: zigo at debian.org (Thomas Goirand) Date: Thu, 6 Jul 2023 11:26:09 +0200 Subject: [designate] About delegating PTR using shared zones on a shared internal network Message-ID: <5059613b-2c76-c5a5-1281-42970a190def@debian.org> Hi there! I have just watched Michael Johnson's video of the presentation at the last Vancouver summit: https://www.youtube.com/watch?v=zdhvNezU1w4 This was very interesting, and thanks Michael, for that awesome feature. In our use case, we created an OpenStack internal network, that is shared for all of our customers, so that they can allocate public IPs assigned directly to their instances, without the need to create a router and use NAT. For the moment (since we don't have the shared zones feature yet), we manage the PTR record through support tickets. We'd like to move forward, and delegate the PTR handling to our customer, whenever they allocate a port on that specific shared network. How can this be done? Should we create a new sink of our own, so that we would automate the the shared zone creation, so that it would delegate the in-addr.arpa. range, when a port on that network is created, and the same way, remove the delegation (and records) whenever the port is deleted? Would this be something similar, probably, we the sink we implemented to automatically delete PTR records in floating IPs, available here [1]? [1] https://salsa.debian.org/openstack-team/services/designate/-/blob/debian/antelope/debian/patches/add-new-floatingip-handler.patch I'd love to have some pointers / examples, so we could implement this directly in Designate, so that the feature could be shared in the OpenStack community. For sure, we aren't the only public cloud provider with the use case... Cheers, Thomas Goirand (zigo) From manchandavishal143 at gmail.com Thu Jul 6 09:33:40 2023 From: manchandavishal143 at gmail.com (vishal manchanda) Date: Thu, 6 Jul 2023 15:03:40 +0530 Subject: [horizon] The state of the gate for the last 1 month at least In-Reply-To: <632bba9f-3743-bc3d-33af-e66f21ec134b@debian.org> References: <632bba9f-3743-bc3d-33af-e66f21ec134b@debian.org> Message-ID: Hello Thomas, FYI gate is failing after cirros image version changed in devstack that's the reason "test_floatingip_associate_disassociate" is failing because the cirros image that the test is using is not available now. We fixed that issue in Horizon by updating cirros image version and then the devstack cirros image version again got updated. we updated again in the horizon. Even if we disable/skip the test "test_floatingip_associate_disassociate" more integration tests start failing which we trying to fix. We are trying hard to fix the gate asap, but the issue is we still didn't know exactly due to which patch we merged it's started failing. Currently, horizon is using the Firefox browser to run the integration jobs and we can't see the browser error log for that. So we are trying to run that job with Chrome so we can see the error log and fix it. We think these issues are happening after we updated the JQuery-migrate version for Horizon [1] but the latest JQuery-migrate version is not compatible with the older version of jQuery, maybe that's why it is failing. I purpose an initial patch in openstack/requirements[2], now waiting for it to get merged, and then will move to the next step. The simpler thing we can do to fix the gate, for now, is to move back to the older JQuery-migrate version but we are planning to update these versions for so long and we already know we are going to hit all these issues because these migrations are not so simple. Thanks & regards, Vishal Manchanda [1] https://review.opendev.org/c/openstack/requirements/+/883402 [2] https://review.opendev.org/c/openstack/requirements/+/887720 On Thu, Jul 6, 2023 at 2:20?PM Thomas Goirand wrote: > Hi team! > > It's been more than a month that it's impossible to merge anything in > Horizon, due to the gate being broken because of one single functional > test being broken: > > FAILED > > openstack_dashboard/test/integration_tests/tests/test_floatingips.py::TestFloatingipAssociateDisassociate::test_floatingip_associate_disassociate > > I do understand that you guys want to fix this, and I do understand how > hard this puzzle is, with selenium, the chrome driver, jquery and all. > It's far from simple, and I very much welcome the team effort. > > However, could we just disable that unit test until this is fixed, so > that simple patches like this one can be merged? > > https://review.opendev.org/c/openstack/horizon/+/885577 > > Or more complex one from my colleague? > > https://review.opendev.org/c/openstack/horizon/+/885570 > > More than a month without merging any patch is a long time... > > Cheers, > > Thomas Goirand (zigo) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lokendrarathour at gmail.com Thu Jul 6 10:13:27 2023 From: lokendrarathour at gmail.com (Lokendra Rathour) Date: Thu, 6 Jul 2023 15:43:27 +0530 Subject: TripleO | OS-net config logs | openstack overcloud node provision Message-ID: Hi Team, Scenario: We were performing overcloud deployment. During this, we were continuously facing the following errors on two of the nodes out of the four nodes: PLAY [Overcloud Node Grow Volumes] ********************************************* 2023-06-02 19:07:33.483504 | 52540049-4965-0177-e681-00000000000a | TASK | Wait for provisioned nodes to boot [WARNING]: Unhandled error in Python interpreter discovery for host overcloud- controller-0: Failed to connect to the host via ssh: ssh: connect to host 30.30.30.219 port 22: No route to host [WARNING]: Unhandled error in Python interpreter discovery for host overcloud- controller-0: Failed to connect to the host via ssh: ssh: connect to host 30.30.30.219 port 22: Connection refused We faced this error when we were performing overcloud node provisioning. To resolve this issue, we had to update the image and create a user to access the nodes. We were able to access the nodes via this user through the BMC console. This implied that OS was getting correctly installed on the node. Once we accessed the logs on the failed node we came across the following logs: 2023-06-13 09:02:16.120 INFO os_net_config.utils._ordered_nics Finding active nics 2023-06-13 09:02:16.120 INFO os_net_config.utils._ordered_nics eno1 is an embedded active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics ens2f0 is an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics eno4 is an embedded active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics eno2 is an embedded active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics ens1f0 is not an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics lo is not an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics ens2f1 is an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics eno3 is an embedded active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics ens1f1 is an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics No DPDK mapping available in path (/var/lib/os-net-config/dpdk_mapping.yaml) 2023-06-13 09:02:16.122 INFO os_net_config.utils._ordered_nics Active nics are ['eno1', 'eno2', 'eno3', 'eno4', 'ens1f1', 'ens2f0', 'ens2f1'] 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic2 mapped to: eno2 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic6 mapped to: ens2f0 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic5 mapped to: ens1f1 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic3 mapped to: eno3 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic7 mapped to: ens2f1 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic1 mapped to: eno1 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic4 mapped to: eno4 As we can see in the above logs, only 7 nics are active. But as per our bond config file, we required 8 interfaces: a. nic-1 for management IP b. nic-3 and nic-4 for the br-tenant-2 network. c. nic-5 and nic-6 for the br-tenant network. d. nic-7 and nic 8 for the br-ex network. The wire was faulty in this case and hence only 7 interfaces were available Query: The logs which are visible while we run "openstack overcloud node provision" are not very informative. Is there any way where we could get these logs on the undercloud node itself? This could save some time for future references -- ~ Lokendra skype: lokendrarathour -------------- next part -------------- An HTML attachment was scrubbed... URL: From knikolla at bu.edu Thu Jul 6 17:05:29 2023 From: knikolla at bu.edu (Nikolla, Kristi) Date: Thu, 6 Jul 2023 17:05:29 +0000 Subject: [cinder][all][tc] Cinder to EOL all EM branches In-Reply-To: <20230705201854.mcr67unio2s3uzly@yuggoth.org> References: <18927432cab.b35b5b43911163.6905573119740994317@ghanshyammann.com> <20230705201854.mcr67unio2s3uzly@yuggoth.org> Message-ID: <8F41F5B6-EFA9-4EC7-9109-7A276801D566@bu.edu> Thanks for making that point Jeremy. As a courtesy to allow the TC to provide a resolution with coordinated guidelines for EOL-ing branches across projects, we please ask that you wait until the end of July at the latest. If no consensus has been reached on approving a new EM policy has been reached until the end of July, it is within your power, as the policy is written today, to EOL all branches based on an internal team decision. As every TC policy resolution requires staying open for at least 1 week, and taking into account 1-2 weeks of deliberation, that is the earliest that we can reasonably approve a new policy. Thank you, Kristi Nikolla > On Jul 5, 2023, at 4:18 PM, Jeremy Stanley wrote: > > On 2023-07-05 11:14:21 -0700 (-0700), Ghanshyam Mann wrote: > [...] >> Even though we were not able to get any solution to existing EM >> issue and TC is working on that and hopefully we will come to the >> agreement soon (even it might be killing the EM process). > [...] >> IMO, we should wait for overall decision on EM so that we can keep >> consistency to our operator. Whatever the agreement will be, it >> will be same for all projects and give a clear consistency to >> Operators. > [...] > > While I appreciate that this decision on the part of the Cinder > contributors puts the overall OpenStack community in a bit of a > quandary, it would be good to let them know how long they're being > asked to wait for official feedback from the TC, so that they can > proceed with their original plan (backed by existing stable > maintenance policy) if the debate continues to drag out. Can the TC > please commit to reaching a conclusion by some specific deadline? > -- > Jeremy Stanley From gmann at ghanshyammann.com Thu Jul 6 17:48:38 2023 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Thu, 06 Jul 2023 10:48:38 -0700 Subject: [cinder][all][tc] Cinder to EOL all EM branches In-Reply-To: <8F41F5B6-EFA9-4EC7-9109-7A276801D566@bu.edu> References: <18927432cab.b35b5b43911163.6905573119740994317@ghanshyammann.com> <20230705201854.mcr67unio2s3uzly@yuggoth.org> <8F41F5B6-EFA9-4EC7-9109-7A276801D566@bu.edu> Message-ID: <1892c51ff05.107fa7d5d4655.142073720558322639@ghanshyammann.com> ---- On Thu, 06 Jul 2023 10:05:29 -0700 Nikolla, Kristi wrote --- > Thanks for making that point Jeremy. > > As a courtesy to allow the TC to provide a resolution with coordinated guidelines for EOL-ing branches across projects, we please ask that you wait until the end of July at the latest. > > If no consensus has been reached on approving a new EM policy has been reached until the end of July, it is within your power, as the policy is written today, to EOL all branches based on an internal team decision. > > As every TC policy resolution requires staying open for at least 1 week, and taking into account 1-2 weeks of deliberation, that is the earliest that we can reasonably approve a new policy. Also, we will be discussing it in next TC meeting in Tuesday video call. if no consensus in that meeting, I will send more call invite so that we can discuss this in speedy way. Everyone are welcome to join the TC call in that discussion. - https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee -gmann > > Thank you, > Kristi Nikolla > > > > On Jul 5, 2023, at 4:18 PM, Jeremy Stanley fungi at yuggoth.org> wrote: > > > > On 2023-07-05 11:14:21 -0700 (-0700), Ghanshyam Mann wrote: > > [...] > >> Even though we were not able to get any solution to existing EM > >> issue and TC is working on that and hopefully we will come to the > >> agreement soon (even it might be killing the EM process). > > [...] > >> IMO, we should wait for overall decision on EM so that we can keep > >> consistency to our operator. Whatever the agreement will be, it > >> will be same for all projects and give a clear consistency to > >> Operators. > > [...] > > > > While I appreciate that this decision on the part of the Cinder > > contributors puts the overall OpenStack community in a bit of a > > quandary, it would be good to let them know how long they're being > > asked to wait for official feedback from the TC, so that they can > > proceed with their original plan (backed by existing stable > > maintenance policy) if the debate continues to drag out. Can the TC > > please commit to reaching a conclusion by some specific deadline? > > -- > > Jeremy Stanley > > > From rosmaita.fossdev at gmail.com Thu Jul 6 18:01:57 2023 From: rosmaita.fossdev at gmail.com (Brian Rosmaita) Date: Thu, 6 Jul 2023 14:01:57 -0400 Subject: [cinder][all][tc] Cinder to EOL all EM branches In-Reply-To: <8F41F5B6-EFA9-4EC7-9109-7A276801D566@bu.edu> References: <18927432cab.b35b5b43911163.6905573119740994317@ghanshyammann.com> <20230705201854.mcr67unio2s3uzly@yuggoth.org> <8F41F5B6-EFA9-4EC7-9109-7A276801D566@bu.edu> Message-ID: On 7/6/23 1:05 PM, Nikolla, Kristi wrote: [snip] > As a courtesy to allow the TC to provide a resolution with coordinated guidelines for EOL-ing branches across projects, we please ask that you wait until the end of July at the latest. [snip] This seems reasonable to me. The Cinder project has announced its intentions, namely, that we will not fix the gates for any of the branches currently in EM, and that the EOL tags will (eventually) be made at the hashes indicated in the proposed release patches: https://review.opendev.org/q/topic:cinder-eol-june2023 If anyone in the wider OpenStack Community has a desire to have backports merged into these branches, or to keep these branches open longer, now would be a good time to step up and do all appropriate work to make that happen. cheers, brian From ralonsoh at redhat.com Fri Jul 7 12:12:49 2023 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Fri, 7 Jul 2023 14:12:49 +0200 Subject: [neutron] Neutron drivers meeting cancelled Message-ID: Hello Neutrinos: Due to the lack of agenda [1], today's meeting is cancelled. Have a nice weekend! [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers -------------- next part -------------- An HTML attachment was scrubbed... URL: From hberaud at redhat.com Fri Jul 7 14:35:56 2023 From: hberaud at redhat.com (Herve Beraud) Date: Fri, 7 Jul 2023 16:35:56 +0200 Subject: [release] Release countdown for week R-12, Jul 10-14 Message-ID: Development Focus ----------------- We are now past the Bobcat-2 milestone, and entering the last development phase of the cycle. Teams should be focused on implementing planned work for the cycle. Now is a good time to review those plans and reprioritize anything if needed based on the what progress has been made and what looks realistic to complete in the next few weeks. General Information ------------------- Looking ahead to the end of the release cycle, please be aware of the feature freeze dates. Those vary depending on deliverable type: * General libraries (except client libraries) need to have their last feature release before Non-client library freeze (August 24). Their stable branches are cut early. * Client libraries (think python-*client libraries) need to have their last feature release before Client library freeze (August 31) * Deliverables following a cycle-with-rc model (that would be most services) observe a Feature freeze on that same date, August 31. Any feature addition beyond that date should be discussed on the mailing-list and get PTL approval. After feature freeze, cycle-with-rc deliverables need to produce a first release candidate (and a stable branch) before RC1 deadline (September 14) * Deliverables following cycle-with-intermediary model can release as necessary, but in all cases before Final RC deadline (September 28) Upcoming Deadlines & Dates -------------------------- Non-client library freeze: August 24 (R-6 week) Client library freeze: August 31 (R-5 week) Bobcat-3 milestone: August 31 (R-5 week) Final 2023.2 Bobcat release: October 4th, 2023 -- Herv? Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From michel.jouvin at ijclab.in2p3.fr Fri Jul 7 14:43:50 2023 From: michel.jouvin at ijclab.in2p3.fr (Michel Jouvin) Date: Fri, 7 Jul 2023 16:43:50 +0200 Subject: Ussuri: how to delete lbaas loadbalancer left over? Message-ID: Hi, We had a few Magnum (K8s) clusters created a couple of years ago (with Rocky and Stein versions) and forgotten. We started to delete them this spring when we where running Train Neutron service. Basically we managed to do this with the following sequence: - openstack coe cluster delete xxx and waiting for DELETE_FAILED - Use openstack coe cluster show / openstack stack resource list -n2 to identify the neutron entry causing the error and pick the corresponding resource ID - Find the ports associated with the router with openstack port list --router previously_found_id - Use the port subnet to find the port corresponding lbaas load balancer ID, use the neutron CLI to delete the load balancer (deleting one by one all the dependencies preventing the load balancer removal) - Rerun openstack coe cluster delete For some reasons, we didn't cleanup all the abandoned clusters and upgraded Neutron to Ussuri. Unfortunately, since then our previous process is no longer working as it seems that the Neutron server doesn't know anymore anything about the LBAAS load balancers (and "neutron lbaas-loadbalancer list" returns nothing). In the neutron server, any attempt to delete the subnet attached to the load balancer (or to list them with Neutron CLI) results in the following errors in Neutron server.log : ------ 2023-07-07 16:27:31.139 14962 WARNING neutron.pecan_wsgi.controllers.root [req-71e712fc-d8a7-4815-90b3-b406c10e0caa a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa 1367c9a4d5da4b229c35789c271dc7aa] No controller found for: lbaas - returning response code 404: pecan.routing.PecanNotFound 2023-07-07 16:27:31.140 14962 INFO neutron.pecan_wsgi.hooks.translation [req-71e712fc-d8a7-4815-90b3-b406c10e0caa a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa 1367c9a4d5da4b229c35789c271dc7aa] GET failed (client error): The resource could not be found. 2023-07-07 16:27:31.141 14962 INFO neutron.wsgi [req-71e712fc-d8a7-4815-90b3-b406c10e0caa a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa 1367c9a4d5da4b229c35789c271dc7aa] 157.136.249.153 "GET /v2.0/lbaas/loadbalancers?name=kube_service_964f7e76-d2d5-4126-ab11-cd689f6dd9f9_runnerdeploy-wm9sm-5h52l_hello-node-x-default-x-runnerdeploy-wm9sm-5h52l HTTP/1.1" status: 404? len: 304 time: 0.0052643 ------ Any suggestion to workaround this problem and be able to successfully delete our old Magnum clusters? Thanks in advance for any help. Best regards, Michel From knikolla at bu.edu Fri Jul 7 15:15:30 2023 From: knikolla at bu.edu (Nikolla, Kristi) Date: Fri, 7 Jul 2023 15:15:30 +0000 Subject: [all][tc] Proposals for reforming extended maintenance Message-ID: Hi all, There have been quite a few discussions related extended maintenance in the last few months[0][1]. I have made a first-go at trying to write a policy for extended maintenance moving forward which tries to preserve the value being brought by extended maintenance while strengthening the requirements for keeping branches under that phase. [2] This was based on multiple discussions that I was present at and feedback that I have received from various teams and community members with which I talked, and is my interpretation of that feedback into something that looks like policy. I have also written two opposing policy proposals about what do to with the current branches under extended maintenance. One which attempts to apply the new requirements defined in the first proposal almost immediately[3], and one that aims for a more gradual EOL[4]. Please provide feedback on the proposals and help shape up the future of extended maintenance. Thank you, Kristi Nikolla [0]. https://lists.openstack.org/pipermail/openstack-discuss/2023-June/033980.html [1]. https://etherpad.opendev.org/p/vancouver-2023-em [2]. https://review.opendev.org/c/openstack/governance/+/887966 [3]. https://review.opendev.org/c/openstack/governance/+/887968 [4]. https://review.opendev.org/c/openstack/governance/+/887969 From eblock at nde.ag Fri Jul 7 16:46:54 2023 From: eblock at nde.ag (Eugen Block) Date: Fri, 07 Jul 2023 16:46:54 +0000 Subject: Ussuri: how to delete lbaas loadbalancer left over? In-Reply-To: Message-ID: <20230707164654.Horde.sUTt6M8NRpBGR8HwFy7KwGv@webmail.nde.ag> Hi, neutron lbaas was deprecated in Queens so you may have to migrate the existing LBs to octavia. I have never done that but I remember reading through the SUSE Docs when one of our customers had to decide whether they wanted to upgrade or reinstall with a newer openstack release. They decided to do the latter, so we set up octavia from scratch and didn't have to migrate anything. There's also a video I've never watched [2], maybe that helps. I can't really tell if a migration is possible to work around your issue but I thought I'd share anyway. Regards, Eugen [1] https://documentation.suse.com/soc/9/single-html/suse-openstack-cloud-crowbar-deployment/#sec-depl-ostack-octavia-migrate-users [2] https://www.youtube.com/watch?v=jj4KMJPA0Pk Zitat von Michel Jouvin : > Hi, > > We had a few Magnum (K8s) clusters created a couple of years ago > (with Rocky and Stein versions) and forgotten. We started to delete > them this spring when we where running Train Neutron service. > Basically we managed to do this with the following sequence: > > - openstack coe cluster delete xxx and waiting for DELETE_FAILED > - Use openstack coe cluster show / openstack stack resource list -n2 > to identify the neutron entry causing the error and pick the > corresponding resource ID > - Find the ports associated with the router with openstack port list > --router previously_found_id > - Use the port subnet to find the port corresponding lbaas load > balancer ID, use the neutron CLI to delete the load balancer > (deleting one by one all the dependencies preventing the load > balancer removal) > - Rerun openstack coe cluster delete > > For some reasons, we didn't cleanup all the abandoned clusters and > upgraded Neutron to Ussuri. Unfortunately, since then our previous > process is no longer working as it seems that the Neutron server > doesn't know anymore anything about the LBAAS load balancers (and > "neutron lbaas-loadbalancer list" returns nothing). In the neutron > server, any attempt to delete the subnet attached to the load > balancer (or to list them with Neutron CLI) results in the following > errors in Neutron server.log : > > ------ > > 2023-07-07 16:27:31.139 14962 WARNING > neutron.pecan_wsgi.controllers.root > [req-71e712fc-d8a7-4815-90b3-b406c10e0caa > a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c > 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa > 1367c9a4d5da4b229c35789c271dc7aa] No controller found for: lbaas - > returning response code 404: pecan.routing.PecanNotFound > 2023-07-07 16:27:31.140 14962 INFO > neutron.pecan_wsgi.hooks.translation > [req-71e712fc-d8a7-4815-90b3-b406c10e0caa > a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c > 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa > 1367c9a4d5da4b229c35789c271dc7aa] GET failed (client error): The > resource could not be found. > 2023-07-07 16:27:31.141 14962 INFO neutron.wsgi > [req-71e712fc-d8a7-4815-90b3-b406c10e0caa > a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c > 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa > 1367c9a4d5da4b229c35789c271dc7aa] 157.136.249.153 "GET > /v2.0/lbaas/loadbalancers?name=kube_service_964f7e76-d2d5-4126-ab11-cd689f6dd9f9_runnerdeploy-wm9sm-5h52l_hello-node-x-default-x-runnerdeploy-wm9sm-5h52l HTTP/1.1" status: 404? len: 304 time: > 0.0052643 > ------ > > Any suggestion to workaround this problem and be able to > successfully delete our old Magnum clusters? > > Thanks in advance for any help. Best regards, > > Michel From molenkam at uwo.ca Fri Jul 7 18:08:47 2023 From: molenkam at uwo.ca (Gary Molenkamp) Date: Fri, 7 Jul 2023 14:08:47 -0400 Subject: SNAT failure with OVN under Antelope In-Reply-To: References: <34841bdd-bb95-ff9a-258b-cfae26558627@uwo.ca> <536805d2-f334-e94f-1415-2984a219cb65@uwo.ca> Message-ID: <8511695f-443b-1593-fc20-04cb82cfc791@uwo.ca> Happy Friday afternoon. I'm still pondering a lack of connectivity in an HA OVN with each compute node acting as a potential gateway chassis. >> The problem is basically that the port of the OVN LRP may not be >> in the same chassis as the VM that failed (since the CR-LRP will >> be where the first VM of that network will be created).?The >> suggestion is to remove the enable-chassis-as-gw from the compute >> nodes to allow the VM to forward traffic via tunneling/Geneve to >> the chassis where the LRP resides. >> > > I forced a similar VM onto the same chassis as the working VM, and > it was able to communicate out.??? If we do want to keep multiple > chassis' as gateways, would that be addressed with the > ovn-bridge-mappings? > > I built a small test cloud to explore this further as I continue to see the same issue:? A vm will only be able to use SNAT outbound if it is on the same chassis as the CR-LRP. In my test cloud, I have one controller, and two compute nodes.? The controller only runs the north and southd in addition to the neutron server.? Each of the two compute nodes is configured as below.? On a tenent network I have three VMs: ??? - #1:? cirros VM with FIP ??? - #2:? cirros VM running on compute node 1 ??? - #3:? cirros VM running on compute node 2 E/W traffic between VMs in the same tenent network are fine.? N/S traffic is fine for the FIP.? N/S traffic only works for the VM whose CR-LRP is active on same chassis.?? Does anything jump out as a mistake in my understanding at to how this should be working? Thanks as always, Gary on each hypervisor: /usr/bin/ovs-vsctl set open . external-ids:ovn-remote=tcp:{{ controllerip }}:6642 /usr/bin/ovs-vsctl set open . external-ids:ovn-encap-type=geneve /usr/bin/ovs-vsctl set open . external-ids:ovn-encap-ip={{ overlaynetip }} /usr/bin/ovs-vsctl set open . external-ids:ovn-cms-options=enable-chassis-as-gw /usr/bin/ovs-vsctl add-br br-provider -- set bridge br-provider protocols=OpenFlow10,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 /usr/bin/ovs-vsctl add-port br-provider {{ provider_nic }} /usr/bin/ovs-vsctl br-set-external-id provider bridge-id br-provider /usr/bin/ovs-vsctl set open . external-ids:ovn-bridge-mappings=provider:br-provider plugin.ini: [ml2] mechanism_drivers = ovn type_drivers = flat,geneve tenant_network_types = geneve extension_drivers = port_security overlay_ip_version = 4 [ml2_type_flat] flat_networks = provider [ml2_type_geneve] vni_ranges = 1:65536 max_header_size = 38 [securitygroup] enable_security_group = True firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver [ovn] ovn_nb_connection = tcp:{{controllerip}}:6641 ovn_sb_connection = tcp:{{controllerip}}:6642 ovn_l3_scheduler = leastloaded ovn_metadata_enabled = True enable_distributed_floating_ip = true -- Gary Molenkamp Science Technology Services Systems Administrator University of Western Ontario molenkam at uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566 -------------- next part -------------- An HTML attachment was scrubbed... URL: From michel.jouvin at ijclab.in2p3.fr Fri Jul 7 21:32:07 2023 From: michel.jouvin at ijclab.in2p3.fr (Michel Jouvin) Date: Fri, 07 Jul 2023 23:32:07 +0200 Subject: Ussuri: how to delete lbaas loadbalancer left over? In-Reply-To: <20230707164654.Horde.sUTt6M8NRpBGR8HwFy7KwGv@webmail.nde.ag> References: <20230707164654.Horde.sUTt6M8NRpBGR8HwFy7KwGv@webmail.nde.ag> Message-ID: <1893244f3d8.27f3.86e99a005b57e9644b7bb922d8d7f665@ijclab.in2p3.fr> Ho Eug?ne, Thanks for your answer. Do you mean that after installing octavia (it is planned) we'll have again the ability to delete the remaining LBAAS instances? Or just that octavia is the LBAAS replacement in terms of functionalities? Best regards, Michel Sent from my mobile Le 7 juillet 2023 18:52:30 Eugen Block a ?crit : > Hi, > > neutron lbaas was deprecated in Queens so you may have to migrate the > existing LBs to octavia. I have never done that but I remember reading > through the SUSE Docs when one of our customers had to decide whether > they wanted to upgrade or reinstall with a newer openstack release. > They decided to do the latter, so we set up octavia from scratch and > didn't have to migrate anything. There's also a video I've never > watched [2], maybe that helps. I can't really tell if a migration is > possible to work around your issue but I thought I'd share anyway. > > Regards, > Eugen > > [1] > https://documentation.suse.com/soc/9/single-html/suse-openstack-cloud-crowbar-deployment/#sec-depl-ostack-octavia-migrate-users > [2] https://www.youtube.com/watch?v=jj4KMJPA0Pk > > Zitat von Michel Jouvin : > >> Hi, >> >> We had a few Magnum (K8s) clusters created a couple of years ago >> (with Rocky and Stein versions) and forgotten. We started to delete >> them this spring when we where running Train Neutron service. >> Basically we managed to do this with the following sequence: >> >> - openstack coe cluster delete xxx and waiting for DELETE_FAILED >> - Use openstack coe cluster show / openstack stack resource list -n2 >> to identify the neutron entry causing the error and pick the >> corresponding resource ID >> - Find the ports associated with the router with openstack port list >> --router previously_found_id >> - Use the port subnet to find the port corresponding lbaas load >> balancer ID, use the neutron CLI to delete the load balancer >> (deleting one by one all the dependencies preventing the load >> balancer removal) >> - Rerun openstack coe cluster delete >> >> For some reasons, we didn't cleanup all the abandoned clusters and >> upgraded Neutron to Ussuri. Unfortunately, since then our previous >> process is no longer working as it seems that the Neutron server >> doesn't know anymore anything about the LBAAS load balancers (and >> "neutron lbaas-loadbalancer list" returns nothing). In the neutron >> server, any attempt to delete the subnet attached to the load >> balancer (or to list them with Neutron CLI) results in the following >> errors in Neutron server.log : >> >> ------ >> >> 2023-07-07 16:27:31.139 14962 WARNING >> neutron.pecan_wsgi.controllers.root >> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >> 1367c9a4d5da4b229c35789c271dc7aa] No controller found for: lbaas - >> returning response code 404: pecan.routing.PecanNotFound >> 2023-07-07 16:27:31.140 14962 INFO >> neutron.pecan_wsgi.hooks.translation >> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >> 1367c9a4d5da4b229c35789c271dc7aa] GET failed (client error): The >> resource could not be found. >> 2023-07-07 16:27:31.141 14962 INFO neutron.wsgi >> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >> 1367c9a4d5da4b229c35789c271dc7aa] 157.136.249.153 "GET >> /v2.0/lbaas/loadbalancers?name=kube_service_964f7e76-d2d5-4126-ab11-cd689f6dd9f9_runnerdeploy-wm9sm-5h52l_hello-node-x-default-x-runnerdeploy-wm9sm-5h52l >> HTTP/1.1" status: 404 len: 304 time: >> 0.0052643 >> ------ >> >> Any suggestion to workaround this problem and be able to >> successfully delete our old Magnum clusters? >> >> Thanks in advance for any help. Best regards, >> >> Michel -------------- next part -------------- An HTML attachment was scrubbed... URL: From gmann at ghanshyammann.com Fri Jul 7 21:57:03 2023 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Fri, 07 Jul 2023 14:57:03 -0700 Subject: [all][tc] Proposals for reforming extended maintenance In-Reply-To: References: Message-ID: <189325bc896.fc1d299689349.7239951418896363270@ghanshyammann.com> ---- On Fri, 07 Jul 2023 08:15:30 -0700 Nikolla, Kristi wrote --- > Hi all, > > There have been quite a few discussions related extended maintenance in the last few months[0][1]. > > I have made a first-go at trying to write a policy for extended maintenance moving forward which tries to preserve the value being brought by extended maintenance while strengthening the requirements for keeping branches under that phase. [2] > This was based on multiple discussions that I was present at and feedback that I have received from various teams and community members with which I talked, and is my interpretation of that feedback into something that looks like policy. > > I have also written two opposing policy proposals about what do to with the current branches under extended maintenance. One which attempts to apply the new requirements defined in the first proposal almost immediately[3], and one that aims for a more gradual EOL[4]. > > Please provide feedback on the proposals and help shape up the future of extended maintenance. Thanks, Kristi for the proposal. I have created the below etherpad for this discussion, where we will first discuss all problems and the discussion of the possible solutions before jumping to the one. I have collected a few of the solutions/ways we discussed till now, which include the 887966 proposal too. Feel Free to add more possible options in etherpad. - https://etherpad.opendev.org/p/openstack-exteneded-maintenance We can use the same etherpad to collect the discussion at a single place, either from TC meetings or any further calls if there will be any. -gmann > > Thank you, > Kristi Nikolla > > [0]. https://lists.openstack.org/pipermail/openstack-discuss/2023-June/033980.html > [1]. https://etherpad.opendev.org/p/vancouver-2023-em > [2]. https://review.opendev.org/c/openstack/governance/+/887966 > [3]. https://review.opendev.org/c/openstack/governance/+/887968 > [4]. https://review.opendev.org/c/openstack/governance/+/887969 > From eblock at nde.ag Fri Jul 7 22:05:12 2023 From: eblock at nde.ag (Eugen Block) Date: Fri, 07 Jul 2023 22:05:12 +0000 Subject: Ussuri: how to delete lbaas loadbalancer left over? In-Reply-To: <1893244f3d8.27f3.86e99a005b57e9644b7bb922d8d7f665@ijclab.in2p3.fr> References: <20230707164654.Horde.sUTt6M8NRpBGR8HwFy7KwGv@webmail.nde.ag> <1893244f3d8.27f3.86e99a005b57e9644b7bb922d8d7f665@ijclab.in2p3.fr> Message-ID: <20230707220512.Horde.10Q-iIqvZct1kSTQs9Gd4-u@webmail.nde.ag> Hi, I mean the latter. Once you have Octavia installed you can create new LBs, but as I understand it you won?t be able to delete the legacy LBs. Did the neutron config change when you upgraded to Ussuri? I wonder if there?s just some config missing to be able to delete the old LBs, I don?t have a clue tbh. Maybe someone else has some more experience and will chime in. Zitat von Michel Jouvin : > Ho Eug?ne, > > Thanks for your answer. Do you mean that after installing octavia > (it is planned) we'll have again the ability to delete the remaining > LBAAS instances? Or just that octavia is the LBAAS replacement in > terms of functionalities? > > Best regards, > > Michel > Sent from my mobile > Le 7 juillet 2023 18:52:30 Eugen Block a ?crit : > >> Hi, >> >> neutron lbaas was deprecated in Queens so you may have to migrate the >> existing LBs to octavia. I have never done that but I remember reading >> through the SUSE Docs when one of our customers had to decide whether >> they wanted to upgrade or reinstall with a newer openstack release. >> They decided to do the latter, so we set up octavia from scratch and >> didn't have to migrate anything. There's also a video I've never >> watched [2], maybe that helps. I can't really tell if a migration is >> possible to work around your issue but I thought I'd share anyway. >> >> Regards, >> Eugen >> >> [1] >> https://documentation.suse.com/soc/9/single-html/suse-openstack-cloud-crowbar-deployment/#sec-depl-ostack-octavia-migrate-users >> [2] https://www.youtube.com/watch?v=jj4KMJPA0Pk >> >> Zitat von Michel Jouvin : >> >>> Hi, >>> >>> We had a few Magnum (K8s) clusters created a couple of years ago >>> (with Rocky and Stein versions) and forgotten. We started to delete >>> them this spring when we where running Train Neutron service. >>> Basically we managed to do this with the following sequence: >>> >>> - openstack coe cluster delete xxx and waiting for DELETE_FAILED >>> - Use openstack coe cluster show / openstack stack resource list -n2 >>> to identify the neutron entry causing the error and pick the >>> corresponding resource ID >>> - Find the ports associated with the router with openstack port list >>> --router previously_found_id >>> - Use the port subnet to find the port corresponding lbaas load >>> balancer ID, use the neutron CLI to delete the load balancer >>> (deleting one by one all the dependencies preventing the load >>> balancer removal) >>> - Rerun openstack coe cluster delete >>> >>> For some reasons, we didn't cleanup all the abandoned clusters and >>> upgraded Neutron to Ussuri. Unfortunately, since then our previous >>> process is no longer working as it seems that the Neutron server >>> doesn't know anymore anything about the LBAAS load balancers (and >>> "neutron lbaas-loadbalancer list" returns nothing). In the neutron >>> server, any attempt to delete the subnet attached to the load >>> balancer (or to list them with Neutron CLI) results in the following >>> errors in Neutron server.log : >>> >>> ------ >>> >>> 2023-07-07 16:27:31.139 14962 WARNING >>> neutron.pecan_wsgi.controllers.root >>> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >>> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >>> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >>> 1367c9a4d5da4b229c35789c271dc7aa] No controller found for: lbaas - >>> returning response code 404: pecan.routing.PecanNotFound >>> 2023-07-07 16:27:31.140 14962 INFO >>> neutron.pecan_wsgi.hooks.translation >>> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >>> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >>> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >>> 1367c9a4d5da4b229c35789c271dc7aa] GET failed (client error): The >>> resource could not be found. >>> 2023-07-07 16:27:31.141 14962 INFO neutron.wsgi >>> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >>> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >>> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >>> 1367c9a4d5da4b229c35789c271dc7aa] 157.136.249.153 "GET >>> /v2.0/lbaas/loadbalancers?name=kube_service_964f7e76-d2d5-4126-ab11-cd689f6dd9f9_runnerdeploy-wm9sm-5h52l_hello-node-x-default-x-runnerdeploy-wm9sm-5h52l HTTP/1.1" status: 404 len: 304 >>> time: >>> 0.0052643 >>> ------ >>> >>> Any suggestion to workaround this problem and be able to >>> successfully delete our old Magnum clusters? >>> >>> Thanks in advance for any help. Best regards, >>> >>> Michel From eblock at nde.ag Fri Jul 7 22:15:44 2023 From: eblock at nde.ag (Eugen Block) Date: Fri, 07 Jul 2023 22:15:44 +0000 Subject: Ussuri: how to delete lbaas loadbalancer left over? In-Reply-To: <20230707220512.Horde.10Q-iIqvZct1kSTQs9Gd4-u@webmail.nde.ag> References: <20230707164654.Horde.sUTt6M8NRpBGR8HwFy7KwGv@webmail.nde.ag> <1893244f3d8.27f3.86e99a005b57e9644b7bb922d8d7f665@ijclab.in2p3.fr> <20230707220512.Horde.10Q-iIqvZct1kSTQs9Gd4-u@webmail.nde.ag> Message-ID: <20230707221544.Horde.tGBcMHNkuxKWyNahfmPmJ9V@webmail.nde.ag> Unfortunately, the link to the migration tool doesn?t work: https://wiki.openstack.org/wiki/Neutron/LBaaS/Deprecation But maybe it can get you in the right direction, neutron-lbaas-proxy seems to be the keyword. But as I already mentioned, I don?t have experience with this path. Zitat von Eugen Block : > Hi, > > I mean the latter. Once you have Octavia installed you can create > new LBs, but as I understand it you won?t be able to delete the > legacy LBs. Did the neutron config change when you upgraded to > Ussuri? I wonder if there?s just some config missing to be able to > delete the old LBs, I don?t have a clue tbh. Maybe someone else has > some more experience and will chime in. > > Zitat von Michel Jouvin : > >> Ho Eug?ne, >> >> Thanks for your answer. Do you mean that after installing octavia >> (it is planned) we'll have again the ability to delete the >> remaining LBAAS instances? Or just that octavia is the LBAAS >> replacement in terms of functionalities? >> >> Best regards, >> >> Michel >> Sent from my mobile >> Le 7 juillet 2023 18:52:30 Eugen Block a ?crit : >> >>> Hi, >>> >>> neutron lbaas was deprecated in Queens so you may have to migrate the >>> existing LBs to octavia. I have never done that but I remember reading >>> through the SUSE Docs when one of our customers had to decide whether >>> they wanted to upgrade or reinstall with a newer openstack release. >>> They decided to do the latter, so we set up octavia from scratch and >>> didn't have to migrate anything. There's also a video I've never >>> watched [2], maybe that helps. I can't really tell if a migration is >>> possible to work around your issue but I thought I'd share anyway. >>> >>> Regards, >>> Eugen >>> >>> [1] >>> https://documentation.suse.com/soc/9/single-html/suse-openstack-cloud-crowbar-deployment/#sec-depl-ostack-octavia-migrate-users >>> [2] https://www.youtube.com/watch?v=jj4KMJPA0Pk >>> >>> Zitat von Michel Jouvin : >>> >>>> Hi, >>>> >>>> We had a few Magnum (K8s) clusters created a couple of years ago >>>> (with Rocky and Stein versions) and forgotten. We started to delete >>>> them this spring when we where running Train Neutron service. >>>> Basically we managed to do this with the following sequence: >>>> >>>> - openstack coe cluster delete xxx and waiting for DELETE_FAILED >>>> - Use openstack coe cluster show / openstack stack resource list -n2 >>>> to identify the neutron entry causing the error and pick the >>>> corresponding resource ID >>>> - Find the ports associated with the router with openstack port list >>>> --router previously_found_id >>>> - Use the port subnet to find the port corresponding lbaas load >>>> balancer ID, use the neutron CLI to delete the load balancer >>>> (deleting one by one all the dependencies preventing the load >>>> balancer removal) >>>> - Rerun openstack coe cluster delete >>>> >>>> For some reasons, we didn't cleanup all the abandoned clusters and >>>> upgraded Neutron to Ussuri. Unfortunately, since then our previous >>>> process is no longer working as it seems that the Neutron server >>>> doesn't know anymore anything about the LBAAS load balancers (and >>>> "neutron lbaas-loadbalancer list" returns nothing). In the neutron >>>> server, any attempt to delete the subnet attached to the load >>>> balancer (or to list them with Neutron CLI) results in the following >>>> errors in Neutron server.log : >>>> >>>> ------ >>>> >>>> 2023-07-07 16:27:31.139 14962 WARNING >>>> neutron.pecan_wsgi.controllers.root >>>> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >>>> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >>>> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >>>> 1367c9a4d5da4b229c35789c271dc7aa] No controller found for: lbaas - >>>> returning response code 404: pecan.routing.PecanNotFound >>>> 2023-07-07 16:27:31.140 14962 INFO >>>> neutron.pecan_wsgi.hooks.translation >>>> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >>>> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >>>> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >>>> 1367c9a4d5da4b229c35789c271dc7aa] GET failed (client error): The >>>> resource could not be found. >>>> 2023-07-07 16:27:31.141 14962 INFO neutron.wsgi >>>> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >>>> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >>>> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >>>> 1367c9a4d5da4b229c35789c271dc7aa] 157.136.249.153 "GET >>>> /v2.0/lbaas/loadbalancers?name=kube_service_964f7e76-d2d5-4126-ab11-cd689f6dd9f9_runnerdeploy-wm9sm-5h52l_hello-node-x-default-x-runnerdeploy-wm9sm-5h52l HTTP/1.1" status: 404 len: 304 >>>> time: >>>> 0.0052643 >>>> ------ >>>> >>>> Any suggestion to workaround this problem and be able to >>>> successfully delete our old Magnum clusters? >>>> >>>> Thanks in advance for any help. Best regards, >>>> >>>> Michel From sbaker at redhat.com Sun Jul 9 21:42:14 2023 From: sbaker at redhat.com (Steve Baker) Date: Mon, 10 Jul 2023 09:42:14 +1200 Subject: TripleO | OS-net config logs | openstack overcloud node provision In-Reply-To: References: Message-ID: The --verbose flag is passed through to ansible runs in the provision command, so this will provide more verbose logging: openstack -vvv overcloud node provision On 6/07/23 22:13, Lokendra Rathour wrote: > > Hi Team, > > Scenario: > We were performing overcloud deployment. > During this, we were continuously facing the following errors on two > of the nodes out of the four nodes: > > ????????????? PLAY [Overcloud Node Grow Volumes] > ********************************************* > ????????????? 2023-06-02 19:07:33.483504 | > 52540049-4965-0177-e681-00000000000a |?????? TASK | Wait for > provisioned nodes to boot > ????????????? [WARNING]: Unhandled error in Python interpreter > discovery for host overcloud- > ????????????? controller-0: Failed to connect to the host via ssh: > ssh: connect to host > ????????????? 30.30.30.219 port 22: No route to host > ????????????? [WARNING]: Unhandled error in Python interpreter > discovery for host overcloud- > ????????????? controller-0: Failed to connect to the host via ssh: > ssh: connect to host > ????????????? 30.30.30.219 port 22: Connection refused > > We faced this error when we were performing overcloud node provisioning. > > To resolve this issue, we had to update the image and create a user to > access the nodes. > We were able to access the nodes via this user through the BMC console. > > This implied that OS was getting correctly installed on the node. > Once we accessed the logs on the failed node we came across the > following logs: > > ????????????? 2023-06-13 09:02:16.120 INFO > os_net_config.utils._ordered_nics Finding active nics > ????????????? 2023-06-13 09:02:16.120 INFO > os_net_config.utils._ordered_nics eno1 is an embedded active nic > ????????????? 2023-06-13 09:02:16.121 INFO > os_net_config.utils._ordered_nics ens2f0 is an active nic > ????????????? 2023-06-13 09:02:16.121 INFO > os_net_config.utils._ordered_nics eno4 is an embedded active nic > ????????????? 2023-06-13 09:02:16.121 INFO > os_net_config.utils._ordered_nics eno2 is an embedded active nic > ????????????? 2023-06-13 09:02:16.121 INFO > os_net_config.utils._ordered_nics ens1f0 is not an active nic > ????????????? 2023-06-13 09:02:16.121 INFO > os_net_config.utils._ordered_nics lo is not an active nic > ????????????? 2023-06-13 09:02:16.121 INFO > os_net_config.utils._ordered_nics ens2f1 is an active nic > ????????????? 2023-06-13 09:02:16.121 INFO > os_net_config.utils._ordered_nics eno3 is an embedded active nic > ????????????? 2023-06-13 09:02:16.121 INFO > os_net_config.utils._ordered_nics ens1f1 is an active nic > ????????????? 2023-06-13 09:02:16.121 INFO > os_net_config.utils._ordered_nics No DPDK mapping available in path > (/var/lib/os-net-config/dpdk_mapping.yaml) > ????????????? 2023-06-13 09:02:16.122 INFO > os_net_config.utils._ordered_nics Active nics are ['eno1', 'eno2', > 'eno3', 'eno4', 'ens1f1', 'ens2f0', 'ens2f1'] > ????????????? 2023-06-13 09:02:16.122 INFO > os_net_config.objects.mapped_nics nic2 mapped to: eno2 > ????????????? 2023-06-13 09:02:16.122 INFO > os_net_config.objects.mapped_nics nic6 mapped to: ens2f0 > ????????????? 2023-06-13 09:02:16.122 INFO > os_net_config.objects.mapped_nics nic5 mapped to: ens1f1 > ????????????? 2023-06-13 09:02:16.122 INFO > os_net_config.objects.mapped_nics nic3 mapped to: eno3 > ????????????? 2023-06-13 09:02:16.122 INFO > os_net_config.objects.mapped_nics nic7 mapped to: ens2f1 > ????????????? 2023-06-13 09:02:16.122 INFO > os_net_config.objects.mapped_nics nic1 mapped to: eno1 > ????????????? 2023-06-13 09:02:16.122 INFO > os_net_config.objects.mapped_nics nic4 mapped to: eno4 > > > As we can see in the above logs, only 7 nics are active. > But as per our bond config file, we required 8 interfaces: > ????????????? a. nic-1 for management IP > ????????????? b. nic-3 and nic-4 for the br-tenant-2 network. > ????????????? c. nic-5 and nic-6 for the br-tenant network. > ????????????? d. nic-7 and nic 8 for the br-ex network. > > The wire was faulty in this case and hence only 7 interfaces were > available > > Query: > The logs which are visible while we run "openstack overcloud node > provision" are not very informative. > Is there any way where we could get these logs on the undercloud node > itself? > This could save some time for future references > > > -- > ~ Lokendra > skype: lokendrarathour > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yuta.kazato at ntt.com Mon Jul 10 12:09:15 2023 From: yuta.kazato at ntt.com (Yuta Kazato) Date: Mon, 10 Jul 2023 12:09:15 +0000 Subject: [tacker][oslo][kolla-ansible] Issue reports for session management in oslo when using kolla-ansible Message-ID: Hi, OpenStack Community. Our Tacker team found an issue with session management in oslo and sqlalchemy when using kolla-ansible. The following report details the issue. [Issue details] - In the case that we build Tacker and other components by using kolla-ansible, an issue with session management in oslo and sqlalchemy occurs when Tacker update the DB with `with context.session.begin(subtransactions=True):` - code: https://github.com/openstack/tacker/blob/7eaf841525d7ead30d7883abbde6c4fb86ae2e4f/tacker/sol_refactored/controller/vnffm_v1.py#L101 - error details === * cli logs tacker at tacker:~/tacker-nic_internal/samples/takt-docker-sample/deployments/tacker$ curl -s -X PATCH http://"10.244.121.249:9890"/vnffm/v1/alarms/f23f39cc-f9a4-45bf-9a87-2f8148541b0a \ -H "Content-type: application/json" \ -H "Version: 1.3.0" \ -d "{\"ackState\": \"ACKNOWLEDGED\"}" \ -w "\n%{http_code}\n" {"status": 500, "detail": "this TransactionFactory is already started"} * tacker logs 2023-06-12 05:28:56.028 1 INFO tacker.service [-] Tacker service started, listening on 0.0.0.0:9890 2023-06-12 05:28:56.028 1 INFO tacker.wsgi [-] (1) wsgi starting up on http://0.0.0.0:9890 2023-06-12 05:30:07.021 1 DEBUG tacker.wsgi [-] (1) accepted ('10.100.2.3', 52508) server /usr/local/lib/python3.10/dist-packages/eventlet/wsgi.py:1004 2023-06-12 05:30:07.027 1 INFO tacker.sol_refactored.api.wsgi [-] PATCH http://10.244.121.249:9890/vnffm/v1/alarms/f23f39cc-f9a4-45bf-9a87-2f8148541b0a 2023-06-12 05:30:07.052 1 DEBUG tacker.sol_refactored.common.coordinate [None req-a19d9e85-ba9f-4f6b-938a-3fde04bce62b - - - - - -] resources f23f39cc-f9a4-45bf-9a87-2f8148541b0a locked. wrapper /usr/local/lib/python3.10/dist-packages/tacker/sol_refactored/common/coordinate.py:95 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi [None req-a19d9e85-ba9f-4f6b-938a-3fde04bce62b - - - - - -] Unknown error: oslo_db.sqlalchemy.enginefacade.AlreadyStartedError: this TransactionFactory is already started 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi Traceback (most recent call last): 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/tacker/sol_refactored/api/wsgi.py", line 134, in __call__ 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi result = self._dispatch(request, action, args) 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/tacker/sol_refactored/api/wsgi.py", line 170, in _dispatch 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi return controller_method(request=request, **action_args) 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/tacker/sol_refactored/api/validator.py", line 48, in wrapper 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi return func(*args, **kwargs) 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/tacker/sol_refactored/common/coordinate.py", line 96, in wrapper 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi return func(*args, **kwargs) 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/tacker/sol_refactored/controller/vnffm_v1.py", line 101, in update 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi with context.session.begin(subtransactions=True): 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/tacker/context.py", line 183, in session 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi self._session = db_api.get_session() 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/tacker/db/api.py", line 46, in get_session 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi facade = _create_facade_lazily() 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/tacker/db/api.py", line 32, in _create_facade_lazily 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi context_manager.configure(sqlite_fk=True, **cfg.CONF.database) 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 812, in configure 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi self._factory.configure(**kw) 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 319, in configure 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi self._configure(False, kw) 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 332, in _configure 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi raise AlreadyStartedError( 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi oslo_db.sqlalchemy.enginefacade.AlreadyStartedError: this TransactionFactory is already started 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi 2023-06-12 05:30:07.159 1 INFO tacker.sol_refactored.api.wsgi [None req-a19d9e85-ba9f-4f6b-938a-3fde04bce62b - - - - - -] http://10.244.121.249:9890/vnffm/v1/alarms/f23f39cc-f9a4-45bf-9a87-2f8148541b0a returned with HTTP 500 2023-06-12 05:30:07.159 1 INFO tacker.wsgi [None req-a19d9e85-ba9f-4f6b-938a-3fde04bce62b - - - - - -] 10.100.2.3 - - [12/Jun/2023 05:30:07] "PATCH /vnffm/v1/alarms/f23f39cc-f9a4-45bf-9a87-2f8148541b0a HTTP/1.1" 500 309 0.135927 === - This issue does not occur with a regular Tacker Deploy (manual deployment or devstack), but kola-ansible or container envs only. [Temporary solution] - Remove the codes, but we investigate to check if there are any other impacts. To solve these problems will require the cooperation of not only Tacker but also oslo and kolla-ansible projects. We invite your feedback on these problems and especially welcome your share of similar situation info and solutions. Best Regards, Yuta -- Yuta Kazato NTT Network Innovation Center. tel: +81-422-59-6754 mail: yuta.kazato at ntt.com From dwilde at redhat.com Mon Jul 10 13:06:55 2023 From: dwilde at redhat.com (Dave Wilde) Date: Mon, 10 Jul 2023 08:06:55 -0500 Subject: [keystone] Weekly Meeting Move Proposal Message-ID: <7f33c15e-f5aa-4f1a-9251-99c64d23c072@Spark> Hello, I'd like to propose moving the Keystone weekly meeting from Tuesdays at 15:00 UTC to Wednesdays at 15:00 UTC.??Please let me know any feedback on this proposal, without objection our first Wednesday meeting will be 19-Jul-2023 at 15:00 UTC. Thank you! /Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdhasman at redhat.com Mon Jul 10 14:30:49 2023 From: rdhasman at redhat.com (Rajat Dhasmana) Date: Mon, 10 Jul 2023 20:00:49 +0530 Subject: [cinder] Extending driver merge deadline to 14th July Message-ID: Hi, As discussed in the last cinder meeting[1], due to less review bandwidth, Cinder drivers proposed for 2023.2 Bobcat didn't get much reviews. As a result, we will be extending the deadline to this week and the new date for driver merge deadline will be 14 July, 2023. Here is a list of all the drivers proposed for 2023.1 Bobcat cycle[2], feel free to review and add your comment even if you are not a core reviewer, all reviews are appreciated. [1] https://meetings.opendev.org/irclogs/%23openstack-meeting-alt/%23openstack-meeting-alt.2023-07-05.log.html#t2023-07-05T14:13:47 [2] https://etherpad.opendev.org/p/cinder-2023-2-bobcat-drivers Thanks Rajat Dhasmana -------------- next part -------------- An HTML attachment was scrubbed... URL: From knikolla at bu.edu Mon Jul 10 20:11:24 2023 From: knikolla at bu.edu (Nikolla, Kristi) Date: Mon, 10 Jul 2023 20:11:24 +0000 Subject: [tc] Technical Committee next weekly meeting on July 11, 2023 Message-ID: Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held on Tuesday, July 11, 2023 at 1800 UTC on Zoom. Meeting link and agenda can be found at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting Please find below the current agenda for the meeting. ? Roll call ? Follow up on past action items ? knikolla will write alternative proposals for Extended Maintenance. ? Extended Maintenance ? https://etherpad.opendev.org/p/openstack-exteneded-maintenance ? Gate health check ? Open Discussion and Reviews ? https://review.opendev.org/q/projects:openstack/governance+is:open Thank you, Kristi Nikolla From johnsomor at gmail.com Mon Jul 10 21:03:23 2023 From: johnsomor at gmail.com (Michael Johnson) Date: Mon, 10 Jul 2023 14:03:23 -0700 Subject: [designate] About delegating PTR using shared zones on a shared internal network In-Reply-To: <5059613b-2c76-c5a5-1281-42970a190def@debian.org> References: <5059613b-2c76-c5a5-1281-42970a190def@debian.org> Message-ID: Hi Thomas, Thanks for the kind words. Using a sink work certainly work. This is kind of the "next step", to understand how people do IPAM and what is the right way to automate this. Ideally you would allocate blocks of IPs to a project, but I realize many deployments are not set up that way. The other thing to think about is if you use one of the neutron extensions that create records based on the network domain and port dns name. You could get into an interesting situation where the configuration in neutron no longer matches the PTR records if you allow users to "own" their PTR records in Designte. Michael On Thu, Jul 6, 2023 at 2:26?AM Thomas Goirand wrote: > > Hi there! > > I have just watched Michael Johnson's video of the presentation at the > last Vancouver summit: > https://www.youtube.com/watch?v=zdhvNezU1w4 > > This was very interesting, and thanks Michael, for that awesome feature. > > In our use case, we created an OpenStack internal network, that is > shared for all of our customers, so that they can allocate public IPs > assigned directly to their instances, without the need to create a > router and use NAT. > > For the moment (since we don't have the shared zones feature yet), we > manage the PTR record through support tickets. We'd like to move > forward, and delegate the PTR handling to our customer, whenever they > allocate a port on that specific shared network. How can this be done? > > Should we create a new sink of our own, so that we would automate the > the shared zone creation, so that it would delegate the in-addr.arpa. > range, when a port on that network is created, and the same way, remove > the delegation (and records) whenever the port is deleted? Would this be > something similar, probably, we the sink we implemented to automatically > delete PTR records in floating IPs, available here [1]? > > [1] > https://salsa.debian.org/openstack-team/services/designate/-/blob/debian/antelope/debian/patches/add-new-floatingip-handler.patch > > I'd love to have some pointers / examples, so we could implement this > directly in Designate, so that the feature could be shared in the > OpenStack community. For sure, we aren't the only public cloud provider > with the use case... > > Cheers, > > Thomas Goirand (zigo) > From kennelson11 at gmail.com Mon Jul 10 23:23:50 2023 From: kennelson11 at gmail.com (Kendall Nelson) Date: Mon, 10 Jul 2023 18:23:50 -0500 Subject: vPTG October 2023 Dates & Registration Kickoff Message-ID: Hello Everyone! The next PTG[1] will take place October 23-27, 2023 virtually! Registration is now open[2] and free for all to attend. Keep an eye out in the coming weeks because I will send along the team sign up to collect groups interested in participating. Thanks! -Kendall Nelson(diablo_rojo) [1] https://openinfra.dev/ptg/ [2] http://ptg2023.openinfra.dev/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ueha.ayumu at fujitsu.com Tue Jul 11 00:08:01 2023 From: ueha.ayumu at fujitsu.com (Ayumu Ueha (Fujitsu)) Date: Tue, 11 Jul 2023 00:08:01 +0000 Subject: [kuryr][tacker] fails to bump the k8s version to v1.26 In-Reply-To: References: Message-ID: Hi kuryr team, Do you know anything about the content of the following thread I sent? https://lists.openstack.org/pipermail/openstack-discuss/2023-July/034326.html I'd appreciate it if you could help us. Thank you. Best Regards, Ayumu From: Ayumu Ueha (Fujitsu) Sent: Tuesday, July 4, 2023 4:54 PM To: 'openstack-discuss at lists.openstack.org' Subject: [kuryr][tacker] fails to bump the k8s version to v1.26 Hi kuryr-kubetenetes team, Tacker uses the kuryr-kubernetes as the setup for the kubernetes Environment. In the Bobcat version, we will bump the supported version of kubenetes to 1.26.6, and when I tried to bump the version in devstack's local.conf by the patch [1], the following error occurred. (please refer the details to Zuul log [2]) ============================== [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [kubelet-check] Initial timeout of 40s passed. Unfortunately, an error has occurred: timed out waiting for the condition This error is likely caused by: - The kubelet is not running - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled) If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands: - 'systemctl status kubelet' - 'journalctl -xeu kubelet' Additionally, a control plane component may have crashed or exited when started by the container runtime. To troubleshoot, list all containers using your preferred container runtimes CLI. Here is one example how you may list all running Kubernetes containers by using crictl: - 'crictl --runtime-endpoint unix:///var/run/crio/crio.sock ps -a | grep kube | grep -v pause' Once you have found the failing container, you can inspect its logs with: - 'crictl --runtime-endpoint unix:///var/run/crio/crio.sock logs CONTAINERID' error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster ============================== I know it is not yet supporting version 1.26 of the K8S at the kuryr-kubernetes as for now, but do you know any way to avoid the above error? I suspect this is due to a change from version 1.25, but I'm not sure which one is affecting... Also, when will you support the K8S v 1.26 at the kuryr-kubernetes? Will the Bobcat release support it? Please kindly let me know if you know anything. Thank you. [1] https://review.opendev.org/c/openstack/tacker/+/886935 Change the parameters: KURYR_KUBERNETES_VERSION: 1.25.6 to 1.26.6 CRIO_VERSION: 1.25 to 1.26 [2] https://zuul.opendev.org/t/openstack/build/1a4061da74e640368da133ba219b54d9/log/controller-k8s/logs/devstacklog.txt#9761-9816 Best Regards, Ueha -------------- next part -------------- An HTML attachment was scrubbed... URL: From masahito.muroi at linecorp.com Tue Jul 11 01:53:05 2023 From: masahito.muroi at linecorp.com (=?utf-8?B?TWFzYWhpdG8gTXVyb2k=?=) Date: Tue, 11 Jul 2023 10:53:05 +0900 Subject: =?utf-8?B?UmU6IFtvc2xvXVtsYXJnZXNjYWxlLXNpZ10gSFRUUCBiYXNlIGRpcmVjdCBSUEMgb3Nsbw==?= =?utf-8?B?Lm1lc3NhZ2luZyBkcml2ZXIgY29udHJpYnV0aW9u?= In-Reply-To: References: <926bd380c381258c1885782ee5735d@cweb02.nmdf.nhnsystem.com> Message-ID: Hi, Takashi, Herve and Stephan, Thank you for the summit wrap-up, Takashi. We have addressed all the unclear points in the spec and updated following the summit offsite talk. I hope the latest spec enhances your thoughts about the new driver details. Please review the spec file again. https://review.opendev.org/c/openstack/oslo-specs/+/885809 Thanks, Masahito -----Original Message----- From: "Takashi Kajinami" To: "Masahito Muroi"; Cc: "Jay Faulkner"; "Julia Kreger"; ; "Herve Beraud"; "Arnaud Morin"; Sent: 2023/06/29(?) 13:46 (GMT+09:00) Subject: Re: [oslo][largescale-sig] HTTP base direct RPC oslo.messaging driver contribution Stephen and I from oslo core met the LINE team at Vancouver and discussed this topic, and in short we(at least Stephen and I) agreed with the proposal. Let me dump some topics we discussed there. # Sorry I thought I posted this somewhere early... Some concerns were raised about adding consul as a new core component, but we agreed with using it as the initial implementation. There were some suggestions to rely on existing service records stored in DB but we need some amount of work, to address the following points. - Not all OpenStack services store service records in DB. (eg ceilometer) - Even though some services store service records in DB, the available information there is not enough - We have to maintain mapping between host and reachable endpoint - Some services spawns worker and need endpoint *per worker*, and we need endpoint list for all workers Random port assignment from range is another topic we discussed there, but we eventually agreed that this is the required approach, mainly because it's hard to allocate port numbers for all services running in a single node(required number of ports can be different based on services/workers/etc). So we agreed that the proposed way to dynamically allocate available ports from the given range is a good solution. Also, LINE already deployed this feature in their production and it has been proven to work well in scale. As we don't see any technical blockers at this moment, we agreed with moving forward with the proposed architecture. There can be some improvement like adding the driver mechanism for cluster backend but, we would avoid expanding our current scope, until someone is really interested in working on such topics. Finally we suggested updating the test plan and adding a devstack(at least single node) job in the gate so that we can properly maintain the feature. Other people might have additional thoughts, but hope the above outcomes make the current proposal more clear to all. Thank you, Takashi On Sat, Jun 10, 2023 at 6:14 PM Masahito Muroi wrote: Hi all, We have pushed the spec. Please feel free to review it. https://review.opendev.org/c/openstack/oslo-specs/+/885809 best regards, Masahito -----Original Message----- From: "Jay Faulkner" To: "Julia Kreger"; Cc: "Masahito Muroi"; "Takashi Kajinami"; ; "Herve Beraud"; "Arnaud Morin"; Sent: 2023/06/07(?) 07:31 (GMT+09:00) Subject: Re: [oslo][largescale-sig] HTTP base direct RPC oslo.messaging driver contribution I'm interested in this as well, please add me to the spec if you need additional brains :). I'll also be at the summit if you'd like to discuss any of it in person. -- Jay Faulkner Ironic PTL On Tue, Jun 6, 2023 at 3:14 PM Julia Kreger wrote: Jumping in because the thread has been rather reminiscent of the json-rpc messaging feature ironic carries so our users don't have to run with rabbit. I suspect Ironic might be happy to propose it to oslo.messaging if this http driver is acceptable. Please feel free to add me as a reviewer on the spec. -Julia On Tue, Jun 6, 2023 at 2:10 PM Masahito Muroi wrote: Hi, Thank you everyone for the kindly reply. I got the PTG situation. Submitting the spec seems to be a nice first step. We don't have public repository of the driver because of internal repository structure reason. The repository is really stick to the current internal repository structure now. Cleaning up repository would take time so that we didn't do the extra tasks. best regards. Masahito -----Original Message----- From: "Takashi Kajinami" To: "Masahito Muroi"; Cc: ; "Herve Beraud"; Sent: 2023/06/06(?) 18:50 (GMT+09:00) Subject: Re: [oslo] HTTP base direct RPC oslo.messaging driver contribution Hello, This is very interesting and I agree having the spec would be the good way to move this forward. We have not requested oslo sessions in the upcoming PTG but Stephen and I are attending it so will be available for the discussion. Because some other cores such as Herve won't be there, we'd need to continue further discussions after PTG in spec review, but if that early in-person discussion sounds helpful for you then I'll reserve a table. Thank you, Takashi On Tue, Jun 6, 2023 at 4:48 PM Herve Beraud wrote: Hello, Indeed, Oslo doesn't have PTG sessions. Best regards Le lun. 5 juin 2023 ? 10:42, Masahito Muroi a ?crit : Hello Herve, Thank you for the quick replying. Let us prepare the spec and submit it. btw, does olso team have PTG in the up-comming summit? We'd like to get a quick feedback of the spec if time is allowed in the PTG. But it looks like oslo team won't have PTG there. best regards, Masahito -----Original Message----- From: "Herve Beraud" To: "????"; Cc: ; Sent: 2023/06/05(?) 17:21 (GMT+09:00) Subject: Re: [oslo] HTTP base direct RPC oslo.messaging driver contribution Hello Masahito, Submission to oslo-spec is a good starting point. Best regards Le lun. 5 juin 2023 ? 10:04, ???? a ?crit : Hi oslo team, We'd like to contribute HTTP base direct RPC driver to the oslo.messaging community. We have developed the HTTP base driver internally. We have been using the driver in the production with over 10K hypervisors now. I checked the IRC meeting log of the oslo team[1], but there is no regluar meeting in 2023. Is it okay to submit oslo-spec[2] to propose the driver directly, or is there another good place to discuss the feature before submitting a spec? 1. https://meetings.opendev.org/#Oslo_Team_Meeting 2. https://opendev.org/openstack/oslo-specs best regards, Masahito -- Herv? Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ -- Herv? Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ygk.kmr at gmail.com Tue Jul 11 07:43:24 2023 From: ygk.kmr at gmail.com (Gk Gk) Date: Tue, 11 Jul 2023 13:13:24 +0530 Subject: Need help Message-ID: Hi Folks, I work on openstack platforms. Can anyone suggest some good books or resources which will help me in becoming a cloud solutions architect ? What would this path look like ? What would be my responsibilities ? Thanks Kumar -------------- next part -------------- An HTML attachment was scrubbed... URL: From noonedeadpunk at gmail.com Tue Jul 11 08:44:40 2023 From: noonedeadpunk at gmail.com (Dmitriy Rabotyagov) Date: Tue, 11 Jul 2023 10:44:40 +0200 Subject: [ptl] Need PTL volunteer for OpenStack Vitrage In-Reply-To: <1877b98576a.f5f7e32e587608.323404367955418499@ghanshyammann.com> References: <1870a6b95c5.105ea64ba994248.8904640311035076666@ghanshyammann.com> <1877b98576a.f5f7e32e587608.323404367955418499@ghanshyammann.com> Message-ID: Hey Eyal, Can you kindly add me to the core team of Vitrage, as I've stepped in as a PTL for this cycle? Thanks in advance. ??, 13 ???. 2023??. ? 19:13, Ghanshyam Mann : > > Thanks Eyal for maintaining it for so long even in your free time. We really > appreciate your effort in this project and understand your priority. > > Let me start calling out for other maintainers if any otherwise last option is to > propose retirement. > > > -gmann > ---- On Thu, 13 Apr 2023 01:12:02 -0700 Eyal B wrote --- > > Hi,Unfortunately I won't be able to continue as vitrage PTL, I don't have the time any more. I have been doing it on my free time for several years now but I cannot continue. > > BR Eyal > > On Wed, 22 Mar 2023, 18:43 Ghanshyam Mann, gmann at ghanshyammann.com> wrote: > > Hi Eyal, > > > > I am reaching out to you as you were PTL for OpenStack Vitrage project in the last cycle. > > > > There is no PTL candidate for the next cycle (2023.2), and it is on the leaderless project list. Please > > check if you or anyone you know would like to lead this project. > > > > - https://etherpad.opendev.org/p/2023.2-leaderless > > > > Also, if anyone else would like to help leading this project, this is time to let TC knows. > > > > -gmann > > > From Nils.Magnus at t-systems.com Tue Jul 11 09:57:25 2023 From: Nils.Magnus at t-systems.com (Nils.Magnus at t-systems.com) Date: Tue, 11 Jul 2023 09:57:25 +0000 Subject: AW: Need help In-Reply-To: References: Message-ID: Hi Kumar, not meaning to be offensive, but having a rough idea how to become an architect is a good first step ? That having said, it?s probably not just reading a good book or booking a good training, but doing practical exercises over and over again. However, there are good news: There?s plenty of material available and there are probably many paths that lead to success. I can only outline four suggestions that helped me personally in the past couple of years: * There are several OpenStack introduction videos available on YouTube. Don?t be afraid watching some videos, even if they are a few years old and seem to be outdated. If you keep an open mind, this is not so important when you actually start your learning journey. Try to get an overview of the structure and top-level-architecture of OpenStack. One example (of many): https://www.youtube.com/watch?v=8kADjGCuSVI * There is one notorious architectural diagram that is available in many places in the Internet. Here is one link: https://www.pinterest.de/pin/1196337376870638/ While this might not be the premier example of creating an easy-to-understand diagram, it has some inner beauty. Make yourself familiar with it. This does not mean that you need to be able to explain every single box or arrow (and there are plenty of them!), but understand the *idea* of it. When you want to attach a volume to a server instance with the command line tool, try to figure out your exact path through this diagram, for example. * You need to exercise. Practically. With your fingertips on the keyboard. Not just reading books. There are basically two easy ways to do so: One is to figure out an existing public cloud that offers some free tier for beginners or students. Even with as little as $100 of virtual money you can do a lot experiments for a start. The alternative is to install a simplified version of OpenStack on your own computer. If you are really new, I don?t recommend to start with installing ?the real thing? directly, as there might be many minor pitfalls that don?t pay into your learning experience. Something I like is Ubuntu?s ?microstack?, which is a ?snap? package for the Linux distro (nowadays also labelled as ?sunbeam? if I understood the session right during the last Summit). Some people like Ubuntu snaps, some don?t. We could engage in a lengthy discussion if they are the ?right? way to package software, but setting up a tiny instance on your single notebook computer was never so easy. It takes you less than 15 minutes to have a setup up and running. If you progress on your journey, the microstack approach may have some limitations, but it?s a great start. * There is a browser-based UI to work with OpenStack (?Horizon?). It?s okay to use that for very first steps, but keep in mind that in the long run you picked the cloud principle to be able to automate tasks. Using the CLI (?OpenStack Client?, ?OSC?, or just ?openstack? on the command line) is an important first step in this direction. Familiarize yourself also with Python and its language bindings to the OpenStack API, called ?OpenStack SDK?. It?s not necessary to write huge programs, but writing a script that just lists all your volumes or servers might be insightful: https://www.youtube.com/watch?v=EKadc-cxZc0 (second talk). I hope that helps! Regards, Nils -- Nils Magnus T-Systems International GmbH Open Telekom Cloud Delivery ? Chapter Architecture Holzhauser Str. 4-8, 13509 Berlin, Germany +49 30 8353 24446 +49 170 4189 377 (mobile) Von: Gk Gk Gesendet: Dienstag, 11. Juli 2023 09:43 An: OpenStack Development Mailing List (not for usage questions) Betreff: Need help Hi Folks, I work on openstack platforms. Can anyone suggest some good books or resources which will help me in becoming a cloud solutions architect ? What would this path look like ? What would be my responsibilities ? Thanks Kumar -------------- next part -------------- An HTML attachment was scrubbed... URL: From ygk.kmr at gmail.com Tue Jul 11 10:52:17 2023 From: ygk.kmr at gmail.com (Gk Gk) Date: Tue, 11 Jul 2023 16:22:17 +0530 Subject: Need help In-Reply-To: References: Message-ID: Thanks Nils... I have been using and working on openstack for nine years now. These days I am stuck with how to shape my career with my present skill set of Linux, Openstack in general and which technology will help me in transitioning to a higher level. I hate management based positions. Instead I love to tackle tech stuff. The above advice sounds good to me. And one other technology I am trying to learn now is Artificial Intelligence and Machine Learning. Will that help me in combining that with Openstack ? Please comment on how that works and what the near future would look like with this ? Appreciate your time and help, Kumar On Tue, Jul 11, 2023 at 3:27?PM wrote: > Hi Kumar, > > > > not meaning to be offensive, but having a rough idea how to become an > architect is a good first step ? > > > > That having said, it?s probably not just reading a good book or booking a > good training, but doing practical exercises over and over again. However, > there are good news: There?s plenty of material available and there are > probably many paths that lead to success. I can only outline *four > suggestions* that helped me personally in the past couple of years: > > > > - There are several OpenStack introduction videos available on > YouTube. Don?t be afraid watching some videos, even if they are a few years > old and seem to be outdated. If you keep an open mind, this is not so > important when you actually start your learning journey. Try to get an > overview of the structure and top-level-architecture of OpenStack. > One example (of many): https://www.youtube.com/watch?v=8kADjGCuSVI > > > > - There is one notorious architectural diagram that is available in > many places in the Internet. Here is one link: > https://www.pinterest.de/pin/1196337376870638/ While this might not be > the premier example of creating an easy-to-understand diagram, it has some > inner beauty. Make yourself familiar with it. This does not mean that you > need to be able to explain every single box or arrow (and there are plenty > of them!), but understand the **idea** of it. When you want to attach > a volume to a server instance with the command line tool, try to figure out > your exact path through this diagram, for example. > > > > - You need to exercise. Practically. With your fingertips on the > keyboard. Not just reading books. There are basically two easy ways to do > so: One is to figure out an existing public cloud that offers some free > tier for beginners or students. Even with as little as $100 of virtual > money you can do a lot experiments for a start. The alternative is to > install a simplified version of OpenStack on your own computer. If you are > really new, I don?t recommend to start with installing ?the real thing? > directly, as there might be many minor pitfalls that don?t pay into your > learning experience. Something I like is Ubuntu?s ?microstack?, which is a > ?snap? package for the Linux distro (nowadays also labelled as ?sunbeam? if > I understood the session right during the last Summit). Some people like > Ubuntu snaps, some don?t. We could engage in a lengthy discussion if they > are the ?right? way to package software, but setting up a tiny instance on > your single notebook computer was never so easy. It takes you less than 15 > minutes to have a setup up and running. If you progress on your journey, > the microstack approach may have some limitations, but it?s a great start. > > > > - There is a browser-based UI to work with OpenStack (?Horizon?). It?s > okay to use that for very first steps, but keep in mind that in the long > run you picked the cloud principle to be able to automate tasks. Using the > CLI (?OpenStack Client?, ?OSC?, or just ?openstack? on the command line) is > an important first step in this direction. Familiarize yourself also with > Python and its language bindings to the OpenStack API, called ?OpenStack > SDK?. It?s not necessary to write huge programs, but writing a script that > just lists all your volumes or servers might be insightful: > https://www.youtube.com/watch?v=EKadc-cxZc0 (second talk). > > > > I hope that helps! > > > > Regards, > > > > Nils > > > > -- > > Nils Magnus > > *T-Systems International GmbH* > Open Telekom Cloud Delivery ? Chapter Architecture > > Holzhauser Str. 4-8, 13509 Berlin, Germany > +49 30 8353 24446 > > +49 170 4189 377 (mobile) > > > > > > > > > > *Von:* Gk Gk > *Gesendet:* Dienstag, 11. Juli 2023 09:43 > *An:* OpenStack Development Mailing List (not for usage questions) < > openstack-dev at lists.openstack.org> > *Betreff:* Need help > > > > Hi Folks, > > > > I work on openstack platforms. Can anyone suggest some good books or > resources which will help me in becoming a cloud solutions architect ? What > would this path look like ? What would be my responsibilities ? > > > > Thanks > > Kumar > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay at gr-oss.io Tue Jul 11 11:06:11 2023 From: jay at gr-oss.io (Jay Faulkner) Date: Tue, 11 Jul 2023 12:06:11 +0100 Subject: [all][tc] Proposals for reforming extended maintenance In-Reply-To: References: Message-ID: There is interesting and useful discussion in the governance patches posted by Kristi. I strongly recommend folks take a look and participate. The more independent opinions we get, the better off the end result will be. Thanks, Jay Faulkner TC Member On Fri, Jul 7, 2023 at 4:25?PM Nikolla, Kristi wrote: > Hi all, > > There have been quite a few discussions related extended maintenance in > the last few months[0][1]. > > I have made a first-go at trying to write a policy for extended > maintenance moving forward which tries to preserve the value being brought > by extended maintenance while strengthening the requirements for keeping > branches under that phase. [2] > This was based on multiple discussions that I was present at and feedback > that I have received from various teams and community members with which I > talked, and is my interpretation of that feedback into something that looks > like policy. > > I have also written two opposing policy proposals about what do to with > the current branches under extended maintenance. One which attempts to > apply the new requirements defined in the first proposal almost > immediately[3], and one that aims for a more gradual EOL[4]. > > Please provide feedback on the proposals and help shape up the future of > extended maintenance. > > Thank you, > Kristi Nikolla > > [0]. > https://lists.openstack.org/pipermail/openstack-discuss/2023-June/033980.html > [1]. https://etherpad.opendev.org/p/vancouver-2023-em > [2]. https://review.opendev.org/c/openstack/governance/+/887966 > [3]. https://review.opendev.org/c/openstack/governance/+/887968 > [4]. https://review.opendev.org/c/openstack/governance/+/887969 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ygk.kmr at gmail.com Tue Jul 11 11:58:29 2023 From: ygk.kmr at gmail.com (Gk Gk) Date: Tue, 11 Jul 2023 17:28:29 +0530 Subject: Need help Message-ID: Hi All, If I use the openstacksdk to connect to an openstack cloud, I have to use clouds.yaml file for specifying the cloud configuration which includes username and password as well. Since its a plain text file, how can I mask the password mentioned in clouds.yaml file for security purposes? Thanks Kumar -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralonsoh at redhat.com Tue Jul 11 13:41:52 2023 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Tue, 11 Jul 2023 15:41:52 +0200 Subject: [neutron] Bug deputy 3 July - 9 July Message-ID: Hello Neutrinos: This is the list of patches opened last week: Critical: * https://bugs.launchpad.net/neutron/+bug/2025753. "Job openstack-tox-py310-with-neutron-lib-master is failing every day 02.07.2023". Patch proposed: https://review.opendev.org/c/openstack/neutron/+/887942 High: * https://bugs.launchpad.net/neutron/+bug/2025618. "[UT] "test_negative_update_floatingip_port_forwarding" does not create the ports in the correct subnet". Patch merged: https://review.opendev.org/c/openstack/neutron/+/887495 * https://bugs.launchpad.net/neutron/+bug/2025946. "Neutron 504 Gateway Timeout Openstack Kolla-Ansible : Ussuri". *Not assigned*. Medium: * https://bugs.launchpad.net/neutron/+bug/2025637. "[ovn-octavia-provider] FIP traffic not distributed for members' FIP after lb cascade deletion". Patch merged: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/886611 * https://bugs.launchpad.net/neutron/+bug/2026122. "neutron-l3-agent permission denied when execute (list_network_namespaces)". Assigned, under investigation right now, there is also a configuration fix tested. * https://bugs.launchpad.net/neutron/+bug/2026182. "Add support for the service role in neutron API policies". Assigned, under development right now. * https://bugs.launchpad.net/neutron/+bug/2026264. " "--target" parameter in "network log object create" is only expected for ML2/OVS". Assigned, under development right now. * https://bugs.launchpad.net/neutron/+bug/2026775. "Metadata agents do not parse X-Forwarded-For headers properly". Patch proposed: https://review.opendev.org/c/openstack/neutron/+/888067. * https://bugs.launchpad.net/neutron/+bug/2026825. " [OVN] Disable LSP_OPTIONS_MCAST_FLOOD_REPORTS for LSPs". Patch proposed: https://review.opendev.org/c/openstack/neutron/+/888127. Low: * https://bugs.launchpad.net/neutron/+bug/2026547. "[RBAC] Remove "AddressScope" shared field". Assigned. Wishlist: * https://bugs.launchpad.net/neutron/+bug/2025969. "bgp os_ken ssh_console support". *Not assigned*. * https://bugs.launchpad.net/neutron/+bug/2026286. "Missing documentation for notification events". *Not assigned.* * https://bugs.launchpad.net/neutron/+bug/2026489. " [RFE][quota] Implement "router_route" quota using the Neutron quota engine, replacing "max_routes" config option". *Not assigned.* Incomplete: * https://bugs.launchpad.net/neutron/+bug/2025587. "neutron request body or parameters cause "Internal Server Error"" Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From swogatpradhan22 at gmail.com Tue Jul 11 18:10:48 2023 From: swogatpradhan22 at gmail.com (Swogat Pradhan) Date: Tue, 11 Jul 2023 23:40:48 +0530 Subject: instance console something went wrong, connection is closed | Wallaby DCN In-Reply-To: References: <55eff3d6-b852-840e-80e0-26cbb5a58aac@gmail.com> <3a7830e4-9f09-210c-c1c5-0ee27d8945ff@gmail.com> Message-ID: Hi Melanie, After searching through i found that no service is exposing port 5901. Only when I migrate the instance or do any kind of activity port 5901 is exposed on the said hypervisor node and the vm console is accessible. So, I believe your point 'Something about redoing the port bindings is helping the situation sometimes.' is correct. But how do I troubleshoot this? With regards, Swogat Pradhan On Fri, Jun 30, 2023 at 12:29?AM melanie witt wrote: > On 06/25/23 03:50, Swogat Pradhan wrote: > > Hi, > > After doing a console url show after migration, I am still unable to > > access the console. > > > > My site consists of 1 central site and 2 DCN sites. Consoles for > > central and DCN02 are working fine without any issues. > > But when i am creating an instance for DCN01 the console for the > > instance is not coming up (attached image for reference). > > > > Today I created 3 different VM's using the same flavor, image, security > > group, the instances were created in the same compute host. Console was > > not accessible, so I shelved and unshelved all 3 instances, after which > > I was able to access the console for 2 of those VM's and am still unable > > to access the console of the 3rd VM no matter what I do. > > Apologies for the delayed reply. > > It sounds like there may be some type of problem with regard to the > network connection from the novnc console proxy service to the DCN01 > site rather than something with the console proxy itself given that > things work fine with DCN02 and central. You're seeing "Cannot write > data: Broken pipe" in the connection between the compute host and the > console proxy showing a connection being broken after being established. > > As for why shelve and unshelve sometimes helps, it may be because the > network port bindings are updated to inactive during the shelve and then > they are updated to active during the unshelve. Something about redoing > the port bindings is helping the situation sometimes. > > It may be worthwhile to check if there is anything different between the > networks/ports the instances with working consoles have vs what the > instances with non-working consoles have. > > -melwitt > > > On Sat, Jun 24, 2023 at 2:00?AM melanie witt > > wrote: > > > > On 06/22/23 20:07, Swogat Pradhan wrote: > > > Hi Mel, > > > Thank you for your response. > > > I am facing issues with the instance console (vnc) in the > openstack > > > dashboard, Most of the time I shelve the instance and unshelve the > > > instance to get the console. > > > But there are some VM's I created which are not working even after > > > shelve/unshelve. > > > > > > I have used the same director to deploy a total of a central and > > 2 edge > > > sites. > > > This issue is happening on a single edge site. > > > Cold Migration also helps in some situations. > > > > OK, you didn't mention whether requesting a new console 'openstack > > console url show --vnc ' gets you a working console after a > > migration (or other event where you see the console stop working). > I'm > > trying to determine whether the behavior you're seeing is expected > or a > > bug. After an instance is moved to a different compute node than the > > one > > it was on when the console was started, that console is not expected > to > > work anymore. And a new console needs to be started. > > > > Can you give steps for reproducing the issue? Maybe that will provide > > more clarity. > > > > -melwitt > > > > > On Fri, Jun 23, 2023 at 12:42?AM melanie witt > > > > >> wrote: > > > > > > On 06/22/23 01:08, Swogat Pradhan wrote: > > > > Hi, > > > > Please find the below log: > > > > [root at dcn01-hci-1 libvirt]# cat virtqemud.log > > > > 2023-06-22 07:40:01.575+0000: 350319: error : > > > virNetSocketReadWire:1804 > > > > : End of file while reading data: Input/output error > > > > 2023-06-22 07:40:01.575+0000: 350319: error : > > > virNetSocketWriteWire:1844 > > > > : Cannot write data: Broken pipe > > > > > > > > I think this is causing the problem of not getting the > > instance > > > console. > > > > > > When you say "instance console" are you referring to an > > interactive > > > console like VNC or are you talking about the console log for > the > > > instance? > > > > > > If it's the interactive console, if you have a console open > > and then > > > migrate the instance, that console will not be moved along > > with the > > > instance. When a user requests a console, the console proxy > > service > > > establishes a connection to the compute host where the > > instance is > > > located. The proxy doesn't know when an instance has been > > moved though, > > > so if the instance is moved, the user will need to request a > new > > > console > > > (which will establish a new connection to the new compute > host). > > > > > > Is that the behavior you are seeing? > > > > > > -melwitt > > > > > > > On Fri, Jun 2, 2023 at 11:27?AM Swogat Pradhan > > > > > > > > > > > > > > > >>> wrote: > > > > > > > > Update: > > > > If the i am performing any activity like migration or > > resize > > > of an > > > > instance whose console is accessible, the console > becomes > > > > inaccessible giving out the following error : > > something went > > > wrong, > > > > connection is closed > > > > > > > > The was 1 other instance whose console was not > > accessible and > > > i did > > > > a shelve and unshelve and suddenly the instance > > console became > > > > accessible. > > > > > > > > This is a peculiar behavior and i don't understand > > where is > > > the issue . > > > > > > > > With regards, > > > > Swogat Pradhan > > > > > > > > On Fri, Jun 2, 2023 at 11:19?AM Swogat Pradhan > > > > > > > > > > > > > > > >>> wrote: > > > > > > > > Hi, > > > > I am creating instances in my DCN site and i am > > unable to get > > > > the console sometimes, error: something went wrong, > > > connection > > > > is closed > > > > > > > > I have 3 instances now running on my hci02 node > > and there is > > > > console access on 1 of the vm's and the rest two i > > am not > > > > getting the console, i have used the same flavor, > > same image > > > > same security group for the VM's. > > > > > > > > Please suggest what can be done. > > > > > > > > With regards, > > > > Swogat Pradhan > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From swogatpradhan22 at gmail.com Tue Jul 11 18:31:06 2023 From: swogatpradhan22 at gmail.com (Swogat Pradhan) Date: Wed, 12 Jul 2023 00:01:06 +0530 Subject: instance console something went wrong, connection is closed | Wallaby DCN In-Reply-To: References: <55eff3d6-b852-840e-80e0-26cbb5a58aac@gmail.com> <3a7830e4-9f09-210c-c1c5-0ee27d8945ff@gmail.com> Message-ID: Please ignore the previous email about the port information. So, I believe your point 'Something about redoing the port bindings is helping the situation sometimes.' is correct. But how do I troubleshoot this? With regards, Swogat Pradhan On Tue, Jul 11, 2023 at 11:40?PM Swogat Pradhan wrote: > Hi Melanie, > After searching through i found that no service is exposing port 5901. > Only when I migrate the instance or do any kind of activity port 5901 is > exposed on the said hypervisor node and the vm console is accessible. > > So, I believe your point 'Something about redoing the port bindings is > helping the situation sometimes.' is correct. > > But how do I troubleshoot this? > > With regards, > Swogat Pradhan > > > On Fri, Jun 30, 2023 at 12:29?AM melanie witt wrote: > >> On 06/25/23 03:50, Swogat Pradhan wrote: >> > Hi, >> > After doing a console url show after migration, I am still unable to >> > access the console. >> > >> > My site consists of 1 central site and 2 DCN sites. Consoles for >> > central and DCN02 are working fine without any issues. >> > But when i am creating an instance for DCN01 the console for the >> > instance is not coming up (attached image for reference). >> > >> > Today I created 3 different VM's using the same flavor, image, security >> > group, the instances were created in the same compute host. Console was >> > not accessible, so I shelved and unshelved all 3 instances, after which >> > I was able to access the console for 2 of those VM's and am still >> unable >> > to access the console of the 3rd VM no matter what I do. >> >> Apologies for the delayed reply. >> >> It sounds like there may be some type of problem with regard to the >> network connection from the novnc console proxy service to the DCN01 >> site rather than something with the console proxy itself given that >> things work fine with DCN02 and central. You're seeing "Cannot write >> data: Broken pipe" in the connection between the compute host and the >> console proxy showing a connection being broken after being established. >> >> As for why shelve and unshelve sometimes helps, it may be because the >> network port bindings are updated to inactive during the shelve and then >> they are updated to active during the unshelve. Something about redoing >> the port bindings is helping the situation sometimes. >> >> It may be worthwhile to check if there is anything different between the >> networks/ports the instances with working consoles have vs what the >> instances with non-working consoles have. >> >> -melwitt >> >> > On Sat, Jun 24, 2023 at 2:00?AM melanie witt > > > wrote: >> > >> > On 06/22/23 20:07, Swogat Pradhan wrote: >> > > Hi Mel, >> > > Thank you for your response. >> > > I am facing issues with the instance console (vnc) in the >> openstack >> > > dashboard, Most of the time I shelve the instance and unshelve >> the >> > > instance to get the console. >> > > But there are some VM's I created which are not working even >> after >> > > shelve/unshelve. >> > > >> > > I have used the same director to deploy a total of a central and >> > 2 edge >> > > sites. >> > > This issue is happening on a single edge site. >> > > Cold Migration also helps in some situations. >> > >> > OK, you didn't mention whether requesting a new console 'openstack >> > console url show --vnc ' gets you a working console after a >> > migration (or other event where you see the console stop working). >> I'm >> > trying to determine whether the behavior you're seeing is expected >> or a >> > bug. After an instance is moved to a different compute node than the >> > one >> > it was on when the console was started, that console is not >> expected to >> > work anymore. And a new console needs to be started. >> > >> > Can you give steps for reproducing the issue? Maybe that will >> provide >> > more clarity. >> > >> > -melwitt >> > >> > > On Fri, Jun 23, 2023 at 12:42?AM melanie witt < >> melwittt at gmail.com >> > >> > > >> wrote: >> > > >> > > On 06/22/23 01:08, Swogat Pradhan wrote: >> > > > Hi, >> > > > Please find the below log: >> > > > [root at dcn01-hci-1 libvirt]# cat virtqemud.log >> > > > 2023-06-22 07:40:01.575+0000: 350319: error : >> > > virNetSocketReadWire:1804 >> > > > : End of file while reading data: Input/output error >> > > > 2023-06-22 07:40:01.575+0000: 350319: error : >> > > virNetSocketWriteWire:1844 >> > > > : Cannot write data: Broken pipe >> > > > >> > > > I think this is causing the problem of not getting the >> > instance >> > > console. >> > > >> > > When you say "instance console" are you referring to an >> > interactive >> > > console like VNC or are you talking about the console log >> for the >> > > instance? >> > > >> > > If it's the interactive console, if you have a console open >> > and then >> > > migrate the instance, that console will not be moved along >> > with the >> > > instance. When a user requests a console, the console proxy >> > service >> > > establishes a connection to the compute host where the >> > instance is >> > > located. The proxy doesn't know when an instance has been >> > moved though, >> > > so if the instance is moved, the user will need to request a >> new >> > > console >> > > (which will establish a new connection to the new compute >> host). >> > > >> > > Is that the behavior you are seeing? >> > > >> > > -melwitt >> > > >> > > > On Fri, Jun 2, 2023 at 11:27?AM Swogat Pradhan >> > > > > > > swogatpradhan22 at gmail.com >> > > >> > > > > >> > > > > >>> wrote: >> > > > >> > > > Update: >> > > > If the i am performing any activity like migration or >> > resize >> > > of an >> > > > instance whose console is accessible, the console >> becomes >> > > > inaccessible giving out the following error : >> > something went >> > > wrong, >> > > > connection is closed >> > > > >> > > > The was 1 other instance whose console was not >> > accessible and >> > > i did >> > > > a shelve and unshelve and suddenly the instance >> > console became >> > > > accessible. >> > > > >> > > > This is a peculiar behavior and i don't understand >> > where is >> > > the issue . >> > > > >> > > > With regards, >> > > > Swogat Pradhan >> > > > >> > > > On Fri, Jun 2, 2023 at 11:19?AM Swogat Pradhan >> > > > > > > swogatpradhan22 at gmail.com >> > > >> > > > > >> > > > > >>> wrote: >> > > > >> > > > Hi, >> > > > I am creating instances in my DCN site and i am >> > unable to get >> > > > the console sometimes, error: something went >> wrong, >> > > connection >> > > > is closed >> > > > >> > > > I have 3 instances now running on my hci02 node >> > and there is >> > > > console access on 1 of the vm's and the rest two i >> > am not >> > > > getting the console, i have used the same flavor, >> > same image >> > > > same security group for the VM's. >> > > > >> > > > Please suggest what can be done. >> > > > >> > > > With regards, >> > > > Swogat Pradhan >> > > > >> > > >> > >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From swogatpradhan22 at gmail.com Tue Jul 11 18:32:44 2023 From: swogatpradhan22 at gmail.com (Swogat Pradhan) Date: Wed, 12 Jul 2023 00:02:44 +0530 Subject: instance console something went wrong, connection is closed | Wallaby DCN In-Reply-To: References: <55eff3d6-b852-840e-80e0-26cbb5a58aac@gmail.com> <3a7830e4-9f09-210c-c1c5-0ee27d8945ff@gmail.com> Message-ID: I am getting the following error in nova-novncproxy.log in controller node: 2023-07-11 18:28:50.464 1080 INFO nova.console.websocketproxy [req-847a31bd-b896-4535-a4b3-6e7d163f0ec8 - - - - -] handler exception: [Errno 107] Transport endpoint is not connected On Wed, Jul 12, 2023 at 12:01?AM Swogat Pradhan wrote: > Please ignore the previous email about the port information. > > So, I believe your point 'Something about redoing the port bindings is > helping the situation sometimes.' is correct. > > But how do I troubleshoot this? > > With regards, > Swogat Pradhan > > On Tue, Jul 11, 2023 at 11:40?PM Swogat Pradhan > wrote: > >> Hi Melanie, >> After searching through i found that no service is exposing port 5901. >> Only when I migrate the instance or do any kind of activity port 5901 is >> exposed on the said hypervisor node and the vm console is accessible. >> >> So, I believe your point 'Something about redoing the port bindings is >> helping the situation sometimes.' is correct. >> >> But how do I troubleshoot this? >> >> With regards, >> Swogat Pradhan >> >> >> On Fri, Jun 30, 2023 at 12:29?AM melanie witt wrote: >> >>> On 06/25/23 03:50, Swogat Pradhan wrote: >>> > Hi, >>> > After doing a console url show after migration, I am still unable to >>> > access the console. >>> > >>> > My site consists of 1 central site and 2 DCN sites. Consoles for >>> > central and DCN02 are working fine without any issues. >>> > But when i am creating an instance for DCN01 the console for the >>> > instance is not coming up (attached image for reference). >>> > >>> > Today I created 3 different VM's using the same flavor, image, >>> security >>> > group, the instances were created in the same compute host. >>> Console was >>> > not accessible, so I shelved and unshelved all 3 instances, after >>> which >>> > I was able to access the console for 2 of those VM's and am still >>> unable >>> > to access the console of the 3rd VM no matter what I do. >>> >>> Apologies for the delayed reply. >>> >>> It sounds like there may be some type of problem with regard to the >>> network connection from the novnc console proxy service to the DCN01 >>> site rather than something with the console proxy itself given that >>> things work fine with DCN02 and central. You're seeing "Cannot write >>> data: Broken pipe" in the connection between the compute host and the >>> console proxy showing a connection being broken after being established. >>> >>> As for why shelve and unshelve sometimes helps, it may be because the >>> network port bindings are updated to inactive during the shelve and then >>> they are updated to active during the unshelve. Something about redoing >>> the port bindings is helping the situation sometimes. >>> >>> It may be worthwhile to check if there is anything different between the >>> networks/ports the instances with working consoles have vs what the >>> instances with non-working consoles have. >>> >>> -melwitt >>> >>> > On Sat, Jun 24, 2023 at 2:00?AM melanie witt >> > > wrote: >>> > >>> > On 06/22/23 20:07, Swogat Pradhan wrote: >>> > > Hi Mel, >>> > > Thank you for your response. >>> > > I am facing issues with the instance console (vnc) in the >>> openstack >>> > > dashboard, Most of the time I shelve the instance and unshelve >>> the >>> > > instance to get the console. >>> > > But there are some VM's I created which are not working even >>> after >>> > > shelve/unshelve. >>> > > >>> > > I have used the same director to deploy a total of a central and >>> > 2 edge >>> > > sites. >>> > > This issue is happening on a single edge site. >>> > > Cold Migration also helps in some situations. >>> > >>> > OK, you didn't mention whether requesting a new console 'openstack >>> > console url show --vnc ' gets you a working console after a >>> > migration (or other event where you see the console stop working). >>> I'm >>> > trying to determine whether the behavior you're seeing is expected >>> or a >>> > bug. After an instance is moved to a different compute node than >>> the >>> > one >>> > it was on when the console was started, that console is not >>> expected to >>> > work anymore. And a new console needs to be started. >>> > >>> > Can you give steps for reproducing the issue? Maybe that will >>> provide >>> > more clarity. >>> > >>> > -melwitt >>> > >>> > > On Fri, Jun 23, 2023 at 12:42?AM melanie witt < >>> melwittt at gmail.com >>> > >>> > > >> wrote: >>> > > >>> > > On 06/22/23 01:08, Swogat Pradhan wrote: >>> > > > Hi, >>> > > > Please find the below log: >>> > > > [root at dcn01-hci-1 libvirt]# cat virtqemud.log >>> > > > 2023-06-22 07:40:01.575+0000: 350319: error : >>> > > virNetSocketReadWire:1804 >>> > > > : End of file while reading data: Input/output error >>> > > > 2023-06-22 07:40:01.575+0000: 350319: error : >>> > > virNetSocketWriteWire:1844 >>> > > > : Cannot write data: Broken pipe >>> > > > >>> > > > I think this is causing the problem of not getting the >>> > instance >>> > > console. >>> > > >>> > > When you say "instance console" are you referring to an >>> > interactive >>> > > console like VNC or are you talking about the console log >>> for the >>> > > instance? >>> > > >>> > > If it's the interactive console, if you have a console open >>> > and then >>> > > migrate the instance, that console will not be moved along >>> > with the >>> > > instance. When a user requests a console, the console proxy >>> > service >>> > > establishes a connection to the compute host where the >>> > instance is >>> > > located. The proxy doesn't know when an instance has been >>> > moved though, >>> > > so if the instance is moved, the user will need to request >>> a new >>> > > console >>> > > (which will establish a new connection to the new compute >>> host). >>> > > >>> > > Is that the behavior you are seeing? >>> > > >>> > > -melwitt >>> > > >>> > > > On Fri, Jun 2, 2023 at 11:27?AM Swogat Pradhan >>> > > > >> > >> swogatpradhan22 at gmail.com >>> > > >>> > > >> > >>> > > >> > >>> wrote: >>> > > > >>> > > > Update: >>> > > > If the i am performing any activity like migration or >>> > resize >>> > > of an >>> > > > instance whose console is accessible, the console >>> becomes >>> > > > inaccessible giving out the following error : >>> > something went >>> > > wrong, >>> > > > connection is closed >>> > > > >>> > > > The was 1 other instance whose console was not >>> > accessible and >>> > > i did >>> > > > a shelve and unshelve and suddenly the instance >>> > console became >>> > > > accessible. >>> > > > >>> > > > This is a peculiar behavior and i don't understand >>> > where is >>> > > the issue . >>> > > > >>> > > > With regards, >>> > > > Swogat Pradhan >>> > > > >>> > > > On Fri, Jun 2, 2023 at 11:19?AM Swogat Pradhan >>> > > > >> > >> swogatpradhan22 at gmail.com >>> > > >>> > > >> > >>> > > >> > >>> wrote: >>> > > > >>> > > > Hi, >>> > > > I am creating instances in my DCN site and i am >>> > unable to get >>> > > > the console sometimes, error: something went >>> wrong, >>> > > connection >>> > > > is closed >>> > > > >>> > > > I have 3 instances now running on my hci02 node >>> > and there is >>> > > > console access on 1 of the vm's and the rest two >>> i >>> > am not >>> > > > getting the console, i have used the same flavor, >>> > same image >>> > > > same security group for the VM's. >>> > > > >>> > > > Please suggest what can be done. >>> > > > >>> > > > With regards, >>> > > > Swogat Pradhan >>> > > > >>> > > >>> > >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Tue Jul 11 19:01:00 2023 From: smooney at redhat.com (smooney at redhat.com) Date: Tue, 11 Jul 2023 20:01:00 +0100 Subject: instance console something went wrong, connection is closed | Wallaby DCN In-Reply-To: References: <55eff3d6-b852-840e-80e0-26cbb5a58aac@gmail.com> <3a7830e4-9f09-210c-c1c5-0ee27d8945ff@gmail.com> Message-ID: <17502e7a10a290e23eb161b86a0c62ee6bc8f187.camel@redhat.com> On Tue, 2023-07-11 at 23:40 +0530, Swogat Pradhan wrote: > Hi Melanie, > After searching through i found that no service is exposing port 5901. > Only when I migrate the instance or do any kind of activity port 5901 is > exposed on the said hypervisor node and the vm console is accessible. > > So, I believe your point 'Something about redoing the port bindings is > helping the situation sometimes.' is correct. > > But how do I troubleshoot this? that does nto really make sense to me 5901-n are used by qemu to expose the vnc console fo the vm but that has nothign to do with neutorn. each vm will have a specific vnc port in the xml if you do a virsh isntance list. that can also change on migrations form host ot host. it is allcoated by libvirt and the prot range is configureed in the libvirt qemu config the port range is contoled by remote_display_port_min and remote_display_port_max in /etc/libvirt/qemu.conf """ # Override the port for creating both VNC and SPICE sessions (min). # This defaults to 5900 and increases for consecutive sessions # or when ports are occupied, until it hits the maximum. # # Minimum must be greater than or equal to 5900 as lower number would # result into negative vnc display number. # # Maximum must be less than 65536, because higher numbers do not make # sense as a port number. # #remote_display_port_min = 5900 #remote_display_port_max = 65535 """ libvirt just increment the number form the min value for each instance on the host. you might need to check your firewall config to ensure that an approare range is openend for each host typically you do not have more then a hounded vms on a give host but a common default is to open 1024 ports to not hit the limit. tripleo opens "5900-6923" https://github.com/openstack/tripleo-heat-templates/blob/1393d39be367db3acb02508e0e858395a4e4fefa/deployment/nova/nova-modular-libvirt-container-puppet.yaml#L391 so you need to ensure that those ports are acceable in the host in the dcn site form the cental site where the nova vnc proxy is located. if you are seeing errors in virtqemud.log related ot Cannot write data: Broken pipe that might point to a differnt error related to virtlogd so you shoudl also see fi there are any related issues there. for example its a know limitation taht if virtlogd is restarted it will break the console https://bugs.launchpad.net/tripleo/+bug/1838272 https://review.opendev.org/c/openstack/puppet-tripleo/+/787771 simple ensure tripleo did not restart the virtlogd contaienr but if anythying else did it would break the console fo runing vms untill they are restart, or moved to a new host. so perhaps you are hitting something similar? > > With regards, > Swogat Pradhan > > > On Fri, Jun 30, 2023 at 12:29?AM melanie witt wrote: > > > On 06/25/23 03:50, Swogat Pradhan wrote: > > > Hi, > > > After doing a console url show after migration, I am still unable to > > > access the console. > > > > > > ? My site consists of 1 central site and 2 DCN sites. Consoles for > > > central and DCN02 are working fine without any issues. > > > But when i am creating an instance for DCN01 the console for the > > > instance is not coming up (attached image for reference). > > > > > > Today I created 3 different VM's using the same flavor, image, security > > > group, the instances were created in the same compute host. Console was > > > not accessible, so I shelved and unshelved all 3 instances, after which > > > I was able to access the console for 2 of those VM's and am still unable > > > to access the console of? the 3rd VM no matter what I do. > > > > Apologies for the delayed reply. > > > > It sounds like there may be some type of problem with regard to the > > network connection from the novnc console proxy service to the DCN01 > > site rather than something with the console proxy itself given that > > things work fine with DCN02 and central. You're seeing "Cannot write > > data: Broken pipe" in the connection between the compute host and the > > console proxy showing a connection being broken after being established. > > > > As for why shelve and unshelve sometimes helps, it may be because the > > network port bindings are updated to inactive during the shelve and then > > they are updated to active during the unshelve. Something about redoing > > the port bindings is helping the situation sometimes. > > > > It may be worthwhile to check if there is anything different between the > > networks/ports the instances with working consoles have vs what the > > instances with non-working consoles have. > > > > -melwitt > > > > > On Sat, Jun 24, 2023 at 2:00?AM melanie witt > > > wrote: > > > > > > ??? On 06/22/23 20:07, Swogat Pradhan wrote: > > > ???? > Hi Mel, > > > ???? > Thank you for your response. > > > ???? > I am facing issues with the instance console (vnc) in the > > openstack > > > ???? > dashboard, Most of the time I shelve the instance and unshelve the > > > ???? > instance to get the console. > > > ???? > But there are some VM's I created which are not working even after > > > ???? > shelve/unshelve. > > > ???? > > > > ???? > I have used the same director to deploy a total of a central and > > > ??? 2 edge > > > ???? > sites. > > > ???? > This issue is happening on a single edge site. > > > ???? > Cold Migration also helps in some situations. > > > > > > ??? OK, you didn't mention whether requesting a new console 'openstack > > > ??? console url show --vnc ' gets you a working console after a > > > ??? migration (or other event where you see the console stop working). > > I'm > > > ??? trying to determine whether the behavior you're seeing is expected > > or a > > > ??? bug. After an instance is moved to a different compute node than the > > > ??? one > > > ??? it was on when the console was started, that console is not expected > > to > > > ??? work anymore. And a new console needs to be started. > > > > > > ??? Can you give steps for reproducing the issue? Maybe that will provide > > > ??? more clarity. > > > > > > ??? -melwitt > > > > > > ???? > On Fri, Jun 23, 2023 at 12:42?AM melanie witt > > ??? > > > ???? > >> wrote: > > > ???? > > > > ???? >???? On 06/22/23 01:08, Swogat Pradhan wrote: > > > ???? >????? > Hi, > > > ???? >????? > Please find the below log: > > > ???? >????? > [root at dcn01-hci-1 libvirt]# cat virtqemud.log > > > ???? >????? > 2023-06-22 07:40:01.575+0000: 350319: error : > > > ???? >???? virNetSocketReadWire:1804 > > > ???? >????? > : End of file while reading data: Input/output error > > > ???? >????? > 2023-06-22 07:40:01.575+0000: 350319: error : > > > ???? >???? virNetSocketWriteWire:1844 > > > ???? >????? > : Cannot write data: Broken pipe > > > ???? >????? > > > > ???? >????? > I think this is causing the problem of not getting the > > > ??? instance > > > ???? >???? console. > > > ???? > > > > ???? >???? When you say "instance console" are you referring to an > > > ??? interactive > > > ???? >???? console like VNC or are you talking about the console log for > > the > > > ???? >???? instance? > > > ???? > > > > ???? >???? If it's the interactive console, if you have a console open > > > ??? and then > > > ???? >???? migrate the instance, that console will not be moved along > > > ??? with the > > > ???? >???? instance. When a user requests a console, the console proxy > > > ??? service > > > ???? >???? establishes a connection to the compute host where the > > > ??? instance is > > > ???? >???? located. The proxy doesn't know when an instance has been > > > ??? moved though, > > > ???? >???? so if the instance is moved, the user will need to request a > > new > > > ???? >???? console > > > ???? >???? (which will establish a new connection to the new compute > > host). > > > ???? > > > > ???? >???? Is that the behavior you are seeing? > > > ???? > > > > ???? >???? -melwitt > > > ???? > > > > ???? >????? > On Fri, Jun 2, 2023 at 11:27?AM Swogat Pradhan > > > ???? >????? > > > ??? > > ??? > > > > ???? >???? > > ??? > > > ???? >???? > > ??? >>> wrote: > > > ???? >????? > > > > ???? >????? >???? Update: > > > ???? >????? >???? If the i am performing any activity like migration or > > > ??? resize > > > ???? >???? of an > > > ???? >????? >???? instance whose console is accessible, the console > > becomes > > > ???? >????? >???? inaccessible giving out the following error : > > > ??? something went > > > ???? >???? wrong, > > > ???? >????? >???? connection is closed > > > ???? >????? > > > > ???? >????? >???? The was 1 other instance whose console was not > > > ??? accessible and > > > ???? >???? i did > > > ???? >????? >???? a shelve and unshelve and suddenly the instance > > > ??? console became > > > ???? >????? >???? accessible. > > > ???? >????? > > > > ???? >????? >???? This is a peculiar behavior and i don't understand > > > ??? where is > > > ???? >???? the issue . > > > ???? >????? > > > > ???? >????? >???? With regards, > > > ???? >????? >???? Swogat Pradhan > > > ???? >????? > > > > ???? >????? >???? On Fri, Jun 2, 2023 at 11:19?AM Swogat Pradhan > > > ???? >????? >???? > > ??? > > ??? > > > > ???? >???? > > ??? > > > ???? >???? > > ??? >>> wrote: > > > ???? >????? > > > > ???? >????? >???????? Hi, > > > ???? >????? >???????? I am creating instances in my DCN site and i am > > > ??? unable to get > > > ???? >????? >???????? the console sometimes, error: something went wrong, > > > ???? >???? connection > > > ???? >????? >???????? is closed > > > ???? >????? > > > > ???? >????? >???????? I have 3 instances now running on my hci02 node > > > ??? and there is > > > ???? >????? >???????? console access on 1 of the vm's and the rest two i > > > ??? am not > > > ???? >????? >???????? getting the console, i have used the same flavor, > > > ??? same image > > > ???? >????? >???????? same security group for the VM's. > > > ???? >????? > > > > ???? >????? >???????? Please suggest what can be done. > > > ???? >????? > > > > ???? >????? >???????? With regards, > > > ???? >????? >???????? Swogat Pradhan > > > ???? >????? > > > > ???? > > > > > > > > From roberto.acosta at luizalabs.com Tue Jul 11 20:49:55 2023 From: roberto.acosta at luizalabs.com (Roberto Bartzen Acosta) Date: Tue, 11 Jul 2023 17:49:55 -0300 Subject: [neutron] unmanaged router resources - OVN interconnect Message-ID: Hello Neutron folks, Regarding the conversation started in March [1] about the use of OVN interconnect with Neutron, I would like to evolve the discussion in relation to resources allocated by OVN-IC and which are not managed by Neutron. In the March PTG [2], the problem related to the db_sync tool was presented, and a proposed solution in which Neutron does not manage these resources. After discussing some architectural designs with Felix/StackIT, it seems to make sense to come up with a proposal to change the mech_driver to validate the external_ids key and not remove resources present in the OVN backend without Neutron "signature". The discussion here is more complex than it seems, because architecturally Neutron does not allow us to interconnect workloads in multiple AZs (with different OpenStacks), but this could be a requirement for a high availability cloud solution (Cloud Region). Additionally, this OVN-IC solution allows interconnecting other cloud backends that use OVN, in the case of kubernetes with ovn-kube. We tested an OVN interconnect integrated with 3 OpenStack installations and it worked very well !!! considering direct L3 traffic at the router level between different network infrastructures. SYNC_REPAIR - problem * Static Routes (learned OVN-IC routes) * Router Port -> Transit Switches Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in OVN but not in Neutron, port_id=rt2-admin-tenant1 Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron Any suggestions on how to resolve this db_sync issue? since all other Neutron modules did not present any problem with OVN-IC (as far as I tested). I remember Terry was keen to suggest some things to help. I believe that before proposing some complex mechanism to solve this simple problem, I would like to hear the community opinion. In our use case, a bit change in db_sync with filter by neutron key in external_ids would be enough. Regards, Roberto [1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg Additional logs: OpenStack 1 root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 IPv4 Routes Route Table
: 20.0.1.0/24 169.254.200.2 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip IPv6 Routes Route Table
: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 IPv4 Routes Route Table
: 10.0.1.0/24 169.254.100.2 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip IPv6 Routes Route Table
: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# OpenStack 2 root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc IPv4 Routes Route Table
: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip IPv6 Routes Route Table
: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list ce45f681-6454-43fe-974f-81344bb8113a IPv4 Routes Route Table
: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip IPv6 Routes Route Table
: ::/0 fc00:ca5a:ca5a:8000:: dst-ip OpenStack 3 root at os-infra-1-neutron-ovn-northd-container-f237db97:~# root at os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 IPv4 Routes Route Table
: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.1.0/24 169.254.200.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip IPv6 Routes Route Table
: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root at os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 IPv4 Routes Route Table
: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.1.0/24 169.254.100.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip IPv6 Routes Route Table
: ::/0 fc00:ca5a:ca5a:8000:: dst-ip OVN-IC Global database root at ovn-global-db1:~# ovn-ic-sbctl show availability-zone osp1 gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 hostname: osp1-gwnode1 type: geneve ip: 192.168.200.28 port admin-rt1-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] port admin-rt1-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] availability-zone osp2 gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 hostname: osp2-gwnode1 type: geneve ip: 192.168.200.128 port admin-rt2-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] port admin-rt2-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] availability-zone osp3 gateway 97595af9-7896-40d0-a883-beadbff1aa5b hostname: osp3-gwnode1 type: geneve ip: 192.168.200.228 port admin-rt3-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] port admin-rt3-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] -- _?Esta mensagem ? direcionada apenas para os endere?os constantes no cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o imediatamente anuladas e proibidas?._ *?**?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o poder? aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos?.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nils.Magnus at t-systems.com Tue Jul 11 21:21:45 2023 From: Nils.Magnus at t-systems.com (Nils.Magnus at t-systems.com) Date: Tue, 11 Jul 2023 21:21:45 +0000 Subject: AW: Need help In-Reply-To: References: Message-ID: Kumar, for the ML/AI field almost the same advice applies: There?s lots of good training material available online (even for free), but you have to practice. Technically ML/AI and cloud computing are just loosely connected, however, if you do reasonable ML/AI, you quickly need some reasonable compute power. And there?s nothing that can manage comput and storage capacities better than a cloud. Regards, Nils Von: Gk Gk Gesendet: Dienstag, 11. Juli 2023 12:52 An: Magnus, Nils Cc: openstack-dev at lists.openstack.org Betreff: Re: Need help Thanks Nils... I have been using and working on openstack for nine years now. These days I am stuck with how to shape my career with my present skill set of Linux, Openstack in general and which technology will help me in transitioning to a higher level. I hate management based positions. Instead I love to tackle tech stuff. The above advice sounds good to me. And one other technology I am trying to learn now is Artificial Intelligence and Machine Learning. Will that help me in combining that with Openstack ? Please comment on how that works and what the near future would look like with this ? Appreciate your time and help, Kumar On Tue, Jul 11, 2023 at 3:27?PM > wrote: Hi Kumar, not meaning to be offensive, but having a rough idea how to become an architect is a good first step ? That having said, it?s probably not just reading a good book or booking a good training, but doing practical exercises over and over again. However, there are good news: There?s plenty of material available and there are probably many paths that lead to success. I can only outline four suggestions that helped me personally in the past couple of years: * There are several OpenStack introduction videos available on YouTube. Don?t be afraid watching some videos, even if they are a few years old and seem to be outdated. If you keep an open mind, this is not so important when you actually start your learning journey. Try to get an overview of the structure and top-level-architecture of OpenStack. One example (of many): https://www.youtube.com/watch?v=8kADjGCuSVI * There is one notorious architectural diagram that is available in many places in the Internet. Here is one link: https://www.pinterest.de/pin/1196337376870638/ While this might not be the premier example of creating an easy-to-understand diagram, it has some inner beauty. Make yourself familiar with it. This does not mean that you need to be able to explain every single box or arrow (and there are plenty of them!), but understand the *idea* of it. When you want to attach a volume to a server instance with the command line tool, try to figure out your exact path through this diagram, for example. * You need to exercise. Practically. With your fingertips on the keyboard. Not just reading books. There are basically two easy ways to do so: One is to figure out an existing public cloud that offers some free tier for beginners or students. Even with as little as $100 of virtual money you can do a lot experiments for a start. The alternative is to install a simplified version of OpenStack on your own computer. If you are really new, I don?t recommend to start with installing ?the real thing? directly, as there might be many minor pitfalls that don?t pay into your learning experience. Something I like is Ubuntu?s ?microstack?, which is a ?snap? package for the Linux distro (nowadays also labelled as ?sunbeam? if I understood the session right during the last Summit). Some people like Ubuntu snaps, some don?t. We could engage in a lengthy discussion if they are the ?right? way to package software, but setting up a tiny instance on your single notebook computer was never so easy. It takes you less than 15 minutes to have a setup up and running. If you progress on your journey, the microstack approach may have some limitations, but it?s a great start. * There is a browser-based UI to work with OpenStack (?Horizon?). It?s okay to use that for very first steps, but keep in mind that in the long run you picked the cloud principle to be able to automate tasks. Using the CLI (?OpenStack Client?, ?OSC?, or just ?openstack? on the command line) is an important first step in this direction. Familiarize yourself also with Python and its language bindings to the OpenStack API, called ?OpenStack SDK?. It?s not necessary to write huge programs, but writing a script that just lists all your volumes or servers might be insightful: https://www.youtube.com/watch?v=EKadc-cxZc0 (second talk). I hope that helps! Regards, Nils -- Nils Magnus T-Systems International GmbH Open Telekom Cloud Delivery ? Chapter Architecture Holzhauser Str. 4-8, 13509 Berlin, Germany +49 30 8353 24446 +49 170 4189 377 (mobile) Von: Gk Gk > Gesendet: Dienstag, 11. Juli 2023 09:43 An: OpenStack Development Mailing List (not for usage questions) > Betreff: Need help Hi Folks, I work on openstack platforms. Can anyone suggest some good books or resources which will help me in becoming a cloud solutions architect ? What would this path look like ? What would be my responsibilities ? Thanks Kumar -------------- next part -------------- An HTML attachment was scrubbed... URL: From melwittt at gmail.com Tue Jul 11 21:26:59 2023 From: melwittt at gmail.com (melanie witt) Date: Tue, 11 Jul 2023 14:26:59 -0700 Subject: Need help In-Reply-To: References: Message-ID: <998b2894-14ed-1462-a868-f09eb8f5fbc6@gmail.com> On 07/11/23 04:58, Gk Gk wrote: > Hi All, > > If I use the openstacksdk to connect to an openstack cloud, I have to > use clouds.yaml file for > specifying the cloud configuration which includes username and password > as well. Since its a plain text file, how can I mask the password > mentioned in clouds.yaml file for security purposes? I don't think there is any way to mask the password in clouds.yaml but the clouds-public.yaml sounds like it might work for you: https://docs.openstack.org/python-openstackclient/latest/configuration/index.html#clouds-public-yaml -melwitt From jamesleong123098 at gmail.com Wed Jul 12 01:21:40 2023 From: jamesleong123098 at gmail.com (James Leong) Date: Tue, 11 Jul 2023 20:21:40 -0500 Subject: [horizon][keystone] Adding different rules in the same protocol for federated logon Message-ID: Hi all, I have yoga version openstack with the deployment tool of kolla-ansible. I am trying to combine different mapping rules such as allowing user to login to different domain. However, I am not able to do that in a single JSON file. When I try to include different rule in the same JSON file, only the first rule is being considered. Is there a way to allow multiple rule to redirect user to their account in a different domain. Best, James -------------- next part -------------- An HTML attachment was scrubbed... URL: From tony at bakeyournoodle.com Wed Jul 12 04:09:03 2023 From: tony at bakeyournoodle.com (Tony Breeds) Date: Wed, 12 Jul 2023 14:09:03 +1000 Subject: Need help In-Reply-To: References: Message-ID: On Tue, 11 Jul 2023 at 22:02, Gk Gk wrote: > > Hi All, > > If I use the openstacksdk to connect to an openstack cloud, I have to use clouds.yaml file for > specifying the cloud configuration which includes username and password as well. Since its a plain text file, how can I mask the password mentioned in clouds.yaml file for security purposes? You can also create and use a token for authentication. -=-=-=-=-=-=- $ openstack \ --os-auth-url "$OS_AUTH_URL" \ --os-user-domain-name "<>" \ --os-region-name "regionOne" \ --os-interface "public" \ --os-identity-api-version 3 \ --os-project-name "$OS_PROJECT_NAME" \ --os-username "$OS_USERNAME" \ --os-project-domain-id "$OS_PROJECT_DOMAIN_ID" \ --os-password "$OS_PASSWORD" \ token issue -f value -c id $ cat ~/.config/openstack/clouds.yaml --- clouds: openstack: auth_type: "token" auth: token: "<>" auth_url: "<>" project_id: "<>" etc etc etc -=-=-=-=-=-=- You will need to generate the token regularly, but it does avoid having the plain text password on disk. Yours Tony. From artem.goncharov at gmail.com Wed Jul 12 07:59:27 2023 From: artem.goncharov at gmail.com (Artem Goncharov) Date: Wed, 12 Jul 2023 08:59:27 +0100 Subject: Need help In-Reply-To: References: Message-ID: There is a support for splitting configuration into clouds.yaml and secret.yaml (read the SDK documentation for details on that). This way you can keep clouds.yaml without username and password to be able to share it freely. Artem On Wed, Jul 12, 2023, 05:12 Tony Breeds wrote: > On Tue, 11 Jul 2023 at 22:02, Gk Gk wrote: > > > > Hi All, > > > > If I use the openstacksdk to connect to an openstack cloud, I have to > use clouds.yaml file for > > specifying the cloud configuration which includes username and password > as well. Since its a plain text file, how can I mask the password mentioned > in clouds.yaml file for security purposes? > > You can also create and use a token for authentication. > -=-=-=-=-=-=- > $ openstack \ > --os-auth-url "$OS_AUTH_URL" \ > --os-user-domain-name "<>" \ > --os-region-name "regionOne" \ > --os-interface "public" \ > --os-identity-api-version 3 \ > --os-project-name "$OS_PROJECT_NAME" \ > --os-username "$OS_USERNAME" \ > --os-project-domain-id "$OS_PROJECT_DOMAIN_ID" \ > --os-password "$OS_PASSWORD" \ > token issue -f value -c id > $ cat ~/.config/openstack/clouds.yaml > --- > clouds: > openstack: > auth_type: "token" > auth: > token: "<>" > auth_url: "<>" > project_id: "<>" > etc etc etc > -=-=-=-=-=-=- > > You will need to generate the token regularly, but it does avoid > having the plain text password on disk. > > Yours Tony. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralonsoh at redhat.com Wed Jul 12 08:10:48 2023 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Wed, 12 Jul 2023 10:10:48 +0200 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: Hello Roberto: We talked about this possible RFE during the PTG [1] but I don't remember having any proposal. Actually what I remember (and was written in the etherpad) is that you were going to present an RFE. Can you explain it briefly? We also talked about the idea, proposed by Felix in the mailing list, of having a different network type only to hold the Transit Switches and thus the LRP. If you have a proposal, please present it. Regards. [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506 On Tue, Jul 11, 2023 at 10:50?PM Roberto Bartzen Acosta < roberto.acosta at luizalabs.com> wrote: > Hello Neutron folks, > > Regarding the conversation started in March [1] about the use of OVN > interconnect with Neutron, I would like to evolve the discussion in > relation to resources allocated by OVN-IC and which are not managed by > Neutron. In the March PTG [2], the problem related to the db_sync tool was > presented, and a proposed solution in which Neutron does not manage these > resources. > > After discussing some architectural designs with Felix/StackIT, it seems > to make sense to come up with a proposal to change the mech_driver to > validate the external_ids key and not remove resources present in the OVN > backend without Neutron "signature". > > The discussion here is more complex than it seems, because architecturally > Neutron does not allow us to interconnect workloads in multiple AZs (with > different OpenStacks), but this could be a requirement for a high > availability cloud solution (Cloud Region). Additionally, this OVN-IC > solution allows interconnecting other cloud backends that use OVN, in the > case of kubernetes with ovn-kube. > > We tested an OVN interconnect integrated with 3 OpenStack installations > and it worked very well !!! considering direct L3 traffic at the router > level between different network infrastructures. > > SYNC_REPAIR - problem > > * Static Routes (learned OVN-IC routes) > * Router Port -> Transit Switches > > Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae > neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING > neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync > [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in > OVN but not in Neutron, port_id=rt2-admin-tenant1 > Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae > neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING > neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync > [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router > 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' > 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '10.0.2.1/24', > 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron > > > Any suggestions on how to resolve this db_sync issue? since all other > Neutron modules did not present any problem with OVN-IC (as far as I > tested). > I remember Terry was keen to suggest some things to help. I believe that > before proposing some complex mechanism to solve this simple problem, I > would like to hear the community opinion. In our use case, a bit change in > db_sync with filter by neutron key in external_ids would be enough. > > Regards, > Roberto > > > > [1] > https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html > [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg > > > > Additional logs: > > OpenStack 1 > > root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# > root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# > root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl > lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 > IPv4 Routes > Route Table
: > 20.0.1.0/24 169.254.200.2 dst-ip (learned) > 20.0.2.0/24 169.254.200.3 dst-ip (learned) > 0.0.0.0/0 200.200.200.1 dst-ip > > IPv6 Routes > Route Table
: > ::/0 fc00:ca5a:ca5a:8000:: dst-ip > root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl > lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 > IPv4 Routes > Route Table
: > 10.0.1.0/24 169.254.100.2 dst-ip (learned) > 10.0.2.0/24 169.254.100.3 dst-ip (learned) > 0.0.0.0/0 200.200.200.1 dst-ip > > IPv6 Routes > Route Table
: > ::/0 fc00:ca5a:ca5a:8000:: dst-ip > root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# > > > OpenStack 2 > > root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl > lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc > IPv4 Routes > Route Table
: > 20.0.0.0/24 169.254.200.1 dst-ip (learned) > 20.0.2.0/24 169.254.200.3 dst-ip (learned) > 0.0.0.0/0 200.200.200.1 dst-ip > > IPv6 Routes > Route Table
: > ::/0 fc00:ca5a:ca5a:8000:: dst-ip > root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl > lr-route-list ce45f681-6454-43fe-974f-81344bb8113a > IPv4 Routes > Route Table
: > 10.0.0.0/24 169.254.100.1 dst-ip (learned) > 10.0.2.0/24 169.254.100.3 dst-ip (learned) > 0.0.0.0/0 200.200.200.1 dst-ip > > IPv6 Routes > Route Table
: > ::/0 fc00:ca5a:ca5a:8000:: dst-ip > > > > OpenStack 3 > > root at os-infra-1-neutron-ovn-northd-container-f237db97:~# > root at os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl > lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 > IPv4 Routes > Route Table
: > 20.0.0.0/24 169.254.200.1 dst-ip (learned) > 20.0.1.0/24 169.254.200.2 dst-ip (learned) > 0.0.0.0/0 200.200.200.1 dst-ip > > IPv6 Routes > Route Table
: > ::/0 fc00:ca5a:ca5a:8000:: dst-ip > root at os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl > lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 > IPv4 Routes > Route Table
: > 10.0.0.0/24 169.254.100.1 dst-ip (learned) > 10.0.1.0/24 169.254.100.2 dst-ip (learned) > 0.0.0.0/0 200.200.200.1 dst-ip > > IPv6 Routes > Route Table
: > ::/0 fc00:ca5a:ca5a:8000:: dst-ip > > > OVN-IC Global database > > root at ovn-global-db1:~# ovn-ic-sbctl show > availability-zone osp1 > gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 > hostname: osp1-gwnode1 > type: geneve > ip: 192.168.200.28 > port admin-rt1-tenant1 > transit switch: admin-tenant1 > address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] > port admin-rt1-tenant1_1 > transit switch: admin-tenant1_1 > address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] > availability-zone osp2 > gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 > hostname: osp2-gwnode1 > type: geneve > ip: 192.168.200.128 > port admin-rt2-tenant1 > transit switch: admin-tenant1 > address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] > port admin-rt2-tenant1_1 > transit switch: admin-tenant1_1 > address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] > availability-zone osp3 > gateway 97595af9-7896-40d0-a883-beadbff1aa5b > hostname: osp3-gwnode1 > type: geneve > ip: 192.168.200.228 > port admin-rt3-tenant1 > transit switch: admin-tenant1 > address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] > port admin-rt3-tenant1_1 > transit switch: admin-tenant1_1 > address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] > > > > > > > > *?Esta mensagem ? direcionada apenas para os endere?os constantes no > cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no > cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa > mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o > imediatamente anuladas e proibidas?.* > > *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para > assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o > poder? aceitar a responsabilidade por quaisquer perdas ou danos causados > por esse e-mail ou por seus anexos?.* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ygk.kmr at gmail.com Wed Jul 12 08:25:06 2023 From: ygk.kmr at gmail.com (Gk Gk) Date: Wed, 12 Jul 2023 13:55:06 +0530 Subject: Need help In-Reply-To: References: Message-ID: Is the file secret.yaml encrypted or plain text ? On Wed, Jul 12, 2023 at 1:29?PM Artem Goncharov wrote: > There is a support for splitting configuration into clouds.yaml and > secret.yaml (read the SDK documentation for details on that). This way you > can keep clouds.yaml without username and password to be able to share it > freely. > > Artem > > On Wed, Jul 12, 2023, 05:12 Tony Breeds wrote: > >> On Tue, 11 Jul 2023 at 22:02, Gk Gk wrote: >> > >> > Hi All, >> > >> > If I use the openstacksdk to connect to an openstack cloud, I have to >> use clouds.yaml file for >> > specifying the cloud configuration which includes username and password >> as well. Since its a plain text file, how can I mask the password mentioned >> in clouds.yaml file for security purposes? >> >> You can also create and use a token for authentication. >> -=-=-=-=-=-=- >> $ openstack \ >> --os-auth-url "$OS_AUTH_URL" \ >> --os-user-domain-name "<>" \ >> --os-region-name "regionOne" \ >> --os-interface "public" \ >> --os-identity-api-version 3 \ >> --os-project-name "$OS_PROJECT_NAME" \ >> --os-username "$OS_USERNAME" \ >> --os-project-domain-id "$OS_PROJECT_DOMAIN_ID" \ >> --os-password "$OS_PASSWORD" \ >> token issue -f value -c id >> $ cat ~/.config/openstack/clouds.yaml >> --- >> clouds: >> openstack: >> auth_type: "token" >> auth: >> token: "<>" >> auth_url: "<>" >> project_id: "<>" >> etc etc etc >> -=-=-=-=-=-=- >> >> You will need to generate the token regularly, but it does avoid >> having the plain text password on disk. >> >> Yours Tony. >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From swogatpradhan22 at gmail.com Wed Jul 12 08:49:31 2023 From: swogatpradhan22 at gmail.com (Swogat Pradhan) Date: Wed, 12 Jul 2023 14:19:31 +0530 Subject: instance console something went wrong, connection is closed | Wallaby DCN In-Reply-To: <17502e7a10a290e23eb161b86a0c62ee6bc8f187.camel@redhat.com> References: <55eff3d6-b852-840e-80e0-26cbb5a58aac@gmail.com> <3a7830e4-9f09-210c-c1c5-0ee27d8945ff@gmail.com> <17502e7a10a290e23eb161b86a0c62ee6bc8f187.camel@redhat.com> Message-ID: Hi Smooney, Please ignore that mail, please check the other emails after that. I can see other ports like 5900 and 5901 are being exposed and for the controller node i am able to telnet also to the said ports. But for some reason i am sometimes able to access the console of the vm's and sometimes not. I want to find a way to fix this issue. With regards, Swogat Pradhan On Wed, Jul 12, 2023 at 12:31?AM wrote: > On Tue, 2023-07-11 at 23:40 +0530, Swogat Pradhan wrote: > > Hi Melanie, > > After searching through i found that no service is exposing port 5901. > > Only when I migrate the instance or do any kind of activity port 5901 is > > exposed on the said hypervisor node and the vm console is accessible. > > > > So, I believe your point 'Something about redoing the port bindings is > > helping the situation sometimes.' is correct. > > > > But how do I troubleshoot this? > that does nto really make sense to me 5901-n are used by qemu to expose > the vnc console fo the vm > but that has nothign to do with neutorn. > > each vm will have a specific vnc port in the xml if you do a virsh > isntance list. > that can also change on migrations form host ot host. > > it is allcoated by libvirt and the prot range is configureed in the > libvirt qemu config > > the port range is contoled by remote_display_port_min and > remote_display_port_max > in /etc/libvirt/qemu.conf > """ > # Override the port for creating both VNC and SPICE sessions (min). > # This defaults to 5900 and increases for consecutive sessions > # or when ports are occupied, until it hits the maximum. > # > # Minimum must be greater than or equal to 5900 as lower number would > # result into negative vnc display number. > # > # Maximum must be less than 65536, because higher numbers do not make > # sense as a port number. > # > #remote_display_port_min = 5900 > #remote_display_port_max = 65535 > """ > > libvirt just increment the number form the min value for each instance on > the host. > > you might need to check your firewall config to ensure that an approare > range is openend for each host > typically you do not have more then a hounded vms on a give host but a > common default is to open 1024 > ports to not hit the limit. > tripleo opens "5900-6923" > > https://github.com/openstack/tripleo-heat-templates/blob/1393d39be367db3acb02508e0e858395a4e4fefa/deployment/nova/nova-modular-libvirt-container-puppet.yaml#L391 > so you need to ensure that those ports are acceable in the host in the dcn > site form the cental site where the nova vnc > proxy is located. > > if you are seeing errors in virtqemud.log related ot Cannot write data: > Broken pipe > that might point to a differnt error related to virtlogd so you shoudl > also see fi there are any related issues there. > for example its a know limitation taht if virtlogd is restarted it will > break the console > https://bugs.launchpad.net/tripleo/+bug/1838272 > https://review.opendev.org/c/openstack/puppet-tripleo/+/787771 simple > ensure tripleo did not restart the virtlogd > contaienr but if anythying else did it would break the console fo runing > vms untill they are restart, or moved to a new > host. so perhaps you are hitting something similar? > > > > > > With regards, > > Swogat Pradhan > > > > > > On Fri, Jun 30, 2023 at 12:29?AM melanie witt > wrote: > > > > > On 06/25/23 03:50, Swogat Pradhan wrote: > > > > Hi, > > > > After doing a console url show after migration, I am still unable to > > > > access the console. > > > > > > > > My site consists of 1 central site and 2 DCN sites. Consoles for > > > > central and DCN02 are working fine without any issues. > > > > But when i am creating an instance for DCN01 the console for the > > > > instance is not coming up (attached image for reference). > > > > > > > > Today I created 3 different VM's using the same flavor, image, > security > > > > group, the instances were created in the same compute host. Console > was > > > > not accessible, so I shelved and unshelved all 3 instances, after > which > > > > I was able to access the console for 2 of those VM's and am still > unable > > > > to access the console of the 3rd VM no matter what I do. > > > > > > Apologies for the delayed reply. > > > > > > It sounds like there may be some type of problem with regard to the > > > network connection from the novnc console proxy service to the DCN01 > > > site rather than something with the console proxy itself given that > > > things work fine with DCN02 and central. You're seeing "Cannot write > > > data: Broken pipe" in the connection between the compute host and the > > > console proxy showing a connection being broken after being > established. > > > > > > As for why shelve and unshelve sometimes helps, it may be because the > > > network port bindings are updated to inactive during the shelve and > then > > > they are updated to active during the unshelve. Something about redoing > > > the port bindings is helping the situation sometimes. > > > > > > It may be worthwhile to check if there is anything different between > the > > > networks/ports the instances with working consoles have vs what the > > > instances with non-working consoles have. > > > > > > -melwitt > > > > > > > On Sat, Jun 24, 2023 at 2:00?AM melanie witt > > > > wrote: > > > > > > > > On 06/22/23 20:07, Swogat Pradhan wrote: > > > > > Hi Mel, > > > > > Thank you for your response. > > > > > I am facing issues with the instance console (vnc) in the > > > openstack > > > > > dashboard, Most of the time I shelve the instance and > unshelve the > > > > > instance to get the console. > > > > > But there are some VM's I created which are not working even > after > > > > > shelve/unshelve. > > > > > > > > > > I have used the same director to deploy a total of a central > and > > > > 2 edge > > > > > sites. > > > > > This issue is happening on a single edge site. > > > > > Cold Migration also helps in some situations. > > > > > > > > OK, you didn't mention whether requesting a new console > 'openstack > > > > console url show --vnc ' gets you a working console > after a > > > > migration (or other event where you see the console stop > working). > > > I'm > > > > trying to determine whether the behavior you're seeing is > expected > > > or a > > > > bug. After an instance is moved to a different compute node than > the > > > > one > > > > it was on when the console was started, that console is not > expected > > > to > > > > work anymore. And a new console needs to be started. > > > > > > > > Can you give steps for reproducing the issue? Maybe that will > provide > > > > more clarity. > > > > > > > > -melwitt > > > > > > > > > On Fri, Jun 23, 2023 at 12:42?AM melanie witt < > melwittt at gmail.com > > > > > > > > > >> > wrote: > > > > > > > > > > On 06/22/23 01:08, Swogat Pradhan wrote: > > > > > > Hi, > > > > > > Please find the below log: > > > > > > [root at dcn01-hci-1 libvirt]# cat virtqemud.log > > > > > > 2023-06-22 07:40:01.575+0000: 350319: error : > > > > > virNetSocketReadWire:1804 > > > > > > : End of file while reading data: Input/output error > > > > > > 2023-06-22 07:40:01.575+0000: 350319: error : > > > > > virNetSocketWriteWire:1844 > > > > > > : Cannot write data: Broken pipe > > > > > > > > > > > > I think this is causing the problem of not getting the > > > > instance > > > > > console. > > > > > > > > > > When you say "instance console" are you referring to an > > > > interactive > > > > > console like VNC or are you talking about the console log > for > > > the > > > > > instance? > > > > > > > > > > If it's the interactive console, if you have a console > open > > > > and then > > > > > migrate the instance, that console will not be moved along > > > > with the > > > > > instance. When a user requests a console, the console > proxy > > > > service > > > > > establishes a connection to the compute host where the > > > > instance is > > > > > located. The proxy doesn't know when an instance has been > > > > moved though, > > > > > so if the instance is moved, the user will need to > request a > > > new > > > > > console > > > > > (which will establish a new connection to the new compute > > > host). > > > > > > > > > > Is that the behavior you are seeing? > > > > > > > > > > -melwitt > > > > > > > > > > > On Fri, Jun 2, 2023 at 11:27?AM Swogat Pradhan > > > > > > > > > swogatpradhan22 at gmail.com > > > > > > > > > > > > > > > > > > > > > >>> wrote: > > > > > > > > > > > > Update: > > > > > > If the i am performing any activity like migration > or > > > > resize > > > > > of an > > > > > > instance whose console is accessible, the console > > > becomes > > > > > > inaccessible giving out the following error : > > > > something went > > > > > wrong, > > > > > > connection is closed > > > > > > > > > > > > The was 1 other instance whose console was not > > > > accessible and > > > > > i did > > > > > > a shelve and unshelve and suddenly the instance > > > > console became > > > > > > accessible. > > > > > > > > > > > > This is a peculiar behavior and i don't understand > > > > where is > > > > > the issue . > > > > > > > > > > > > With regards, > > > > > > Swogat Pradhan > > > > > > > > > > > > On Fri, Jun 2, 2023 at 11:19?AM Swogat Pradhan > > > > > > > > > swogatpradhan22 at gmail.com > > > > > > > > > > > > > > > > > > > > > >>> wrote: > > > > > > > > > > > > Hi, > > > > > > I am creating instances in my DCN site and i am > > > > unable to get > > > > > > the console sometimes, error: something went > wrong, > > > > > connection > > > > > > is closed > > > > > > > > > > > > I have 3 instances now running on my hci02 node > > > > and there is > > > > > > console access on 1 of the vm's and the rest > two i > > > > am not > > > > > > getting the console, i have used the same > flavor, > > > > same image > > > > > > same security group for the VM's. > > > > > > > > > > > > Please suggest what can be done. > > > > > > > > > > > > With regards, > > > > > > Swogat Pradhan > > > > > > > > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From senrique at redhat.com Wed Jul 12 11:29:51 2023 From: senrique at redhat.com (Sofia Enriquez) Date: Wed, 12 Jul 2023 12:29:51 +0100 Subject: Cinder Bug Report 2023-07-12 Message-ID: Hello Argonauts, Cinder Bug Meeting Etherpad High - Cannot create a volume from a backup . - *Status*: Fix proposed to master , but it needs a release note. - Cinder backup appears as down . - *Status*: Unassigned. Medium - [api] Sort by boot and pass in the marker Exception . - *Status*: Unassigned. - [os-brick] _get_host_uuid incompatible with btrfs snapshots . - *Status*: Unassigned. Workaround on the launchpad bug report. Low - HPE 3par: Issue observed during retype/migrate. - *Status*: Unassigned. Cheers, -- Sof?a Enriquez she/her Software Engineer Red Hat PnT IRC: @enriquetaso @RedHat Red Hat Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.acosta at luizalabs.com Wed Jul 12 12:05:54 2023 From: roberto.acosta at luizalabs.com (Roberto Bartzen Acosta) Date: Wed, 12 Jul 2023 09:05:54 -0300 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: Hi Rodolfo, Thanks for the feedback. I liked the idea of having a different network type only to hold the Transit Switches and thus the LRP but it depends on some factors like multi-tenancy. >From the OVN perspective, the TS is global and will only be linked to a tenant when it is plugged into the tenant router port. My concern here is that if Neutron manages the TS, it will need to assume the dynamic characteristics of this TS function, ovn-ic creates and removes the TS in NB_Global, in addition to plugging and removing ports in this switch according to the network topology. Additionally, Neutron would still need to handle the learned remote routes (which are listed as static routes from the tenant router perspective). This is an open discussion, Felix can add ideas here. In general, it seems to me that we have no alternatives for this type of solution other than OVN-IC (note that ovn-bgp-agent does not learn remote routes at the SDN level / OVN). OpenStack seems to be designed to run like a "bubble" and this traffic from one VM to another always needs to be routed at the FIP level and external routing layers. Regards, Roberto Em qua., 12 de jul. de 2023 ?s 05:11, Rodolfo Alonso Hernandez < ralonsoh at redhat.com> escreveu: > Hello Roberto: > > We talked about this possible RFE during the PTG [1] but I don't remember > having any proposal. Actually what I remember (and was written in the > etherpad) is that you were going to present an RFE. Can you explain it > briefly? > > We also talked about the idea, proposed by Felix in the mailing list, of > having a different network type only to hold the Transit Switches and thus > the LRP. If you have a proposal, please present it. > > Regards. > > [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506 > > On Tue, Jul 11, 2023 at 10:50?PM Roberto Bartzen Acosta < > roberto.acosta at luizalabs.com> wrote: > >> Hello Neutron folks, >> >> Regarding the conversation started in March [1] about the use of OVN >> interconnect with Neutron, I would like to evolve the discussion in >> relation to resources allocated by OVN-IC and which are not managed by >> Neutron. In the March PTG [2], the problem related to the db_sync tool was >> presented, and a proposed solution in which Neutron does not manage these >> resources. >> >> After discussing some architectural designs with Felix/StackIT, it seems >> to make sense to come up with a proposal to change the mech_driver to >> validate the external_ids key and not remove resources present in the OVN >> backend without Neutron "signature". >> >> The discussion here is more complex than it seems, because >> architecturally Neutron does not allow us to interconnect workloads in >> multiple AZs (with different OpenStacks), but this could be a requirement >> for a high availability cloud solution (Cloud Region). Additionally, this >> OVN-IC solution allows interconnecting other cloud backends that use OVN, >> in the case of kubernetes with ovn-kube. >> >> We tested an OVN interconnect integrated with 3 OpenStack installations >> and it worked very well !!! considering direct L3 traffic at the router >> level between different network infrastructures. >> >> SYNC_REPAIR - problem >> >> * Static Routes (learned OVN-IC routes) >> * Router Port -> Transit Switches >> >> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in >> OVN but not in Neutron, port_id=rt2-admin-tenant1 >> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router >> 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' >> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '10.0.2.1/24', >> 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron >> >> >> Any suggestions on how to resolve this db_sync issue? since all other >> Neutron modules did not present any problem with OVN-IC (as far as I >> tested). >> I remember Terry was keen to suggest some things to help. I believe that >> before proposing some complex mechanism to solve this simple problem, I >> would like to hear the community opinion. In our use case, a bit change in >> db_sync with filter by neutron key in external_ids would be enough. >> >> Regards, >> Roberto >> >> >> >> [1] >> https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html >> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg >> >> >> >> Additional logs: >> >> OpenStack 1 >> >> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl >> lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 >> IPv4 Routes >> Route Table
: >> 20.0.1.0/24 169.254.200.2 dst-ip (learned) >> 20.0.2.0/24 169.254.200.3 dst-ip (learned) >> 0.0.0.0/0 200.200.200.1 dst-ip >> >> IPv6 Routes >> Route Table
: >> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl >> lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 >> IPv4 Routes >> Route Table
: >> 10.0.1.0/24 169.254.100.2 dst-ip (learned) >> 10.0.2.0/24 169.254.100.3 dst-ip (learned) >> 0.0.0.0/0 200.200.200.1 dst-ip >> >> IPv6 Routes >> Route Table
: >> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >> >> >> OpenStack 2 >> >> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl >> lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc >> IPv4 Routes >> Route Table
: >> 20.0.0.0/24 169.254.200.1 dst-ip (learned) >> 20.0.2.0/24 169.254.200.3 dst-ip (learned) >> 0.0.0.0/0 200.200.200.1 dst-ip >> >> IPv6 Routes >> Route Table
: >> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl >> lr-route-list ce45f681-6454-43fe-974f-81344bb8113a >> IPv4 Routes >> Route Table
: >> 10.0.0.0/24 169.254.100.1 dst-ip (learned) >> 10.0.2.0/24 169.254.100.3 dst-ip (learned) >> 0.0.0.0/0 200.200.200.1 dst-ip >> >> IPv6 Routes >> Route Table
: >> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> >> >> >> OpenStack 3 >> >> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl >> lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 >> IPv4 Routes >> Route Table
: >> 20.0.0.0/24 169.254.200.1 dst-ip (learned) >> 20.0.1.0/24 169.254.200.2 dst-ip (learned) >> 0.0.0.0/0 200.200.200.1 dst-ip >> >> IPv6 Routes >> Route Table
: >> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl >> lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 >> IPv4 Routes >> Route Table
: >> 10.0.0.0/24 169.254.100.1 dst-ip (learned) >> 10.0.1.0/24 169.254.100.2 dst-ip (learned) >> 0.0.0.0/0 200.200.200.1 dst-ip >> >> IPv6 Routes >> Route Table
: >> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> >> >> OVN-IC Global database >> >> root at ovn-global-db1:~# ovn-ic-sbctl show >> availability-zone osp1 >> gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 >> hostname: osp1-gwnode1 >> type: geneve >> ip: 192.168.200.28 >> port admin-rt1-tenant1 >> transit switch: admin-tenant1 >> address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] >> port admin-rt1-tenant1_1 >> transit switch: admin-tenant1_1 >> address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] >> availability-zone osp2 >> gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 >> hostname: osp2-gwnode1 >> type: geneve >> ip: 192.168.200.128 >> port admin-rt2-tenant1 >> transit switch: admin-tenant1 >> address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] >> port admin-rt2-tenant1_1 >> transit switch: admin-tenant1_1 >> address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] >> availability-zone osp3 >> gateway 97595af9-7896-40d0-a883-beadbff1aa5b >> hostname: osp3-gwnode1 >> type: geneve >> ip: 192.168.200.228 >> port admin-rt3-tenant1 >> transit switch: admin-tenant1 >> address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] >> port admin-rt3-tenant1_1 >> transit switch: admin-tenant1_1 >> address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] >> >> >> >> >> >> >> >> *?Esta mensagem ? direcionada apenas para os endere?os constantes no >> cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no >> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa >> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o >> imediatamente anuladas e proibidas?.* >> >> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para >> assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o >> poder? aceitar a responsabilidade por quaisquer perdas ou danos causados >> por esse e-mail ou por seus anexos?.* >> > -- _?Esta mensagem ? direcionada apenas para os endere?os constantes no cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o imediatamente anuladas e proibidas?._ *?**?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o poder? aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos?.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Wed Jul 12 12:07:45 2023 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 12 Jul 2023 12:07:45 +0000 Subject: [sdks][security-sig] Protecting client password (was: Need help) In-Reply-To: References: Message-ID: <20230712120744.xiyer3lzqukaa7fa@yuggoth.org> On 2023-07-12 13:55:06 +0530 (+0530), Gk Gk wrote: > Is the file secret.yaml encrypted or plain text ? [...] It's plain text, but you could for example LUKS mount an encrypted file on a loopback and store it inside that. The bigger question is, if you encrypt the file with your password in it, then where do you safely store the decryption key? Without knowing more about your use case, it sounds like you're back to the same problem you had with the password. If you're only using the software interactively anyway then just don't put the password in your configuration, enter it when prompted instead. You can also supply it as an environment variable (OS_PASSWORD) or command line argument (--os-password) if you don't want to be prompted but also don't want to put it in your configuration. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From roberto.acosta at luizalabs.com Wed Jul 12 12:11:41 2023 From: roberto.acosta at luizalabs.com (Roberto Bartzen Acosta) Date: Wed, 12 Jul 2023 09:11:41 -0300 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: Em qua., 12 de jul. de 2023 ?s 09:05, Roberto Bartzen Acosta < roberto.acosta at luizalabs.com> escreveu: > Hi Rodolfo, > > Thanks for the feedback. > > I liked the idea of having a different network type only to hold the > Transit Switches and thus the LRP but it depends on some factors like > multi-tenancy. > From the OVN perspective, the TS is global and will only be linked to a > tenant when it is plugged into the tenant router port. My concern here is > that if Neutron manages the TS, it will need to assume the dynamic > characteristics of this TS function, ovn-ic creates and removes the TS in > NB_Global, in addition to plugging and removing ports in this switch > according to the network topology. Additionally, Neutron would still need > to handle the learned remote routes (which are listed as static routes from > the tenant router perspective). > sorry: NB_Global -> Northbound Database. > This is an open discussion, Felix can add ideas here. > > In general, it seems to me that we have no alternatives for this type of > solution other than OVN-IC (note that ovn-bgp-agent does not learn remote > routes at the SDN level / OVN). > OpenStack seems to be designed to run like a "bubble" and this traffic > from one VM to another always needs to be routed at the FIP level and > external routing layers. > > Regards, > Roberto > > Em qua., 12 de jul. de 2023 ?s 05:11, Rodolfo Alonso Hernandez < > ralonsoh at redhat.com> escreveu: > >> Hello Roberto: >> >> We talked about this possible RFE during the PTG [1] but I don't remember >> having any proposal. Actually what I remember (and was written in the >> etherpad) is that you were going to present an RFE. Can you explain it >> briefly? >> >> We also talked about the idea, proposed by Felix in the mailing list, of >> having a different network type only to hold the Transit Switches and thus >> the LRP. If you have a proposal, please present it. >> >> Regards. >> >> [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506 >> >> On Tue, Jul 11, 2023 at 10:50?PM Roberto Bartzen Acosta < >> roberto.acosta at luizalabs.com> wrote: >> >>> Hello Neutron folks, >>> >>> Regarding the conversation started in March [1] about the use of OVN >>> interconnect with Neutron, I would like to evolve the discussion in >>> relation to resources allocated by OVN-IC and which are not managed by >>> Neutron. In the March PTG [2], the problem related to the db_sync tool was >>> presented, and a proposed solution in which Neutron does not manage these >>> resources. >>> >>> After discussing some architectural designs with Felix/StackIT, it seems >>> to make sense to come up with a proposal to change the mech_driver to >>> validate the external_ids key and not remove resources present in the OVN >>> backend without Neutron "signature". >>> >>> The discussion here is more complex than it seems, because >>> architecturally Neutron does not allow us to interconnect workloads in >>> multiple AZs (with different OpenStacks), but this could be a requirement >>> for a high availability cloud solution (Cloud Region). Additionally, this >>> OVN-IC solution allows interconnecting other cloud backends that use OVN, >>> in the case of kubernetes with ovn-kube. >>> >>> We tested an OVN interconnect integrated with 3 OpenStack installations >>> and it worked very well !!! considering direct L3 traffic at the router >>> level between different network infrastructures. >>> >>> SYNC_REPAIR - problem >>> >>> * Static Routes (learned OVN-IC routes) >>> * Router Port -> Transit Switches >>> >>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in >>> OVN but not in Neutron, port_id=rt2-admin-tenant1 >>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router >>> 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' >>> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '10.0.2.1/24', >>> 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron >>> >>> >>> Any suggestions on how to resolve this db_sync issue? since all other >>> Neutron modules did not present any problem with OVN-IC (as far as I >>> tested). >>> I remember Terry was keen to suggest some things to help. I believe that >>> before proposing some complex mechanism to solve this simple problem, I >>> would like to hear the community opinion. In our use case, a bit change in >>> db_sync with filter by neutron key in external_ids would be enough. >>> >>> Regards, >>> Roberto >>> >>> >>> >>> [1] >>> https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html >>> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg >>> >>> >>> >>> Additional logs: >>> >>> OpenStack 1 >>> >>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl >>> lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 >>> IPv4 Routes >>> Route Table
: >>> 20.0.1.0/24 169.254.200.2 dst-ip (learned) >>> 20.0.2.0/24 169.254.200.3 dst-ip (learned) >>> 0.0.0.0/0 200.200.200.1 dst-ip >>> >>> IPv6 Routes >>> Route Table
: >>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl >>> lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 >>> IPv4 Routes >>> Route Table
: >>> 10.0.1.0/24 169.254.100.2 dst-ip (learned) >>> 10.0.2.0/24 169.254.100.3 dst-ip (learned) >>> 0.0.0.0/0 200.200.200.1 dst-ip >>> >>> IPv6 Routes >>> Route Table
: >>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>> >>> >>> OpenStack 2 >>> >>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl >>> lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc >>> IPv4 Routes >>> Route Table
: >>> 20.0.0.0/24 169.254.200.1 dst-ip (learned) >>> 20.0.2.0/24 169.254.200.3 dst-ip (learned) >>> 0.0.0.0/0 200.200.200.1 dst-ip >>> >>> IPv6 Routes >>> Route Table
: >>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl >>> lr-route-list ce45f681-6454-43fe-974f-81344bb8113a >>> IPv4 Routes >>> Route Table
: >>> 10.0.0.0/24 169.254.100.1 dst-ip (learned) >>> 10.0.2.0/24 169.254.100.3 dst-ip (learned) >>> 0.0.0.0/0 200.200.200.1 dst-ip >>> >>> IPv6 Routes >>> Route Table
: >>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>> >>> >>> >>> OpenStack 3 >>> >>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl >>> lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 >>> IPv4 Routes >>> Route Table
: >>> 20.0.0.0/24 169.254.200.1 dst-ip (learned) >>> 20.0.1.0/24 169.254.200.2 dst-ip (learned) >>> 0.0.0.0/0 200.200.200.1 dst-ip >>> >>> IPv6 Routes >>> Route Table
: >>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl >>> lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 >>> IPv4 Routes >>> Route Table
: >>> 10.0.0.0/24 169.254.100.1 dst-ip (learned) >>> 10.0.1.0/24 169.254.100.2 dst-ip (learned) >>> 0.0.0.0/0 200.200.200.1 dst-ip >>> >>> IPv6 Routes >>> Route Table
: >>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>> >>> >>> OVN-IC Global database >>> >>> root at ovn-global-db1:~# ovn-ic-sbctl show >>> availability-zone osp1 >>> gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 >>> hostname: osp1-gwnode1 >>> type: geneve >>> ip: 192.168.200.28 >>> port admin-rt1-tenant1 >>> transit switch: admin-tenant1 >>> address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] >>> port admin-rt1-tenant1_1 >>> transit switch: admin-tenant1_1 >>> address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] >>> availability-zone osp2 >>> gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 >>> hostname: osp2-gwnode1 >>> type: geneve >>> ip: 192.168.200.128 >>> port admin-rt2-tenant1 >>> transit switch: admin-tenant1 >>> address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] >>> port admin-rt2-tenant1_1 >>> transit switch: admin-tenant1_1 >>> address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] >>> availability-zone osp3 >>> gateway 97595af9-7896-40d0-a883-beadbff1aa5b >>> hostname: osp3-gwnode1 >>> type: geneve >>> ip: 192.168.200.228 >>> port admin-rt3-tenant1 >>> transit switch: admin-tenant1 >>> address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] >>> port admin-rt3-tenant1_1 >>> transit switch: admin-tenant1_1 >>> address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] >>> >>> >>> >>> >>> >>> >>> >>> *?Esta mensagem ? direcionada apenas para os endere?os constantes no >>> cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no >>> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa >>> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o >>> imediatamente anuladas e proibidas?.* >>> >>> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para >>> assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o >>> poder? aceitar a responsabilidade por quaisquer perdas ou danos causados >>> por esse e-mail ou por seus anexos?.* >>> >> -- _?Esta mensagem ? direcionada apenas para os endere?os constantes no cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o imediatamente anuladas e proibidas?._ *?**?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o poder? aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos?.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From pdeore at redhat.com Wed Jul 12 13:17:00 2023 From: pdeore at redhat.com (Pranali Deore) Date: Wed, 12 Jul 2023 18:47:00 +0530 Subject: [Glance] Cancelling Weekly Meeting 13th July Message-ID: Hello, I will not be around during the meeting time due to some personal reasons and there is nothing much on agenda, so cancelling the meeting for this week. Let's meet next week on 20th July !! Thanks, Pranali D -------------- next part -------------- An HTML attachment was scrubbed... URL: From molenkam at uwo.ca Wed Jul 12 14:21:21 2023 From: molenkam at uwo.ca (Gary Molenkamp) Date: Wed, 12 Jul 2023 10:21:21 -0400 Subject: SNAT failure with OVN under Antelope In-Reply-To: <8511695f-443b-1593-fc20-04cb82cfc791@uwo.ca> References: <34841bdd-bb95-ff9a-258b-cfae26558627@uwo.ca> <536805d2-f334-e94f-1415-2984a219cb65@uwo.ca> <8511695f-443b-1593-fc20-04cb82cfc791@uwo.ca> Message-ID: <9c6b3c42-bf44-a59e-0fe8-b0f0430edf76@uwo.ca> For comparison, I looked at how openstack-ansible was setting up OVN and I don't see any major differences other than O-A configures a manager for ovs: ????? ovs-vsctl --id @manager create Manager "target=\ .... I don't believe this is the point of failure (but feel free to correct me if I'm wrong ;) ). ovn-trace on both VM's inports shows the same trace for the working VM and the non-working VM. ie: ovn-trace --db=$SB --ovs default_net 'inport == "f4cbc8c7-e7bf-47f3-9fea-a1663f6eb34d" && eth.src==fa:16:3e:a6:62:8e && ip4.src == 172.31.101.168 && ip4.dst == ' On 2023-07-07 14:08, Gary Molenkamp wrote: > Happy Friday afternoon. > > I'm still pondering a lack of connectivity in an HA OVN with each > compute node acting as a potential gateway chassis. > >>> The problem is basically that the port of the OVN LRP may not be >>> in the same chassis as the VM that failed (since the CR-LRP will >>> be where the first VM of that network will be created).?The >>> suggestion is to remove the enable-chassis-as-gw from the >>> compute nodes to allow the VM to forward traffic via >>> tunneling/Geneve to the chassis where the LRP resides. >>> >> >> I forced a similar VM onto the same chassis as the working VM, >> and it was able to communicate out.??? If we do want to keep >> multiple chassis' as gateways, would that be addressed with the >> ovn-bridge-mappings? >> >> > > I built a small test cloud to explore this further as I continue to > see the same issue:? A vm will only be able to use SNAT outbound if it > is on the same chassis as the CR-LRP. > > In my test cloud, I have one controller, and two compute nodes. The > controller only runs the north and southd in addition to the neutron > server.? Each of the two compute nodes is configured as below.? On a > tenent network I have three VMs: > ??? - #1:? cirros VM with FIP > ??? - #2:? cirros VM running on compute node 1 > ??? - #3:? cirros VM running on compute node 2 > > E/W traffic between VMs in the same tenent network are fine.? N/S > traffic is fine for the FIP.? N/S traffic only works for the VM whose > CR-LRP is active on same chassis.?? Does anything jump out as a > mistake in my understanding at to how this should be working? > > Thanks as always, > Gary > > > on each hypervisor: > > /usr/bin/ovs-vsctl set open . external-ids:ovn-remote=tcp:{{ > controllerip }}:6642 > /usr/bin/ovs-vsctl set open . external-ids:ovn-encap-type=geneve > /usr/bin/ovs-vsctl set open . external-ids:ovn-encap-ip={{ overlaynetip }} > /usr/bin/ovs-vsctl set open . > external-ids:ovn-cms-options=enable-chassis-as-gw > /usr/bin/ovs-vsctl add-br br-provider -- set bridge br-provider > protocols=OpenFlow10,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 > /usr/bin/ovs-vsctl add-port br-provider {{ provider_nic }} > /usr/bin/ovs-vsctl br-set-external-id provider bridge-id br-provider > /usr/bin/ovs-vsctl set open . > external-ids:ovn-bridge-mappings=provider:br-provider > > plugin.ini: > [ml2] > mechanism_drivers = ovn > type_drivers = flat,geneve > tenant_network_types = geneve > extension_drivers = port_security > overlay_ip_version = 4 > [ml2_type_flat] > flat_networks = provider > [ml2_type_geneve] > vni_ranges = 1:65536 > max_header_size = 38 > [securitygroup] > enable_security_group = True > firewall_driver = > neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver > [ovn] > ovn_nb_connection = tcp:{{controllerip}}:6641 > ovn_sb_connection = tcp:{{controllerip}}:6642 > ovn_l3_scheduler = leastloaded > ovn_metadata_enabled = True > enable_distributed_floating_ip = true > > > > > -- > Gary Molenkamp Science Technology Services > Systems Administrator University of Western Ontario > molenkam at uwo.ca http://sts.sci.uwo.ca > (519) 661-2111 x86882 (519) 661-3566 -- Gary Molenkamp Science Technology Services Systems Engineer University of Western Ontario molenkam at uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566 -------------- next part -------------- An HTML attachment was scrubbed... URL: From molenkam at uwo.ca Wed Jul 12 15:43:12 2023 From: molenkam at uwo.ca (Gary Molenkamp) Date: Wed, 12 Jul 2023 11:43:12 -0400 Subject: SNAT failure with OVN under Antelope In-Reply-To: <9c6b3c42-bf44-a59e-0fe8-b0f0430edf76@uwo.ca> References: <34841bdd-bb95-ff9a-258b-cfae26558627@uwo.ca> <536805d2-f334-e94f-1415-2984a219cb65@uwo.ca> <8511695f-443b-1593-fc20-04cb82cfc791@uwo.ca> <9c6b3c42-bf44-a59e-0fe8-b0f0430edf76@uwo.ca> Message-ID: A little progress, but I may be tripping over bug https://bugs.launchpad.net/neutron/+bug/2003455 If I remove the provider bridge from the second hypervisor: ??? ovs-vsctl remove open . external-ids ovn-cms-options="enable-chassis-as-gw" ??? ovs-vsctl remove open . external-ids ovn-bridge-mappings ??? ip link set br-provider down ? ? ovs-vsctl del-br br-provider and disable ??? enable_distributed_floating_ip Then both VMs using SNAT on each compute server work. Turning the second chassis back on as a gateway immediately breaks the VM on the second compute server: ??? ovs-vsctl set open . external-ids:ovn-cms-options=enable-chassis-as-gw ??? ovs-vsctl add-br br-provider ??? ovs-vsctl set open . external-ids:ovn-bridge-mappings=provider:br-provider ? ? ovs-vsctl add-port br-provider ens256 ??? systemctl restart ovn-controller openvswitch.service I am running neutron 22.0.1 but maybe something related? python3-neutron-22.0.1-1.el9s.noarch openstack-neutron-common-22.0.1-1.el9s.noarch openstack-neutron-22.0.1-1.el9s.noarch openstack-neutron-ml2-22.0.1-1.el9s.noarch openstack-neutron-openvswitch-22.0.1-1.el9s.noarch openstack-neutron-ovn-metadata-agent-22.0.1-1.el9s.noarch On 2023-07-12 10:21, Gary Molenkamp wrote: > For comparison, I looked at how openstack-ansible was setting up OVN > and I don't see any major differences other than O-A configures a > manager for ovs: > ????? ovs-vsctl --id @manager create Manager "target=\ .... > I don't believe this is the point of failure (but feel free to correct > me if I'm wrong ;) ). > > ovn-trace on both VM's inports shows the same trace for the working VM > and the non-working VM. ie: > > ovn-trace --db=$SB --ovs default_net 'inport == > "f4cbc8c7-e7bf-47f3-9fea-a1663f6eb34d" && eth.src==fa:16:3e:a6:62:8e > && ip4.src == 172.31.101.168 && ip4.dst == ' > > > > On 2023-07-07 14:08, Gary Molenkamp wrote: >> Happy Friday afternoon. >> >> I'm still pondering a lack of connectivity in an HA OVN with each >> compute node acting as a potential gateway chassis. >> >>>> The problem is basically that the port of the OVN LRP may not >>>> be in the same chassis as the VM that failed (since the CR-LRP >>>> will be where the first VM of that network will be >>>> created).?The suggestion is to remove the enable-chassis-as-gw >>>> from the compute nodes to allow the VM to forward traffic via >>>> tunneling/Geneve to the chassis where the LRP resides. >>>> >>> >>> I forced a similar VM onto the same chassis as the working VM, >>> and it was able to communicate out.??? If we do want to keep >>> multiple chassis' as gateways, would that be addressed with the >>> ovn-bridge-mappings? >>> >>> >> >> I built a small test cloud to explore this further as I continue to >> see the same issue:? A vm will only be able to use SNAT outbound if >> it is on the same chassis as the CR-LRP. >> >> In my test cloud, I have one controller, and two compute nodes. The >> controller only runs the north and southd in addition to the neutron >> server.? Each of the two compute nodes is configured as below.? On a >> tenent network I have three VMs: >> ??? - #1:? cirros VM with FIP >> ??? - #2:? cirros VM running on compute node 1 >> ??? - #3:? cirros VM running on compute node 2 >> >> E/W traffic between VMs in the same tenent network are fine. N/S >> traffic is fine for the FIP.? N/S traffic only works for the VM whose >> CR-LRP is active on same chassis.?? Does anything jump out as a >> mistake in my understanding at to how this should be working? >> >> Thanks as always, >> Gary >> >> >> on each hypervisor: >> >> /usr/bin/ovs-vsctl set open . external-ids:ovn-remote=tcp:{{ >> controllerip }}:6642 >> /usr/bin/ovs-vsctl set open . external-ids:ovn-encap-type=geneve >> /usr/bin/ovs-vsctl set open . external-ids:ovn-encap-ip={{ >> overlaynetip }} >> /usr/bin/ovs-vsctl set open . >> external-ids:ovn-cms-options=enable-chassis-as-gw >> /usr/bin/ovs-vsctl add-br br-provider -- set bridge br-provider >> protocols=OpenFlow10,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 >> /usr/bin/ovs-vsctl add-port br-provider {{ provider_nic }} >> /usr/bin/ovs-vsctl br-set-external-id provider bridge-id br-provider >> /usr/bin/ovs-vsctl set open . >> external-ids:ovn-bridge-mappings=provider:br-provider >> >> plugin.ini: >> [ml2] >> mechanism_drivers = ovn >> type_drivers = flat,geneve >> tenant_network_types = geneve >> extension_drivers = port_security >> overlay_ip_version = 4 >> [ml2_type_flat] >> flat_networks = provider >> [ml2_type_geneve] >> vni_ranges = 1:65536 >> max_header_size = 38 >> [securitygroup] >> enable_security_group = True >> firewall_driver = >> neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver >> [ovn] >> ovn_nb_connection = tcp:{{controllerip}}:6641 >> ovn_sb_connection = tcp:{{controllerip}}:6642 >> ovn_l3_scheduler = leastloaded >> ovn_metadata_enabled = True >> enable_distributed_floating_ip = true >> >> >> >> >> -- >> Gary Molenkamp Science Technology Services >> Systems Administrator University of Western Ontario >> molenkam at uwo.ca http://sts.sci.uwo.ca >> (519) 661-2111 x86882 (519) 661-3566 > > -- > Gary Molenkamp Science Technology Services > Systems Engineer University of Western Ontario > molenkam at uwo.ca http://sts.sci.uwo.ca > (519) 661-2111 x86882 (519) 661-3566 -- Gary Molenkamp Science Technology Services Systems Engineer University of Western Ontario molenkam at uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566 -------------- next part -------------- An HTML attachment was scrubbed... URL: From nguyenhuukhoinw at gmail.com Wed Jul 12 16:46:39 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Wed, 12 Jul 2023 23:46:39 +0700 Subject: [kolla-ansible][octavia] Message-ID: Hello guys. I am using openstack yoga by kolla-ansible but when i reconfigure octavia then it stucked at TASK [octavia : Create loadbalancer management network] \"Unable to create the network. The VLAN xxx on physical network physnet1 is in use.\ I see that it cannot create lb network because someone used it. Will we need to change some kolla-ansible tasks to fix it? Thank you very much. Nguyen Huu Khoi -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafaelweingartner at gmail.com Wed Jul 12 16:50:59 2023 From: rafaelweingartner at gmail.com (=?UTF-8?Q?Rafael_Weing=C3=A4rtner?=) Date: Wed, 12 Jul 2023 13:50:59 -0300 Subject: [horizon][keystone] Adding different rules in the same protocol for federated logon In-Reply-To: References: Message-ID: The mapping is one to one. You will not be able to easily map N domains that come as attributes from the IdP to a user in Keystone via the current identity federation implementation. We started an initiative to make that more flexible, but the specs were never accepted. You can see specs [1] and [2]. The spec [1] is not about this per se, but it is the base to enable us to better evolve the attribute mapping process without causing backwards impacts. However, it was never accepted. Also, the spec [2] is something that we did to achieve what you want with the domain, but applied at a project level. Therefore, if we had those in, it would be easy to expand to other use cases, such as the one you are describing. [1] https://review.opendev.org/c/openstack/keystone-specs/+/748042?usp=search [2] https://review.opendev.org/c/openstack/keystone-specs/+/748748?usp=search On Tue, Jul 11, 2023 at 10:26?PM James Leong wrote: > Hi all, > > I have yoga version openstack with the deployment tool of kolla-ansible. I > am trying to combine different mapping rules such as allowing user to login > to different domain. However, I am not able to do that in a single JSON > file. When I try to include different rule in the same JSON file, only the > first rule is being considered. Is there a way to allow multiple rule to > redirect user to their account in a different domain. > > Best, > James > -- Rafael Weing?rtner -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.acosta at luizalabs.com Wed Jul 12 12:03:09 2023 From: roberto.acosta at luizalabs.com (Roberto Bartzen Acosta) Date: Wed, 12 Jul 2023 09:03:09 -0300 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: Hi Rodolfo, Thanks for the feedback. I liked the idea of having a different network type only to hold the Transit Switches and thus the LRP but it depends on some factors like multi-tenancy. >From the OVN perspective, the TS is global and will only be linked to a tenant when it is plugged into the tenant router port. My concern here is that if Neutron manages the TS, it will need to assume the dynamic characteristics of this TS function, ovn-ic creates and removes the TS in NB_Global, in addition to plugging and removing ports in this switch according to the network topology. Additionally, Neutron would still need to handle the learned remote routes (which are listed as static routes from the tenant router perspective). This is an open discussion, Felix can add ideas here. In general, it seems to me that we have no alternatives for this type of solution other than OVN-IC (note that ovn-bgp-agent does not learn remote routes at the SDN level / OVN). OpenStack seems to be designed to run like a "bubble" and this traffic from one VM to another always needs to be routed at the FIP level and external routing layers. Regards, Roberto [image: image.png] Em qua., 12 de jul. de 2023 ?s 05:11, Rodolfo Alonso Hernandez < ralonsoh at redhat.com> escreveu: > Hello Roberto: > > We talked about this possible RFE during the PTG [1] but I don't remember > having any proposal. Actually what I remember (and was written in the > etherpad) is that you were going to present an RFE. Can you explain it > briefly? > > We also talked about the idea, proposed by Felix in the mailing list, of > having a different network type only to hold the Transit Switches and thus > the LRP. If you have a proposal, please present it. > > Regards. > > [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506 > > On Tue, Jul 11, 2023 at 10:50?PM Roberto Bartzen Acosta < > roberto.acosta at luizalabs.com> wrote: > >> Hello Neutron folks, >> >> Regarding the conversation started in March [1] about the use of OVN >> interconnect with Neutron, I would like to evolve the discussion in >> relation to resources allocated by OVN-IC and which are not managed by >> Neutron. In the March PTG [2], the problem related to the db_sync tool was >> presented, and a proposed solution in which Neutron does not manage these >> resources. >> >> After discussing some architectural designs with Felix/StackIT, it seems >> to make sense to come up with a proposal to change the mech_driver to >> validate the external_ids key and not remove resources present in the OVN >> backend without Neutron "signature". >> >> The discussion here is more complex than it seems, because >> architecturally Neutron does not allow us to interconnect workloads in >> multiple AZs (with different OpenStacks), but this could be a requirement >> for a high availability cloud solution (Cloud Region). Additionally, this >> OVN-IC solution allows interconnecting other cloud backends that use OVN, >> in the case of kubernetes with ovn-kube. >> >> We tested an OVN interconnect integrated with 3 OpenStack installations >> and it worked very well !!! considering direct L3 traffic at the router >> level between different network infrastructures. >> >> SYNC_REPAIR - problem >> >> * Static Routes (learned OVN-IC routes) >> * Router Port -> Transit Switches >> >> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in >> OVN but not in Neutron, port_id=rt2-admin-tenant1 >> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router >> 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' >> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '10.0.2.1/24', >> 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron >> >> >> Any suggestions on how to resolve this db_sync issue? since all other >> Neutron modules did not present any problem with OVN-IC (as far as I >> tested). >> I remember Terry was keen to suggest some things to help. I believe that >> before proposing some complex mechanism to solve this simple problem, I >> would like to hear the community opinion. In our use case, a bit change in >> db_sync with filter by neutron key in external_ids would be enough. >> >> Regards, >> Roberto >> >> >> >> [1] >> https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html >> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg >> >> >> >> Additional logs: >> >> OpenStack 1 >> >> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl >> lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 >> IPv4 Routes >> Route Table
: >> 20.0.1.0/24 169.254.200.2 dst-ip (learned) >> 20.0.2.0/24 169.254.200.3 dst-ip (learned) >> 0.0.0.0/0 200.200.200.1 dst-ip >> >> IPv6 Routes >> Route Table
: >> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl >> lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 >> IPv4 Routes >> Route Table
: >> 10.0.1.0/24 169.254.100.2 dst-ip (learned) >> 10.0.2.0/24 169.254.100.3 dst-ip (learned) >> 0.0.0.0/0 200.200.200.1 dst-ip >> >> IPv6 Routes >> Route Table
: >> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >> >> >> OpenStack 2 >> >> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl >> lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc >> IPv4 Routes >> Route Table
: >> 20.0.0.0/24 169.254.200.1 dst-ip (learned) >> 20.0.2.0/24 169.254.200.3 dst-ip (learned) >> 0.0.0.0/0 200.200.200.1 dst-ip >> >> IPv6 Routes >> Route Table
: >> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl >> lr-route-list ce45f681-6454-43fe-974f-81344bb8113a >> IPv4 Routes >> Route Table
: >> 10.0.0.0/24 169.254.100.1 dst-ip (learned) >> 10.0.2.0/24 169.254.100.3 dst-ip (learned) >> 0.0.0.0/0 200.200.200.1 dst-ip >> >> IPv6 Routes >> Route Table
: >> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> >> >> >> OpenStack 3 >> >> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl >> lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 >> IPv4 Routes >> Route Table
: >> 20.0.0.0/24 169.254.200.1 dst-ip (learned) >> 20.0.1.0/24 169.254.200.2 dst-ip (learned) >> 0.0.0.0/0 200.200.200.1 dst-ip >> >> IPv6 Routes >> Route Table
: >> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl >> lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 >> IPv4 Routes >> Route Table
: >> 10.0.0.0/24 169.254.100.1 dst-ip (learned) >> 10.0.1.0/24 169.254.100.2 dst-ip (learned) >> 0.0.0.0/0 200.200.200.1 dst-ip >> >> IPv6 Routes >> Route Table
: >> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> >> >> OVN-IC Global database >> >> root at ovn-global-db1:~# ovn-ic-sbctl show >> availability-zone osp1 >> gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 >> hostname: osp1-gwnode1 >> type: geneve >> ip: 192.168.200.28 >> port admin-rt1-tenant1 >> transit switch: admin-tenant1 >> address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] >> port admin-rt1-tenant1_1 >> transit switch: admin-tenant1_1 >> address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] >> availability-zone osp2 >> gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 >> hostname: osp2-gwnode1 >> type: geneve >> ip: 192.168.200.128 >> port admin-rt2-tenant1 >> transit switch: admin-tenant1 >> address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] >> port admin-rt2-tenant1_1 >> transit switch: admin-tenant1_1 >> address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] >> availability-zone osp3 >> gateway 97595af9-7896-40d0-a883-beadbff1aa5b >> hostname: osp3-gwnode1 >> type: geneve >> ip: 192.168.200.228 >> port admin-rt3-tenant1 >> transit switch: admin-tenant1 >> address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] >> port admin-rt3-tenant1_1 >> transit switch: admin-tenant1_1 >> address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] >> >> >> >> >> >> >> >> *?Esta mensagem ? direcionada apenas para os endere?os constantes no >> cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no >> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa >> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o >> imediatamente anuladas e proibidas?.* >> >> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para >> assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o >> poder? aceitar a responsabilidade por quaisquer perdas ou danos causados >> por esse e-mail ou por seus anexos?.* >> > -- _?Esta mensagem ? direcionada apenas para os endere?os constantes no cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o imediatamente anuladas e proibidas?._ *?**?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o poder? aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos?.* -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 102681 bytes Desc: not available URL: From jamesleong123098 at gmail.com Wed Jul 12 22:02:57 2023 From: jamesleong123098 at gmail.com (James Leong) Date: Wed, 12 Jul 2023 17:02:57 -0500 Subject: [horizon][keystone] Adding different rules in the same protocol for federated logon In-Reply-To: References: Message-ID: Thanks for the explanation. I was thinking to make the domain name as part of the oidc-organization, so it would map to the domain dynamically. Best, James On Wed, 12 Jul 2023, 11:51 am Rafael Weing?rtner, < rafaelweingartner at gmail.com> wrote: > The mapping is one to one. You will not be able to easily map N domains > that come as attributes from the IdP to a user in Keystone via the current > identity federation implementation. We started an initiative to make that > more flexible, but the specs were never accepted. You can see specs [1] and > [2]. The spec [1] is not about this per se, but it is the base to enable us > to better evolve the attribute mapping process without causing backwards > impacts. However, it was never accepted. Also, the spec [2] is something > that we did to achieve what you want with the domain, but applied at a > project level. Therefore, if we had those in, it would be easy to expand to > other use cases, such as the one you are describing. > > [1] > https://review.opendev.org/c/openstack/keystone-specs/+/748042?usp=search > [2] > https://review.opendev.org/c/openstack/keystone-specs/+/748748?usp=search > > On Tue, Jul 11, 2023 at 10:26?PM James Leong > wrote: > >> Hi all, >> >> I have yoga version openstack with the deployment tool of kolla-ansible. >> I am trying to combine different mapping rules such as allowing user to >> login to different domain. However, I am not able to do that in a single >> JSON file. When I try to include different rule in the same JSON file, only >> the first rule is being considered. Is there a way to allow multiple rule >> to redirect user to their account in a different domain. >> >> Best, >> James >> > > > -- > Rafael Weing?rtner > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rafaelweingartner at gmail.com Wed Jul 12 22:18:57 2023 From: rafaelweingartner at gmail.com (=?UTF-8?Q?Rafael_Weing=C3=A4rtner?=) Date: Wed, 12 Jul 2023 19:18:57 -0300 Subject: [horizon][keystone] Adding different rules in the same protocol for federated logon In-Reply-To: References: Message-ID: Yes, it makes sense. However, it would work only for single domain mapping. If you need something more dynamic, then with the current implementation that is not possible. On Wed, Jul 12, 2023 at 7:03?PM James Leong wrote: > Thanks for the explanation. I was thinking to make the domain name as part > of the oidc-organization, so it would map to the domain dynamically. > > Best, > James > > On Wed, 12 Jul 2023, 11:51 am Rafael Weing?rtner, < > rafaelweingartner at gmail.com> wrote: > >> The mapping is one to one. You will not be able to easily map N domains >> that come as attributes from the IdP to a user in Keystone via the current >> identity federation implementation. We started an initiative to make that >> more flexible, but the specs were never accepted. You can see specs [1] and >> [2]. The spec [1] is not about this per se, but it is the base to enable us >> to better evolve the attribute mapping process without causing backwards >> impacts. However, it was never accepted. Also, the spec [2] is something >> that we did to achieve what you want with the domain, but applied at a >> project level. Therefore, if we had those in, it would be easy to expand to >> other use cases, such as the one you are describing. >> >> [1] >> https://review.opendev.org/c/openstack/keystone-specs/+/748042?usp=search >> [2] >> https://review.opendev.org/c/openstack/keystone-specs/+/748748?usp=search >> >> On Tue, Jul 11, 2023 at 10:26?PM James Leong >> wrote: >> >>> Hi all, >>> >>> I have yoga version openstack with the deployment tool of kolla-ansible. >>> I am trying to combine different mapping rules such as allowing user to >>> login to different domain. However, I am not able to do that in a single >>> JSON file. When I try to include different rule in the same JSON file, only >>> the first rule is being considered. Is there a way to allow multiple rule >>> to redirect user to their account in a different domain. >>> >>> Best, >>> James >>> >> >> >> -- >> Rafael Weing?rtner >> > -- Rafael Weing?rtner -------------- next part -------------- An HTML attachment was scrubbed... URL: From jamesleong123098 at gmail.com Thu Jul 13 01:42:02 2023 From: jamesleong123098 at gmail.com (James Leong) Date: Wed, 12 Jul 2023 20:42:02 -0500 Subject: [horizon][keystone] Adding different rules in the same protocol for federated logon In-Reply-To: References: Message-ID: Ok, thanks for the clarification. On Wed, 12 Jul 2023, 5:19 pm Rafael Weing?rtner, < rafaelweingartner at gmail.com> wrote: > Yes, it makes sense. However, it would work only for single domain > mapping. If you need something more dynamic, then with the current > implementation that is not possible. > > On Wed, Jul 12, 2023 at 7:03?PM James Leong > wrote: > >> Thanks for the explanation. I was thinking to make the domain name as >> part of the oidc-organization, so it would map to the domain dynamically. >> >> Best, >> James >> >> On Wed, 12 Jul 2023, 11:51 am Rafael Weing?rtner, < >> rafaelweingartner at gmail.com> wrote: >> >>> The mapping is one to one. You will not be able to easily map N domains >>> that come as attributes from the IdP to a user in Keystone via the current >>> identity federation implementation. We started an initiative to make that >>> more flexible, but the specs were never accepted. You can see specs [1] and >>> [2]. The spec [1] is not about this per se, but it is the base to enable us >>> to better evolve the attribute mapping process without causing backwards >>> impacts. However, it was never accepted. Also, the spec [2] is something >>> that we did to achieve what you want with the domain, but applied at a >>> project level. Therefore, if we had those in, it would be easy to expand to >>> other use cases, such as the one you are describing. >>> >>> [1] >>> https://review.opendev.org/c/openstack/keystone-specs/+/748042?usp=search >>> [2] >>> https://review.opendev.org/c/openstack/keystone-specs/+/748748?usp=search >>> >>> On Tue, Jul 11, 2023 at 10:26?PM James Leong >>> wrote: >>> >>>> Hi all, >>>> >>>> I have yoga version openstack with the deployment tool of >>>> kolla-ansible. I am trying to combine different mapping rules such as >>>> allowing user to login to different domain. However, I am not able to do >>>> that in a single JSON file. When I try to include different rule in the >>>> same JSON file, only the first rule is being considered. Is there a way to >>>> allow multiple rule to redirect user to their account in a different domain. >>>> >>>> Best, >>>> James >>>> >>> >>> >>> -- >>> Rafael Weing?rtner >>> >> > > -- > Rafael Weing?rtner > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wchy1001 at gmail.com Thu Jul 13 01:58:01 2023 From: wchy1001 at gmail.com (chunyang wu) Date: Thu, 13 Jul 2023 09:58:01 +0800 Subject: [kolla-ansible][octavia] In-Reply-To: References: Message-ID: Hi, you should check which network is using the vlan xxx in neutron, if the network is not intended for octavia management, you should delete it or change a new vlan id for octavia. otherwise, you could try to stop creating management network in kolla deployment by the flowing steps: I saw you perform a reconfigure action, so i assume you have an already worked cluster. 1. set octavia_auto_configure to false in you global.yaml. 2. set the flowing opts in gloabl.yaml octavia_amp_image_owner_id: the value you can find in your exists config file(e.g. /etc/kolla/octavia-api/octavia.conf : amp_image_owner_id) octavia_amp_boot_network_list : the vale of `amp_boot_network_list in octavia.conf octavia_amp_secgroup_list: the value of `amp_secgroup_list` in octavia.conf octavia_amp_flavor_id: the value of `amp_flavor_id ` in "octavia.config" 3. re-run reconfigure action. refer to: https://github.com/openstack/kolla-ansible/blob/master/etc/kolla/globals.yml#L778 thanks. Nguy?n H?u Kh?i ?2023?7?13??? 00:48??? > Hello guys. > > I am using openstack yoga by kolla-ansible but when i reconfigure octavia > then it stucked at > > TASK [octavia : Create loadbalancer management network] > > \"Unable to create the network. The VLAN xxx on physical network physnet1 > is in use.\ > > I see that it cannot create lb network because someone used it. > > Will we need to change some kolla-ansible tasks to fix it? > > > Thank you very much. > > Nguyen Huu Khoi > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nguyenhuukhoinw at gmail.com Thu Jul 13 03:33:39 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Thu, 13 Jul 2023 10:33:39 +0700 Subject: [kolla-ansible][octavia] In-Reply-To: References: Message-ID: Thank very much. I got it, I will try and feedback :) Nguyen Huu Khoi On Thu, Jul 13, 2023 at 8:58?AM chunyang wu wrote: > Hi, > you should check which network is using the vlan xxx in neutron, if the > network is not intended for octavia management, you should delete it or > change a new vlan id for octavia. otherwise, you could try to stop creating > management network in kolla deployment by the flowing steps: > I saw you perform a reconfigure action, so i assume you have an > already worked cluster. > 1. set octavia_auto_configure to false in you global.yaml. > 2. set the flowing opts in gloabl.yaml > octavia_amp_image_owner_id: the value you can find in your exists > config file(e.g. /etc/kolla/octavia-api/octavia.conf : amp_image_owner_id) > octavia_amp_boot_network_list : the vale of `amp_boot_network_list in > octavia.conf > octavia_amp_secgroup_list: the value of `amp_secgroup_list` in > octavia.conf > octavia_amp_flavor_id: the value of `amp_flavor_id ` in > "octavia.config" > 3. re-run reconfigure action. > > refer to: > https://github.com/openstack/kolla-ansible/blob/master/etc/kolla/globals.yml#L778 > > thanks. > > Nguy?n H?u Kh?i ?2023?7?13??? 00:48??? > >> Hello guys. >> >> I am using openstack yoga by kolla-ansible but when i reconfigure octavia >> then it stucked at >> >> TASK [octavia : Create loadbalancer management network] >> >> \"Unable to create the network. The VLAN xxx on physical network physnet1 >> is in use.\ >> >> I see that it cannot create lb network because someone used it. >> >> Will we need to change some kolla-ansible tasks to fix it? >> >> >> Thank you very much. >> >> Nguyen Huu Khoi >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdulko at redhat.com Thu Jul 13 06:47:44 2023 From: mdulko at redhat.com (=?UTF-8?Q?Micha=C5=82?= Dulko) Date: Thu, 13 Jul 2023 08:47:44 +0200 Subject: [kuryr][tacker] fails to bump the k8s version to v1.26 In-Reply-To: References: Message-ID: On Tue, 2023-07-04 at 07:53 +0000, Ayumu Ueha (Fujitsu) wrote: > > > > Hi kuryr-kubetenetes team, > ? > Tacker uses the kuryr-kubernetes as the setup for the kubernetes > Environment. > In the Bobcat version, we will bump the supported version of > kubenetes to 1.26.6, > and when I tried to bump the version in devstack's local.conf by the > patch [1], > the following error occurred. (please refer the details to Zuul log > [2]) > ============================== > [wait-control-plane] Waiting for the kubelet to boot up the control > plane as static Pods from directory "/etc/kubernetes/manifests". This > can take up to 4m0s > [kubelet-check] Initial timeout of 40s passed. > ? > Unfortunately, an error has occurred: > ????????????? timed out waiting for the condition > ? > This error is likely caused by: > ????????????? - The kubelet is not running > ????????????? - The kubelet is unhealthy due to a misconfiguration of > the node in some way (required cgroups disabled) > ? > If you are on a systemd-powered system, you can try to troubleshoot > the error with the following commands: > ????????????? - 'systemctl status kubelet' > ????????????? - 'journalctl -xeu kubelet' > ? > Additionally, a control plane component may have crashed or exited > when started by the container runtime. > To troubleshoot, list all containers using your preferred container > runtimes CLI. > Here is one example how you may list all running Kubernetes > containers by using crictl: > ????????????? - 'crictl --runtime-endpoint > unix:///var/run/crio/crio.sock ps -a | grep kube | grep -v pause' > ????????????? Once you have found the failing container, you can > inspect its logs with: > ????????????? - 'crictl --runtime-endpoint > unix:///var/run/crio/crio.sock logs CONTAINERID' > error execution phase wait-control-plane: couldn't initialize a > Kubernetes cluster > ============================== > ? > I know it is not yet supporting version 1.26 of the K8S at the kuryr- > kubernetes as for now, but do you know any way to avoid the above > error? > I suspect this is due to a change from version 1.25, but I'm not sure > which one is affecting... > ? > Also, when will you support the K8S v 1.26 at the kuryr-kubernetes? > Will the Bobcat release support it? > Please kindly let me know if you know anything. Thank you. Kuryr-kubernetes itself supports K8s 1.26 and 1.27 as we test it with these versions as part of OpenShift. What doesn't seem to work is the DevStack plugin, I guess due to some breaking changes in kubelet or kubeadm. I can advise to try to deploy DevStack with these settings on your local machine and then you should have access to all the logs required to figure out what is causing the problem with the version bump. > [1] https://review.opendev.org/c/openstack/tacker/+/886935 > ??Change the parameters: > ?????KURYR_KUBERNETES_VERSION: 1.25.6 to 1.26.6 > ?????CRIO_VERSION: 1.25 to 1.26 > [2] > https://zuul.opendev.org/t/openstack/build/1a4061da74e640368da133ba219b54d9/log/controller-k8s/logs/devstacklog.txt#9761-9816 > ? > Best Regards, > Ueha From geguileo at redhat.com Thu Jul 13 10:09:54 2023 From: geguileo at redhat.com (Gorka Eguileor) Date: Thu, 13 Jul 2023 12:09:54 +0200 Subject: [cinder]cinder qos question. In-Reply-To: References: Message-ID: <20230713100954.uktezddliezilack@localhost> On 27/06, Nguy?n H?u Kh?i wrote: > Hello guys. > I am trying to use cinder qos. > But i have question. > - I added qos after a volume was create and I see that qos did not apply to > this volume. > - i create new volume with assosiate qos then it is ok but when i changed > extra specs in qos, volume keep old specs. > Is there any way to make qos apply with two above cases. > Many thanks. Hi, I'm surprised you can even do it, I would expect this would be similar to changing a volume type when there's already a volume using it, which is not allowed. I assume you are using 'front-end' or 'both' QoS provider, because the situation is even worse for 'back-end' QoS providers. For front-end qos the values will NOT be updated in Nova until a new attachment is created, but it must be one that creates new connection information. Changing that would be a new feature requiring changes in Cinder and Nova, since Cinder would need to go through all the volumes that use volumes types associated with that QoS, check which ones are currently attached, update their connection_information in the cinder DB, and then report to Nova the new values (this API doesn't exist). For back-end QoS it would be driver dependent. I assume in most cases a retype would be necessary, but in some drivers they may be updating the QoS on the backed for all volumes once a new volume with that volume type or QoS is created. Cheers, Gorka. From geguileo at redhat.com Thu Jul 13 10:18:08 2023 From: geguileo at redhat.com (Gorka Eguileor) Date: Thu, 13 Jul 2023 12:18:08 +0200 Subject: [cinder-backup][kolla] mix cinder-volume and cinder-backup types (nfs, ceph) In-Reply-To: References: Message-ID: <20230713101808.xylu4vngan4mkqek@localhost> On 23/06, garcetto wrote: > good evening, > it is possible to mix cinder-volume backend (say ceph) and cinder-backup > backend (say nfs)? > and also do incrementals backup? > thank you Hi, In general the answer is YES, you can backup from any volume backend to any backup backend. Unfortunately Ceph is slightly different (it was the first one to do incremental and is the most efficient if you do Ceph to Ceph), so there have been some bugs related to backing to non Ceph backends in the past. I believe these have been fixed already, but I'm not completely sure and in which releases. The best way would be to test in your desired release. Cheers, Gorka. From mdulko at redhat.com Thu Jul 13 10:52:57 2023 From: mdulko at redhat.com (=?UTF-8?Q?Micha=C5=82?= Dulko) Date: Thu, 13 Jul 2023 12:52:57 +0200 Subject: [kuryr][tacker] fails to bump the k8s version to v1.26 In-Reply-To: References: Message-ID: <828063778057244c6f52b3d8c1eb99c730f664fd.camel@redhat.com> On Thu, 2023-07-13 at 10:16 +0000, Ayumu Ueha (Fujitsu) wrote: > Hi Micha?, > > Thanks for your reply, > > I trying it in my local machine and check logs, but I am not familiar > with troubleshooting with kubelet/kubeadm, so I do not know why it > errored as for now. > We will continue to investigate, but it would be helpful if you could > share any small findings. > > For your reference, I will attach a log of the kubelet.service when > it failed in my local environment. Looks like kube-apiserver and kube-scheduler are failing to start. Checking logs of them would be my next idea. You might want to use `crictl` to find them. > Thank you. > > Best Regards, > Ueha > > -----Original Message----- > From: Micha? Dulko > Sent: Thursday, July 13, 2023 3:48 PM > To: openstack-discuss at lists.openstack.org > Subject: Re: [kuryr][tacker] fails to bump the k8s version to v1.26 > > On Tue, 2023-07-04 at 07:53 +0000, Ayumu Ueha (Fujitsu) wrote: > > > > > > > > Hi kuryr-kubetenetes team, > > ? > > Tacker uses the kuryr-kubernetes as the setup for the kubernetes > > Environment. > > In the Bobcat version, we will bump the supported version of > > kubenetes > > to 1.26.6, and when I tried to bump the version in devstack's > > local.conf by the patch [1], the following error occurred. (please > > refer the details to Zuul log > > [2]) > > ============================== > > [wait-control-plane] Waiting for the kubelet to boot up the control > > plane as static Pods from directory "/etc/kubernetes/manifests". > > This > > can take up to 4m0s [kubelet-check] Initial timeout of 40s passed. > > ? > > Unfortunately, an error has occurred: > > ????????????? timed out waiting for the condition > > ? > > This error is likely caused by: > > ????????????? - The kubelet is not running > > ????????????? - The kubelet is unhealthy due to a misconfiguration > > of > > the node in some way (required cgroups disabled) > > ? > > If you are on a systemd-powered system, you can try to troubleshoot > > the error with the following commands: > > ????????????? - 'systemctl status kubelet' > > ????????????? - 'journalctl -xeu kubelet' > > ? > > Additionally, a control plane component may have crashed or exited > > when started by the container runtime. > > To troubleshoot, list all containers using your preferred container > > runtimes CLI. > > Here is one example how you may list all running Kubernetes > > containers > > by using crictl: > > ????????????? - 'crictl --runtime-endpoint > > unix:///var/run/crio/crio.sock ps -a | grep kube | grep -v pause' > > ????????????? Once you have found the failing container, you can > > inspect its logs with: > > ????????????? - 'crictl --runtime-endpoint > > unix:///var/run/crio/crio.sock logs CONTAINERID' > > error execution phase wait-control-plane: couldn't initialize a > > Kubernetes cluster ============================== > > ? > > I know it is not yet supporting version 1.26 of the K8S at the > > kuryr- > > kubernetes as for now, but do you know any way to avoid the above > > error? > > I suspect this is due to a change from version 1.25, but I'm not > > sure > > which one is affecting... > > ? > > Also, when will you support the K8S v 1.26 at the kuryr-kubernetes? > > Will the Bobcat release support it? > > Please kindly let me know if you know anything. Thank you. > > Kuryr-kubernetes itself supports K8s 1.26 and 1.27 as we test it with > these versions as part of OpenShift. What doesn't seem to work is the > DevStack plugin, I guess due to some breaking changes in kubelet or > kubeadm. > > I can advise to try to deploy DevStack with these settings on your > local machine and then you should have access to all the logs > required to figure out what is causing the problem with the version > bump. > > > [1] https://review.opendev.org/c/openstack/tacker/+/886935 > > ??Change the parameters: > > ?????KURYR_KUBERNETES_VERSION: 1.25.6 to 1.26.6 > > ?????CRIO_VERSION: 1.25 to 1.26 > > [2] > > https://zuul.opendev.org/t/openstack/build/1a4061da74e640368da133ba219 > > b54d9/log/controller-k8s/logs/devstacklog.txt#9761-9816 > > ? > > Best Regards, > > Ueha > > From nguyenhuukhoinw at gmail.com Thu Jul 13 11:17:38 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Thu, 13 Jul 2023 18:17:38 +0700 Subject: [cinder]cinder qos question. In-Reply-To: <20230713100954.uktezddliezilack@localhost> References: <20230713100954.uktezddliezilack@localhost> Message-ID: hello. Thank you for your valuable information, I got this problem. Can you help me see how nova will be updated? I ask about something which I can touch or modify at nova manually, Nguyen Huu Khoi On Thu, Jul 13, 2023 at 5:10?PM Gorka Eguileor wrote: > On 27/06, Nguy?n H?u Kh?i wrote: > > Hello guys. > > I am trying to use cinder qos. > > But i have question. > > - I added qos after a volume was create and I see that qos did not apply > to > > this volume. > > - i create new volume with assosiate qos then it is ok but when i changed > > extra specs in qos, volume keep old specs. > > Is there any way to make qos apply with two above cases. > > Many thanks. > > Hi, > > I'm surprised you can even do it, I would expect this would be similar > to changing a volume type when there's already a volume using it, which > is not allowed. > > I assume you are using 'front-end' or 'both' QoS provider, because the > situation is even worse for 'back-end' QoS providers. > > For front-end qos the values will NOT be updated in Nova until a new > attachment is created, but it must be one that creates new connection > information. > > Changing that would be a new feature requiring changes in Cinder and > Nova, since Cinder would need to go through all the volumes that use > volumes types associated with that QoS, check which ones are currently > attached, update their connection_information in the cinder DB, and then > report to Nova the new values (this API doesn't exist). > > For back-end QoS it would be driver dependent. I assume in most cases a > retype would be necessary, but in some drivers they may be updating the > QoS on the backed for all volumes once a new volume with that volume > type or QoS is created. > > Cheers, > Gorka. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From geguileo at redhat.com Thu Jul 13 11:44:56 2023 From: geguileo at redhat.com (Gorka Eguileor) Date: Thu, 13 Jul 2023 13:44:56 +0200 Subject: [cinder]cinder qos question. In-Reply-To: References: <20230713100954.uktezddliezilack@localhost> Message-ID: <20230713114456.qisuhinipfhcq5zd@localhost> On 13/07, Nguy?n H?u Kh?i wrote: > hello. > Thank you for your valuable information, I got this problem. > Can you help me see how nova will be updated? I ask about something which I > can touch or modify at nova manually, > > Nguyen Huu Khoi > Hi, This is not something that can be easily fixed in the OpenStack code. There's always the manual hacking mechanism of changing things manually in libvirt using virsh without Nova knowing about it. I don't know the exact commands to run because I work on Cinder, but it's probably related to this old spec [1]. Making the code changes in OpenStack is non trivial, as it would require: - Add a new external event type to the Nova code to change the QoS for an instance. - Changing Cinder to do the following when QoS is updated: - Check volume types that use the updated QoS - Check volumes using those volume types - Check which volumes are connected to instances - For each of those volumes: - Update the connection information in the DB - Call Nova with the new external event type to update the instance. - If it's multi attached it may require making multiple calls for the same volume. Cheers, Gorka. [1]: https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/enhanced-kvm-storage-qos.html > > On Thu, Jul 13, 2023 at 5:10?PM Gorka Eguileor wrote: > > > On 27/06, Nguy?n H?u Kh?i wrote: > > > Hello guys. > > > I am trying to use cinder qos. > > > But i have question. > > > - I added qos after a volume was create and I see that qos did not apply > > to > > > this volume. > > > - i create new volume with assosiate qos then it is ok but when i changed > > > extra specs in qos, volume keep old specs. > > > Is there any way to make qos apply with two above cases. > > > Many thanks. > > > > Hi, > > > > I'm surprised you can even do it, I would expect this would be similar > > to changing a volume type when there's already a volume using it, which > > is not allowed. > > > > I assume you are using 'front-end' or 'both' QoS provider, because the > > situation is even worse for 'back-end' QoS providers. > > > > For front-end qos the values will NOT be updated in Nova until a new > > attachment is created, but it must be one that creates new connection > > information. > > > > Changing that would be a new feature requiring changes in Cinder and > > Nova, since Cinder would need to go through all the volumes that use > > volumes types associated with that QoS, check which ones are currently > > attached, update their connection_information in the cinder DB, and then > > report to Nova the new values (this API doesn't exist). > > > > For back-end QoS it would be driver dependent. I assume in most cases a > > retype would be necessary, but in some drivers they may be updating the > > QoS on the backed for all volumes once a new volume with that volume > > type or QoS is created. > > > > Cheers, > > Gorka. > > > > > > From nguyenhuukhoinw at gmail.com Thu Jul 13 11:47:40 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Thu, 13 Jul 2023 18:47:40 +0700 Subject: [cinder]cinder qos question. In-Reply-To: <20230713114456.qisuhinipfhcq5zd@localhost> References: <20230713100954.uktezddliezilack@localhost> <20230713114456.qisuhinipfhcq5zd@localhost> Message-ID: Thank you much. I try to understand it. I will feedback if I can do something, Nguyen Huu Khoi On Thu, Jul 13, 2023 at 6:45?PM Gorka Eguileor wrote: > On 13/07, Nguy?n H?u Kh?i wrote: > > hello. > > Thank you for your valuable information, I got this problem. > > Can you help me see how nova will be updated? I ask about something > which I > > can touch or modify at nova manually, > > > > Nguyen Huu Khoi > > > > Hi, > > This is not something that can be easily fixed in the OpenStack code. > > There's always the manual hacking mechanism of changing things manually > in libvirt using virsh without Nova knowing about it. I don't know the > exact commands to run because I work on Cinder, but it's probably > related to this old spec [1]. > > Making the code changes in OpenStack is non trivial, as it would > require: > > - Add a new external event type to the Nova code to change the QoS for > an instance. > > - Changing Cinder to do the following when QoS is updated: > > - Check volume types that use the updated QoS > - Check volumes using those volume types > - Check which volumes are connected to instances > - For each of those volumes: > - Update the connection information in the DB > - Call Nova with the new external event type to update the > instance. > - If it's multi attached it may require making multiple calls > for the same volume. > > Cheers, > Gorka. > > [1]: > https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/enhanced-kvm-storage-qos.html > > > > > On Thu, Jul 13, 2023 at 5:10?PM Gorka Eguileor > wrote: > > > > > On 27/06, Nguy?n H?u Kh?i wrote: > > > > Hello guys. > > > > I am trying to use cinder qos. > > > > But i have question. > > > > - I added qos after a volume was create and I see that qos did not > apply > > > to > > > > this volume. > > > > - i create new volume with assosiate qos then it is ok but when i > changed > > > > extra specs in qos, volume keep old specs. > > > > Is there any way to make qos apply with two above cases. > > > > Many thanks. > > > > > > Hi, > > > > > > I'm surprised you can even do it, I would expect this would be similar > > > to changing a volume type when there's already a volume using it, which > > > is not allowed. > > > > > > I assume you are using 'front-end' or 'both' QoS provider, because the > > > situation is even worse for 'back-end' QoS providers. > > > > > > For front-end qos the values will NOT be updated in Nova until a new > > > attachment is created, but it must be one that creates new connection > > > information. > > > > > > Changing that would be a new feature requiring changes in Cinder and > > > Nova, since Cinder would need to go through all the volumes that use > > > volumes types associated with that QoS, check which ones are currently > > > attached, update their connection_information in the cinder DB, and > then > > > report to Nova the new values (this API doesn't exist). > > > > > > For back-end QoS it would be driver dependent. I assume in most cases a > > > retype would be necessary, but in some drivers they may be updating the > > > QoS on the backed for all volumes once a new volume with that volume > > > type or QoS is created. > > > > > > Cheers, > > > Gorka. > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ykarel at redhat.com Thu Jul 13 15:43:05 2023 From: ykarel at redhat.com (Yatin Karel) Date: Thu, 13 Jul 2023 21:13:05 +0530 Subject: SNAT failure with OVN under Antelope In-Reply-To: References: <34841bdd-bb95-ff9a-258b-cfae26558627@uwo.ca> <536805d2-f334-e94f-1415-2984a219cb65@uwo.ca> <8511695f-443b-1593-fc20-04cb82cfc791@uwo.ca> <9c6b3c42-bf44-a59e-0fe8-b0f0430edf76@uwo.ca> Message-ID: Hi Gary, On Wed, Jul 12, 2023 at 9:22?PM Gary Molenkamp wrote: > A little progress, but I may be tripping over bug > https://bugs.launchpad.net/neutron/+bug/2003455 > > That bug was mostly targeting vlan provider networks but you mentioned you using geneve and flat networks so this might not be related. Multiple components involved so it would be difficult to narrow it down here without much details as functionality wise it would have just worked(in my Train OVN environment i checked it worked fine). So I think it would be best to start with a bug report at https://bugs.launchpad.net/neutron/ with details(by reverting the env to previous state bridges, ovn-cms options configured and DVR enabled). Good to include details like:- - Environment details:- - Number of controller, computes nodes - Nodes are virtual or physical - Deployment tool used, Operating System - Neutron version - OVN/OVS versions - Share ovn-controller logs from the compute and controller node - Share OVN Northbound and Southbound DB files from the controller node and ovs conf.db from compute nodes - Output of resources involved:- - openstack network agent list - openstack server list --long - openstack port list --router - Reproduction steps along with output from the operations(both with good and bad vms) - Output of below commands from controller and compute nodes:- - iptables -L - netstat -i - ip addr show - ovs-vsctl show - ovs-vsctl list open . > If I remove the provider bridge from the second hypervisor: > ovs-vsctl remove open . external-ids > ovn-cms-options="enable-chassis-as-gw" > ovs-vsctl remove open . external-ids ovn-bridge-mappings > ip link set br-provider down > ovs-vsctl del-br br-provider > and disable > enable_distributed_floating_ip > > Then both VMs using SNAT on each compute server work. > > This looks interesting. Would be good to also check the behavior when no VM has FIP attached. > Turning the second chassis back on as a gateway immediately breaks the VM > on the second compute server: > > ovs-vsctl set open . external-ids:ovn-cms-options=enable-chassis-as-gw > ovs-vsctl add-br br-provider > ovs-vsctl set open . > external-ids:ovn-bridge-mappings=provider:br-provider > ovs-vsctl add-port br-provider ens256 > systemctl restart ovn-controller openvswitch.service > > Here it would be interesting to check where exactly traffic drops using tcpdump. > I am running neutron 22.0.1 but maybe something related? > > python3-neutron-22.0.1-1.el9s.noarch > openstack-neutron-common-22.0.1-1.el9s.noarch > openstack-neutron-22.0.1-1.el9s.noarch > openstack-neutron-ml2-22.0.1-1.el9s.noarch > openstack-neutron-openvswitch-22.0.1-1.el9s.noarch > openstack-neutron-ovn-metadata-agent-22.0.1-1.el9s.noarch > > > > > > > On 2023-07-12 10:21, Gary Molenkamp wrote: > > For comparison, I looked at how openstack-ansible was setting up OVN and I > don't see any major differences other than O-A configures a manager for ovs: > ovs-vsctl --id @manager create Manager "target=\ .... > I don't believe this is the point of failure (but feel free to correct me > if I'm wrong ;) ). > > ovn-trace on both VM's inports shows the same trace for the working VM and > the non-working VM. ie: > > ovn-trace --db=$SB --ovs default_net 'inport == > "f4cbc8c7-e7bf-47f3-9fea-a1663f6eb34d" && eth.src==fa:16:3e:a6:62:8e && > ip4.src == 172.31.101.168 && ip4.dst == ' > > > > On 2023-07-07 14:08, Gary Molenkamp wrote: > > Happy Friday afternoon. > > I'm still pondering a lack of connectivity in an HA OVN with each compute > node acting as a potential gateway chassis. > > The problem is basically that the port of the OVN LRP may not be in the >> same chassis as the VM that failed (since the CR-LRP will be where the >> first VM of that network will be created). The suggestion is to remove the >> enable-chassis-as-gw from the compute nodes to allow the VM to forward >> traffic via tunneling/Geneve to the chassis where the LRP resides. >> >> >> I forced a similar VM onto the same chassis as the working VM, and it was >> able to communicate out. If we do want to keep multiple chassis' as >> gateways, would that be addressed with the ovn-bridge-mappings? >> > > > I built a small test cloud to explore this further as I continue to see > the same issue: A vm will only be able to use SNAT outbound if it is on > the same chassis as the CR-LRP. > > In my test cloud, I have one controller, and two compute nodes. The > controller only runs the north and southd in addition to the neutron > server. Each of the two compute nodes is configured as below. On a tenent > network I have three VMs: > - #1: cirros VM with FIP > - #2: cirros VM running on compute node 1 > - #3: cirros VM running on compute node 2 > > E/W traffic between VMs in the same tenent network are fine. N/S traffic > is fine for the FIP. N/S traffic only works for the VM whose CR-LRP is > active on same chassis. Does anything jump out as a mistake in my > understanding at to how this should be working? > > Thanks as always, > Gary > > > on each hypervisor: > > /usr/bin/ovs-vsctl set open . external-ids:ovn-remote=tcp:{{ controllerip > }}:6642 > /usr/bin/ovs-vsctl set open . external-ids:ovn-encap-type=geneve > /usr/bin/ovs-vsctl set open . external-ids:ovn-encap-ip={{ overlaynetip }} > /usr/bin/ovs-vsctl set open . > external-ids:ovn-cms-options=enable-chassis-as-gw > /usr/bin/ovs-vsctl add-br br-provider -- set bridge br-provider > protocols=OpenFlow10,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 > /usr/bin/ovs-vsctl add-port br-provider {{ provider_nic }} > /usr/bin/ovs-vsctl br-set-external-id provider bridge-id br-provider > /usr/bin/ovs-vsctl set open . > external-ids:ovn-bridge-mappings=provider:br-provider > > plugin.ini: > [ml2] > mechanism_drivers = ovn > type_drivers = flat,geneve > tenant_network_types = geneve > extension_drivers = port_security > overlay_ip_version = 4 > [ml2_type_flat] > flat_networks = provider > [ml2_type_geneve] > vni_ranges = 1:65536 > max_header_size = 38 > [securitygroup] > enable_security_group = True > firewall_driver = > neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver > [ovn] > ovn_nb_connection = tcp:{{controllerip}}:6641 > ovn_sb_connection = tcp:{{controllerip}}:6642 > ovn_l3_scheduler = leastloaded > ovn_metadata_enabled = True > enable_distributed_floating_ip = true > > > > > -- > Gary Molenkamp Science Technology Services > Systems Administrator University of Western Ontariomolenkam at uwo.ca http://sts.sci.uwo.ca > (519) 661-2111 x86882 (519) 661-3566 > > > -- > Gary Molenkamp Science Technology Services > Systems Engineer University of Western Ontariomolenkam at uwo.ca http://sts.sci.uwo.ca > (519) 661-2111 x86882 (519) 661-3566 > > > -- > Gary Molenkamp Science Technology Services > Systems Engineer University of Western Ontariomolenkam at uwo.ca http://sts.sci.uwo.ca > (519) 661-2111 x86882 (519) 661-3566 > > Thanks and Regards Yatin Karel -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristin at openinfra.dev Thu Jul 13 16:12:27 2023 From: kristin at openinfra.dev (Kristin Barrientos) Date: Thu, 13 Jul 2023 11:12:27 -0500 Subject: OpenStack "C" Release Voting Message-ID: Hi everyone, It?s time to vote for the OpenStack ?C? Release! The nominees are: - Celine - Caracal - Cashew Get your votes in! Vote here: https://civs1.civs.us/cgi-bin/vote.pl?id=E_852cf3e53467d72e&akey=1504cb6ad9d24667 Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralonsoh at redhat.com Thu Jul 13 16:44:58 2023 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Thu, 13 Jul 2023 18:44:58 +0200 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: Hello Roberto: I think you have provided a very detailed explanation of the issue but you still didn't present any possible solution. I would ask you at least for a proposal and for specific details of what is failing in the OVN sync tool. For example, if you add a TS between two clouds, you'll need to create (1) routers (NAT registers), (2) router ports (LRP) and (3) some static routes (LRSR). All these elements are monitored by the DB sync tool and will fail because the counterparts in the Neutron DB don't exist. I guess that you have some script or tool or at least a document to manually add these resources; sharing it could help to find a solution. If these elements are manually created during the deployment phase, you can also control the information provided. I'm thinking about the "external_ids" field; we can push some defined constant that will make these registers transparent to the OVN DB sync tool. Please check this idea and if it is feasible. In that case, please add a topic to the Neutron drivers meeting agenda [1] to discuss it. Regards. [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers On Wed, Jul 12, 2023 at 2:12?PM Roberto Bartzen Acosta < roberto.acosta at luizalabs.com> wrote: > > > Em qua., 12 de jul. de 2023 ?s 09:05, Roberto Bartzen Acosta < > roberto.acosta at luizalabs.com> escreveu: > >> Hi Rodolfo, >> >> Thanks for the feedback. >> >> I liked the idea of having a different network type only to hold the >> Transit Switches and thus the LRP but it depends on some factors like >> multi-tenancy. >> From the OVN perspective, the TS is global and will only be linked to a >> tenant when it is plugged into the tenant router port. My concern here is >> that if Neutron manages the TS, it will need to assume the dynamic >> characteristics of this TS function, ovn-ic creates and removes the TS in >> NB_Global, in addition to plugging and removing ports in this switch >> according to the network topology. Additionally, Neutron would still need >> to handle the learned remote routes (which are listed as static routes from >> the tenant router perspective). >> > sorry: NB_Global -> Northbound Database. > > >> This is an open discussion, Felix can add ideas here. >> >> In general, it seems to me that we have no alternatives for this type of >> solution other than OVN-IC (note that ovn-bgp-agent does not learn remote >> routes at the SDN level / OVN). >> OpenStack seems to be designed to run like a "bubble" and this traffic >> from one VM to another always needs to be routed at the FIP level and >> external routing layers. >> >> Regards, >> Roberto >> >> Em qua., 12 de jul. de 2023 ?s 05:11, Rodolfo Alonso Hernandez < >> ralonsoh at redhat.com> escreveu: >> >>> Hello Roberto: >>> >>> We talked about this possible RFE during the PTG [1] but I don't >>> remember having any proposal. Actually what I remember (and was written in >>> the etherpad) is that you were going to present an RFE. Can you explain it >>> briefly? >>> >>> We also talked about the idea, proposed by Felix in the mailing list, of >>> having a different network type only to hold the Transit Switches and thus >>> the LRP. If you have a proposal, please present it. >>> >>> Regards. >>> >>> [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506 >>> >>> On Tue, Jul 11, 2023 at 10:50?PM Roberto Bartzen Acosta < >>> roberto.acosta at luizalabs.com> wrote: >>> >>>> Hello Neutron folks, >>>> >>>> Regarding the conversation started in March [1] about the use of OVN >>>> interconnect with Neutron, I would like to evolve the discussion in >>>> relation to resources allocated by OVN-IC and which are not managed by >>>> Neutron. In the March PTG [2], the problem related to the db_sync tool was >>>> presented, and a proposed solution in which Neutron does not manage these >>>> resources. >>>> >>>> After discussing some architectural designs with Felix/StackIT, it >>>> seems to make sense to come up with a proposal to change the mech_driver to >>>> validate the external_ids key and not remove resources present in the OVN >>>> backend without Neutron "signature". >>>> >>>> The discussion here is more complex than it seems, because >>>> architecturally Neutron does not allow us to interconnect workloads in >>>> multiple AZs (with different OpenStacks), but this could be a requirement >>>> for a high availability cloud solution (Cloud Region). Additionally, this >>>> OVN-IC solution allows interconnecting other cloud backends that use OVN, >>>> in the case of kubernetes with ovn-kube. >>>> >>>> We tested an OVN interconnect integrated with 3 OpenStack installations >>>> and it worked very well !!! considering direct L3 traffic at the router >>>> level between different network infrastructures. >>>> >>>> SYNC_REPAIR - problem >>>> >>>> * Static Routes (learned OVN-IC routes) >>>> * Router Port -> Transit Switches >>>> >>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in >>>> OVN but not in Neutron, port_id=rt2-admin-tenant1 >>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router >>>> 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' >>>> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '10.0.2.1/24', >>>> 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron >>>> >>>> >>>> Any suggestions on how to resolve this db_sync issue? since all other >>>> Neutron modules did not present any problem with OVN-IC (as far as I >>>> tested). >>>> I remember Terry was keen to suggest some things to help. I believe >>>> that before proposing some complex mechanism to solve this simple problem, >>>> I would like to hear the community opinion. In our use case, a bit change >>>> in db_sync with filter by neutron key in external_ids would be enough. >>>> >>>> Regards, >>>> Roberto >>>> >>>> >>>> >>>> [1] >>>> https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html >>>> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg >>>> >>>> >>>> >>>> Additional logs: >>>> >>>> OpenStack 1 >>>> >>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl >>>> lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 >>>> IPv4 Routes >>>> Route Table
: >>>> 20.0.1.0/24 169.254.200.2 dst-ip (learned) >>>> 20.0.2.0/24 169.254.200.3 dst-ip (learned) >>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>> >>>> IPv6 Routes >>>> Route Table
: >>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl >>>> lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 >>>> IPv4 Routes >>>> Route Table
: >>>> 10.0.1.0/24 169.254.100.2 dst-ip (learned) >>>> 10.0.2.0/24 169.254.100.3 dst-ip (learned) >>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>> >>>> IPv6 Routes >>>> Route Table
: >>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>> >>>> >>>> OpenStack 2 >>>> >>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl >>>> lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc >>>> IPv4 Routes >>>> Route Table
: >>>> 20.0.0.0/24 169.254.200.1 dst-ip (learned) >>>> 20.0.2.0/24 169.254.200.3 dst-ip (learned) >>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>> >>>> IPv6 Routes >>>> Route Table
: >>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl >>>> lr-route-list ce45f681-6454-43fe-974f-81344bb8113a >>>> IPv4 Routes >>>> Route Table
: >>>> 10.0.0.0/24 169.254.100.1 dst-ip (learned) >>>> 10.0.2.0/24 169.254.100.3 dst-ip (learned) >>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>> >>>> IPv6 Routes >>>> Route Table
: >>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>> >>>> >>>> >>>> OpenStack 3 >>>> >>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl >>>> lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 >>>> IPv4 Routes >>>> Route Table
: >>>> 20.0.0.0/24 169.254.200.1 dst-ip (learned) >>>> 20.0.1.0/24 169.254.200.2 dst-ip (learned) >>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>> >>>> IPv6 Routes >>>> Route Table
: >>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl >>>> lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 >>>> IPv4 Routes >>>> Route Table
: >>>> 10.0.0.0/24 169.254.100.1 dst-ip (learned) >>>> 10.0.1.0/24 169.254.100.2 dst-ip (learned) >>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>> >>>> IPv6 Routes >>>> Route Table
: >>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>> >>>> >>>> OVN-IC Global database >>>> >>>> root at ovn-global-db1:~# ovn-ic-sbctl show >>>> availability-zone osp1 >>>> gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 >>>> hostname: osp1-gwnode1 >>>> type: geneve >>>> ip: 192.168.200.28 >>>> port admin-rt1-tenant1 >>>> transit switch: admin-tenant1 >>>> address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] >>>> port admin-rt1-tenant1_1 >>>> transit switch: admin-tenant1_1 >>>> address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] >>>> availability-zone osp2 >>>> gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 >>>> hostname: osp2-gwnode1 >>>> type: geneve >>>> ip: 192.168.200.128 >>>> port admin-rt2-tenant1 >>>> transit switch: admin-tenant1 >>>> address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] >>>> port admin-rt2-tenant1_1 >>>> transit switch: admin-tenant1_1 >>>> address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] >>>> availability-zone osp3 >>>> gateway 97595af9-7896-40d0-a883-beadbff1aa5b >>>> hostname: osp3-gwnode1 >>>> type: geneve >>>> ip: 192.168.200.228 >>>> port admin-rt3-tenant1 >>>> transit switch: admin-tenant1 >>>> address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] >>>> port admin-rt3-tenant1_1 >>>> transit switch: admin-tenant1_1 >>>> address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> *?Esta mensagem ? direcionada apenas para os endere?os constantes no >>>> cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no >>>> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa >>>> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o >>>> imediatamente anuladas e proibidas?.* >>>> >>>> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para >>>> assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o >>>> poder? aceitar a responsabilidade por quaisquer perdas ou danos causados >>>> por esse e-mail ou por seus anexos?.* >>>> >>> > > *?Esta mensagem ? direcionada apenas para os endere?os constantes no > cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no > cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa > mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o > imediatamente anuladas e proibidas?.* > > *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para > assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o > poder? aceitar a responsabilidade por quaisquer perdas ou danos causados > por esse e-mail ou por seus anexos?.* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marc.gariepy at calculquebec.ca Thu Jul 13 17:14:22 2023 From: marc.gariepy at calculquebec.ca (Marc Gariepy) Date: Thu, 13 Jul 2023 13:14:22 -0400 Subject: SNAT failure with OVN under Antelope In-Reply-To: <8511695f-443b-1593-fc20-04cb82cfc791@uwo.ca> References: <34841bdd-bb95-ff9a-258b-cfae26558627@uwo.ca> <536805d2-f334-e94f-1415-2984a219cb65@uwo.ca> <8511695f-443b-1593-fc20-04cb82cfc791@uwo.ca> Message-ID: hello, might have something to do with the firewall_driver in your plugin.ini ? > > plugin.ini: > [ml2] > mechanism_drivers = ovn > type_drivers = flat,geneve > tenant_network_types = geneve > extension_drivers = port_security > overlay_ip_version = 4 > [ml2_type_flat] > flat_networks = provider > [ml2_type_geneve] > vni_ranges = 1:65536 > max_header_size = 38 > [securitygroup] > enable_security_group = True > firewall_driver = > neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver > [ovn] > ovn_nb_connection = tcp:{{controllerip}}:6641 > ovn_sb_connection = tcp:{{controllerip}}:6642 > ovn_l3_scheduler = leastloaded > ovn_metadata_enabled = True > enable_distributed_floating_ip = true > Marc -------------- next part -------------- An HTML attachment was scrubbed... URL: From molenkam at uwo.ca Thu Jul 13 17:36:21 2023 From: molenkam at uwo.ca (Gary Molenkamp) Date: Thu, 13 Jul 2023 13:36:21 -0400 Subject: SNAT failure with OVN under Antelope In-Reply-To: References: <34841bdd-bb95-ff9a-258b-cfae26558627@uwo.ca> <536805d2-f334-e94f-1415-2984a219cb65@uwo.ca> <8511695f-443b-1593-fc20-04cb82cfc791@uwo.ca> <9c6b3c42-bf44-a59e-0fe8-b0f0430edf76@uwo.ca> Message-ID: <9da67399-bd7b-dff9-67eb-ca94903a6394@uwo.ca> Thanks Yartin,?? I will put together a bug report. I have found that if I disable enable_distributed_floating_ip, but leave the entire OVN/OVS setup as below for redundancy, then traffic flows as expected. As soon as I set enable_distributed_floating_ip to true, E/W remains, but the N/S traffic stops for the VMs not on the host with the CR-LRP. I can't say for sure why as ovn-trace/flow debugging is still new to me, but the north and south dbs look correct. Gary On 2023-07-13 11:43, Yatin Karel wrote: > Hi Gary, > > On Wed, Jul 12, 2023 at 9:22?PM Gary Molenkamp wrote: > > A little progress, but I may be tripping over bug > https://bugs.launchpad.net/neutron/+bug/2003455 > > That bug was mostly targeting vlan provider networks but you mentioned > you using geneve and flat networks so this might not be related. > > Multiple components involved so it would be difficult to narrow it > down here without much details as functionality wise it would have > just worked(in my Train OVN environment i checked it worked fine). So > I think it would be best to start with a bug report at > https://bugs.launchpad.net/neutron/ with details(by reverting the env > to previous state bridges, ovn-cms options configured and DVR > enabled). Good to include details like:- > > - Environment details:- > ??- Number of controller, computes nodes > ? - Nodes are virtual or physical > ? - Deployment tool used, Operating System > ??- Neutron version > ? - OVN/OVS versions > - Share ovn-controller logs from the compute and controller node > - Share OVN Northbound and Southbound DB files from the controller > node and ovs conf.db from compute nodes > - Output of resources involved:- > ? - openstack network agent list > ? - openstack server list --long > ? - openstack port list --router > ? - Reproduction steps along with output from the operations(both with > good and bad vms) > - Output of below commands from controller and compute nodes:- > ? - iptables -L > ? - netstat -i > ? - ip addr show > ? - ovs-vsctl show > ? - ovs-vsctl list open . > > If I remove the provider bridge from the second hypervisor: > ??? ovs-vsctl remove open . external-ids > ovn-cms-options="enable-chassis-as-gw" > ??? ovs-vsctl remove open . external-ids ovn-bridge-mappings > ??? ip link set br-provider down > ? ? ovs-vsctl del-br br-provider > and disable > ??? enable_distributed_floating_ip > > Then both VMs using SNAT on each compute server work. > > This looks interesting. Would be good to also check the behavior when > no VM has FIP attached. > > Turning the second chassis back on as a gateway immediately breaks > the VM on the second compute server: > > ??? ovs-vsctl set open . > external-ids:ovn-cms-options=enable-chassis-as-gw > ??? ovs-vsctl add-br br-provider > ??? ovs-vsctl set open . > external-ids:ovn-bridge-mappings=provider:br-provider > ? ? ovs-vsctl add-port br-provider ens256 > ??? systemctl restart ovn-controller openvswitch.service > > Here it would be interesting to check where exactly traffic drops > using tcpdump. > > I am running neutron 22.0.1 but maybe something related? > > python3-neutron-22.0.1-1.el9s.noarch > openstack-neutron-common-22.0.1-1.el9s.noarch > openstack-neutron-22.0.1-1.el9s.noarch > openstack-neutron-ml2-22.0.1-1.el9s.noarch > openstack-neutron-openvswitch-22.0.1-1.el9s.noarch > openstack-neutron-ovn-metadata-agent-22.0.1-1.el9s.noarch > > > > > > > On 2023-07-12 10:21, Gary Molenkamp wrote: >> For comparison, I looked at how openstack-ansible was setting up >> OVN and I don't see any major differences other than O-A >> configures a manager for ovs: >> ????? ovs-vsctl --id @manager create Manager "target=\ .... >> I don't believe this is the point of failure (but feel free to >> correct me if I'm wrong ;) ). >> >> ovn-trace on both VM's inports shows the same trace for the >> working VM and the non-working VM. ie: >> >> ovn-trace --db=$SB --ovs default_net 'inport == >> "f4cbc8c7-e7bf-47f3-9fea-a1663f6eb34d" && >> eth.src==fa:16:3e:a6:62:8e && ip4.src == 172.31.101.168 && >> ip4.dst == ' >> >> >> >> On 2023-07-07 14:08, Gary Molenkamp wrote: >>> Happy Friday afternoon. >>> >>> I'm still pondering a lack of connectivity in an HA OVN with >>> each compute node acting as a potential gateway chassis. >>> >>>>> The problem is basically that the port of the OVN LRP may >>>>> not be in the same chassis as the VM that failed (since >>>>> the CR-LRP will be where the first VM of that network will >>>>> be created).?The suggestion is to remove the >>>>> enable-chassis-as-gw from the compute nodes to allow the >>>>> VM to forward traffic via tunneling/Geneve to the chassis >>>>> where the LRP resides. >>>>> >>>> >>>> I forced a similar VM onto the same chassis as the working >>>> VM, and it was able to communicate out.??? If we do want to >>>> keep multiple chassis' as gateways, would that be addressed >>>> with the ovn-bridge-mappings? >>>> >>>> >>> >>> I built a small test cloud to explore this further as I continue >>> to see the same issue:? A vm will only be able to use SNAT >>> outbound if it is on the same chassis as the CR-LRP. >>> >>> In my test cloud, I have one controller, and two compute nodes.? >>> The controller only runs the north and southd in addition to the >>> neutron server.? Each of the two compute nodes is configured as >>> below.? On a tenent network I have three VMs: >>> ??? - #1:? cirros VM with FIP >>> ??? - #2:? cirros VM running on compute node 1 >>> ??? - #3:? cirros VM running on compute node 2 >>> >>> E/W traffic between VMs in the same tenent network are fine.? >>> N/S traffic is fine for the FIP.? N/S traffic only works for the >>> VM whose CR-LRP is active on same chassis.?? Does anything jump >>> out as a mistake in my understanding at to how this should be >>> working? >>> >>> Thanks as always, >>> Gary >>> >>> >>> on each hypervisor: >>> >>> /usr/bin/ovs-vsctl set open . external-ids:ovn-remote=tcp:{{ >>> controllerip }}:6642 >>> /usr/bin/ovs-vsctl set open . external-ids:ovn-encap-type=geneve >>> /usr/bin/ovs-vsctl set open . external-ids:ovn-encap-ip={{ >>> overlaynetip }} >>> /usr/bin/ovs-vsctl set open . >>> external-ids:ovn-cms-options=enable-chassis-as-gw >>> /usr/bin/ovs-vsctl add-br br-provider -- set bridge br-provider >>> protocols=OpenFlow10,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 >>> /usr/bin/ovs-vsctl add-port br-provider {{ provider_nic }} >>> /usr/bin/ovs-vsctl br-set-external-id provider bridge-id br-provider >>> /usr/bin/ovs-vsctl set open . >>> external-ids:ovn-bridge-mappings=provider:br-provider >>> >>> plugin.ini: >>> [ml2] >>> mechanism_drivers = ovn >>> type_drivers = flat,geneve >>> tenant_network_types = geneve >>> extension_drivers = port_security >>> overlay_ip_version = 4 >>> [ml2_type_flat] >>> flat_networks = provider >>> [ml2_type_geneve] >>> vni_ranges = 1:65536 >>> max_header_size = 38 >>> [securitygroup] >>> enable_security_group = True >>> firewall_driver = >>> neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver >>> [ovn] >>> ovn_nb_connection = tcp:{{controllerip}}:6641 >>> ovn_sb_connection = tcp:{{controllerip}}:6642 >>> ovn_l3_scheduler = leastloaded >>> ovn_metadata_enabled = True >>> enable_distributed_floating_ip = true >>> >>> >>> >>> >>> -- >>> Gary Molenkamp Science Technology Services >>> Systems Administrator University of Western Ontario >>> molenkam at uwo.ca http://sts.sci.uwo.ca >>> (519) 661-2111 x86882 (519) 661-3566 >> >> -- >> Gary Molenkamp Science Technology Services >> Systems Engineer University of Western Ontario >> molenkam at uwo.ca http://sts.sci.uwo.ca >> (519) 661-2111 x86882 (519) 661-3566 > > -- > Gary Molenkamp Science Technology Services > Systems Engineer University of Western Ontario > molenkam at uwo.ca http://sts.sci.uwo.ca > (519) 661-2111 x86882 (519) 661-3566 > > > Thanks and Regards > Yatin Karel -- Gary Molenkamp Science Technology Services Systems Engineer University of Western Ontario molenkam at uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566 -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.acosta at luizalabs.com Thu Jul 13 22:29:51 2023 From: roberto.acosta at luizalabs.com (Roberto Bartzen Acosta) Date: Thu, 13 Jul 2023 19:29:51 -0300 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: Hi Rodolfo, Em qui., 13 de jul. de 2023 ?s 13:45, Rodolfo Alonso Hernandez < ralonsoh at redhat.com> escreveu: > Hello Roberto: > > I think you have provided a very detailed explanation of the issue but you > still didn't present any possible solution. I would ask you at least for a > proposal and for specific details of what is failing in the OVN sync tool. > > For example, if you add a TS between two clouds, you'll need to create (1) > routers (NAT registers), (2) router ports (LRP) and (3) some static routes > (LRSR). All these elements are monitored by the DB sync tool and will fail > because the counterparts in the Neutron DB don't exist. I guess that you > have some script or tool or at least a document to manually add these > resources; sharing it could help to find a solution. > > If these elements are manually created during the deployment phase, you > can also control the information provided. I'm thinking about the > "external_ids" field; we can push some defined constant that will make > these registers transparent to the OVN DB sync tool. > > Please check this idea and if it is feasible. In that case, please add a > topic to the Neutron drivers meeting agenda [1] to discuss it. > That sounds good, thanks. https://bugs.launchpad.net/neutron/+bug/2027742 Regards, Roberto > > Regards. > > [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers > > > On Wed, Jul 12, 2023 at 2:12?PM Roberto Bartzen Acosta < > roberto.acosta at luizalabs.com> wrote: > >> >> >> Em qua., 12 de jul. de 2023 ?s 09:05, Roberto Bartzen Acosta < >> roberto.acosta at luizalabs.com> escreveu: >> >>> Hi Rodolfo, >>> >>> Thanks for the feedback. >>> >>> I liked the idea of having a different network type only to hold the >>> Transit Switches and thus the LRP but it depends on some factors like >>> multi-tenancy. >>> From the OVN perspective, the TS is global and will only be linked to a >>> tenant when it is plugged into the tenant router port. My concern here is >>> that if Neutron manages the TS, it will need to assume the dynamic >>> characteristics of this TS function, ovn-ic creates and removes the TS in >>> NB_Global, in addition to plugging and removing ports in this switch >>> according to the network topology. Additionally, Neutron would still need >>> to handle the learned remote routes (which are listed as static routes from >>> the tenant router perspective). >>> >> sorry: NB_Global -> Northbound Database. >> >> >>> This is an open discussion, Felix can add ideas here. >>> >>> In general, it seems to me that we have no alternatives for this type of >>> solution other than OVN-IC (note that ovn-bgp-agent does not learn remote >>> routes at the SDN level / OVN). >>> OpenStack seems to be designed to run like a "bubble" and this traffic >>> from one VM to another always needs to be routed at the FIP level and >>> external routing layers. >>> >>> Regards, >>> Roberto >>> >>> Em qua., 12 de jul. de 2023 ?s 05:11, Rodolfo Alonso Hernandez < >>> ralonsoh at redhat.com> escreveu: >>> >>>> Hello Roberto: >>>> >>>> We talked about this possible RFE during the PTG [1] but I don't >>>> remember having any proposal. Actually what I remember (and was written in >>>> the etherpad) is that you were going to present an RFE. Can you explain it >>>> briefly? >>>> >>>> We also talked about the idea, proposed by Felix in the mailing list, >>>> of having a different network type only to hold the Transit Switches and >>>> thus the LRP. If you have a proposal, please present it. >>>> >>>> Regards. >>>> >>>> [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506 >>>> >>>> On Tue, Jul 11, 2023 at 10:50?PM Roberto Bartzen Acosta < >>>> roberto.acosta at luizalabs.com> wrote: >>>> >>>>> Hello Neutron folks, >>>>> >>>>> Regarding the conversation started in March [1] about the use of OVN >>>>> interconnect with Neutron, I would like to evolve the discussion in >>>>> relation to resources allocated by OVN-IC and which are not managed by >>>>> Neutron. In the March PTG [2], the problem related to the db_sync tool was >>>>> presented, and a proposed solution in which Neutron does not manage these >>>>> resources. >>>>> >>>>> After discussing some architectural designs with Felix/StackIT, it >>>>> seems to make sense to come up with a proposal to change the mech_driver to >>>>> validate the external_ids key and not remove resources present in the OVN >>>>> backend without Neutron "signature". >>>>> >>>>> The discussion here is more complex than it seems, because >>>>> architecturally Neutron does not allow us to interconnect workloads in >>>>> multiple AZs (with different OpenStacks), but this could be a requirement >>>>> for a high availability cloud solution (Cloud Region). Additionally, this >>>>> OVN-IC solution allows interconnecting other cloud backends that use OVN, >>>>> in the case of kubernetes with ovn-kube. >>>>> >>>>> We tested an OVN interconnect integrated with 3 OpenStack >>>>> installations and it worked very well !!! considering direct L3 traffic at >>>>> the router level between different network infrastructures. >>>>> >>>>> SYNC_REPAIR - problem >>>>> >>>>> * Static Routes (learned OVN-IC routes) >>>>> * Router Port -> Transit Switches >>>>> >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in >>>>> OVN but not in Neutron, port_id=rt2-admin-tenant1 >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router >>>>> 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' >>>>> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': ' >>>>> 10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in >>>>> Neutron >>>>> >>>>> >>>>> Any suggestions on how to resolve this db_sync issue? since all other >>>>> Neutron modules did not present any problem with OVN-IC (as far as I >>>>> tested). >>>>> I remember Terry was keen to suggest some things to help. I believe >>>>> that before proposing some complex mechanism to solve this simple problem, >>>>> I would like to hear the community opinion. In our use case, a bit change >>>>> in db_sync with filter by neutron key in external_ids would be enough. >>>>> >>>>> Regards, >>>>> Roberto >>>>> >>>>> >>>>> >>>>> [1] >>>>> https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html >>>>> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg >>>>> >>>>> >>>>> >>>>> Additional logs: >>>>> >>>>> OpenStack 1 >>>>> >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl >>>>> lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 >>>>> IPv4 Routes >>>>> Route Table
: >>>>> 20.0.1.0/24 169.254.200.2 dst-ip (learned) >>>>> 20.0.2.0/24 169.254.200.3 dst-ip (learned) >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>> >>>>> IPv6 Routes >>>>> Route Table
: >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl >>>>> lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 >>>>> IPv4 Routes >>>>> Route Table
: >>>>> 10.0.1.0/24 169.254.100.2 dst-ip (learned) >>>>> 10.0.2.0/24 169.254.100.3 dst-ip (learned) >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>> >>>>> IPv6 Routes >>>>> Route Table
: >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>>> >>>>> >>>>> OpenStack 2 >>>>> >>>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl >>>>> lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc >>>>> IPv4 Routes >>>>> Route Table
: >>>>> 20.0.0.0/24 169.254.200.1 dst-ip (learned) >>>>> 20.0.2.0/24 169.254.200.3 dst-ip (learned) >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>> >>>>> IPv6 Routes >>>>> Route Table
: >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl >>>>> lr-route-list ce45f681-6454-43fe-974f-81344bb8113a >>>>> IPv4 Routes >>>>> Route Table
: >>>>> 10.0.0.0/24 169.254.100.1 dst-ip (learned) >>>>> 10.0.2.0/24 169.254.100.3 dst-ip (learned) >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>> >>>>> IPv6 Routes >>>>> Route Table
: >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>> >>>>> >>>>> >>>>> OpenStack 3 >>>>> >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl >>>>> lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 >>>>> IPv4 Routes >>>>> Route Table
: >>>>> 20.0.0.0/24 169.254.200.1 dst-ip (learned) >>>>> 20.0.1.0/24 169.254.200.2 dst-ip (learned) >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>> >>>>> IPv6 Routes >>>>> Route Table
: >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl >>>>> lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 >>>>> IPv4 Routes >>>>> Route Table
: >>>>> 10.0.0.0/24 169.254.100.1 dst-ip (learned) >>>>> 10.0.1.0/24 169.254.100.2 dst-ip (learned) >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>> >>>>> IPv6 Routes >>>>> Route Table
: >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>> >>>>> >>>>> OVN-IC Global database >>>>> >>>>> root at ovn-global-db1:~# ovn-ic-sbctl show >>>>> availability-zone osp1 >>>>> gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 >>>>> hostname: osp1-gwnode1 >>>>> type: geneve >>>>> ip: 192.168.200.28 >>>>> port admin-rt1-tenant1 >>>>> transit switch: admin-tenant1 >>>>> address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] >>>>> port admin-rt1-tenant1_1 >>>>> transit switch: admin-tenant1_1 >>>>> address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] >>>>> availability-zone osp2 >>>>> gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 >>>>> hostname: osp2-gwnode1 >>>>> type: geneve >>>>> ip: 192.168.200.128 >>>>> port admin-rt2-tenant1 >>>>> transit switch: admin-tenant1 >>>>> address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] >>>>> port admin-rt2-tenant1_1 >>>>> transit switch: admin-tenant1_1 >>>>> address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] >>>>> availability-zone osp3 >>>>> gateway 97595af9-7896-40d0-a883-beadbff1aa5b >>>>> hostname: osp3-gwnode1 >>>>> type: geneve >>>>> ip: 192.168.200.228 >>>>> port admin-rt3-tenant1 >>>>> transit switch: admin-tenant1 >>>>> address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] >>>>> port admin-rt3-tenant1_1 >>>>> transit switch: admin-tenant1_1 >>>>> address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *?Esta mensagem ? direcionada apenas para os endere?os constantes no >>>>> cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no >>>>> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa >>>>> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o >>>>> imediatamente anuladas e proibidas?.* >>>>> >>>>> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para >>>>> assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o >>>>> poder? aceitar a responsabilidade por quaisquer perdas ou danos causados >>>>> por esse e-mail ou por seus anexos?.* >>>>> >>>> >> >> *?Esta mensagem ? direcionada apenas para os endere?os constantes no >> cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no >> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa >> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o >> imediatamente anuladas e proibidas?.* >> >> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para >> assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o >> poder? aceitar a responsabilidade por quaisquer perdas ou danos causados >> por esse e-mail ou por seus anexos?.* >> > -- _?Esta mensagem ? direcionada apenas para os endere?os constantes no cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o imediatamente anuladas e proibidas?._ *?**?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o poder? aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos?.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From jobernar at redhat.com Fri Jul 14 01:47:02 2023 From: jobernar at redhat.com (Jon Bernard) Date: Thu, 13 Jul 2023 21:47:02 -0400 Subject: [cinder] I/O throttling using blkio cgroups Message-ID: I started to look at a cinder bug [1] to migrate our cgroup usage from v1 to v2 and I'm wondering: is anyone using this feature to throttle cinder's local I/O per cinder-volume instance? Docker since 20.10 and kubernetes since version 1.19 support cgroup v2. Since cinder services are deployed within containers, I think it makes sense to consider implementing I/O throttling at the container-level as opposed to application-specific code within a particular service (cinder in this case, but I believe glance also leverages cgroups). That would mean instead of updating our implementation to use the v2 interface, we could drop our cgroup code entirely in favor of a more generic solution that could be applied to any container that performs local I/O. Would anyone be opposed to this idea? Thoughts? 1: https://bugs.launchpad.net/cinder/+bug/1942203 -- Jon From christian.rohmann at inovex.de Fri Jul 14 07:37:04 2023 From: christian.rohmann at inovex.de (Christian Rohmann) Date: Fri, 14 Jul 2023 09:37:04 +0200 Subject: [all] [oslo.messaging] Interest in collaboration on a NATS driver In-Reply-To: References: <52AA12A0-AE67-4EF7-B924-DE1F2873B909@binero.com> <0E347D5A-967D-42E3-AC00-56034907053B@binero.com> <8F7F790F-0F5C-4257-845A-1BCE671AF844@binero.com> Message-ID: <3b2f711e-a2f1-71da-92ba-d36c229420a2@inovex.de> On 12/09/2022 09:05, Tobias Urdin wrote: > Hello, > > Since I haven?t heard any objections we?ll move the meeting to the Oslo > meeting on the 19th of September. See you there! [1] > > [1] https://wiki.openstack.org/wiki/Meetings/Oslo > Could you maybe give an update on the current state on adding NATS to oslo.messaging? I know there was some positive exchange during an Oslo Meeting [1], but now all but the spec change ?[2] are abandoned? Is the work on the spec [3] still going then? Regards Christian [1] https://meetings.opendev.org/meetings/oslo/2022/oslo.2022-09-19-15.12.log.txt [2] https://review.opendev.org/q/topic:asyncio-nats [3] https://review.opendev.org/c/openstack/oslo-specs/+/692784 From ralonsoh at redhat.com Fri Jul 14 08:48:35 2023 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Fri, 14 Jul 2023 10:48:35 +0200 Subject: [neutron] Neutron drivers meeting Message-ID: Hello Neutrinos: Today's meeting will be cancelled due to the lack of quorum. At least three of our drivers members [1] are in PTO today, thus we'll postpone the requested RFE discussion [2] to next week. Sorry for the inconveniences. Regards. [1]https://launchpad.net/~neutron-drivers/+members [2]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers -------------- next part -------------- An HTML attachment was scrubbed... URL: From zigo at debian.org Fri Jul 14 09:31:39 2023 From: zigo at debian.org (Thomas Goirand) Date: Fri, 14 Jul 2023 11:31:39 +0200 Subject: [designate] About delegating PTR using shared zones on a shared internal network In-Reply-To: References: <5059613b-2c76-c5a5-1281-42970a190def@debian.org> Message-ID: On 7/10/23 23:03, Michael Johnson wrote: > Ideally you would allocate blocks of IPs to a project, but I realize > many deployments are not set up that way. You are right, that's not the way it works in our deployment. As we implemented a public cloud, and have thousands of customers, it's not doable to manually assign a block of IPs to a customer with an admin command. What we have is a network that we called "ext-net1". It is an internal OpenStack network (with a router and its default gateway owned by the admin user), on which we added multiple /24 subnet blocks of public IPs. Typically, a user would create a port on this "ext-net1" and use it when doing "openstack server create" to optain a (random) public IP on that network. Or, using "openstack server create --nic net-id=ext-net1". In this scenario, what we want is automatically assigning the PTR subzone matching the individual IP address to be: 1/ created (as admin) 2/ shared with the customers project (openstack zone share create), so the customer can add a PTR record 3/ if possible, also automatically create a PTR record on the behalf of the customer, so there's a default value in place. When the port is deleted, we would like the subzone to be deleted automatically as well. > The other thing to think about is if you use one of the neutron > extensions that create records based on the network domain and port > dns name. We are using scenario 3a, so customers can do whatever they want with the IP address they own. > You could get into an interesting situation where the > configuration in neutron no longer matches the PTR records if you > allow users to "own" their PTR records in Designte. Well, if the port name and domain doesn't match the PTR record, it may look ugly, but I don't think that's a real problem. I couldn't care less, if at least it's doing what we expect (ie: let our customers create forward and reverse entries, in a way that can be shared with other customers without any conflict). As we're still in Victoria (currently planning the upgrade to Zed), we wont be able to contribute what I described above in a foreseeable future. The situation *may* change at the end of this year though, if we successfully upgrade to Zed, and if I can backport the shared zone patches. BTW, it would be great if, like we discuss in the summit, you could send me a list of patch, in the correct order, so I could backport the shared zones feature to Zed. Of course only do that if it doesn't take too much of your time... Also, did you backport the bugfix you told me was missing to Antelope? Thanks for all of your work, Cheers, Thomas Goirand (zigo) From christian.rohmann at inovex.de Fri Jul 14 10:27:29 2023 From: christian.rohmann at inovex.de (Christian Rohmann) Date: Fri, 14 Jul 2023 12:27:29 +0200 Subject: [OPENSTACK][rabbitmq] using quorum queues In-Reply-To: References: Message-ID: <002ad34d-a1c4-3958-3865-730d221c3059@inovex.de> On 30/06/2023 11:45, Dmitriy Rabotyagov wrote: > 1. I'm not sure if that's a proper community goal, simply because it > boils down to individual deployments and affects a really minority of > projects, like oslo.messaging and deployment projects. Moreover, this > affects only rabbit driver, which is the most popular option in > oslo.messaging but not the only one. Usually community goal is used > when you need to apply changes to the absolute majority of services, > like in case with SQLAlchemy. So here I think we should discuss more > default behaviour of oslo.messaging for rabbit driver and default > value of rabbit_quorum_queue, which is basically up to oslo team to > decide. In case all of the queue type specifics only live in oslo.messaging and are abstracted away from the application I believe you are totally right. I was simply looking at [1] in which certain features, if used, might require changes within the application. This led me to believe this required a broader effort across the projects to make this change. But in the end the recommendation of a migration to quorum queues, however done technically, to me should rather be a goal and general release note for an OpenStack release. With the deprecation of classic mirrored queues within RabbitMQ this migration needs to happen sooner or later anyways. To me it makes no sense to change for one component, but not for another. Choice in migration speed and config options are always nice (like [3]), but in the end it's a required cut over from one type to another with no need to keep support for classic queues after a "bridge release" or two supporting both. > 2. In OpenStack-Ansible we already agreed on the latest PTG to make > Quorum queues a new default and introduce migration to them. So it > should be inlcuded in 2023.2 (bobcat) release, with keeping the > upgrade path at very least to 2024.1 which will be the next SLURP. Do you have a pointer to those efforts for me to follow? A Gerrit topic or bug? [1] https://www.rabbitmq.com/migrate-mcq-to-qq.html#mcq-changes-way-queue-is-used which [2] https://review.opendev.org/c/openstack/oslo.messaging/+/888479 [3] https://review.opendev.org/c/openstack/oslo.messaging/+/888479 Regards Christian From noonedeadpunk at gmail.com Fri Jul 14 10:37:41 2023 From: noonedeadpunk at gmail.com (Dmitriy Rabotyagov) Date: Fri, 14 Jul 2023 12:37:41 +0200 Subject: [OPENSTACK][rabbitmq] using quorum queues In-Reply-To: <002ad34d-a1c4-3958-3865-730d221c3059@inovex.de> References: <002ad34d-a1c4-3958-3865-730d221c3059@inovex.de> Message-ID: > > 2. In OpenStack-Ansible we already agreed on the latest PTG to make > > Quorum queues a new default and introduce migration to them. So it > > should be inlcuded in 2023.2 (bobcat) release, with keeping the > > upgrade path at very least to 2024.1 which will be the next SLURP. > > Do you have a pointer to those efforts for me to follow? A Gerrit topic > or bug? Sure, you can follow this topic: https://review.opendev.org/q/topic:osa%252Fquorum_queues I haven't used as it's not merged yet in oslo (and I was unaware of this patch). So it sounds like it is worth waiting for it before proceeding with the topic. > [1] > https://www.rabbitmq.com/migrate-mcq-to-qq.html#mcq-changes-way-queue-is-used > which > [2] https://review.opendev.org/c/openstack/oslo.messaging/+/888479 > [3] https://review.opendev.org/c/openstack/oslo.messaging/+/888479 > PS: your [2] and [3] links are exactly the same. From christian.rohmann at inovex.de Fri Jul 14 11:06:34 2023 From: christian.rohmann at inovex.de (Christian Rohmann) Date: Fri, 14 Jul 2023 13:06:34 +0200 Subject: [OPENSTACK][rabbitmq] using quorum queues In-Reply-To: References: <002ad34d-a1c4-3958-3865-730d221c3059@inovex.de> Message-ID: <6692d72c-a6f3-45e8-938b-db7246b7bfc5@inovex.de> On 14/07/2023 12:37, Dmitriy Rabotyagov wrote: > PS: your [2] and [3] links are exactly the same. Sorry. I really should have sent out the email after lunch ;-) [2] was supposed to be https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/#removal-of-classic-queue-mirroring in regards to be saying: > With the deprecation of classic mirrored queues within RabbitMQ this > migration needs to happen sooner or later anyways. In short: before RabbitMQ 4.x the classic queues need to be gone for good. No more config option, no looking back. Regards Christian From noonedeadpunk at gmail.com Fri Jul 14 11:13:27 2023 From: noonedeadpunk at gmail.com (Dmitriy Rabotyagov) Date: Fri, 14 Jul 2023 13:13:27 +0200 Subject: [OPENSTACK][rabbitmq] using quorum queues In-Reply-To: <6692d72c-a6f3-45e8-938b-db7246b7bfc5@inovex.de> References: <002ad34d-a1c4-3958-3865-730d221c3059@inovex.de> <6692d72c-a6f3-45e8-938b-db7246b7bfc5@inovex.de> Message-ID: Aha, ok, the same link I'm referring to in a patch [1] then :D [1] https://review.opendev.org/c/openstack/openstack-ansible/+/873618 ??, 14 ???. 2023??. ? 13:06, Christian Rohmann : > > On 14/07/2023 12:37, Dmitriy Rabotyagov wrote: > > PS: your [2] and [3] links are exactly the same. > Sorry. I really should have sent out the email after lunch ;-) > > [2] was supposed to be > https://blog.rabbitmq.com/posts/2021/08/4.0-deprecation-announcements/#removal-of-classic-queue-mirroring > in regards to be saying: > > > With the deprecation of classic mirrored queues within RabbitMQ this > > migration needs to happen sooner or later anyways. > > > In short: before RabbitMQ 4.x the classic queues need to be gone for > good. No more config option, no looking back. > > Regards > > Christian > > From michel.jouvin at ijclab.in2p3.fr Fri Jul 14 21:24:07 2023 From: michel.jouvin at ijclab.in2p3.fr (Michel Jouvin) Date: Fri, 14 Jul 2023 23:24:07 +0200 Subject: Ussuri: how to delete lbaas loadbalancer left over? In-Reply-To: <20230707221544.Horde.tGBcMHNkuxKWyNahfmPmJ9V@webmail.nde.ag> References: <20230707164654.Horde.sUTt6M8NRpBGR8HwFy7KwGv@webmail.nde.ag> <1893244f3d8.27f3.86e99a005b57e9644b7bb922d8d7f665@ijclab.in2p3.fr> <20230707220512.Horde.10Q-iIqvZct1kSTQs9Gd4-u@webmail.nde.ag> <20230707221544.Horde.tGBcMHNkuxKWyNahfmPmJ9V@webmail.nde.ag> Message-ID: <1a342d65-191b-c367-9872-1d72f83479f6@ijclab.in2p3.fr> Hi Eugen, I found some time to look at my issue in the light of your advices! In my research, I found the presentation https://opendev.org/openstack/neutron-lbaas which is both synthetic and clear, with a list of the migration options offered (at the time of the presentation). This includes the DB migration tool, nlbaas2octavia.py. Following your broken link, I manage to find it, it is still there but hidden. You need to checkout the repo https://opendev.org/openstack/neutron-lbaas and go back a couple of revisions (it is explained in the README). It is Python2 but at least it is a starting point, either installing a Python2 venv or migrating the script which is not very long and seems to have very few dependcies, apart from oslo. I'm afraid that the neutron-lbaas-proxy is no longer an option as, as far as I understood, it relies on some LBAAS code still being present in the Neutron server and this part of the Neutron code has been completely removed I think in Ussuri release (it's the reason of my problems in fact, too bad, it was a matter of a few days, just missed the very old announcement that it would be the case). If I succeed to clean the things (again I don't really want to migrate the existing LBAAS, I just want to delete them), I'll report in this thread in case it is useful to somebody else... Best regards, Michel Le 08/07/2023 ? 00:15, Eugen Block a ?crit?: > Unfortunately, the link to the migration tool doesn?t work: > > https://wiki.openstack.org/wiki/Neutron/LBaaS/Deprecation > > But maybe it can get you in the right direction, neutron-lbaas-proxy > seems to be the keyword. But as I already mentioned, I don?t have > experience with this path. > > Zitat von Eugen Block : > >> Hi, >> >> I mean the latter. Once you have Octavia installed you can create new >> LBs, but as I understand it you won?t be able to delete the legacy >> LBs. Did the neutron config change when you upgraded to Ussuri? I >> wonder if there?s just some config missing to be able to delete the >> old LBs, I don?t have a clue tbh. Maybe someone else has some more >> experience and will chime in. >> >> Zitat von Michel Jouvin : >> >>> Ho Eug?ne, >>> >>> Thanks for your answer. Do you mean that after installing octavia >>> (it is planned) we'll have again the ability to delete the remaining >>> LBAAS instances? Or just that octavia is the LBAAS replacement in >>> terms of functionalities? >>> >>> Best regards, >>> >>> Michel >>> Sent from my mobile >>> Le 7 juillet 2023 18:52:30 Eugen Block a ?crit : >>> >>>> Hi, >>>> >>>> neutron lbaas was deprecated in Queens so you may have to migrate the >>>> existing LBs to octavia. I have never done that but I remember reading >>>> through the SUSE Docs when one of our customers had to decide whether >>>> they wanted to upgrade or reinstall with a newer openstack release. >>>> They decided to do the latter, so we set up octavia from scratch and >>>> didn't have to migrate anything. There's also a video I've never >>>> watched [2], maybe that helps. I can't really tell if a migration is >>>> possible to work around your issue but I thought I'd share anyway. >>>> >>>> Regards, >>>> Eugen >>>> >>>> [1] >>>> https://documentation.suse.com/soc/9/single-html/suse-openstack-cloud-crowbar-deployment/#sec-depl-ostack-octavia-migrate-users >>>> >>>> [2] https://www.youtube.com/watch?v=jj4KMJPA0Pk >>>> >>>> Zitat von Michel Jouvin : >>>> >>>>> Hi, >>>>> >>>>> We had a few Magnum (K8s) clusters created a couple of years ago >>>>> (with Rocky and Stein versions) and forgotten. We started to delete >>>>> them this spring when we where running Train Neutron service. >>>>> Basically we managed to do this with the following sequence: >>>>> >>>>> - openstack coe cluster delete xxx and waiting for DELETE_FAILED >>>>> - Use openstack coe cluster show / openstack stack resource list -n2 >>>>> to identify the neutron entry causing the error and pick the >>>>> corresponding resource ID >>>>> - Find the ports associated with the router with openstack port list >>>>> --router previously_found_id >>>>> - Use the port subnet to find the port corresponding lbaas load >>>>> balancer ID, use the neutron CLI to delete the load balancer >>>>> (deleting one by one all the dependencies preventing the load >>>>> balancer removal) >>>>> - Rerun openstack coe cluster delete >>>>> >>>>> For some reasons, we didn't cleanup all the abandoned clusters and >>>>> upgraded Neutron to Ussuri. Unfortunately, since then our previous >>>>> process is no longer working as it seems that the Neutron server >>>>> doesn't know anymore anything about the LBAAS load balancers (and >>>>> "neutron lbaas-loadbalancer list" returns nothing). In the neutron >>>>> server, any attempt to delete the subnet attached to the load >>>>> balancer (or to list them with Neutron CLI) results in the following >>>>> errors in Neutron server.log : >>>>> >>>>> ------ >>>>> >>>>> 2023-07-07 16:27:31.139 14962 WARNING >>>>> neutron.pecan_wsgi.controllers.root >>>>> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >>>>> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >>>>> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >>>>> 1367c9a4d5da4b229c35789c271dc7aa] No controller found for: lbaas - >>>>> returning response code 404: pecan.routing.PecanNotFound >>>>> 2023-07-07 16:27:31.140 14962 INFO >>>>> neutron.pecan_wsgi.hooks.translation >>>>> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >>>>> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >>>>> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >>>>> 1367c9a4d5da4b229c35789c271dc7aa] GET failed (client error): The >>>>> resource could not be found. >>>>> 2023-07-07 16:27:31.141 14962 INFO neutron.wsgi >>>>> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >>>>> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >>>>> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >>>>> 1367c9a4d5da4b229c35789c271dc7aa] 157.136.249.153 "GET >>>>> /v2.0/lbaas/loadbalancers?name=kube_service_964f7e76-d2d5-4126-ab11-cd689f6dd9f9_runnerdeploy-wm9sm-5h52l_hello-node-x-default-x-runnerdeploy-wm9sm-5h52l >>>>> HTTP/1.1" status: 404? len: 304 time: >>>>> 0.0052643 >>>>> ------ >>>>> >>>>> Any suggestion to workaround this problem and be able to >>>>> successfully delete our old Magnum clusters? >>>>> >>>>> Thanks in advance for any help. Best regards, >>>>> >>>>> Michel > > > > From nguyenhuukhoinw at gmail.com Sat Jul 15 01:02:48 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Sat, 15 Jul 2023 08:02:48 +0700 Subject: [cinder] cinder-backup volume stuck in creating In-Reply-To: <20230201120237.5p35swkzyxsg2koj@localhost> References: <20230201120237.5p35swkzyxsg2koj@localhost> Message-ID: Hello guys, I had the same problem, there is no more error log. I resolved by turning off cinder-api, cinder-scheduler and cinder-backup then removing these queues, such as cinder, cinder-scheduler and cinder-backup.. I enabled debug log on rabbitmq and cinder, however, I cannot see any related logs, 2023-07-13 20:57:22.903 666 INFO cinder.backup.manager [req-019d2459-6f43-4839-8ad6-71c805dbd708 6371ebfe0fce499983b1f07e7c34bf6d 183e47a6dd14489991db5c3cf4132a2a - - -] Create backup started, backup: 2cebd275-6fc3-47ae-8c8d-3337f9227057 volume: d8446fb4-032a-44a1-b47d-f922c4a0020e. 2023-07-13 20:57:22.919 666 INFO cinder.backup.manager [req-019d2459-6f43-4839-8ad6-71c805dbd708 6371ebfe0fce499983b1f07e7c34bf6d 183e47a6dd14489991db5c3cf4132a2a - - -] Call Volume Manager to get_backup_device for Backup(availability_zone=None,container=None,created_at=2023-07-13T13:57:22Z,data_timestamp=2023-07-13T13:57:22Z,deleted=False,deleted_at=None,display_description='',display_name='ijoo',encryption_key_id=None,fail_reason=None,host='controller01',id=2cebd275-6fc3-47ae-8c8d-3337f9227057,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='183e47a6dd14489991db5c3cf4132a2a',restore_volume_id=None,service='cinder.backup.drivers.nfs.NFSBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='6371ebfe0fce499983b1f07e7c34bf6d',volume_id=d8446fb4-032a-44a1-b47d-f922c4a0020e) It is hard to reproduce this problems. Nguyen Huu Khoi On Wed, Feb 1, 2023 at 7:08?PM Gorka Eguileor wrote: > On 27/01, Satish Patel wrote: > > Thank you Jon/Sofia, > > > > Biggest issue is even if I turn on debugging, it's not producing enough > > logs to see what is going on. See following output. > > > > https://paste.opendev.org/show/bh9OF9l2OrozrNMglv2Y/ > > Hi, > > I don't see the cinder-volume logs, which are necessary to debug this > issue, because we can see in the backup logs that it is doing an RPC > call to the cinder-volume service "Call Volume Manager to > get_backup_device". > > What is the value of the "host" field of the source volume? > > Because if it's anything other than "kolla-infra-1.example.com at rbd-1", > then the problem is that the cinder-volume service for that backend is > currently down. > > Cheers, > Gorka. > > > > > On Fri, Jan 27, 2023 at 10:50 AM Jon Bernard > wrote: > > > > > Without the logs themselves it's really hard to say. One way to > proceed > > > would be to file a bug [1] and the team can work with you there. You > > > could also enable debugging (debug = True), reproduce the failure, and > > > upload the relevant logs there as well. > > > > > > [1]: https://bugs.launchpad.net/cinder/+filebug > > > > > > -- > > > Jon > > > > > > On Thu, Jan 26, 2023 at 2:20 PM Satish Patel > wrote: > > > > > >> Folks, > > >> > > >> I have configured nova and cinder with ceph storage. VMs running on > ceph > > >> storage but now when i am trying to create a backup of cinder volume > its > > >> getting stuck on creating and doing nothing. Logs also do not give any > > >> indication of bad. > > >> > > >> My cinder.conf > > >> > > >> [DEFAULT] > > >> > > >> enabled_backends = rbd-1 > > >> backup_driver = cinder.backup.drivers.ceph.CephBackupDriver > > >> backup_ceph_conf = /etc/ceph/ceph.conf > > >> backup_ceph_user = cinder-backup > > >> backup_ceph_chunk_size = 134217728 > > >> backup_ceph_pool = backups > > >> backup_ceph_stripe_unit = 0 > > >> backup_ceph_stripe_count = 0 > > >> restore_discard_excess_bytes = true > > >> osapi_volume_listen = 10.73.0.181 > > >> osapi_volume_listen_port = 8776 > > >> > > >> > > >> Output of "openstack volume service list" showing cinder-backup > service > > >> is up but when i create a backup it's getting stuck in this stage and > no > > >> activity. I am not seeing anything getting transferred to the ceph > backups > > >> pool also. Any clue? or method to debug? > > >> > > >> # openstack volume backup list --all > > >> > > >> > +--------------------------------------+------+-------------+----------+------+ > > >> | ID | Name | Description | Status > | > > >> Size | > > >> > > >> > +--------------------------------------+------+-------------+----------+------+ > > >> | bc844d55-8c5a-4bd3-b0e9-7c4c780c95ad | foo1 | | > creating | > > >> 20 | > > >> > > >> > +--------------------------------------+------+-------------+----------+------+ > > >> > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nguyenhuukhoinw at gmail.com Sat Jul 15 01:10:41 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Sat, 15 Jul 2023 08:10:41 +0700 Subject: [kolla-ansible][octavia] In-Reply-To: References: Message-ID: Hello Chunyang, I followed your guide and it worked. I checked this network and it is only for octavia management, as you said, I had a cluster already. Is there another way to reconfigure without the above steps, I am curious. Could we enhance this behavior by customizing playbooks? Thank you. Regards Nguyen Huu Khoi On Thu, Jul 13, 2023 at 10:33?AM Nguy?n H?u Kh?i wrote: > Thank very much. > I got it, I will try and feedback :) > Nguyen Huu Khoi > > > On Thu, Jul 13, 2023 at 8:58?AM chunyang wu wrote: > >> Hi, >> you should check which network is using the vlan xxx in neutron, if >> the network is not intended for octavia management, you should delete it >> or change a new vlan id for octavia. otherwise, you could try to stop >> creating management network in kolla deployment by the flowing steps: >> I saw you perform a reconfigure action, so i assume you have an >> already worked cluster. >> 1. set octavia_auto_configure to false in you global.yaml. >> 2. set the flowing opts in gloabl.yaml >> octavia_amp_image_owner_id: the value you can find in your exists >> config file(e.g. /etc/kolla/octavia-api/octavia.conf : amp_image_owner_id) >> octavia_amp_boot_network_list : the vale of `amp_boot_network_list >> in octavia.conf >> octavia_amp_secgroup_list: the value of `amp_secgroup_list` in >> octavia.conf >> octavia_amp_flavor_id: the value of `amp_flavor_id ` in >> "octavia.config" >> 3. re-run reconfigure action. >> >> refer to: >> https://github.com/openstack/kolla-ansible/blob/master/etc/kolla/globals.yml#L778 >> >> thanks. >> >> Nguy?n H?u Kh?i ?2023?7?13??? 00:48??? >> >>> Hello guys. >>> >>> I am using openstack yoga by kolla-ansible but when i reconfigure >>> octavia then it stucked at >>> >>> TASK [octavia : Create loadbalancer management network] >>> >>> \"Unable to create the network. The VLAN xxx on physical network >>> physnet1 is in use.\ >>> >>> I see that it cannot create lb network because someone used it. >>> >>> Will we need to change some kolla-ansible tasks to fix it? >>> >>> >>> Thank you very much. >>> >>> Nguyen Huu Khoi >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From tobias.urdin at binero.com Sat Jul 15 14:27:20 2023 From: tobias.urdin at binero.com (Tobias Urdin) Date: Sat, 15 Jul 2023 14:27:20 +0000 Subject: [all] [oslo.messaging] Interest in collaboration on a NATS driver In-Reply-To: <3b2f711e-a2f1-71da-92ba-d36c229420a2@inovex.de> References: <52AA12A0-AE67-4EF7-B924-DE1F2873B909@binero.com> <0E347D5A-967D-42E3-AC00-56034907053B@binero.com> <8F7F790F-0F5C-4257-845A-1BCE671AF844@binero.com> <3b2f711e-a2f1-71da-92ba-d36c229420a2@inovex.de> Message-ID: <12EB84AA-A70A-4246-ABEC-4A4BC9E7B682@binero.com> Hello Christian, Sure, no problem I can give you a status update on the situation as of now! The effort is mostly now to get/find a supported third-party Python library that does not use asyncio for usage in a oslo.messaging driver. The investigation into introducing asyncio support in the ecosystem (oslo and then all separate projects) was a too massive effort. Since then I?ve opened an issue [1] on the nats.py GitHub and some proof-of-concept patches [2] on introducing synchronous support in nats.py that we could then use in a driver. The oslo.messaging driver patch [3] has Devstack passing when using [2] but it?s also more of a proof-of-concept patch that would need more work in itself. Unfortunately I didn?t get any feedback from the company maintaining the nats.py library on supported developer time implemeting the sync client feature, I assume (my view) that they didn?t like that we cannot guarantee that NATS would become a core component (or used, or the primary bus) in the ecosystem (since we don?t, and don?t want to for that matter, force a backend/driver on operators). Any help is of course appreciated, right now this is not a top priority and I don?t spend any full-time effort into it. Best regards Tobias [1] https://github.com/nats-io/nats.py/issues/462 [2] https://github.com/tobias-urdin/nats.py/commits/sync-client [3] https://review.opendev.org/c/openstack/oslo.messaging/+/878588 > On 14 Jul 2023, at 09:37, Christian Rohmann wrote: > > On 12/09/2022 09:05, Tobias Urdin wrote: >> Hello, >> >> Since I haven?t heard any objections we?ll move the meeting to the Oslo >> meeting on the 19th of September. See you there! [1] >> >> [1] https://wiki.openstack.org/wiki/Meetings/Oslo >> > Could you maybe give an update on the current state on adding NATS to oslo.messaging? > > I know there was some positive exchange during an Oslo Meeting [1], but now all but the spec change > [2] are abandoned? Is the work on the spec [3] still going then? > > > Regards > > Christian > > > [1] https://meetings.opendev.org/meetings/oslo/2022/oslo.2022-09-19-15.12.log.txt > [2] https://review.opendev.org/q/topic:asyncio-nats > [3] https://review.opendev.org/c/openstack/oslo-specs/+/692784 > From murilo at evocorp.com.br Sat Jul 15 14:58:33 2023 From: murilo at evocorp.com.br (Murilo Morais) Date: Sat, 15 Jul 2023 11:58:33 -0300 Subject: [OSA][NEUTRON] Failed to bind port XXX on host YYY for vnic_type normal using segments... Message-ID: Hey guys. I'm having problems with a new deploy. I'm relatively new to the project, so I still don't know how to guide myself in solving some errors. When I create some Network and try to upload a VM the following errors appear in the Neutron container: ############################### Jul 15 11:31:50 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: 2023-07-15 11:31:50.988 5676 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:31:51 dcn2-neutron-server-container-a7122a16 neutron-api[5683]: 2023-07-15 11:31:51.009 5683 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] ECONNRESET (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] ECONNRESET Jul 15 11:31:51 dcn2-neutron-server-container-a7122a16 uwsgi[5683]: Sat Jul 15 11:31:51 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/availability_zones (ip 172.29.237.22) !!! Jul 15 11:33:53 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: 2023-07-15 11:33:53.860 5676 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] ECONNRESET (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] ECONNRESET Jul 15 11:33:58 dcn2-neutron-server-container-a7122a16 neutron-api[5678]: 2023-07-15 11:33:58.733 5678 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:33:58 dcn2-neutron-server-container-a7122a16 uwsgi[5678]: Sat Jul 15 11:33:58 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnetpools (ip 172.29.237.22) !!! Jul 15 11:33:58 dcn2-neutron-server-container-a7122a16 neutron-api[5677]: 2023-07-15 11:33:58.959 5677 INFO neutron.db.segments_db [None req-61542ff4-867d-402f-8239-b45392b2c188 f8da6c7abaa64116948db1d2aaa829f1 86e3eede43b940d69d2b43ccba1b256b - - default default] Added segment 16ec3850-a4f2-40f3-9dc8-3b4650053935 of type vlan for network 6887bbfb-4680-4ed7-910b-90a65f7731dc Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: 2023-07-15 11:33:59.095 5680 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 uwsgi[5680]: Sat Jul 15 11:33:59 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnets (ip 172.29.237.22) !!! Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: 2023-07-15 11:33:59.584 5680 WARNING neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api [None req-5ad409c5-c073-4726-bb26-5024d9c9c65b f8da6c7abaa64116948db1d2aaa829f1 86e3eede43b940d69d2b43ccba1b256b - - default default] Unable to schedule network 6887bbfb-4680-4ed7-910b-90a65f7731dc: no agents available; will retry on subsequent port and subnet creation events. Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:33:59.684 5688 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:41:32 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: 2023-07-15 11:41:32.961 5680 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: conn = connection.create_connection( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise err Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: sock.connect(sa) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/eventlet/greenio/base.py", line 256, in connect Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: socket_checkerr(fd) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/eventlet/greenio/base.py", line 54, in socket_checkerr Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise socket.error(err, errno.errorcode[err]) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: BrokenPipeError: [Errno 32] EPIPE Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During handling of the above exception, another exception occurred: Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: httplib_response = self._make_request( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connectionpool.py", line 398, in _make_request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: conn.request(method, url, **httplib_request_kw) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", line 239, in request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: super(HTTPConnection, self).request(method, url, body=body, headers=headers) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 1255, in request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self._send_request(method, url, body, headers, encode_chunked) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 1301, in _send_request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self.endheaders(body, encode_chunked=encode_chunked) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 1250, in endheaders Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self._send_output(message_body, encode_chunked=encode_chunked) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 1010, in _send_output Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self.send(msg) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 950, in send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self.connect() Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", line 205, in connect Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: conn = self._new_conn() Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise NewConnectionError( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 32] EPIPE Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During handling of the above exception, another exception occurred: Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/adapters.py", line 489, in send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: resp = conn.urlopen( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: retries = retries.increment( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise MaxRetryError(_pool, url, error or ResponseError(cause)) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.29.236.2', port=8780): Max retries exceeded with url: /resource_providers/dd7e74fb-b494-448e-b16a-712a7ad60e6c/aggregates (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 32] EPIPE')) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During handling of the above exception, another exception occurred: Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", line 1022, in _send_request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: resp = self.session.request(method, url, **kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/sessions.py", line 587, in request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: resp = self.send(prep, **send_kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/sessions.py", line 701, in send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: r = adapter.send(request, **kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/adapters.py", line 565, in send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise ConnectionError(e, request=request) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.29.236.2', port=8780): Max retries exceeded with url: /resource_providers/dd7e74fb-b494-448e-b16a-712a7ad60e6c/aggregates (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 32] EPIPE')) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During handling of the above exception, another exception occurred: Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/eventlet/hubs/poll.py", line 111, in wait Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: listener.cb(fileno) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/common/utils.py", line 924, in wrapper Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return func(*args, **kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/notifiers/batch_notifier.py", line 58, in synced_send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self._notify() Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/notifiers/batch_notifier.py", line 69, in _notify Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self.callback(batched_events) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/services/segments/plugin.py", line 211, in _send_notifications Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: event.method(event) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/services/segments/plugin.py", line 383, in _delete_nova_inventory Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: aggregate_id = self._get_aggregate_id(event.segment_id) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/services/segments/plugin.py", line 370, in _get_aggregate_id Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: aggregate_uuid = self.p_client.list_aggregates( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron_lib/placement/client.py", line 58, in wrapper Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return f(self, *a, **k) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron_lib/placement/client.py", line 554, in list_aggregates Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return self._get(url).json() Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron_lib/placement/client.py", line 190, in _get Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return self._client.get(url, endpoint_filter=self._ks_filter, Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", line 1141, in get Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return self.request(url, 'GET', **kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", line 931, in request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: resp = send(**kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", line 1038, in _send_request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise exceptions.ConnectFailure(msg) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:41:33.144 5688 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] ECONNRESET (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] ECONNRESET Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://172.29.236.2:8780/resource_providers/dd7e74fb-b494-448e-b16a-712a7ad60e6c/aggregates: HTTPConnectionPool(host='172.29.236.2', port=8780): Max retries exceeded with url: /resource_providers/dd7e74fb-b494-448e-b16a-712a7ad60e6c/aggregates (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 32] EPIPE')) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Removing descriptor: 13 Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 neutron-api[5689]: 2023-07-15 11:41:33.542 5689 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:42:36 dcn2-neutron-server-container-a7122a16 neutron-api[5678]: 2023-07-15 11:42:36.571 5678 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] Connection reset by peer (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] Connection reset by peer Jul 15 11:42:40 dcn2-neutron-server-container-a7122a16 neutron-api[5689]: 2023-07-15 11:42:40.942 5689 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] Connection reset by peer (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] Connection reset by peer Jul 15 11:42:41 dcn2-neutron-server-container-a7122a16 uwsgi[5683]: Sat Jul 15 11:42:41 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnets (ip 172.29.237.22) !!! Jul 15 11:42:42 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:42:42.053 5688 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] Connection reset by peer (retrying in 3.0 seconds): ConnectionResetError: [Errno 104] Connection reset by peer Jul 15 11:42:45 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:42:45.240 5688 WARNING neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-0270e149-44c9-49c4-af20-47918e00c4ea f8da6c7abaa64116948db1d2aaa829f1 86e3eede43b940d69d2b43ccba1b256b - - default default] Unable to schedule network 6887bbfb-4680-4ed7-910b-90a65f7731dc: no agents available; will retry on subsequent port and subnet creation events. Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.563 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.564 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 2 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.641 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.641 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 3 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.695 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.695 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 4 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.743 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.743 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 5 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.798 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.799 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 6 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.856 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.856 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 7 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.904 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.904 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 8 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.952 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.952 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 9 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:47 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:47.000 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:47 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:47.001 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 10 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:47 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:47.056 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:54 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: 2023-07-15 11:42:54.511 5676 INFO neutron.notifiers.nova [-] Nova event matching ['req-0ffbf4aa-051f-4707-afe7-2908726310c5'] response: {'server_uuid': '9f206d9f-6c96-479f-bbaa-116d967129dc', 'name': 'network-vif-deleted', 'tag': 'caa01668-d5c0-4b37-a185-0ebc458326f7', 'status': 'completed', 'code': 200} Jul 15 11:48:24 dcn2-neutron-server-container-a7122a16 neutron-api[5678]: 2023-07-15 11:48:24.062 5678 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:31 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: 2023-07-15 11:48:31.762 5676 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:31 dcn2-neutron-server-container-a7122a16 neutron-api[5677]: 2023-07-15 11:48:31.828 5677 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:48:31 dcn2-neutron-server-container-a7122a16 uwsgi[5677]: Sat Jul 15 11:48:31 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnets (ip 172.29.237.22) !!! Jul 15 11:48:32 dcn2-neutron-server-container-a7122a16 neutron-api[5686]: 2023-07-15 11:48:32.108 5686 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:48:32 dcn2-neutron-server-container-a7122a16 uwsgi[5686]: Sat Jul 15 11:48:32 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/networks?limit=21&sort_dir=asc&sort_key=id&router%3Aexternal=True&shared=False (ip 172.29.237.22) !!! Jul 15 11:48:34 dcn2-neutron-server-container-a7122a16 neutron-api[5689]: 2023-07-15 11:48:34.625 5689 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:34 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:48:34.641 5688 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:35 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:48:35.978 5682 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:36 dcn2-neutron-server-container-a7122a16 neutron-api[5683]: 2023-07-15 11:48:36.052 5683 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:36 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: 2023-07-15 11:48:36.809 5680 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:48:36 dcn2-neutron-server-container-a7122a16 uwsgi[5680]: Sat Jul 15 11:48:36 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnetpools (ip 172.29.237.22) !!! ############################### After the errors appear, the Horizon interface reports the following error: Error: Failed to perform requested operation on instance "test", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9f206d9f-6c96-479f-bbaa-116d967129dc.]. What could be happening? I believe I'm missing something to configure, but I don't know where to start. -------------- next part -------------- An HTML attachment was scrubbed... URL: From noonedeadpunk at gmail.com Sun Jul 16 05:44:11 2023 From: noonedeadpunk at gmail.com (Dmitriy Rabotyagov) Date: Sun, 16 Jul 2023 07:44:11 +0200 Subject: [OSA][NEUTRON] Failed to bind port XXX on host YYY for vnic_type normal using segments... In-Reply-To: References: Message-ID: Hey, Can you kindly elaborate what neutron driver do you use for the deployment (OVN/OVS/LXB)? Or, if you unsure and was just relying on the default, what osa version you are using? Also can you kindly provide output for "openstack network agent list" and "openstack compute service list"? -------------- next part -------------- An HTML attachment was scrubbed... URL: From miguel at mlavalle.com Mon Jul 17 00:19:10 2023 From: miguel at mlavalle.com (Miguel Lavalle) Date: Sun, 16 Jul 2023 19:19:10 -0500 Subject: [neutron] Bugs deputy report July 10th - 16th Message-ID: Hi, Here's last week's bugs deputy report: Critical ====== https://bugs.launchpad.net/neutron/+bug/2027817 neutron slow jobs before xena broken with ERROR: unknown environment 'slow'. Assigned to Ghanshyam Mann. Proposed fix: https://review.opendev.org/c/openstack/tempest/+/888590. In progress High ==== https://bugs.launchpad.net/neutron/+bug/2027605 [OVN][Trunk] Sometimes subport doesn't reach status ACTIVE after live-migration. Requires owner Medium ====== https://bugs.launchpad.net/neutron/+bug/2027595 [sqlalchemy-20] Check represent "network" model regardless of the elements order. Assigned to Rodolfo Alonso. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/888286. Merged. https://bugs.launchpad.net/neutron/+bug/2027602 [sqlalchemy-20] "FromClause.join" accepts only one model and an optional expression to define the join. Assigned to Rodolfo Alonso. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/888295. Merged. https://bugs.launchpad.net/neutron/+bug/2027604 [sqlalchemy-20] The Query.get() method is considered legacy as of the 1.x series of SQLAlchemy and becomes a legacy construct in 2.0. Assigned to Rodolfo Alonso https://bugs.launchpad.net/neutron/+bug/2027610 [sqlalchemy-20] Error in "test_notify_port_status_all_values" using an wrong OVO parameter Assigned to Rodolfo Alonso. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/888305. Merged RFEs ==== https://bugs.launchpad.net/neutron/+bug/2027742 [RFE] unmanaged dynamic router resources - OVN Heads up from the Ironic team: ======================== https://bugs.launchpad.net/neutron/+bug/2026757. dnsmasq on Ubuntu Jammy/Lunar crashes on neutron-dhcp-agent updates. Julia Kreger indicates that most likely this is not a neutron bug. However, she believes we might encounter it at some point. Yatin Karel is watching it. Bringing it up just for visibility to the Neutron team -------------- next part -------------- An HTML attachment was scrubbed... URL: From felix.huettner at mail.schwarz Mon Jul 17 06:43:12 2023 From: felix.huettner at mail.schwarz (Felix Huettner) Date: Mon, 17 Jul 2023 08:43:12 +0200 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: Hi everyone, On Thu, Jul 13, 2023 at 07:29:51PM -0300, Roberto Bartzen Acosta wrote: > Hi Rodolfo, > > Em qui., 13 de jul. de 2023 ?s 13:45, Rodolfo Alonso Hernandez < > ralonsoh at redhat.com> escreveu: > > > Hello Roberto: > > > > I think you have provided a very detailed explanation of the issue but you > > still didn't present any possible solution. I would ask you at least for a > > proposal and for specific details of what is failing in the OVN sync tool. > > > > For example, if you add a TS between two clouds, you'll need to create (1) > > routers (NAT registers), (2) router ports (LRP) and (3) some static routes > > (LRSR). All these elements are monitored by the DB sync tool and will fail > > because the counterparts in the Neutron DB don't exist. I guess that you > > have some script or tool or at least a document to manually add these > > resources; sharing it could help to find a solution. > > > > If these elements are manually created during the deployment phase, you > > can also control the information provided. I'm thinking about the > > "external_ids" field; we can push some defined constant that will make > > these registers transparent to the OVN DB sync tool. As these resources are also created by tools not necessarily controlled by us we need to avoid changing them. I would therefor propose to not add a "ignore" constant to the "external_ids", to to rather check for the existence of the neutron keys in "external_ids" (i think thats also what is described in the bug below). > > > > Please check this idea and if it is feasible. In that case, please add a > > topic to the Neutron drivers meeting agenda [1] to discuss it. > > > > That sounds good, thanks. > https://bugs.launchpad.net/neutron/+bug/2027742 >From my perspective there is also an additional step after the above REF. Currently it just makes it possible to use ovn-ic without neutron killing it. However if we could get neutron to treat a Transit Switch as a external network we would skip the need to implement the NAT, LRP and LRSR logic in custom tooling and could use the existing neutron code. This would probably require creating a custom type of provider network and then specifying the transit switch name in the provider options. Is that something that from your perspective we should take a look at right away, or rather first focus und neutron not burning the transit switch. Regards Felix > > Regards, > Roberto > > > > > > Regards. > > > > [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers > > > > > > On Wed, Jul 12, 2023 at 2:12?PM Roberto Bartzen Acosta < > > roberto.acosta at luizalabs.com> wrote: > > > >> > >> > >> Em qua., 12 de jul. de 2023 ?s 09:05, Roberto Bartzen Acosta < > >> roberto.acosta at luizalabs.com> escreveu: > >> > >>> Hi Rodolfo, > >>> > >>> Thanks for the feedback. > >>> > >>> I liked the idea of having a different network type only to hold the > >>> Transit Switches and thus the LRP but it depends on some factors like > >>> multi-tenancy. > >>> From the OVN perspective, the TS is global and will only be linked to a > >>> tenant when it is plugged into the tenant router port. My concern here is > >>> that if Neutron manages the TS, it will need to assume the dynamic > >>> characteristics of this TS function, ovn-ic creates and removes the TS in > >>> NB_Global, in addition to plugging and removing ports in this switch > >>> according to the network topology. Additionally, Neutron would still need > >>> to handle the learned remote routes (which are listed as static routes from > >>> the tenant router perspective). > >>> > >> sorry: NB_Global -> Northbound Database. > >> > >> > >>> This is an open discussion, Felix can add ideas here. > >>> > >>> In general, it seems to me that we have no alternatives for this type of > >>> solution other than OVN-IC (note that ovn-bgp-agent does not learn remote > >>> routes at the SDN level / OVN). > >>> OpenStack seems to be designed to run like a "bubble" and this traffic > >>> from one VM to another always needs to be routed at the FIP level and > >>> external routing layers. > >>> > >>> Regards, > >>> Roberto > >>> > >>> Em qua., 12 de jul. de 2023 ?s 05:11, Rodolfo Alonso Hernandez < > >>> ralonsoh at redhat.com> escreveu: > >>> > >>>> Hello Roberto: > >>>> > >>>> We talked about this possible RFE during the PTG [1] but I don't > >>>> remember having any proposal. Actually what I remember (and was written in > >>>> the etherpad) is that you were going to present an RFE. Can you explain it > >>>> briefly? > >>>> > >>>> We also talked about the idea, proposed by Felix in the mailing list, > >>>> of having a different network type only to hold the Transit Switches and > >>>> thus the LRP. If you have a proposal, please present it. > >>>> > >>>> Regards. > >>>> > >>>> [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506 > >>>> > >>>> On Tue, Jul 11, 2023 at 10:50?PM Roberto Bartzen Acosta < > >>>> roberto.acosta at luizalabs.com> wrote: > >>>> > >>>>> Hello Neutron folks, > >>>>> > >>>>> Regarding the conversation started in March [1] about the use of OVN > >>>>> interconnect with Neutron, I would like to evolve the discussion in > >>>>> relation to resources allocated by OVN-IC and which are not managed by > >>>>> Neutron. In the March PTG [2], the problem related to the db_sync tool was > >>>>> presented, and a proposed solution in which Neutron does not manage these > >>>>> resources. > >>>>> > >>>>> After discussing some architectural designs with Felix/StackIT, it > >>>>> seems to make sense to come up with a proposal to change the mech_driver to > >>>>> validate the external_ids key and not remove resources present in the OVN > >>>>> backend without Neutron "signature". > >>>>> > >>>>> The discussion here is more complex than it seems, because > >>>>> architecturally Neutron does not allow us to interconnect workloads in > >>>>> multiple AZs (with different OpenStacks), but this could be a requirement > >>>>> for a high availability cloud solution (Cloud Region). Additionally, this > >>>>> OVN-IC solution allows interconnecting other cloud backends that use OVN, > >>>>> in the case of kubernetes with ovn-kube. > >>>>> > >>>>> We tested an OVN interconnect integrated with 3 OpenStack > >>>>> installations and it worked very well !!! considering direct L3 traffic at > >>>>> the router level between different network infrastructures. > >>>>> > >>>>> SYNC_REPAIR - problem > >>>>> > >>>>> * Static Routes (learned OVN-IC routes) > >>>>> * Router Port -> Transit Switches > >>>>> > >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae > >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING > >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync > >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in > >>>>> OVN but not in Neutron, port_id=rt2-admin-tenant1 > >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae > >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING > >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync > >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router > >>>>> 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' > >>>>> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': ' > >>>>> 10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in > >>>>> Neutron > >>>>> > >>>>> > >>>>> Any suggestions on how to resolve this db_sync issue? since all other > >>>>> Neutron modules did not present any problem with OVN-IC (as far as I > >>>>> tested). > >>>>> I remember Terry was keen to suggest some things to help. I believe > >>>>> that before proposing some complex mechanism to solve this simple problem, > >>>>> I would like to hear the community opinion. In our use case, a bit change > >>>>> in db_sync with filter by neutron key in external_ids would be enough. > >>>>> > >>>>> Regards, > >>>>> Roberto > >>>>> > >>>>> > >>>>> > >>>>> [1] > >>>>> https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html > >>>>> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg > >>>>> > >>>>> > >>>>> > >>>>> Additional logs: > >>>>> > >>>>> OpenStack 1 > >>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl > >>>>> lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 > >>>>> IPv4 Routes > >>>>> Route Table
: > >>>>> 20.0.1.0/24 169.254.200.2 dst-ip (learned) > >>>>> 20.0.2.0/24 169.254.200.3 dst-ip (learned) > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip > >>>>> > >>>>> IPv6 Routes > >>>>> Route Table
: > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl > >>>>> lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 > >>>>> IPv4 Routes > >>>>> Route Table
: > >>>>> 10.0.1.0/24 169.254.100.2 dst-ip (learned) > >>>>> 10.0.2.0/24 169.254.100.3 dst-ip (learned) > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip > >>>>> > >>>>> IPv6 Routes > >>>>> Route Table
: > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# > >>>>> > >>>>> > >>>>> OpenStack 2 > >>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl > >>>>> lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc > >>>>> IPv4 Routes > >>>>> Route Table
: > >>>>> 20.0.0.0/24 169.254.200.1 dst-ip (learned) > >>>>> 20.0.2.0/24 169.254.200.3 dst-ip (learned) > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip > >>>>> > >>>>> IPv6 Routes > >>>>> Route Table
: > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip > >>>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl > >>>>> lr-route-list ce45f681-6454-43fe-974f-81344bb8113a > >>>>> IPv4 Routes > >>>>> Route Table
: > >>>>> 10.0.0.0/24 169.254.100.1 dst-ip (learned) > >>>>> 10.0.2.0/24 169.254.100.3 dst-ip (learned) > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip > >>>>> > >>>>> IPv6 Routes > >>>>> Route Table
: > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip > >>>>> > >>>>> > >>>>> > >>>>> OpenStack 3 > >>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl > >>>>> lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 > >>>>> IPv4 Routes > >>>>> Route Table
: > >>>>> 20.0.0.0/24 169.254.200.1 dst-ip (learned) > >>>>> 20.0.1.0/24 169.254.200.2 dst-ip (learned) > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip > >>>>> > >>>>> IPv6 Routes > >>>>> Route Table
: > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl > >>>>> lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 > >>>>> IPv4 Routes > >>>>> Route Table
: > >>>>> 10.0.0.0/24 169.254.100.1 dst-ip (learned) > >>>>> 10.0.1.0/24 169.254.100.2 dst-ip (learned) > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip > >>>>> > >>>>> IPv6 Routes > >>>>> Route Table
: > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip > >>>>> > >>>>> > >>>>> OVN-IC Global database > >>>>> > >>>>> root at ovn-global-db1:~# ovn-ic-sbctl show > >>>>> availability-zone osp1 > >>>>> gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 > >>>>> hostname: osp1-gwnode1 > >>>>> type: geneve > >>>>> ip: 192.168.200.28 > >>>>> port admin-rt1-tenant1 > >>>>> transit switch: admin-tenant1 > >>>>> address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] > >>>>> port admin-rt1-tenant1_1 > >>>>> transit switch: admin-tenant1_1 > >>>>> address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] > >>>>> availability-zone osp2 > >>>>> gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 > >>>>> hostname: osp2-gwnode1 > >>>>> type: geneve > >>>>> ip: 192.168.200.128 > >>>>> port admin-rt2-tenant1 > >>>>> transit switch: admin-tenant1 > >>>>> address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] > >>>>> port admin-rt2-tenant1_1 > >>>>> transit switch: admin-tenant1_1 > >>>>> address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] > >>>>> availability-zone osp3 > >>>>> gateway 97595af9-7896-40d0-a883-beadbff1aa5b > >>>>> hostname: osp3-gwnode1 > >>>>> type: geneve > >>>>> ip: 192.168.200.228 > >>>>> port admin-rt3-tenant1 > >>>>> transit switch: admin-tenant1 > >>>>> address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] > >>>>> port admin-rt3-tenant1_1 > >>>>> transit switch: admin-tenant1_1 > >>>>> address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> *?Esta mensagem ? direcionada apenas para os endere?os constantes no > >>>>> cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no > >>>>> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa > >>>>> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o > >>>>> imediatamente anuladas e proibidas?.* > >>>>> > >>>>> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para > >>>>> assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o > >>>>> poder? aceitar a responsabilidade por quaisquer perdas ou danos causados > >>>>> por esse e-mail ou por seus anexos?.* > >>>>> > >>>> > >> > >> *?Esta mensagem ? direcionada apenas para os endere?os constantes no > >> cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no > >> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa > >> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o > >> imediatamente anuladas e proibidas?.* > >> > >> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para > >> assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o > >> poder? aceitar a responsabilidade por quaisquer perdas ou danos causados > >> por esse e-mail ou por seus anexos?.* > >> > > > > -- > > > > > _?Esta mensagem ? direcionada apenas para os endere?os constantes no > cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no > cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa > mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o > imediatamente anuladas e proibidas?._ > > > * **?Apesar do Magazine Luiza tomar > todas as precau??es razo?veis para assegurar que nenhum v?rus esteja > presente nesse e-mail, a empresa n?o poder? aceitar a responsabilidade por > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos?.* > > > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r die Verwertung durch den vorgesehenen Empf?nger bestimmt. Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender bitte unverz?glich in Kenntnis und l?schen diese E Mail. Hinweise zum Datenschutz finden Sie hier. This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail. Information on data protection can be found here. From frode.nordahl at canonical.com Mon Jul 17 09:01:46 2023 From: frode.nordahl at canonical.com (Frode Nordahl) Date: Mon, 17 Jul 2023 11:01:46 +0200 Subject: [charms] Nominate Martin Kalcok for charms-core Message-ID: Hello all, I would like to nominate Martin Kalcok for charms-core. He has been part of the extended charms teams for some time now, and has provided many contributions to OpenStack, Ceph and OVN charms [0]. He is diligent and thorough in his contributions, and has recently become a full time member of the focused OVN team. 0: https://review.opendev.org/q/owner:martin.kalcok%2540canonical.com -- Frode Nordahl From fkr at osb-alliance.com Mon Jul 17 11:58:55 2023 From: fkr at osb-alliance.com (Felix Kronlage-Dammers) Date: Mon, 17 Jul 2023 13:58:55 +0200 Subject: [publiccloud-sig] Reminder - next meeting July 19th - 0700 UTC Message-ID: <0D29A2FA-C0C2-42B1-B897-DE4FABB0BB0A@osb-alliance.com> Hi everyone, this Wednesday the next meeting of the public cloud sig is going to happen. We meet on IRC in #openstack-operators at 0700 UTC. This time, we will switch to a jitsi-based video-call directly after start. The url will be posted on #openstack-operators at the time of the meeting. A preliminary agenda can be found in the pad: https://etherpad.opendev.org/p/publiccloud-sig-meeting See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG read^see you on Wednesday! felix -- Felix Kronlage-Dammers Product Owner IaaS & Operations Sovereign Cloud Stack Sovereign Cloud Stack ? standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband f?r digitale Souver?nit?t e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr at osb-alliance.com From noonedeadpunk at gmail.com Sun Jul 16 05:42:34 2023 From: noonedeadpunk at gmail.com (Dmitriy Rabotyagov) Date: Sun, 16 Jul 2023 07:42:34 +0200 Subject: [OSA][NEUTRON] Failed to bind port XXX on host YYY for vnic_type normal using segments... In-Reply-To: References: Message-ID: Hey, Can you kindly elaborate what neutron driver do you use for the deployment (OVN/OVS/LXB)? Or, if you unsure and was just relying on the default, what osa version you are using? Also can you kindly provide output for "openstack network agent list" and "openstack compute service list"? On Sat, Jul 15, 2023, 22:14 Murilo Morais wrote: > Hey guys. I'm having problems with a new deploy. I'm relatively new to the > project, so I still don't know how to guide myself in solving some errors. > > When I create some Network and try to upload a VM the following errors > appear in the Neutron container: > > ############################### > > Jul 15 11:31:50 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: > 2023-07-15 11:31:50.988 5676 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: Server > unexpectedly closed connection > Jul 15 11:31:51 dcn2-neutron-server-container-a7122a16 neutron-api[5683]: > 2023-07-15 11:31:51.009 5683 ERROR oslo.messaging._drivers.impl_rabbit [-] > Connection failed: [Errno 104] ECONNRESET (retrying in 1.0 seconds): > ConnectionResetError: [Errno 104] ECONNRESET > Jul 15 11:31:51 dcn2-neutron-server-container-a7122a16 uwsgi[5683]: Sat > Jul 15 11:31:51 2023 - SIGPIPE: writing to a closed pipe/socket/fd > (probably the client disconnected) on request /v2.0/availability_zones (ip > 172.29.237.22) !!! > Jul 15 11:33:53 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: > 2023-07-15 11:33:53.860 5676 ERROR oslo.messaging._drivers.impl_rabbit [-] > Connection failed: [Errno 104] ECONNRESET (retrying in 1.0 seconds): > ConnectionResetError: [Errno 104] ECONNRESET > Jul 15 11:33:58 dcn2-neutron-server-container-a7122a16 neutron-api[5678]: > 2023-07-15 11:33:58.733 5678 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: [Errno > 104] Connection reset by peer > Jul 15 11:33:58 dcn2-neutron-server-container-a7122a16 uwsgi[5678]: Sat > Jul 15 11:33:58 2023 - SIGPIPE: writing to a closed pipe/socket/fd > (probably the client disconnected) on request /v2.0/subnetpools (ip > 172.29.237.22) !!! > Jul 15 11:33:58 dcn2-neutron-server-container-a7122a16 neutron-api[5677]: > 2023-07-15 11:33:58.959 5677 INFO neutron.db.segments_db [None > req-61542ff4-867d-402f-8239-b45392b2c188 f8da6c7abaa64116948db1d2aaa829f1 > 86e3eede43b940d69d2b43ccba1b256b - - default default] Added segment > 16ec3850-a4f2-40f3-9dc8-3b4650053935 of type vlan for network > 6887bbfb-4680-4ed7-910b-90a65f7731dc > Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: > 2023-07-15 11:33:59.095 5680 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: [Errno > 104] Connection reset by peer > Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 uwsgi[5680]: Sat > Jul 15 11:33:59 2023 - SIGPIPE: writing to a closed pipe/socket/fd > (probably the client disconnected) on request /v2.0/subnets (ip > 172.29.237.22) !!! > Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: > 2023-07-15 11:33:59.584 5680 WARNING > neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api [None > req-5ad409c5-c073-4726-bb26-5024d9c9c65b f8da6c7abaa64116948db1d2aaa829f1 > 86e3eede43b940d69d2b43ccba1b256b - - default default] Unable to schedule > network 6887bbfb-4680-4ed7-910b-90a65f7731dc: no agents available; will > retry on subsequent port and subnet creation events. > Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: > 2023-07-15 11:33:59.684 5688 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: Server > unexpectedly closed connection > Jul 15 11:41:32 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: > 2023-07-15 11:41:32.961 5680 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: Server > unexpectedly closed connection > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > Traceback (most recent call last): > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", > line 174, in _new_conn > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > conn = connection.create_connection( > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/util/connection.py", > line 95, in create_connection > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > raise err > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/util/connection.py", > line 85, in create_connection > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > sock.connect(sa) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/eventlet/greenio/base.py", > line 256, in connect > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > socket_checkerr(fd) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/eventlet/greenio/base.py", > line 54, in socket_checkerr > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > raise socket.error(err, errno.errorcode[err]) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > BrokenPipeError: [Errno 32] EPIPE > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During > handling of the above exception, another exception occurred: > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > Traceback (most recent call last): > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connectionpool.py", > line 703, in urlopen > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > httplib_response = self._make_request( > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connectionpool.py", > line 398, in _make_request > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > conn.request(method, url, **httplib_request_kw) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", > line 239, in request > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > super(HTTPConnection, self).request(method, url, body=body, headers=headers) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/usr/lib/python3.9/http/client.py", line 1255, in request > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > self._send_request(method, url, body, headers, encode_chunked) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/usr/lib/python3.9/http/client.py", line 1301, in _send_request > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > self.endheaders(body, encode_chunked=encode_chunked) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/usr/lib/python3.9/http/client.py", line 1250, in endheaders > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > self._send_output(message_body, encode_chunked=encode_chunked) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/usr/lib/python3.9/http/client.py", line 1010, in _send_output > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > self.send(msg) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/usr/lib/python3.9/http/client.py", line 950, in send > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > self.connect() > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", > line 205, in connect > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > conn = self._new_conn() > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", > line 186, in _new_conn > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > raise NewConnectionError( > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > urllib3.exceptions.NewConnectionError: object at 0x7efd5f36ca30>: Failed to establish a new connection: [Errno 32] > EPIPE > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During > handling of the above exception, another exception occurred: > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > Traceback (most recent call last): > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/adapters.py", > line 489, in send > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > resp = conn.urlopen( > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connectionpool.py", > line 787, in urlopen > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > retries = retries.increment( > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/util/retry.py", > line 592, in increment > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > raise MaxRetryError(_pool, url, error or ResponseError(cause)) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.29.236.2', > port=8780): Max retries exceeded with url: > /resource_providers/dd7e74fb-b494-448e-b16a-712a7ad60e6c/aggregates (Caused > by NewConnectionError(' 0x7efd5f36ca30>: Failed to establish a new connection: [Errno 32] EPIPE')) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During > handling of the above exception, another exception occurred: > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > Traceback (most recent call last): > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", > line 1022, in _send_request > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > resp = self.session.request(method, url, **kwargs) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/sessions.py", > line 587, in request > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > resp = self.send(prep, **send_kwargs) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/sessions.py", > line 701, in send > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: r > = adapter.send(request, **kwargs) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/adapters.py", > line 565, in send > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > raise ConnectionError(e, request=request) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > requests.exceptions.ConnectionError: > HTTPConnectionPool(host='172.29.236.2', port=8780): Max retries exceeded > with url: > /resource_providers/dd7e74fb-b494-448e-b16a-712a7ad60e6c/aggregates (Caused > by NewConnectionError(' 0x7efd5f36ca30>: Failed to establish a new connection: [Errno 32] EPIPE')) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During > handling of the above exception, another exception occurred: > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > Traceback (most recent call last): > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/eventlet/hubs/poll.py", > line 111, in wait > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > listener.cb(fileno) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/common/utils.py", > line 924, in wrapper > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > return func(*args, **kwargs) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/notifiers/batch_notifier.py", > line 58, in synced_send > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > self._notify() > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/notifiers/batch_notifier.py", > line 69, in _notify > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > self.callback(batched_events) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/services/segments/plugin.py", > line 211, in _send_notifications > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > event.method(event) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/services/segments/plugin.py", > line 383, in _delete_nova_inventory > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > aggregate_id = self._get_aggregate_id(event.segment_id) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/services/segments/plugin.py", > line 370, in _get_aggregate_id > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > aggregate_uuid = self.p_client.list_aggregates( > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron_lib/placement/client.py", > line 58, in wrapper > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > return f(self, *a, **k) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron_lib/placement/client.py", > line 554, in list_aggregates > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > return self._get(url).json() > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron_lib/placement/client.py", > line 190, in _get > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > return self._client.get(url, endpoint_filter=self._ks_filter, > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", > line 1141, in get > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > return self.request(url, 'GET', **kwargs) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", > line 931, in request > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > resp = send(**kwargs) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File > "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", > line 1038, in _send_request > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > raise exceptions.ConnectFailure(msg) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: > 2023-07-15 11:41:33.144 5688 ERROR oslo.messaging._drivers.impl_rabbit [-] > Connection failed: [Errno 104] ECONNRESET (retrying in 1.0 seconds): > ConnectionResetError: [Errno 104] ECONNRESET > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish > connection to > http://172.29.236.2:8780/resource_providers/dd7e74fb-b494-448e-b16a-712a7ad60e6c/aggregates: > HTTPConnectionPool(host='172.29.236.2', port=8780): Max retries exceeded > with url: > /resource_providers/dd7e74fb-b494-448e-b16a-712a7ad60e6c/aggregates (Caused > by NewConnectionError(' 0x7efd5f36ca30>: Failed to establish a new connection: [Errno 32] EPIPE')) > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: > Removing descriptor: 13 > Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 neutron-api[5689]: > 2023-07-15 11:41:33.542 5689 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: Server > unexpectedly closed connection > Jul 15 11:42:36 dcn2-neutron-server-container-a7122a16 neutron-api[5678]: > 2023-07-15 11:42:36.571 5678 ERROR oslo.messaging._drivers.impl_rabbit [-] > Connection failed: [Errno 104] Connection reset by peer (retrying in 1.0 > seconds): ConnectionResetError: [Errno 104] Connection reset by peer > Jul 15 11:42:40 dcn2-neutron-server-container-a7122a16 neutron-api[5689]: > 2023-07-15 11:42:40.942 5689 ERROR oslo.messaging._drivers.impl_rabbit [-] > Connection failed: [Errno 104] Connection reset by peer (retrying in 1.0 > seconds): ConnectionResetError: [Errno 104] Connection reset by peer > Jul 15 11:42:41 dcn2-neutron-server-container-a7122a16 uwsgi[5683]: Sat > Jul 15 11:42:41 2023 - SIGPIPE: writing to a closed pipe/socket/fd > (probably the client disconnected) on request /v2.0/subnets (ip > 172.29.237.22) !!! > Jul 15 11:42:42 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: > 2023-07-15 11:42:42.053 5688 ERROR oslo.messaging._drivers.impl_rabbit [-] > Connection failed: [Errno 104] Connection reset by peer (retrying in 3.0 > seconds): ConnectionResetError: [Errno 104] Connection reset by peer > Jul 15 11:42:45 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: > 2023-07-15 11:42:45.240 5688 WARNING > neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-0270e149-44c9-49c4-af20-47918e00c4ea f8da6c7abaa64116948db1d2aaa829f1 > 86e3eede43b940d69d2b43ccba1b256b - - default default] Unable to schedule > network 6887bbfb-4680-4ed7-910b-90a65f7731dc: no agents available; will > retry on subsequent port and subnet creation events. > Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:46.563 5682 ERROR neutron.plugins.ml2.managers > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port > caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal > using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', > 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, > 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] > Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:46.564 5682 INFO neutron.plugins.ml2.plugin > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 2 to bind > port caa01668-d5c0-4b37-a185-0ebc458326f7 > Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:46.641 5682 ERROR neutron.plugins.ml2.managers > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port > caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal > using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', > 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, > 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] > Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:46.641 5682 INFO neutron.plugins.ml2.plugin > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 3 to bind > port caa01668-d5c0-4b37-a185-0ebc458326f7 > Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:46.695 5682 ERROR neutron.plugins.ml2.managers > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port > caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal > using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', > 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, > 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] > Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:46.695 5682 INFO neutron.plugins.ml2.plugin > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 4 to bind > port caa01668-d5c0-4b37-a185-0ebc458326f7 > Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:46.743 5682 ERROR neutron.plugins.ml2.managers > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port > caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal > using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', > 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, > 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] > Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:46.743 5682 INFO neutron.plugins.ml2.plugin > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 5 to bind > port caa01668-d5c0-4b37-a185-0ebc458326f7 > Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:46.798 5682 ERROR neutron.plugins.ml2.managers > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port > caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal > using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', > 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, > 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] > Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:46.799 5682 INFO neutron.plugins.ml2.plugin > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 6 to bind > port caa01668-d5c0-4b37-a185-0ebc458326f7 > Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:46.856 5682 ERROR neutron.plugins.ml2.managers > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port > caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal > using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', > 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, > 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] > Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:46.856 5682 INFO neutron.plugins.ml2.plugin > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 7 to bind > port caa01668-d5c0-4b37-a185-0ebc458326f7 > Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:46.904 5682 ERROR neutron.plugins.ml2.managers > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port > caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal > using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', > 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, > 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] > Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:46.904 5682 INFO neutron.plugins.ml2.plugin > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 8 to bind > port caa01668-d5c0-4b37-a185-0ebc458326f7 > Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:46.952 5682 ERROR neutron.plugins.ml2.managers > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port > caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal > using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', > 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, > 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] > Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:46.952 5682 INFO neutron.plugins.ml2.plugin > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 9 to bind > port caa01668-d5c0-4b37-a185-0ebc458326f7 > Jul 15 11:42:47 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:47.000 5682 ERROR neutron.plugins.ml2.managers > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port > caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal > using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', > 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, > 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] > Jul 15 11:42:47 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:47.001 5682 INFO neutron.plugins.ml2.plugin > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 10 to bind > port caa01668-d5c0-4b37-a185-0ebc458326f7 > Jul 15 11:42:47 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:42:47.056 5682 ERROR neutron.plugins.ml2.managers > [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 > req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 > 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port > caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal > using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', > 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, > 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] > Jul 15 11:42:54 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: > 2023-07-15 11:42:54.511 5676 INFO neutron.notifiers.nova [-] Nova event > matching ['req-0ffbf4aa-051f-4707-afe7-2908726310c5'] response: > {'server_uuid': '9f206d9f-6c96-479f-bbaa-116d967129dc', 'name': > 'network-vif-deleted', 'tag': 'caa01668-d5c0-4b37-a185-0ebc458326f7', > 'status': 'completed', 'code': 200} > Jul 15 11:48:24 dcn2-neutron-server-container-a7122a16 neutron-api[5678]: > 2023-07-15 11:48:24.062 5678 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: Server > unexpectedly closed connection > Jul 15 11:48:31 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: > 2023-07-15 11:48:31.762 5676 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: Server > unexpectedly closed connection > Jul 15 11:48:31 dcn2-neutron-server-container-a7122a16 neutron-api[5677]: > 2023-07-15 11:48:31.828 5677 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: [Errno > 104] Connection reset by peer > Jul 15 11:48:31 dcn2-neutron-server-container-a7122a16 uwsgi[5677]: Sat > Jul 15 11:48:31 2023 - SIGPIPE: writing to a closed pipe/socket/fd > (probably the client disconnected) on request /v2.0/subnets (ip > 172.29.237.22) !!! > Jul 15 11:48:32 dcn2-neutron-server-container-a7122a16 neutron-api[5686]: > 2023-07-15 11:48:32.108 5686 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: [Errno > 104] Connection reset by peer > Jul 15 11:48:32 dcn2-neutron-server-container-a7122a16 uwsgi[5686]: Sat > Jul 15 11:48:32 2023 - SIGPIPE: writing to a closed pipe/socket/fd > (probably the client disconnected) on request > /v2.0/networks?limit=21&sort_dir=asc&sort_key=id&router%3Aexternal=True&shared=False > (ip 172.29.237.22) !!! > Jul 15 11:48:34 dcn2-neutron-server-container-a7122a16 neutron-api[5689]: > 2023-07-15 11:48:34.625 5689 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: Server > unexpectedly closed connection > Jul 15 11:48:34 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: > 2023-07-15 11:48:34.641 5688 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: Server > unexpectedly closed connection > Jul 15 11:48:35 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: > 2023-07-15 11:48:35.978 5682 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: Server > unexpectedly closed connection > Jul 15 11:48:36 dcn2-neutron-server-container-a7122a16 neutron-api[5683]: > 2023-07-15 11:48:36.052 5683 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: Server > unexpectedly closed connection > Jul 15 11:48:36 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: > 2023-07-15 11:48:36.809 5680 INFO oslo.messaging._drivers.impl_rabbit [-] A > recoverable connection/channel error occurred, trying to reconnect: [Errno > 104] Connection reset by peer > Jul 15 11:48:36 dcn2-neutron-server-container-a7122a16 uwsgi[5680]: Sat > Jul 15 11:48:36 2023 - SIGPIPE: writing to a closed pipe/socket/fd > (probably the client disconnected) on request /v2.0/subnetpools (ip > 172.29.237.22) !!! > > ############################### > > After the errors appear, the Horizon interface reports the following error: > > Error: Failed to perform requested operation on instance "test", the > instance has an error status: Please try again later [Error: Exceeded > maximum number of retries. Exhausted all hosts available for retrying build > failures for instance 9f206d9f-6c96-479f-bbaa-116d967129dc.]. > > What could be happening? I believe I'm missing something to configure, but > I don't know where to start. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rumman.octanex at gmail.com Sun Jul 16 18:09:19 2023 From: rumman.octanex at gmail.com (A.R. Rumman) Date: Mon, 17 Jul 2023 00:09:19 +0600 Subject: UX Message-ID: Hi, I am a UX Designer. I would love to contribute to OpenStack. So what's the next step or process? Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralonsoh at redhat.com Mon Jul 17 08:39:17 2023 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Mon, 17 Jul 2023 10:39:17 +0200 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: Hello Felix: That's the point: the launchpad bug describes the registers created by the IC configuration but these registers do not have any identification that informs about their relationship with the IC TS. Rather than modifying the Neutron resources (a network is a generic resource that is used by multiple backends), I would try to create a NB-IC resource map and the corresponding NB registers (NAT, LRP and LRSR). If you can link a NB-IC TS with the corresponding NB resources, then the problem is fixed because knowing them we can explicitly skip them from the DB sync tool. If that is not the case, then please present an alternative with the suggested idea of creating an external network type. BTW, I don't understand why you first say that "created by tools not necessarily controlled by us we need to avoid changing them", but then you are talking about refactoring these tools. Are we talking about the tools that create these IC configurations? Can you share them? Regards. On Mon, Jul 17, 2023 at 8:43?AM Felix Huettner wrote: > Hi everyone, > > On Thu, Jul 13, 2023 at 07:29:51PM -0300, Roberto Bartzen Acosta wrote: > > Hi Rodolfo, > > > > Em qui., 13 de jul. de 2023 ?s 13:45, Rodolfo Alonso Hernandez < > > ralonsoh at redhat.com> escreveu: > > > > > Hello Roberto: > > > > > > I think you have provided a very detailed explanation of the issue but > you > > > still didn't present any possible solution. I would ask you at least > for a > > > proposal and for specific details of what is failing in the OVN sync > tool. > > > > > > For example, if you add a TS between two clouds, you'll need to create > (1) > > > routers (NAT registers), (2) router ports (LRP) and (3) some static > routes > > > (LRSR). All these elements are monitored by the DB sync tool and will > fail > > > because the counterparts in the Neutron DB don't exist. I guess that > you > > > have some script or tool or at least a document to manually add these > > > resources; sharing it could help to find a solution. > > > > > > If these elements are manually created during the deployment phase, you > > > can also control the information provided. I'm thinking about the > > > "external_ids" field; we can push some defined constant that will make > > > these registers transparent to the OVN DB sync tool. > > As these resources are also created by tools not necessarily controlled > by us we need to avoid changing them. > I would therefor propose to not add a "ignore" constant to the > "external_ids", to to rather check for the existence of the neutron keys > in "external_ids" (i think thats also what is described in the bug > below). > > > > > > > Please check this idea and if it is feasible. In that case, please add > a > > > topic to the Neutron drivers meeting agenda [1] to discuss it. > > > > > > > That sounds good, thanks. > > https://bugs.launchpad.net/neutron/+bug/2027742 > > From my perspective there is also an additional step after the above > REF. Currently it just makes it possible to use ovn-ic without neutron > killing it. > However if we could get neutron to treat a Transit Switch as a external > network we would skip the need to implement the NAT, LRP and LRSR logic > in custom tooling and could use the existing neutron code. This would > probably require creating a custom type of provider network and then > specifying the transit switch name in the provider options. > > Is that something that from your perspective we should take a look at > right away, or rather first focus und neutron not burning the transit > switch. > > Regards > Felix > > > > > Regards, > > Roberto > > > > > > > > > > Regards. > > > > > > [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers > > > > > > > > > On Wed, Jul 12, 2023 at 2:12?PM Roberto Bartzen Acosta < > > > roberto.acosta at luizalabs.com> wrote: > > > > > >> > > >> > > >> Em qua., 12 de jul. de 2023 ?s 09:05, Roberto Bartzen Acosta < > > >> roberto.acosta at luizalabs.com> escreveu: > > >> > > >>> Hi Rodolfo, > > >>> > > >>> Thanks for the feedback. > > >>> > > >>> I liked the idea of having a different network type only to hold the > > >>> Transit Switches and thus the LRP but it depends on some factors like > > >>> multi-tenancy. > > >>> From the OVN perspective, the TS is global and will only be linked > to a > > >>> tenant when it is plugged into the tenant router port. My concern > here is > > >>> that if Neutron manages the TS, it will need to assume the dynamic > > >>> characteristics of this TS function, ovn-ic creates and removes the > TS in > > >>> NB_Global, in addition to plugging and removing ports in this switch > > >>> according to the network topology. Additionally, Neutron would still > need > > >>> to handle the learned remote routes (which are listed as static > routes from > > >>> the tenant router perspective). > > >>> > > >> sorry: NB_Global -> Northbound Database. > > >> > > >> > > >>> This is an open discussion, Felix can add ideas here. > > >>> > > >>> In general, it seems to me that we have no alternatives for this > type of > > >>> solution other than OVN-IC (note that ovn-bgp-agent does not learn > remote > > >>> routes at the SDN level / OVN). > > >>> OpenStack seems to be designed to run like a "bubble" and this > traffic > > >>> from one VM to another always needs to be routed at the FIP level and > > >>> external routing layers. > > >>> > > >>> Regards, > > >>> Roberto > > >>> > > >>> Em qua., 12 de jul. de 2023 ?s 05:11, Rodolfo Alonso Hernandez < > > >>> ralonsoh at redhat.com> escreveu: > > >>> > > >>>> Hello Roberto: > > >>>> > > >>>> We talked about this possible RFE during the PTG [1] but I don't > > >>>> remember having any proposal. Actually what I remember (and was > written in > > >>>> the etherpad) is that you were going to present an RFE. Can you > explain it > > >>>> briefly? > > >>>> > > >>>> We also talked about the idea, proposed by Felix in the mailing > list, > > >>>> of having a different network type only to hold the Transit > Switches and > > >>>> thus the LRP. If you have a proposal, please present it. > > >>>> > > >>>> Regards. > > >>>> > > >>>> [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506 > > >>>> > > >>>> On Tue, Jul 11, 2023 at 10:50?PM Roberto Bartzen Acosta < > > >>>> roberto.acosta at luizalabs.com> wrote: > > >>>> > > >>>>> Hello Neutron folks, > > >>>>> > > >>>>> Regarding the conversation started in March [1] about the use of > OVN > > >>>>> interconnect with Neutron, I would like to evolve the discussion in > > >>>>> relation to resources allocated by OVN-IC and which are not > managed by > > >>>>> Neutron. In the March PTG [2], the problem related to the db_sync > tool was > > >>>>> presented, and a proposed solution in which Neutron does not > manage these > > >>>>> resources. > > >>>>> > > >>>>> After discussing some architectural designs with Felix/StackIT, it > > >>>>> seems to make sense to come up with a proposal to change the > mech_driver to > > >>>>> validate the external_ids key and not remove resources present in > the OVN > > >>>>> backend without Neutron "signature". > > >>>>> > > >>>>> The discussion here is more complex than it seems, because > > >>>>> architecturally Neutron does not allow us to interconnect > workloads in > > >>>>> multiple AZs (with different OpenStacks), but this could be a > requirement > > >>>>> for a high availability cloud solution (Cloud Region). > Additionally, this > > >>>>> OVN-IC solution allows interconnecting other cloud backends that > use OVN, > > >>>>> in the case of kubernetes with ovn-kube. > > >>>>> > > >>>>> We tested an OVN interconnect integrated with 3 OpenStack > > >>>>> installations and it worked very well !!! considering direct L3 > traffic at > > >>>>> the router level between different network infrastructures. > > >>>>> > > >>>>> SYNC_REPAIR - problem > > >>>>> > > >>>>> * Static Routes (learned OVN-IC routes) > > >>>>> * Router Port -> Transit Switches > > >>>>> > > >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae > > >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING > > >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync > > >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port > found in > > >>>>> OVN but not in Neutron, port_id=rt2-admin-tenant1 > > >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae > > >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING > > >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync > > >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router > > >>>>> 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes > [{'destination': ' > > >>>>> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': ' > > >>>>> 10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in > > >>>>> Neutron > > >>>>> > > >>>>> > > >>>>> Any suggestions on how to resolve this db_sync issue? since all > other > > >>>>> Neutron modules did not present any problem with OVN-IC (as far as > I > > >>>>> tested). > > >>>>> I remember Terry was keen to suggest some things to help. I believe > > >>>>> that before proposing some complex mechanism to solve this simple > problem, > > >>>>> I would like to hear the community opinion. In our use case, a bit > change > > >>>>> in db_sync with filter by neutron key in external_ids would be > enough. > > >>>>> > > >>>>> Regards, > > >>>>> Roberto > > >>>>> > > >>>>> > > >>>>> > > >>>>> [1] > > >>>>> > https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html > > >>>>> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg > > >>>>> > > >>>>> > > >>>>> > > >>>>> Additional logs: > > >>>>> > > >>>>> OpenStack 1 > > >>>>> > > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# > > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# > > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl > > >>>>> lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 > > >>>>> IPv4 Routes > > >>>>> Route Table
: > > >>>>> 20.0.1.0/24 169.254.200.2 dst-ip > (learned) > > >>>>> 20.0.2.0/24 169.254.200.3 dst-ip > (learned) > > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip > > >>>>> > > >>>>> IPv6 Routes > > >>>>> Route Table
: > > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip > > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl > > >>>>> lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 > > >>>>> IPv4 Routes > > >>>>> Route Table
: > > >>>>> 10.0.1.0/24 169.254.100.2 dst-ip > (learned) > > >>>>> 10.0.2.0/24 169.254.100.3 dst-ip > (learned) > > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip > > >>>>> > > >>>>> IPv6 Routes > > >>>>> Route Table
: > > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip > > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# > > >>>>> > > >>>>> > > >>>>> OpenStack 2 > > >>>>> > > >>>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl > > >>>>> lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc > > >>>>> IPv4 Routes > > >>>>> Route Table
: > > >>>>> 20.0.0.0/24 169.254.200.1 dst-ip > (learned) > > >>>>> 20.0.2.0/24 169.254.200.3 dst-ip > (learned) > > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip > > >>>>> > > >>>>> IPv6 Routes > > >>>>> Route Table
: > > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip > > >>>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl > > >>>>> lr-route-list ce45f681-6454-43fe-974f-81344bb8113a > > >>>>> IPv4 Routes > > >>>>> Route Table
: > > >>>>> 10.0.0.0/24 169.254.100.1 dst-ip > (learned) > > >>>>> 10.0.2.0/24 169.254.100.3 dst-ip > (learned) > > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip > > >>>>> > > >>>>> IPv6 Routes > > >>>>> Route Table
: > > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip > > >>>>> > > >>>>> > > >>>>> > > >>>>> OpenStack 3 > > >>>>> > > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# > > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl > > >>>>> lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 > > >>>>> IPv4 Routes > > >>>>> Route Table
: > > >>>>> 20.0.0.0/24 169.254.200.1 dst-ip > (learned) > > >>>>> 20.0.1.0/24 169.254.200.2 dst-ip > (learned) > > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip > > >>>>> > > >>>>> IPv6 Routes > > >>>>> Route Table
: > > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip > > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl > > >>>>> lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 > > >>>>> IPv4 Routes > > >>>>> Route Table
: > > >>>>> 10.0.0.0/24 169.254.100.1 dst-ip > (learned) > > >>>>> 10.0.1.0/24 169.254.100.2 dst-ip > (learned) > > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip > > >>>>> > > >>>>> IPv6 Routes > > >>>>> Route Table
: > > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip > > >>>>> > > >>>>> > > >>>>> OVN-IC Global database > > >>>>> > > >>>>> root at ovn-global-db1:~# ovn-ic-sbctl show > > >>>>> availability-zone osp1 > > >>>>> gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 > > >>>>> hostname: osp1-gwnode1 > > >>>>> type: geneve > > >>>>> ip: 192.168.200.28 > > >>>>> port admin-rt1-tenant1 > > >>>>> transit switch: admin-tenant1 > > >>>>> address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 > fe80::1/64"] > > >>>>> port admin-rt1-tenant1_1 > > >>>>> transit switch: admin-tenant1_1 > > >>>>> address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] > > >>>>> availability-zone osp2 > > >>>>> gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 > > >>>>> hostname: osp2-gwnode1 > > >>>>> type: geneve > > >>>>> ip: 192.168.200.128 > > >>>>> port admin-rt2-tenant1 > > >>>>> transit switch: admin-tenant1 > > >>>>> address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 > fe80::2/64"] > > >>>>> port admin-rt2-tenant1_1 > > >>>>> transit switch: admin-tenant1_1 > > >>>>> address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] > > >>>>> availability-zone osp3 > > >>>>> gateway 97595af9-7896-40d0-a883-beadbff1aa5b > > >>>>> hostname: osp3-gwnode1 > > >>>>> type: geneve > > >>>>> ip: 192.168.200.228 > > >>>>> port admin-rt3-tenant1 > > >>>>> transit switch: admin-tenant1 > > >>>>> address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 > fe80::3/64"] > > >>>>> port admin-rt3-tenant1_1 > > >>>>> transit switch: admin-tenant1_1 > > >>>>> address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> *?Esta mensagem ? direcionada apenas para os endere?os constantes > no > > >>>>> cabe?alho inicial. Se voc? n?o est? listado nos endere?os > constantes no > > >>>>> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do > dessa > > >>>>> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es > citadas est?o > > >>>>> imediatamente anuladas e proibidas?.* > > >>>>> > > >>>>> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis > para > > >>>>> assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa > n?o > > >>>>> poder? aceitar a responsabilidade por quaisquer perdas ou danos > causados > > >>>>> por esse e-mail ou por seus anexos?.* > > >>>>> > > >>>> > > >> > > >> *?Esta mensagem ? direcionada apenas para os endere?os constantes no > > >> cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes > no > > >> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa > > >> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas > est?o > > >> imediatamente anuladas e proibidas?.* > > >> > > >> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para > > >> assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o > > >> poder? aceitar a responsabilidade por quaisquer perdas ou danos > causados > > >> por esse e-mail ou por seus anexos?.* > > >> > > > > > > > -- > > > > > > > > > > _?Esta mensagem ? direcionada apenas para os endere?os constantes no > > cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no > > cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa > > mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas > est?o > > imediatamente anuladas e proibidas?._ > > > > > > * **?Apesar do Magazine Luiza tomar > > todas as precau??es razo?veis para assegurar que nenhum v?rus esteja > > presente nesse e-mail, a empresa n?o poder? aceitar a responsabilidade > por > > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos?.* > > > > > > > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r > die Verwertung durch den vorgesehenen Empf?nger bestimmt. > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender > bitte unverz?glich in Kenntnis und l?schen diese E Mail. > > Hinweise zum Datenschutz finden Sie hier. > > > This e-mail may contain confidential content and is intended only for the > specified recipient/s. > If you are not the intended recipient, please inform the sender > immediately and delete this e-mail. > > Information on data protection can be found here< > https://www.datenschutz.schwarz>. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.acosta at luizalabs.com Mon Jul 17 13:10:41 2023 From: roberto.acosta at luizalabs.com (Roberto Bartzen Acosta) Date: Mon, 17 Jul 2023 10:10:41 -0300 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: Hi Felix and Rodolfo, Em seg., 17 de jul. de 2023 ?s 05:39, Rodolfo Alonso Hernandez < ralonsoh at redhat.com> escreveu: > Hello Felix: > > That's the point: the launchpad bug describes the registers created by the > IC configuration but these registers do not have any identification that > informs about their relationship with the IC TS. > > Rather than modifying the Neutron resources (a network is a generic > resource that is used by multiple backends), I would try to create a NB-IC > resource map and the corresponding NB registers (NAT, LRP and LRSR). If you > can link a NB-IC TS with the corresponding NB resources, then the problem > is fixed because knowing them we can explicitly skip them from the DB sync > tool. > > If that is not the case, then please present an alternative with the > suggested idea of creating an external network type. > > BTW, I don't understand why you first say that "created by tools not > necessarily controlled by us we need to avoid changing them", but then you > are talking about refactoring these tools. Are we talking about the tools > that create these IC configurations? Can you share them? > > I think we are talking about two different cases. 1 - This proposed RFE intends to solve the problem of removing external resources and that is enough for the OVN-IC to work in generic scenarios: multi tenancy, single external provider, multiple external providers, etc. because the OVN-IC will not relate to Neutron's network topology or capabilities. 2- Felix' suggestion is related to integrating the TS to a network on Neutron (some kind of mapping), and in this case, we have a hybrid solution with the TS managed by an external tool, and the management of the NAT, LRP / TS link with the tenant being done directly on Neutron. For this proposal, I believe that the port filter would not be necessary because Neutron would know the resource mappings in the OVN-NB, but it seems to me a more complex solution. The question about case 2 is related to which level we are talking about modifying the generic network to map the TS. If it is a network provider level, the discussion is one - this case would solve the schedule/lrp-set-gateway-chassis problem as it would be a gateway port. but if it is a tenant' network level linked to a router / no gateway port (the discussion would be another) - does not solve the gateway port problem, but it seems closer to what the OVN-IC proposes to do at the network L3 interconnect level). BTW: in case the TS is connected to an external network, we would need an additional router for external/internet connectivity not related to the OVN-IC, or maybe create a router with multiple gateway ports (that doesn't exist yet)... Regards, Roberto > Regards. > > On Mon, Jul 17, 2023 at 8:43?AM Felix Huettner > wrote: > >> Hi everyone, >> >> On Thu, Jul 13, 2023 at 07:29:51PM -0300, Roberto Bartzen Acosta wrote: >> > Hi Rodolfo, >> > >> > Em qui., 13 de jul. de 2023 ?s 13:45, Rodolfo Alonso Hernandez < >> > ralonsoh at redhat.com> escreveu: >> > >> > > Hello Roberto: >> > > >> > > I think you have provided a very detailed explanation of the issue >> but you >> > > still didn't present any possible solution. I would ask you at least >> for a >> > > proposal and for specific details of what is failing in the OVN sync >> tool. >> > > >> > > For example, if you add a TS between two clouds, you'll need to >> create (1) >> > > routers (NAT registers), (2) router ports (LRP) and (3) some static >> routes >> > > (LRSR). All these elements are monitored by the DB sync tool and will >> fail >> > > because the counterparts in the Neutron DB don't exist. I guess that >> you >> > > have some script or tool or at least a document to manually add these >> > > resources; sharing it could help to find a solution. >> > > >> > > If these elements are manually created during the deployment phase, >> you >> > > can also control the information provided. I'm thinking about the >> > > "external_ids" field; we can push some defined constant that will make >> > > these registers transparent to the OVN DB sync tool. >> >> As these resources are also created by tools not necessarily controlled >> by us we need to avoid changing them. >> I would therefor propose to not add a "ignore" constant to the >> "external_ids", to to rather check for the existence of the neutron keys >> in "external_ids" (i think thats also what is described in the bug >> below). >> >> Thanks for the contribution Felix. The idea is not to add (or at least not expect it to be externally added) any identification in the external_ids, Neutron should only filter for DB consistency validation what it knows and has created by Neuron -> Neutron in the external ids key. > > > >> > > Please check this idea and if it is feasible. In that case, please >> add a >> > > topic to the Neutron drivers meeting agenda [1] to discuss it. >> > > >> > >> > That sounds good, thanks. >> > https://bugs.launchpad.net/neutron/+bug/2027742 >> >> From my perspective there is also an additional step after the above >> REF. Currently it just makes it possible to use ovn-ic without neutron >> killing it. >> However if we could get neutron to treat a Transit Switch as a external >> network we would skip the need to implement the NAT, LRP and LRSR logic >> in custom tooling and could use the existing neutron code. This would >> probably require creating a custom type of provider network and then >> specifying the transit switch name in the provider options. >> >> Is that something that from your perspective we should take a look at >> right away, or rather first focus und neutron not burning the transit >> switch. >> >> Regards >> Felix >> >> > >> > Regards, >> > Roberto >> > >> > >> > > >> > > Regards. >> > > >> > > [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers >> > > >> > > >> > > On Wed, Jul 12, 2023 at 2:12?PM Roberto Bartzen Acosta < >> > > roberto.acosta at luizalabs.com> wrote: >> > > >> > >> >> > >> >> > >> Em qua., 12 de jul. de 2023 ?s 09:05, Roberto Bartzen Acosta < >> > >> roberto.acosta at luizalabs.com> escreveu: >> > >> >> > >>> Hi Rodolfo, >> > >>> >> > >>> Thanks for the feedback. >> > >>> >> > >>> I liked the idea of having a different network type only to hold the >> > >>> Transit Switches and thus the LRP but it depends on some factors >> like >> > >>> multi-tenancy. >> > >>> From the OVN perspective, the TS is global and will only be linked >> to a >> > >>> tenant when it is plugged into the tenant router port. My concern >> here is >> > >>> that if Neutron manages the TS, it will need to assume the dynamic >> > >>> characteristics of this TS function, ovn-ic creates and removes the >> TS in >> > >>> NB_Global, in addition to plugging and removing ports in this switch >> > >>> according to the network topology. Additionally, Neutron would >> still need >> > >>> to handle the learned remote routes (which are listed as static >> routes from >> > >>> the tenant router perspective). >> > >>> >> > >> sorry: NB_Global -> Northbound Database. >> > >> >> > >> >> > >>> This is an open discussion, Felix can add ideas here. >> > >>> >> > >>> In general, it seems to me that we have no alternatives for this >> type of >> > >>> solution other than OVN-IC (note that ovn-bgp-agent does not learn >> remote >> > >>> routes at the SDN level / OVN). >> > >>> OpenStack seems to be designed to run like a "bubble" and this >> traffic >> > >>> from one VM to another always needs to be routed at the FIP level >> and >> > >>> external routing layers. >> > >>> >> > >>> Regards, >> > >>> Roberto >> > >>> >> > >>> Em qua., 12 de jul. de 2023 ?s 05:11, Rodolfo Alonso Hernandez < >> > >>> ralonsoh at redhat.com> escreveu: >> > >>> >> > >>>> Hello Roberto: >> > >>>> >> > >>>> We talked about this possible RFE during the PTG [1] but I don't >> > >>>> remember having any proposal. Actually what I remember (and was >> written in >> > >>>> the etherpad) is that you were going to present an RFE. Can you >> explain it >> > >>>> briefly? >> > >>>> >> > >>>> We also talked about the idea, proposed by Felix in the mailing >> list, >> > >>>> of having a different network type only to hold the Transit >> Switches and >> > >>>> thus the LRP. If you have a proposal, please present it. >> > >>>> >> > >>>> Regards. >> > >>>> >> > >>>> [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506 >> > >>>> >> > >>>> On Tue, Jul 11, 2023 at 10:50?PM Roberto Bartzen Acosta < >> > >>>> roberto.acosta at luizalabs.com> wrote: >> > >>>> >> > >>>>> Hello Neutron folks, >> > >>>>> >> > >>>>> Regarding the conversation started in March [1] about the use of >> OVN >> > >>>>> interconnect with Neutron, I would like to evolve the discussion >> in >> > >>>>> relation to resources allocated by OVN-IC and which are not >> managed by >> > >>>>> Neutron. In the March PTG [2], the problem related to the db_sync >> tool was >> > >>>>> presented, and a proposed solution in which Neutron does not >> manage these >> > >>>>> resources. >> > >>>>> >> > >>>>> After discussing some architectural designs with Felix/StackIT, it >> > >>>>> seems to make sense to come up with a proposal to change the >> mech_driver to >> > >>>>> validate the external_ids key and not remove resources present in >> the OVN >> > >>>>> backend without Neutron "signature". >> > >>>>> >> > >>>>> The discussion here is more complex than it seems, because >> > >>>>> architecturally Neutron does not allow us to interconnect >> workloads in >> > >>>>> multiple AZs (with different OpenStacks), but this could be a >> requirement >> > >>>>> for a high availability cloud solution (Cloud Region). >> Additionally, this >> > >>>>> OVN-IC solution allows interconnecting other cloud backends that >> use OVN, >> > >>>>> in the case of kubernetes with ovn-kube. >> > >>>>> >> > >>>>> We tested an OVN interconnect integrated with 3 OpenStack >> > >>>>> installations and it worked very well !!! considering direct L3 >> traffic at >> > >>>>> the router level between different network infrastructures. >> > >>>>> >> > >>>>> SYNC_REPAIR - problem >> > >>>>> >> > >>>>> * Static Routes (learned OVN-IC routes) >> > >>>>> * Router Port -> Transit Switches >> > >>>>> >> > >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >> > >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >> > >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >> > >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port >> found in >> > >>>>> OVN but not in Neutron, port_id=rt2-admin-tenant1 >> > >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >> > >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >> > >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >> > >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router >> > >>>>> 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes >> [{'destination': ' >> > >>>>> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': ' >> > >>>>> 10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not >> in >> > >>>>> Neutron >> > >>>>> >> > >>>>> >> > >>>>> Any suggestions on how to resolve this db_sync issue? since all >> other >> > >>>>> Neutron modules did not present any problem with OVN-IC (as far >> as I >> > >>>>> tested). >> > >>>>> I remember Terry was keen to suggest some things to help. I >> believe >> > >>>>> that before proposing some complex mechanism to solve this simple >> problem, >> > >>>>> I would like to hear the community opinion. In our use case, a >> bit change >> > >>>>> in db_sync with filter by neutron key in external_ids would be >> enough. >> > >>>>> >> > >>>>> Regards, >> > >>>>> Roberto >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> [1] >> > >>>>> >> https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html >> > >>>>> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> Additional logs: >> > >>>>> >> > >>>>> OpenStack 1 >> > >>>>> >> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >> ovn-nbctl >> > >>>>> lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 >> > >>>>> IPv4 Routes >> > >>>>> Route Table
: >> > >>>>> 20.0.1.0/24 169.254.200.2 dst-ip >> (learned) >> > >>>>> 20.0.2.0/24 169.254.200.3 dst-ip >> (learned) >> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >> > >>>>> >> > >>>>> IPv6 Routes >> > >>>>> Route Table
: >> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >> ovn-nbctl >> > >>>>> lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 >> > >>>>> IPv4 Routes >> > >>>>> Route Table
: >> > >>>>> 10.0.1.0/24 169.254.100.2 dst-ip >> (learned) >> > >>>>> 10.0.2.0/24 169.254.100.3 dst-ip >> (learned) >> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >> > >>>>> >> > >>>>> IPv6 Routes >> > >>>>> Route Table
: >> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >> > >>>>> >> > >>>>> >> > >>>>> OpenStack 2 >> > >>>>> >> > >>>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# >> ovn-nbctl >> > >>>>> lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc >> > >>>>> IPv4 Routes >> > >>>>> Route Table
: >> > >>>>> 20.0.0.0/24 169.254.200.1 dst-ip >> (learned) >> > >>>>> 20.0.2.0/24 169.254.200.3 dst-ip >> (learned) >> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >> > >>>>> >> > >>>>> IPv6 Routes >> > >>>>> Route Table
: >> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> > >>>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# >> ovn-nbctl >> > >>>>> lr-route-list ce45f681-6454-43fe-974f-81344bb8113a >> > >>>>> IPv4 Routes >> > >>>>> Route Table
: >> > >>>>> 10.0.0.0/24 169.254.100.1 dst-ip >> (learned) >> > >>>>> 10.0.2.0/24 169.254.100.3 dst-ip >> (learned) >> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >> > >>>>> >> > >>>>> IPv6 Routes >> > >>>>> Route Table
: >> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> OpenStack 3 >> > >>>>> >> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >> ovn-nbctl >> > >>>>> lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 >> > >>>>> IPv4 Routes >> > >>>>> Route Table
: >> > >>>>> 20.0.0.0/24 169.254.200.1 dst-ip >> (learned) >> > >>>>> 20.0.1.0/24 169.254.200.2 dst-ip >> (learned) >> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >> > >>>>> >> > >>>>> IPv6 Routes >> > >>>>> Route Table
: >> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >> ovn-nbctl >> > >>>>> lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 >> > >>>>> IPv4 Routes >> > >>>>> Route Table
: >> > >>>>> 10.0.0.0/24 169.254.100.1 dst-ip >> (learned) >> > >>>>> 10.0.1.0/24 169.254.100.2 dst-ip >> (learned) >> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >> > >>>>> >> > >>>>> IPv6 Routes >> > >>>>> Route Table
: >> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >> > >>>>> >> > >>>>> >> > >>>>> OVN-IC Global database >> > >>>>> >> > >>>>> root at ovn-global-db1:~# ovn-ic-sbctl show >> > >>>>> availability-zone osp1 >> > >>>>> gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 >> > >>>>> hostname: osp1-gwnode1 >> > >>>>> type: geneve >> > >>>>> ip: 192.168.200.28 >> > >>>>> port admin-rt1-tenant1 >> > >>>>> transit switch: admin-tenant1 >> > >>>>> address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 >> fe80::1/64"] >> > >>>>> port admin-rt1-tenant1_1 >> > >>>>> transit switch: admin-tenant1_1 >> > >>>>> address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] >> > >>>>> availability-zone osp2 >> > >>>>> gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 >> > >>>>> hostname: osp2-gwnode1 >> > >>>>> type: geneve >> > >>>>> ip: 192.168.200.128 >> > >>>>> port admin-rt2-tenant1 >> > >>>>> transit switch: admin-tenant1 >> > >>>>> address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 >> fe80::2/64"] >> > >>>>> port admin-rt2-tenant1_1 >> > >>>>> transit switch: admin-tenant1_1 >> > >>>>> address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] >> > >>>>> availability-zone osp3 >> > >>>>> gateway 97595af9-7896-40d0-a883-beadbff1aa5b >> > >>>>> hostname: osp3-gwnode1 >> > >>>>> type: geneve >> > >>>>> ip: 192.168.200.228 >> > >>>>> port admin-rt3-tenant1 >> > >>>>> transit switch: admin-tenant1 >> > >>>>> address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 >> fe80::3/64"] >> > >>>>> port admin-rt3-tenant1_1 >> > >>>>> transit switch: admin-tenant1_1 >> > >>>>> address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> *?Esta mensagem ? direcionada apenas para os endere?os constantes >> no >> > >>>>> cabe?alho inicial. Se voc? n?o est? listado nos endere?os >> constantes no >> > >>>>> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do >> dessa >> > >>>>> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es >> citadas est?o >> > >>>>> imediatamente anuladas e proibidas?.* >> > >>>>> >> > >>>>> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis >> para >> > >>>>> assegurar que nenhum v?rus esteja presente nesse e-mail, a >> empresa n?o >> > >>>>> poder? aceitar a responsabilidade por quaisquer perdas ou danos >> causados >> > >>>>> por esse e-mail ou por seus anexos?.* >> > >>>>> >> > >>>> >> > >> >> > >> *?Esta mensagem ? direcionada apenas para os endere?os constantes no >> > >> cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes >> no >> > >> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do >> dessa >> > >> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es >> citadas est?o >> > >> imediatamente anuladas e proibidas?.* >> > >> >> > >> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para >> > >> assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa >> n?o >> > >> poder? aceitar a responsabilidade por quaisquer perdas ou danos >> causados >> > >> por esse e-mail ou por seus anexos?.* >> > >> >> > > >> > >> > -- >> > >> > >> > >> > >> > _?Esta mensagem ? direcionada apenas para os endere?os constantes no >> > cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no >> > cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa >> > mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas >> est?o >> > imediatamente anuladas e proibidas?._ >> > >> > >> > * **?Apesar do Magazine Luiza tomar >> > todas as precau??es razo?veis para assegurar que nenhum v?rus esteja >> > presente nesse e-mail, a empresa n?o poder? aceitar a responsabilidade >> por >> > quaisquer perdas ou danos causados por esse e-mail ou por seus anexos?.* >> > >> > >> > >> Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r >> die Verwertung durch den vorgesehenen Empf?nger bestimmt. >> Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender >> bitte unverz?glich in Kenntnis und l?schen diese E Mail. >> >> Hinweise zum Datenschutz finden Sie hier> >. >> >> >> This e-mail may contain confidential content and is intended only for the >> specified recipient/s. >> If you are not the intended recipient, please inform the sender >> immediately and delete this e-mail. >> >> Information on data protection can be found here< >> https://www.datenschutz.schwarz>. >> >> -- _?Esta mensagem ? direcionada apenas para os endere?os constantes no cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o imediatamente anuladas e proibidas?._ *?**?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o poder? aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos?.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristin at openinfra.dev Mon Jul 17 15:20:08 2023 From: kristin at openinfra.dev (Kristin Barrientos) Date: Mon, 17 Jul 2023 10:20:08 -0500 Subject: OpenStack "C" Release Voting Reminder Message-ID: <8B93A940-A597-4226-8F93-3515D6E1DCA5@openinfra.dev> Hi everyone, Just a reminder to get your votes in for the OpenStack ?C? Release. Voting closes Tuesday, July 18! Vote here: https://civs1.civs.us/cgi-bin/vote.pl?id=E_852cf3e53467d72e&akey=1504cb6ad9d24667 Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation -------------- next part -------------- An HTML attachment was scrubbed... URL: From michal.arbet at ultimum.io Mon Jul 17 15:21:22 2023 From: michal.arbet at ultimum.io (Michal Arbet) Date: Mon, 17 Jul 2023 17:21:22 +0200 Subject: UX In-Reply-To: References: Message-ID: Hi, Just start to contribute :) https://wiki.openstack.org/wiki/How_To_Contribute Michal Arbet Openstack Engineer Ultimum Technologies a.s. Na Po???? 1047/26, 11000 Praha 1 Czech Republic +420 604 228 897 michal.arbet at ultimum.io *https://ultimum.io * LinkedIn | Twitter | Facebook po 17. 7. 2023 v 15:49 odes?latel A.R. Rumman napsal: > Hi, > I am a UX Designer. I would love to contribute to OpenStack. So what's the > next step or process? > > Thank you. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnsomor at gmail.com Mon Jul 17 16:17:00 2023 From: johnsomor at gmail.com (Michael Johnson) Date: Mon, 17 Jul 2023 09:17:00 -0700 Subject: [designate] About delegating PTR using shared zones on a shared internal network In-Reply-To: References: <5059613b-2c76-c5a5-1281-42970a190def@debian.org> Message-ID: Yes, I have fixed and backported the bug I mentioned at the summit. Here is the list of patches: https://review.opendev.org/c/openstack/designate/+/887425 https://review.opendev.org/c/openstack/designate/+/884554 https://review.opendev.org/c/openstack/designate/+/884347 https://review.opendev.org/c/openstack/designate/+/880691 https://review.opendev.org/c/openstack/designate/+/879208 https://review.opendev.org/c/openstack/designate/+/726334 Michael On Fri, Jul 14, 2023 at 2:31?AM Thomas Goirand wrote: > > On 7/10/23 23:03, Michael Johnson wrote: > > Ideally you would allocate blocks of IPs to a project, but I realize > > many deployments are not set up that way. > > You are right, that's not the way it works in our deployment. As we > implemented a public cloud, and have thousands of customers, it's not > doable to manually assign a block of IPs to a customer with an admin > command. > > What we have is a network that we called "ext-net1". It is an internal > OpenStack network (with a router and its default gateway owned by the > admin user), on which we added multiple /24 subnet blocks of public IPs. > > Typically, a user would create a port on this "ext-net1" and use it when > doing "openstack server create" to optain a (random) public IP on that > network. Or, using "openstack server create --nic net-id=ext-net1". In > this scenario, what we want is automatically assigning the PTR subzone > matching the individual IP address to be: > 1/ created (as admin) > 2/ shared with the customers project (openstack zone share create), so > the customer can add a PTR record > 3/ if possible, also automatically create a PTR record on the behalf of > the customer, so there's a default value in place. > > When the port is deleted, we would like the subzone to be deleted > automatically as well. > > > The other thing to think about is if you use one of the neutron > > extensions that create records based on the network domain and port > > dns name. > > We are using scenario 3a, so customers can do whatever they want with > the IP address they own. > > > You could get into an interesting situation where the > > configuration in neutron no longer matches the PTR records if you > > allow users to "own" their PTR records in Designte. > > Well, if the port name and domain doesn't match the PTR record, it may > look ugly, but I don't think that's a real problem. I couldn't care > less, if at least it's doing what we expect (ie: let our customers > create forward and reverse entries, in a way that can be shared with > other customers without any conflict). > > As we're still in Victoria (currently planning the upgrade to Zed), we > wont be able to contribute what I described above in a foreseeable > future. The situation *may* change at the end of this year though, if we > successfully upgrade to Zed, and if I can backport the shared zone patches. > > BTW, it would be great if, like we discuss in the summit, you could send > me a list of patch, in the correct order, so I could backport the shared > zones feature to Zed. Of course only do that if it doesn't take too much > of your time... Also, did you backport the bugfix you told me was > missing to Antelope? > > Thanks for all of your work, > Cheers, > > Thomas Goirand (zigo) > From corey.bryant at canonical.com Mon Jul 17 16:19:27 2023 From: corey.bryant at canonical.com (Corey Bryant) Date: Mon, 17 Jul 2023 12:19:27 -0400 Subject: CVE-2023-2088 for Ubuntu Openstack Message-ID: Bug LP:#2004555 (CVE-2023-2088) identified that ?an unauthorised access to a volume could occur when an iSCSI or FC connection from a host is severed due to a volume being unmapped on the storage system and the device is later reused for another volume on the same host?. While the bug affects iSCSI and FC, the fix introduces a breaking change for configurations upgrading to the fixed packages if they do not have service token support enabled. The new code introduces mandatory use of service tokens to identify requests coming from a trusted control plane service. Service tokens are issued by Keystone for service users. For example, a token can be issued to the Nova service user. They have a particular trust level, defined by the service, that allows them to perform trusted actions. In the case of Nova, the action is detaching volumes from instances. Versions of Ubuntu OpenStack for Yoga and later will have new packages versions released with CVE patches applied that require the use of service tokens to enable the detaching and deletion of volumes on instances. These packages will be released on July 24, 2023. Versions of Ubuntu OpenStack for Xena and earlier will not have package updates released and will still be vulnerable to the ?unauthorised detach of volumes? issue until a mitigation policy is applied. The mitigation policy provided requires the use of service tokens. = Patched packages = The following package versions include patches for CVE-2023-2088. These package versions deliver services that require the use of service tokens. Package versions prior to those listed will not be patched for CVE-2023-2088 and will therefore require policy updates for mitigation. Ubuntu 23.04 (Lunar) * nova >= 3:27.0.0-0ubuntu1.3 * cinder >= 2:22.0.0-0ubuntu1.3 * python-os-brick >= 6.2.0-0ubuntu2.3 * ironic >= 1:21.4.0-0ubuntu1.1 * python-glance-store >= 4.3.0-0ubuntu1.3 Ubuntu Cloud Archive Antelope * nova >= 3:27.0.0-0ubuntu1.3~cloud0 * cinder >= 2:22.0.0-0ubuntu1.3~cloud0 * python-os-brick >= 6.2.0-0ubuntu2.3~cloud0 * ironic >= 1:21.4.0-0ubuntu1.1~cloud0 * python-glance-store >= 4.3.0-0ubuntu1.3~cloud0 Ubuntu 22.10 (Kinetic) * nova >= 3:26.1.1-0ubuntu1.1 * cinder >= 2:21.2.0-0ubuntu1.1 * python-os-brick >= os-brick - 6.1.0-0ubuntu1.3 * ironic >= 1:21.1.0-0ubuntu1.1 * python-glance-store >= 4.1.0-0ubuntu1.3 Ubuntu Cloud Archive Zed * nova >= 3:26.1.1-0ubuntu1.1~cloud0 * cinder >= 2:21.2.0-0ubuntu1.1~cloud0 * python-os-brick >= os-brick - 6.1.0-0ubuntu1.3~cloud0 * ironic >= 1:21.1.0-0ubuntu1.1~cloud0 * python-glance-store >= 4.1.0-0ubuntu1.3~cloud0 Ubuntu 22.04 (Jammy) * nova >= 3:25.1.1-0ubuntu1.1 * cinder >= 2:20.2.0-0ubuntu1.1 * python-os-brick >= 5.2.2-0ubuntu1.2 * ironic >= 1:20.1.0-0ubuntu1.1 * python-glance-store >= 3.0.0-0ubuntu1.3 Ubuntu Cloud Archive Yoga * nova >= 3:25.1.1-0ubuntu1.1~cloud0 * cinder >= 2:20.2.0-0ubuntu1.1~cloud0 * python-os-brick >= 5.2.2-0ubuntu1.2~cloud0 * ironic >= 1:20.1.0-0ubuntu1.1~cloud0 * python-glance-store >= 3.0.0-0ubuntu1.3~cloud0 = Using a custom Cinder policy as a mitigation = It?s possible to restrict the access to the operations affected by this CVE by providing a custom Cinder policy file that will prevent users from deleting attachments, detach and force detach. In order to use this mitigation, service token support must be enabled. The exact method for customising this depends on the deployment tooling being used. The following steps can be used as a guildeline: 1) Identify the nova service user id running the following command openstack user show --domain service_domain -f value -c id nova The command output looks like this ae034831d6bf41099327bf9d34988fbc 2) Create and apply the following policy to cinder, using the output of the previous command: "is_service": "role:service or service_user_id:ae034831d6bf41099327bf9d34988fbc" "legacy_system_admin_or_project_member": "(role:admin) or (role:member and project_id:%(project_id)s)" "volume:attachment_delete": "rule:legacy_system_admin_or_project_member and rule:is_service" "volume_extension:volume_actions:terminate_connection": "rule:legacy_system_admin_or_project_member and rule:is_service" "volume_extension:volume_actions:detach": "rule:legacy_system_admin_or_project_member and rule:is_service" "volume_extension:volume_admin_actions:force_detach": "!" = Frequently Asked Questions = Q: What can I do if my deployed cloud doesn?t have service tokens enabled and my deployment tooling doesn?t have the option to enable them? A: Pin the version of the affected packages to block the upgrades to the versions listed in ?Affected packages? section or newer, until service tokens are enabled. = References = * CVE-2023-2088: https://ubuntu.com/security/CVE-2023-2088 * OSSA-2023-003 Unauthorized volume access through deleted volume attachments: https://security.openstack.org/ossa/OSSA-2023-003.html * OSSN-0092 Using Configuration as a Short-Term Mitigation for OSSA-2023-003: https://wiki.openstack.org/wiki/OSSN/OSSN-0092 * Bug 2004555: https://bugs.launchpad.net/nova/+bug/2004555 -------------- next part -------------- An HTML attachment was scrubbed... URL: From corey.bryant at canonical.com Mon Jul 17 16:32:19 2023 From: corey.bryant at canonical.com (Corey Bryant) Date: Mon, 17 Jul 2023 12:32:19 -0400 Subject: CVE-2023-2088 for Charmed Openstack Message-ID: Bug LP:#2004555 (CVE-2023-2088) identified that ?an unauthorised access to a volume could occur when an iSCSI or FC connection from a host is severed due to a volume being unmapped on the storage system and the device is later reused for another volume on the same host?. While the bug affects iSCSI and FC, the fix introduces a breaking change for configurations upgrading to the fixed packages if they do not have service token support enabled. The new code introduces mandatory use of service tokens to identify requests coming from a trusted control plane service. Service tokens are issued by Keystone for service users. For example, a token can be issued to the Nova service user. They have a particular trust level, defined by the service, that allows them to perform trusted actions. In the case of Nova, the action is detaching volumes from instances. Versions of Ubuntu OpenStack for Yoga and later will have new packages versions released with CVE patches applied that require the use of service tokens to enable the detaching and deletion of volumes on instances. These packages will be released on July 24, 2023. Versions of Ubuntu OpenStack for Xena and earlier will not have package updates released and will still be vulnerable to the ?unauthorised detach of volumes? issue until a mitigation policy is applied. The mitigation policy provided requires the use of service tokens. For Charmed OpenStack releases, this means that some charms (identified below) must be upgraded to enable service tokens prior to the packages being upgraded and before the mitigation policy is applied. If not, users will lose the ability to detach volumes from their instances. = Patched packages = The following package versions include patches for CVE-2023-2088. These package versions deliver services that require the use of service tokens. Package versions prior to those listed will not be patched for CVE-2023-2088 and will therefore require policy updates for mitigation. Ubuntu 23.04 (Lunar) * nova >= 3:27.0.0-0ubuntu1.3 * cinder >= 2:22.0.0-0ubuntu1.3 * python-os-brick >= 6.2.0-0ubuntu2.3 * ironic >= 1:21.4.0-0ubuntu1.1 * python-glance-store >= 4.3.0-0ubuntu1.3 Ubuntu Cloud Archive Antelope * nova >= 3:27.0.0-0ubuntu1.3~cloud0 * cinder >= 2:22.0.0-0ubuntu1.3~cloud0 * python-os-brick >= 6.2.0-0ubuntu2.3~cloud0 * ironic >= 1:21.4.0-0ubuntu1.1~cloud0 * python-glance-store >= 4.3.0-0ubuntu1.3~cloud0 Ubuntu 22.10 (Kinetic) * nova >= 3:26.1.1-0ubuntu1.1 * cinder >= 2:21.2.0-0ubuntu1.1 * python-os-brick >= os-brick - 6.1.0-0ubuntu1.3 * ironic >= 1:21.1.0-0ubuntu1.1 * python-glance-store >= 4.1.0-0ubuntu1.3 Ubuntu Cloud Archive Zed * nova >= 3:26.1.1-0ubuntu1.1~cloud0 * cinder >= 2:21.2.0-0ubuntu1.1~cloud0 * python-os-brick >= os-brick - 6.1.0-0ubuntu1.3~cloud0 * ironic >= 1:21.1.0-0ubuntu1.1~cloud0 * python-glance-store >= 4.1.0-0ubuntu1.3~cloud0 Ubuntu 22.04 (Jammy) * nova >= 3:25.1.1-0ubuntu1.1 * cinder >= 2:20.2.0-0ubuntu1.1 * python-os-brick >= 5.2.2-0ubuntu1.2 * ironic >= 1:20.1.0-0ubuntu1.1 * python-glance-store >= 3.0.0-0ubuntu1.3 Ubuntu Cloud Archive Yoga * nova >= 3:25.1.1-0ubuntu1.1~cloud0 * cinder >= 2:20.2.0-0ubuntu1.1~cloud0 * python-os-brick >= 5.2.2-0ubuntu1.2~cloud0 * ironic >= 1:20.1.0-0ubuntu1.1~cloud0 * python-glance-store >= 3.0.0-0ubuntu1.3~cloud0 = Charms with Service Tokens Enabled by Default = Following is the list of charms and earliest revision that supports service tokens: Channel: 2023.1/stable * Keystone revision >= 645 * Cinder revision >= 639 * Nova-compute revision >= 678 * Nova-cloud-controller revision >= 674 * Ironic-api revision >= 54 * Ironic-conductor - s390x revision >= 151 - arm64 revision >= 152 - ppc64el revision >= 153 - amd64 revision >= 149 Channel: zed/stable * Keystone revision >= 646 * Cinder revision >= 647 * Nova-compute revision >= 683 * Nova-cloud-controller revision >= 676 * Ironic-api revision >= 53 * Ironic-conductor - s390x revision >= 148 - arm64 revision >= 154 - ppc64el revision >= 150 - amd64 revision >= 147 Channel: yoga/stable * Keystone revision >= 647 * Cinder revision >= 638 * Nova-compute revision >= 681 * Nova-cloud-controller revision >= 679 * Ironic-api revision >= 48 * Ironic-conductor revision >= 145 Channel: xena/stable * Keystone revision >= 651 * Cinder revision >= 640 * Nova-compute revision >= 680 * Nova-cloud-controller revision >= 677 * Ironic-api revision >= 50 * Ironic-conductor revision >= 143 Channel: wallaby/stable * Keystone revision >= 653 * Cinder revision >= 648 * Nova-compute revision >= 684 * Nova-cloud-controller revision >= 675 * Ironic-api revision >= 52 * Ironic-conductor revision >= 144 Channel: victoria/stable * Keystone revision >= 650 * Cinder revision >= 646 * Nova-compute revision >= 679 * Nova-cloud-controller revision >= 678 * Ironic-api revision >= 49 * Ironic-conductor revision >= 142 Channel: ussuri/stable * Keystone revision >= 652 * Cinder revision >= 645 * Nova-compute revision >= 682 * Nova-cloud-controller revision >= 680 * Ironic-api revision >= 51 * Ironic-conductor revision >= 155 = Using a custom Cinder policy as a mitigation = It?s possible to restrict the access to the operations affected by this CVE by providing a custom Cinder policy file that will prevent users from deleting attachments, detach and force detach. In order to use this mitigation, service token support must be enabled. The exact method for customising this depends on the deployment tooling being used. For Charmed OpenStack users it is the following: 1) Identify the nova service user id running the following command openstack user show --domain service_domain -f value -c id nova The command output looks like this ae034831d6bf41099327bf9d34988fbc 2) Create a policy file named cve-2023-2088-cinder.yaml with the following content: "is_service": "role:service or service_user_id:" "legacy_system_admin_or_project_member": "(role:admin) or (role:member and project_id:%(project_id)s)" "volume:attachment_delete": "rule:legacy_system_admin_or_project_member and rule:is_service" "volume_extension:volume_actions:terminate_connection": "rule:legacy_system_admin_or_project_member and rule:is_service" "volume_extension:volume_actions:detach": "rule:legacy_system_admin_or_project_member and rule:is_service" "volume_extension:volume_admin_actions:force_detach": "!" Replacing ?? would result in a snippet that looks like this: "is_service": "role:service or service_user_id:ae034831d6bf41099327bf9d34988fbc" "legacy_system_admin_or_project_member": "(role:admin) or (role:member and project_id:%(project_id)s)" "volume:attachment_delete": "rule:legacy_system_admin_or_project_member and rule:is_service" "volume_extension:volume_actions:terminate_connection": "rule:legacy_system_admin_or_project_member and rule:is_service" "volume_extension:volume_actions:detach": "rule:legacy_system_admin_or_project_member and rule:is_service" "volume_extension:volume_admin_actions:force_detach": "!" 3) Create zip files from the yaml file: zip cve-2023-2088-cinder.zip cve-2023-2088-cinder.yaml 4) Attach the zip file as a resource to the cinder applications: juju attach-resource cinder policyd-override=cve-2023-2088-cinder.zip 5) Enable the overrides via the use-policyd-override charm option: juju config cinder use-policyd-override=true For more details on using policy overrides see the OpenStack Charm Guide. = Frequently Asked Questions = Q: What can I do if my deployed cloud doesn?t have service tokens enabled and my deployment tooling doesn?t have the option to enable them? A: Pin the version of the affected packages to block the upgrades to the versions listed in ?Affected packages? section or newer, until service tokens are enabled. Q: For Charmed OpenStack, what?s the upgrade order for charms and deb packages? A: Upgrade the charms to the latest versions available in the charm channel tracked by your environment or at the least to the revisions listed in section ?Charms with Service Token Enabled By Default?. After that, upgrade the deb packages with your preferred method (e.g. Landscape, apt-get, etc). = References = * CVE-2023-2088: https://ubuntu.com/security/CVE-2023-2088 * OSSA-2023-003 Unauthorized volume access through deleted volume attachments: https://security.openstack.org/ossa/OSSA-2023-003.html * OSSN-0092 Using Configuration as a Short-Term Mitigation for OSSA-2023-003: https://wiki.openstack.org/wiki/OSSN/OSSN-0092 * Bug 2004555: https://bugs.launchpad.net/nova/+bug/2004555 * OpenStack Charm Guide Policy overrides: https://docs.openstack.org/charm-guide/latest/admin/policy-overrides.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralonsoh at redhat.com Mon Jul 17 14:30:07 2023 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Mon, 17 Jul 2023 16:30:07 +0200 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: Ok, I'll ask again for the tool/script/manual used to make the IC connection. I will also ask again for the NB-IC database resources and the NB resources needed, and how are related. If you want Neutron to handle these resources, then you don't need anything special: you can add an external network, handled only by one project. Inside this project you can create the needed resources: networks, routers, etc. This project/user (controlled by the sysadmin), won't create any VM on these networks/routers. Then you can use these resources to connect the TS. On Mon, Jul 17, 2023 at 3:10?PM Roberto Bartzen Acosta < roberto.acosta at luizalabs.com> wrote: > Hi Felix and Rodolfo, > > > Em seg., 17 de jul. de 2023 ?s 05:39, Rodolfo Alonso Hernandez < > ralonsoh at redhat.com> escreveu: > >> Hello Felix: >> >> That's the point: the launchpad bug describes the registers created by >> the IC configuration but these registers do not have any identification >> that informs about their relationship with the IC TS. >> >> Rather than modifying the Neutron resources (a network is a generic >> resource that is used by multiple backends), I would try to create a NB-IC >> resource map and the corresponding NB registers (NAT, LRP and LRSR). If you >> can link a NB-IC TS with the corresponding NB resources, then the problem >> is fixed because knowing them we can explicitly skip them from the DB sync >> tool. >> >> If that is not the case, then please present an alternative with the >> suggested idea of creating an external network type. >> >> BTW, I don't understand why you first say that "created by tools not >> necessarily controlled by us we need to avoid changing them", but then you >> are talking about refactoring these tools. Are we talking about the tools >> that create these IC configurations? Can you share them? >> >> > I think we are talking about two different cases. > > 1 - This proposed RFE intends to solve the problem of removing external > resources and that is enough for the OVN-IC to work in generic scenarios: > multi tenancy, single external provider, multiple external providers, etc. > because the OVN-IC will not relate to Neutron's network topology or > capabilities. > > 2- Felix' suggestion is related to integrating the TS to a network on > Neutron (some kind of mapping), and in this case, we have a hybrid solution > with the TS managed by an external tool, and the management of the NAT, LRP > / TS link with the tenant being done directly on Neutron. For this > proposal, I believe that the port filter would not be necessary because > Neutron would know the resource mappings in the OVN-NB, but it seems to me > a more complex solution. > > The question about case 2 is related to which level we are talking about > modifying the generic network to map the TS. If it is a network provider > level, the discussion is one - this case would solve the > schedule/lrp-set-gateway-chassis problem as it would be a gateway port. > but if it is a tenant' network level linked to a router / no gateway port > (the discussion would be another) - does not solve the gateway port > problem, but it seems closer to what the OVN-IC proposes to do at the > network L3 interconnect level). > > BTW: in case the TS is connected to an external network, we would need an > additional router for external/internet connectivity not related to the > OVN-IC, or maybe create a router with multiple gateway ports (that doesn't > exist yet)... > > Regards, > Roberto > > > >> Regards. >> >> On Mon, Jul 17, 2023 at 8:43?AM Felix Huettner >> wrote: >> >>> Hi everyone, >>> >>> On Thu, Jul 13, 2023 at 07:29:51PM -0300, Roberto Bartzen Acosta wrote: >>> > Hi Rodolfo, >>> > >>> > Em qui., 13 de jul. de 2023 ?s 13:45, Rodolfo Alonso Hernandez < >>> > ralonsoh at redhat.com> escreveu: >>> > >>> > > Hello Roberto: >>> > > >>> > > I think you have provided a very detailed explanation of the issue >>> but you >>> > > still didn't present any possible solution. I would ask you at least >>> for a >>> > > proposal and for specific details of what is failing in the OVN sync >>> tool. >>> > > >>> > > For example, if you add a TS between two clouds, you'll need to >>> create (1) >>> > > routers (NAT registers), (2) router ports (LRP) and (3) some static >>> routes >>> > > (LRSR). All these elements are monitored by the DB sync tool and >>> will fail >>> > > because the counterparts in the Neutron DB don't exist. I guess that >>> you >>> > > have some script or tool or at least a document to manually add these >>> > > resources; sharing it could help to find a solution. >>> > > >>> > > If these elements are manually created during the deployment phase, >>> you >>> > > can also control the information provided. I'm thinking about the >>> > > "external_ids" field; we can push some defined constant that will >>> make >>> > > these registers transparent to the OVN DB sync tool. >>> >>> As these resources are also created by tools not necessarily controlled >>> by us we need to avoid changing them. >>> I would therefor propose to not add a "ignore" constant to the >>> "external_ids", to to rather check for the existence of the neutron keys >>> in "external_ids" (i think thats also what is described in the bug >>> below). >>> >>> > Thanks for the contribution Felix. The idea is not to add (or at least not > expect it to be externally added) any identification in the external_ids, > Neutron should only filter for DB consistency validation what it knows and > has created by Neuron -> Neutron in the external ids key. > > >> > > >>> > > Please check this idea and if it is feasible. In that case, please >>> add a >>> > > topic to the Neutron drivers meeting agenda [1] to discuss it. >>> > > >>> > >>> > That sounds good, thanks. >>> > https://bugs.launchpad.net/neutron/+bug/2027742 >>> >>> From my perspective there is also an additional step after the above >>> REF. Currently it just makes it possible to use ovn-ic without neutron >>> killing it. >>> However if we could get neutron to treat a Transit Switch as a external >>> network we would skip the need to implement the NAT, LRP and LRSR logic >>> in custom tooling and could use the existing neutron code. This would >>> probably require creating a custom type of provider network and then >>> specifying the transit switch name in the provider options. >>> >>> Is that something that from your perspective we should take a look at >>> right away, or rather first focus und neutron not burning the transit >>> switch. >>> >>> Regards >>> Felix >>> >>> > >>> > Regards, >>> > Roberto >>> > >>> > >>> > > >>> > > Regards. >>> > > >>> > > [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers >>> > > >>> > > >>> > > On Wed, Jul 12, 2023 at 2:12?PM Roberto Bartzen Acosta < >>> > > roberto.acosta at luizalabs.com> wrote: >>> > > >>> > >> >>> > >> >>> > >> Em qua., 12 de jul. de 2023 ?s 09:05, Roberto Bartzen Acosta < >>> > >> roberto.acosta at luizalabs.com> escreveu: >>> > >> >>> > >>> Hi Rodolfo, >>> > >>> >>> > >>> Thanks for the feedback. >>> > >>> >>> > >>> I liked the idea of having a different network type only to hold >>> the >>> > >>> Transit Switches and thus the LRP but it depends on some factors >>> like >>> > >>> multi-tenancy. >>> > >>> From the OVN perspective, the TS is global and will only be linked >>> to a >>> > >>> tenant when it is plugged into the tenant router port. My concern >>> here is >>> > >>> that if Neutron manages the TS, it will need to assume the dynamic >>> > >>> characteristics of this TS function, ovn-ic creates and removes >>> the TS in >>> > >>> NB_Global, in addition to plugging and removing ports in this >>> switch >>> > >>> according to the network topology. Additionally, Neutron would >>> still need >>> > >>> to handle the learned remote routes (which are listed as static >>> routes from >>> > >>> the tenant router perspective). >>> > >>> >>> > >> sorry: NB_Global -> Northbound Database. >>> > >> >>> > >> >>> > >>> This is an open discussion, Felix can add ideas here. >>> > >>> >>> > >>> In general, it seems to me that we have no alternatives for this >>> type of >>> > >>> solution other than OVN-IC (note that ovn-bgp-agent does not learn >>> remote >>> > >>> routes at the SDN level / OVN). >>> > >>> OpenStack seems to be designed to run like a "bubble" and this >>> traffic >>> > >>> from one VM to another always needs to be routed at the FIP level >>> and >>> > >>> external routing layers. >>> > >>> >>> > >>> Regards, >>> > >>> Roberto >>> > >>> >>> > >>> Em qua., 12 de jul. de 2023 ?s 05:11, Rodolfo Alonso Hernandez < >>> > >>> ralonsoh at redhat.com> escreveu: >>> > >>> >>> > >>>> Hello Roberto: >>> > >>>> >>> > >>>> We talked about this possible RFE during the PTG [1] but I don't >>> > >>>> remember having any proposal. Actually what I remember (and was >>> written in >>> > >>>> the etherpad) is that you were going to present an RFE. Can you >>> explain it >>> > >>>> briefly? >>> > >>>> >>> > >>>> We also talked about the idea, proposed by Felix in the mailing >>> list, >>> > >>>> of having a different network type only to hold the Transit >>> Switches and >>> > >>>> thus the LRP. If you have a proposal, please present it. >>> > >>>> >>> > >>>> Regards. >>> > >>>> >>> > >>>> [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506 >>> > >>>> >>> > >>>> On Tue, Jul 11, 2023 at 10:50?PM Roberto Bartzen Acosta < >>> > >>>> roberto.acosta at luizalabs.com> wrote: >>> > >>>> >>> > >>>>> Hello Neutron folks, >>> > >>>>> >>> > >>>>> Regarding the conversation started in March [1] about the use of >>> OVN >>> > >>>>> interconnect with Neutron, I would like to evolve the discussion >>> in >>> > >>>>> relation to resources allocated by OVN-IC and which are not >>> managed by >>> > >>>>> Neutron. In the March PTG [2], the problem related to the >>> db_sync tool was >>> > >>>>> presented, and a proposed solution in which Neutron does not >>> manage these >>> > >>>>> resources. >>> > >>>>> >>> > >>>>> After discussing some architectural designs with Felix/StackIT, >>> it >>> > >>>>> seems to make sense to come up with a proposal to change the >>> mech_driver to >>> > >>>>> validate the external_ids key and not remove resources present >>> in the OVN >>> > >>>>> backend without Neutron "signature". >>> > >>>>> >>> > >>>>> The discussion here is more complex than it seems, because >>> > >>>>> architecturally Neutron does not allow us to interconnect >>> workloads in >>> > >>>>> multiple AZs (with different OpenStacks), but this could be a >>> requirement >>> > >>>>> for a high availability cloud solution (Cloud Region). >>> Additionally, this >>> > >>>>> OVN-IC solution allows interconnecting other cloud backends that >>> use OVN, >>> > >>>>> in the case of kubernetes with ovn-kube. >>> > >>>>> >>> > >>>>> We tested an OVN interconnect integrated with 3 OpenStack >>> > >>>>> installations and it worked very well !!! considering direct L3 >>> traffic at >>> > >>>>> the router level between different network infrastructures. >>> > >>>>> >>> > >>>>> SYNC_REPAIR - problem >>> > >>>>> >>> > >>>>> * Static Routes (learned OVN-IC routes) >>> > >>>>> * Router Port -> Transit Switches >>> > >>>>> >>> > >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>> > >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>> > >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>> > >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port >>> found in >>> > >>>>> OVN but not in Neutron, port_id=rt2-admin-tenant1 >>> > >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>> > >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>> > >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>> > >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router >>> > >>>>> 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes >>> [{'destination': ' >>> > >>>>> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': ' >>> > >>>>> 10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not >>> in >>> > >>>>> Neutron >>> > >>>>> >>> > >>>>> >>> > >>>>> Any suggestions on how to resolve this db_sync issue? since all >>> other >>> > >>>>> Neutron modules did not present any problem with OVN-IC (as far >>> as I >>> > >>>>> tested). >>> > >>>>> I remember Terry was keen to suggest some things to help. I >>> believe >>> > >>>>> that before proposing some complex mechanism to solve this >>> simple problem, >>> > >>>>> I would like to hear the community opinion. In our use case, a >>> bit change >>> > >>>>> in db_sync with filter by neutron key in external_ids would be >>> enough. >>> > >>>>> >>> > >>>>> Regards, >>> > >>>>> Roberto >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> [1] >>> > >>>>> >>> https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html >>> > >>>>> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> Additional logs: >>> > >>>>> >>> > >>>>> OpenStack 1 >>> > >>>>> >>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>> ovn-nbctl >>> > >>>>> lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 >>> > >>>>> IPv4 Routes >>> > >>>>> Route Table
: >>> > >>>>> 20.0.1.0/24 169.254.200.2 dst-ip >>> (learned) >>> > >>>>> 20.0.2.0/24 169.254.200.3 dst-ip >>> (learned) >>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>> > >>>>> >>> > >>>>> IPv6 Routes >>> > >>>>> Route Table
: >>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>> ovn-nbctl >>> > >>>>> lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 >>> > >>>>> IPv4 Routes >>> > >>>>> Route Table
: >>> > >>>>> 10.0.1.0/24 169.254.100.2 dst-ip >>> (learned) >>> > >>>>> 10.0.2.0/24 169.254.100.3 dst-ip >>> (learned) >>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>> > >>>>> >>> > >>>>> IPv6 Routes >>> > >>>>> Route Table
: >>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>> > >>>>> >>> > >>>>> >>> > >>>>> OpenStack 2 >>> > >>>>> >>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# >>> ovn-nbctl >>> > >>>>> lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc >>> > >>>>> IPv4 Routes >>> > >>>>> Route Table
: >>> > >>>>> 20.0.0.0/24 169.254.200.1 dst-ip >>> (learned) >>> > >>>>> 20.0.2.0/24 169.254.200.3 dst-ip >>> (learned) >>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>> > >>>>> >>> > >>>>> IPv6 Routes >>> > >>>>> Route Table
: >>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# >>> ovn-nbctl >>> > >>>>> lr-route-list ce45f681-6454-43fe-974f-81344bb8113a >>> > >>>>> IPv4 Routes >>> > >>>>> Route Table
: >>> > >>>>> 10.0.0.0/24 169.254.100.1 dst-ip >>> (learned) >>> > >>>>> 10.0.2.0/24 169.254.100.3 dst-ip >>> (learned) >>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>> > >>>>> >>> > >>>>> IPv6 Routes >>> > >>>>> Route Table
: >>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> OpenStack 3 >>> > >>>>> >>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >>> ovn-nbctl >>> > >>>>> lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 >>> > >>>>> IPv4 Routes >>> > >>>>> Route Table
: >>> > >>>>> 20.0.0.0/24 169.254.200.1 dst-ip >>> (learned) >>> > >>>>> 20.0.1.0/24 169.254.200.2 dst-ip >>> (learned) >>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>> > >>>>> >>> > >>>>> IPv6 Routes >>> > >>>>> Route Table
: >>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >>> ovn-nbctl >>> > >>>>> lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 >>> > >>>>> IPv4 Routes >>> > >>>>> Route Table
: >>> > >>>>> 10.0.0.0/24 169.254.100.1 dst-ip >>> (learned) >>> > >>>>> 10.0.1.0/24 169.254.100.2 dst-ip >>> (learned) >>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>> > >>>>> >>> > >>>>> IPv6 Routes >>> > >>>>> Route Table
: >>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>> > >>>>> >>> > >>>>> >>> > >>>>> OVN-IC Global database >>> > >>>>> >>> > >>>>> root at ovn-global-db1:~# ovn-ic-sbctl show >>> > >>>>> availability-zone osp1 >>> > >>>>> gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 >>> > >>>>> hostname: osp1-gwnode1 >>> > >>>>> type: geneve >>> > >>>>> ip: 192.168.200.28 >>> > >>>>> port admin-rt1-tenant1 >>> > >>>>> transit switch: admin-tenant1 >>> > >>>>> address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 >>> fe80::1/64"] >>> > >>>>> port admin-rt1-tenant1_1 >>> > >>>>> transit switch: admin-tenant1_1 >>> > >>>>> address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] >>> > >>>>> availability-zone osp2 >>> > >>>>> gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 >>> > >>>>> hostname: osp2-gwnode1 >>> > >>>>> type: geneve >>> > >>>>> ip: 192.168.200.128 >>> > >>>>> port admin-rt2-tenant1 >>> > >>>>> transit switch: admin-tenant1 >>> > >>>>> address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 >>> fe80::2/64"] >>> > >>>>> port admin-rt2-tenant1_1 >>> > >>>>> transit switch: admin-tenant1_1 >>> > >>>>> address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] >>> > >>>>> availability-zone osp3 >>> > >>>>> gateway 97595af9-7896-40d0-a883-beadbff1aa5b >>> > >>>>> hostname: osp3-gwnode1 >>> > >>>>> type: geneve >>> > >>>>> ip: 192.168.200.228 >>> > >>>>> port admin-rt3-tenant1 >>> > >>>>> transit switch: admin-tenant1 >>> > >>>>> address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 >>> fe80::3/64"] >>> > >>>>> port admin-rt3-tenant1_1 >>> > >>>>> transit switch: admin-tenant1_1 >>> > >>>>> address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> *?Esta mensagem ? direcionada apenas para os endere?os >>> constantes no >>> > >>>>> cabe?alho inicial. Se voc? n?o est? listado nos endere?os >>> constantes no >>> > >>>>> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do >>> dessa >>> > >>>>> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es >>> citadas est?o >>> > >>>>> imediatamente anuladas e proibidas?.* >>> > >>>>> >>> > >>>>> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis >>> para >>> > >>>>> assegurar que nenhum v?rus esteja presente nesse e-mail, a >>> empresa n?o >>> > >>>>> poder? aceitar a responsabilidade por quaisquer perdas ou danos >>> causados >>> > >>>>> por esse e-mail ou por seus anexos?.* >>> > >>>>> >>> > >>>> >>> > >> >>> > >> *?Esta mensagem ? direcionada apenas para os endere?os constantes no >>> > >> cabe?alho inicial. Se voc? n?o est? listado nos endere?os >>> constantes no >>> > >> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do >>> dessa >>> > >> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es >>> citadas est?o >>> > >> imediatamente anuladas e proibidas?.* >>> > >> >>> > >> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para >>> > >> assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa >>> n?o >>> > >> poder? aceitar a responsabilidade por quaisquer perdas ou danos >>> causados >>> > >> por esse e-mail ou por seus anexos?.* >>> > >> >>> > > >>> > >>> > -- >>> > >>> > >>> > >>> > >>> > _?Esta mensagem ? direcionada apenas para os endere?os constantes no >>> > cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no >>> > cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa >>> > mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas >>> est?o >>> > imediatamente anuladas e proibidas?._ >>> > >>> > >>> > * **?Apesar do Magazine Luiza tomar >>> > todas as precau??es razo?veis para assegurar que nenhum v?rus esteja >>> > presente nesse e-mail, a empresa n?o poder? aceitar a responsabilidade >>> por >>> > quaisquer perdas ou danos causados por esse e-mail ou por seus >>> anexos?.* >>> > >>> > >>> > >>> Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r >>> die Verwertung durch den vorgesehenen Empf?nger bestimmt. >>> Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den >>> Absender bitte unverz?glich in Kenntnis und l?schen diese E Mail. >>> >>> Hinweise zum Datenschutz finden Sie hier>> >. >>> >>> >>> This e-mail may contain confidential content and is intended only for >>> the specified recipient/s. >>> If you are not the intended recipient, please inform the sender >>> immediately and delete this e-mail. >>> >>> Information on data protection can be found here< >>> https://www.datenschutz.schwarz>. >>> >>> > > *?Esta mensagem ? direcionada apenas para os endere?os constantes no > cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no > cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa > mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o > imediatamente anuladas e proibidas?.* > > *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para > assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o > poder? aceitar a responsabilidade por quaisquer perdas ou danos causados > por esse e-mail ou por seus anexos?.* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralonsoh at redhat.com Mon Jul 17 16:33:38 2023 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Mon, 17 Jul 2023 18:33:38 +0200 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: > In my opinion the answer would be: yes and no, this approach would work for isolated cases, but think on a larger scale: how many external networks would need to be managed at the IP Fabric level? This is exactly what you are proposing in your previous mails: "Felix' suggestion is related to integrating the TS to a network on Neutron (some kind of mapping)". So at this point I'm curious about your proposal. And about the static routes, if these routes are created from a Neutron defined router, as I proposed, they won't be deleted. > About the proposed RFE, do you see any problem if Neutron deletes from the OVN only what it manages? Yes, a huge one: OVN is a service for Neutron, same as RabbitMQ or MySQL. These services are 100% controlled by Neutron because the resources can clash. You can't manually create an IPAM register in the DB because that will disable one IP address. The same for OVN: you can't manually create a LRP in a Logical Switch that provides a Neutron network because this LRP will have an IP address. If a new Neutron port is created and it uses the same IP address of the existing LRP (that doesn't exist in Neutron), the new port will be considered virtual and won't be able to be used in a VM (this is an actual problem I found in a customer last week). So no, Neutron should not allow other resources in OVN apart from what Neutron creates. On Mon, Jul 17, 2023 at 6:13?PM Roberto Bartzen Acosta < roberto.acosta at luizalabs.com> wrote: > Hi Rodolfo, > > Em seg., 17 de jul. de 2023 ?s 11:30, Rodolfo Alonso Hernandez < > ralonsoh at redhat.com> escreveu: > >> Ok, I'll ask again for the tool/script/manual used to make the IC >> connection. I will also ask again for the NB-IC database resources and the >> NB resources needed, and how are related. >> > > To be clear, this tool doesn't exist yet and we are working on the design > of the software solution [1]. For now, the tests were performed using > manual commands, such as those described in the RFE link and very similar > to the case applied by the ovn-kube to use OVN-IC with OpenStack - Neutron > [2]. > The main idea is to develop an external layer interface with an API/SDK > and an internal layer interface with the OVN-IC-NB global database and the > OpenStack/AZ OVN-NB database [1]. > > [1] > https://drive.google.com/file/d/1yx66YcxNMH_oB11--0-_0eZzmBJHu-Ox/view?usp=sharing > [2] > https://github.com/kubeovn/kube-ovn/blob/v1.11.0/docs/cluster-interconnection.md > > >> If you want Neutron to handle these resources, then you don't need >> anything special: you can add an external network, handled only by one >> project. Inside this project you can create the needed resources: networks, >> routers, etc. This project/user (controlled by the sysadmin), won't create >> any VM on these networks/routers. Then you can use these resources to >> connect the TS. >> > > In my opinion the answer would be: yes and no, this approach would work > for isolated cases, but think on a larger scale: how many external networks > would need to be managed at the IP Fabric level? what is the service type? > vlan (we would have segmentation limits) vxlan (we would have to extend the > L2 domain across the entire IP fabric) virtual? . I don't know, there are > many questions here. Felix can explain better and propose an RFE in the > future. Even so, Neutron would need a filter to not delete the static > routes learned by the router (the different proposals can be > complementary...) > > > About the proposed RFE, do you see any problem if Neutron deletes from the > OVN only what it manages? > > >> >> On Mon, Jul 17, 2023 at 3:10?PM Roberto Bartzen Acosta < >> roberto.acosta at luizalabs.com> wrote: >> >>> Hi Felix and Rodolfo, >>> >>> >>> Em seg., 17 de jul. de 2023 ?s 05:39, Rodolfo Alonso Hernandez < >>> ralonsoh at redhat.com> escreveu: >>> >>>> Hello Felix: >>>> >>>> That's the point: the launchpad bug describes the registers created by >>>> the IC configuration but these registers do not have any identification >>>> that informs about their relationship with the IC TS. >>>> >>>> Rather than modifying the Neutron resources (a network is a generic >>>> resource that is used by multiple backends), I would try to create a NB-IC >>>> resource map and the corresponding NB registers (NAT, LRP and LRSR). If you >>>> can link a NB-IC TS with the corresponding NB resources, then the problem >>>> is fixed because knowing them we can explicitly skip them from the DB sync >>>> tool. >>>> >>>> If that is not the case, then please present an alternative with the >>>> suggested idea of creating an external network type. >>>> >>>> BTW, I don't understand why you first say that "created by tools not >>>> necessarily controlled by us we need to avoid changing them", but then you >>>> are talking about refactoring these tools. Are we talking about the tools >>>> that create these IC configurations? Can you share them? >>>> >>>> >>> I think we are talking about two different cases. >>> >>> 1 - This proposed RFE intends to solve the problem of removing external >>> resources and that is enough for the OVN-IC to work in generic scenarios: >>> multi tenancy, single external provider, multiple external providers, etc. >>> because the OVN-IC will not relate to Neutron's network topology or >>> capabilities. >>> >>> 2- Felix' suggestion is related to integrating the TS to a network on >>> Neutron (some kind of mapping), and in this case, we have a hybrid solution >>> with the TS managed by an external tool, and the management of the NAT, LRP >>> / TS link with the tenant being done directly on Neutron. For this >>> proposal, I believe that the port filter would not be necessary because >>> Neutron would know the resource mappings in the OVN-NB, but it seems to me >>> a more complex solution. >>> >>> The question about case 2 is related to which level we are talking about >>> modifying the generic network to map the TS. If it is a network provider >>> level, the discussion is one - this case would solve the >>> schedule/lrp-set-gateway-chassis problem as it would be a gateway port. >>> but if it is a tenant' network level linked to a router / no gateway >>> port (the discussion would be another) - does not solve the gateway port >>> problem, but it seems closer to what the OVN-IC proposes to do at the >>> network L3 interconnect level). >>> >>> BTW: in case the TS is connected to an external network, we would need >>> an additional router for external/internet connectivity not related to the >>> OVN-IC, or maybe create a router with multiple gateway ports (that doesn't >>> exist yet)... >>> >>> Regards, >>> Roberto >>> >>> >>> >>>> Regards. >>>> >>>> On Mon, Jul 17, 2023 at 8:43?AM Felix Huettner >>>> wrote: >>>> >>>>> Hi everyone, >>>>> >>>>> On Thu, Jul 13, 2023 at 07:29:51PM -0300, Roberto Bartzen Acosta wrote: >>>>> > Hi Rodolfo, >>>>> > >>>>> > Em qui., 13 de jul. de 2023 ?s 13:45, Rodolfo Alonso Hernandez < >>>>> > ralonsoh at redhat.com> escreveu: >>>>> > >>>>> > > Hello Roberto: >>>>> > > >>>>> > > I think you have provided a very detailed explanation of the issue >>>>> but you >>>>> > > still didn't present any possible solution. I would ask you at >>>>> least for a >>>>> > > proposal and for specific details of what is failing in the OVN >>>>> sync tool. >>>>> > > >>>>> > > For example, if you add a TS between two clouds, you'll need to >>>>> create (1) >>>>> > > routers (NAT registers), (2) router ports (LRP) and (3) some >>>>> static routes >>>>> > > (LRSR). All these elements are monitored by the DB sync tool and >>>>> will fail >>>>> > > because the counterparts in the Neutron DB don't exist. I guess >>>>> that you >>>>> > > have some script or tool or at least a document to manually add >>>>> these >>>>> > > resources; sharing it could help to find a solution. >>>>> > > >>>>> > > If these elements are manually created during the deployment >>>>> phase, you >>>>> > > can also control the information provided. I'm thinking about the >>>>> > > "external_ids" field; we can push some defined constant that will >>>>> make >>>>> > > these registers transparent to the OVN DB sync tool. >>>>> >>>>> As these resources are also created by tools not necessarily controlled >>>>> by us we need to avoid changing them. >>>>> I would therefor propose to not add a "ignore" constant to the >>>>> "external_ids", to to rather check for the existence of the neutron >>>>> keys >>>>> in "external_ids" (i think thats also what is described in the bug >>>>> below). >>>>> >>>>> >>> Thanks for the contribution Felix. The idea is not to add (or at least >>> not expect it to be externally added) any identification in the >>> external_ids, Neutron should only filter for DB consistency validation what >>> it knows and has created by Neuron -> Neutron in the external ids key. >>> >>> >>>> > > >>>>> > > Please check this idea and if it is feasible. In that case, please >>>>> add a >>>>> > > topic to the Neutron drivers meeting agenda [1] to discuss it. >>>>> > > >>>>> > >>>>> > That sounds good, thanks. >>>>> > https://bugs.launchpad.net/neutron/+bug/2027742 >>>>> >>>>> From my perspective there is also an additional step after the above >>>>> REF. Currently it just makes it possible to use ovn-ic without neutron >>>>> killing it. >>>>> However if we could get neutron to treat a Transit Switch as a external >>>>> network we would skip the need to implement the NAT, LRP and LRSR logic >>>>> in custom tooling and could use the existing neutron code. This would >>>>> probably require creating a custom type of provider network and then >>>>> specifying the transit switch name in the provider options. >>>>> >>>>> Is that something that from your perspective we should take a look at >>>>> right away, or rather first focus und neutron not burning the transit >>>>> switch. >>>>> >>>>> Regards >>>>> Felix >>>>> >>>>> > >>>>> > Regards, >>>>> > Roberto >>>>> > >>>>> > >>>>> > > >>>>> > > Regards. >>>>> > > >>>>> > > [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers >>>>> > > >>>>> > > >>>>> > > On Wed, Jul 12, 2023 at 2:12?PM Roberto Bartzen Acosta < >>>>> > > roberto.acosta at luizalabs.com> wrote: >>>>> > > >>>>> > >> >>>>> > >> >>>>> > >> Em qua., 12 de jul. de 2023 ?s 09:05, Roberto Bartzen Acosta < >>>>> > >> roberto.acosta at luizalabs.com> escreveu: >>>>> > >> >>>>> > >>> Hi Rodolfo, >>>>> > >>> >>>>> > >>> Thanks for the feedback. >>>>> > >>> >>>>> > >>> I liked the idea of having a different network type only to hold >>>>> the >>>>> > >>> Transit Switches and thus the LRP but it depends on some factors >>>>> like >>>>> > >>> multi-tenancy. >>>>> > >>> From the OVN perspective, the TS is global and will only be >>>>> linked to a >>>>> > >>> tenant when it is plugged into the tenant router port. My >>>>> concern here is >>>>> > >>> that if Neutron manages the TS, it will need to assume the >>>>> dynamic >>>>> > >>> characteristics of this TS function, ovn-ic creates and removes >>>>> the TS in >>>>> > >>> NB_Global, in addition to plugging and removing ports in this >>>>> switch >>>>> > >>> according to the network topology. Additionally, Neutron would >>>>> still need >>>>> > >>> to handle the learned remote routes (which are listed as static >>>>> routes from >>>>> > >>> the tenant router perspective). >>>>> > >>> >>>>> > >> sorry: NB_Global -> Northbound Database. >>>>> > >> >>>>> > >> >>>>> > >>> This is an open discussion, Felix can add ideas here. >>>>> > >>> >>>>> > >>> In general, it seems to me that we have no alternatives for this >>>>> type of >>>>> > >>> solution other than OVN-IC (note that ovn-bgp-agent does not >>>>> learn remote >>>>> > >>> routes at the SDN level / OVN). >>>>> > >>> OpenStack seems to be designed to run like a "bubble" and this >>>>> traffic >>>>> > >>> from one VM to another always needs to be routed at the FIP >>>>> level and >>>>> > >>> external routing layers. >>>>> > >>> >>>>> > >>> Regards, >>>>> > >>> Roberto >>>>> > >>> >>>>> > >>> Em qua., 12 de jul. de 2023 ?s 05:11, Rodolfo Alonso Hernandez < >>>>> > >>> ralonsoh at redhat.com> escreveu: >>>>> > >>> >>>>> > >>>> Hello Roberto: >>>>> > >>>> >>>>> > >>>> We talked about this possible RFE during the PTG [1] but I don't >>>>> > >>>> remember having any proposal. Actually what I remember (and was >>>>> written in >>>>> > >>>> the etherpad) is that you were going to present an RFE. Can you >>>>> explain it >>>>> > >>>> briefly? >>>>> > >>>> >>>>> > >>>> We also talked about the idea, proposed by Felix in the mailing >>>>> list, >>>>> > >>>> of having a different network type only to hold the Transit >>>>> Switches and >>>>> > >>>> thus the LRP. If you have a proposal, please present it. >>>>> > >>>> >>>>> > >>>> Regards. >>>>> > >>>> >>>>> > >>>> [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506 >>>>> > >>>> >>>>> > >>>> On Tue, Jul 11, 2023 at 10:50?PM Roberto Bartzen Acosta < >>>>> > >>>> roberto.acosta at luizalabs.com> wrote: >>>>> > >>>> >>>>> > >>>>> Hello Neutron folks, >>>>> > >>>>> >>>>> > >>>>> Regarding the conversation started in March [1] about the use >>>>> of OVN >>>>> > >>>>> interconnect with Neutron, I would like to evolve the >>>>> discussion in >>>>> > >>>>> relation to resources allocated by OVN-IC and which are not >>>>> managed by >>>>> > >>>>> Neutron. In the March PTG [2], the problem related to the >>>>> db_sync tool was >>>>> > >>>>> presented, and a proposed solution in which Neutron does not >>>>> manage these >>>>> > >>>>> resources. >>>>> > >>>>> >>>>> > >>>>> After discussing some architectural designs with >>>>> Felix/StackIT, it >>>>> > >>>>> seems to make sense to come up with a proposal to change the >>>>> mech_driver to >>>>> > >>>>> validate the external_ids key and not remove resources present >>>>> in the OVN >>>>> > >>>>> backend without Neutron "signature". >>>>> > >>>>> >>>>> > >>>>> The discussion here is more complex than it seems, because >>>>> > >>>>> architecturally Neutron does not allow us to interconnect >>>>> workloads in >>>>> > >>>>> multiple AZs (with different OpenStacks), but this could be a >>>>> requirement >>>>> > >>>>> for a high availability cloud solution (Cloud Region). >>>>> Additionally, this >>>>> > >>>>> OVN-IC solution allows interconnecting other cloud backends >>>>> that use OVN, >>>>> > >>>>> in the case of kubernetes with ovn-kube. >>>>> > >>>>> >>>>> > >>>>> We tested an OVN interconnect integrated with 3 OpenStack >>>>> > >>>>> installations and it worked very well !!! considering direct >>>>> L3 traffic at >>>>> > >>>>> the router level between different network infrastructures. >>>>> > >>>>> >>>>> > >>>>> SYNC_REPAIR - problem >>>>> > >>>>> >>>>> > >>>>> * Static Routes (learned OVN-IC routes) >>>>> > >>>>> * Router Port -> Transit Switches >>>>> > >>>>> >>>>> > >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>>>> > >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>>>> > >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>>>> > >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router >>>>> Port found in >>>>> > >>>>> OVN but not in Neutron, port_id=rt2-admin-tenant1 >>>>> > >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>>>> > >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>>>> > >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>>>> > >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router >>>>> > >>>>> 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes >>>>> [{'destination': ' >>>>> > >>>>> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': ' >>>>> > >>>>> 10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but >>>>> not in >>>>> > >>>>> Neutron >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> Any suggestions on how to resolve this db_sync issue? since >>>>> all other >>>>> > >>>>> Neutron modules did not present any problem with OVN-IC (as >>>>> far as I >>>>> > >>>>> tested). >>>>> > >>>>> I remember Terry was keen to suggest some things to help. I >>>>> believe >>>>> > >>>>> that before proposing some complex mechanism to solve this >>>>> simple problem, >>>>> > >>>>> I would like to hear the community opinion. In our use case, a >>>>> bit change >>>>> > >>>>> in db_sync with filter by neutron key in external_ids would be >>>>> enough. >>>>> > >>>>> >>>>> > >>>>> Regards, >>>>> > >>>>> Roberto >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> [1] >>>>> > >>>>> >>>>> https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html >>>>> > >>>>> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> Additional logs: >>>>> > >>>>> >>>>> > >>>>> OpenStack 1 >>>>> > >>>>> >>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>>> ovn-nbctl >>>>> > >>>>> lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 >>>>> > >>>>> IPv4 Routes >>>>> > >>>>> Route Table
: >>>>> > >>>>> 20.0.1.0/24 169.254.200.2 dst-ip >>>>> (learned) >>>>> > >>>>> 20.0.2.0/24 169.254.200.3 dst-ip >>>>> (learned) >>>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>> > >>>>> >>>>> > >>>>> IPv6 Routes >>>>> > >>>>> Route Table
: >>>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>>> ovn-nbctl >>>>> > >>>>> lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 >>>>> > >>>>> IPv4 Routes >>>>> > >>>>> Route Table
: >>>>> > >>>>> 10.0.1.0/24 169.254.100.2 dst-ip >>>>> (learned) >>>>> > >>>>> 10.0.2.0/24 169.254.100.3 dst-ip >>>>> (learned) >>>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>> > >>>>> >>>>> > >>>>> IPv6 Routes >>>>> > >>>>> Route Table
: >>>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> OpenStack 2 >>>>> > >>>>> >>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# >>>>> ovn-nbctl >>>>> > >>>>> lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc >>>>> > >>>>> IPv4 Routes >>>>> > >>>>> Route Table
: >>>>> > >>>>> 20.0.0.0/24 169.254.200.1 dst-ip >>>>> (learned) >>>>> > >>>>> 20.0.2.0/24 169.254.200.3 dst-ip >>>>> (learned) >>>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>> > >>>>> >>>>> > >>>>> IPv6 Routes >>>>> > >>>>> Route Table
: >>>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# >>>>> ovn-nbctl >>>>> > >>>>> lr-route-list ce45f681-6454-43fe-974f-81344bb8113a >>>>> > >>>>> IPv4 Routes >>>>> > >>>>> Route Table
: >>>>> > >>>>> 10.0.0.0/24 169.254.100.1 dst-ip >>>>> (learned) >>>>> > >>>>> 10.0.2.0/24 169.254.100.3 dst-ip >>>>> (learned) >>>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>> > >>>>> >>>>> > >>>>> IPv6 Routes >>>>> > >>>>> Route Table
: >>>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> OpenStack 3 >>>>> > >>>>> >>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >>>>> ovn-nbctl >>>>> > >>>>> lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 >>>>> > >>>>> IPv4 Routes >>>>> > >>>>> Route Table
: >>>>> > >>>>> 20.0.0.0/24 169.254.200.1 dst-ip >>>>> (learned) >>>>> > >>>>> 20.0.1.0/24 169.254.200.2 dst-ip >>>>> (learned) >>>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>> > >>>>> >>>>> > >>>>> IPv6 Routes >>>>> > >>>>> Route Table
: >>>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >>>>> ovn-nbctl >>>>> > >>>>> lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 >>>>> > >>>>> IPv4 Routes >>>>> > >>>>> Route Table
: >>>>> > >>>>> 10.0.0.0/24 169.254.100.1 dst-ip >>>>> (learned) >>>>> > >>>>> 10.0.1.0/24 169.254.100.2 dst-ip >>>>> (learned) >>>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>> > >>>>> >>>>> > >>>>> IPv6 Routes >>>>> > >>>>> Route Table
: >>>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> OVN-IC Global database >>>>> > >>>>> >>>>> > >>>>> root at ovn-global-db1:~# ovn-ic-sbctl show >>>>> > >>>>> availability-zone osp1 >>>>> > >>>>> gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 >>>>> > >>>>> hostname: osp1-gwnode1 >>>>> > >>>>> type: geneve >>>>> > >>>>> ip: 192.168.200.28 >>>>> > >>>>> port admin-rt1-tenant1 >>>>> > >>>>> transit switch: admin-tenant1 >>>>> > >>>>> address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 >>>>> fe80::1/64"] >>>>> > >>>>> port admin-rt1-tenant1_1 >>>>> > >>>>> transit switch: admin-tenant1_1 >>>>> > >>>>> address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] >>>>> > >>>>> availability-zone osp2 >>>>> > >>>>> gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 >>>>> > >>>>> hostname: osp2-gwnode1 >>>>> > >>>>> type: geneve >>>>> > >>>>> ip: 192.168.200.128 >>>>> > >>>>> port admin-rt2-tenant1 >>>>> > >>>>> transit switch: admin-tenant1 >>>>> > >>>>> address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 >>>>> fe80::2/64"] >>>>> > >>>>> port admin-rt2-tenant1_1 >>>>> > >>>>> transit switch: admin-tenant1_1 >>>>> > >>>>> address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] >>>>> > >>>>> availability-zone osp3 >>>>> > >>>>> gateway 97595af9-7896-40d0-a883-beadbff1aa5b >>>>> > >>>>> hostname: osp3-gwnode1 >>>>> > >>>>> type: geneve >>>>> > >>>>> ip: 192.168.200.228 >>>>> > >>>>> port admin-rt3-tenant1 >>>>> > >>>>> transit switch: admin-tenant1 >>>>> > >>>>> address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 >>>>> fe80::3/64"] >>>>> > >>>>> port admin-rt3-tenant1_1 >>>>> > >>>>> transit switch: admin-tenant1_1 >>>>> > >>>>> address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> *?Esta mensagem ? direcionada apenas para os endere?os >>>>> constantes no >>>>> > >>>>> cabe?alho inicial. Se voc? n?o est? listado nos endere?os >>>>> constantes no >>>>> > >>>>> cabe?alho, pedimos-lhe que desconsidere completamente o >>>>> conte?do dessa >>>>> > >>>>> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es >>>>> citadas est?o >>>>> > >>>>> imediatamente anuladas e proibidas?.* >>>>> > >>>>> >>>>> > >>>>> *?Apesar do Magazine Luiza tomar todas as precau??es >>>>> razo?veis para >>>>> > >>>>> assegurar que nenhum v?rus esteja presente nesse e-mail, a >>>>> empresa n?o >>>>> > >>>>> poder? aceitar a responsabilidade por quaisquer perdas ou >>>>> danos causados >>>>> > >>>>> por esse e-mail ou por seus anexos?.* >>>>> > >>>>> >>>>> > >>>> >>>>> > >> >>>>> > >> *?Esta mensagem ? direcionada apenas para os endere?os constantes >>>>> no >>>>> > >> cabe?alho inicial. Se voc? n?o est? listado nos endere?os >>>>> constantes no >>>>> > >> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do >>>>> dessa >>>>> > >> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es >>>>> citadas est?o >>>>> > >> imediatamente anuladas e proibidas?.* >>>>> > >> >>>>> > >> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis >>>>> para >>>>> > >> assegurar que nenhum v?rus esteja presente nesse e-mail, a >>>>> empresa n?o >>>>> > >> poder? aceitar a responsabilidade por quaisquer perdas ou danos >>>>> causados >>>>> > >> por esse e-mail ou por seus anexos?.* >>>>> > >> >>>>> > > >>>>> > >>>>> > -- >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > _?Esta mensagem ? direcionada apenas para os endere?os constantes no >>>>> > cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes >>>>> no >>>>> > cabe?alho, pedimos-lhe que desconsidere completamente o conte?do >>>>> dessa >>>>> > mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es >>>>> citadas est?o >>>>> > imediatamente anuladas e proibidas?._ >>>>> > >>>>> > >>>>> > * **?Apesar do Magazine Luiza tomar >>>>> > todas as precau??es razo?veis para assegurar que nenhum v?rus esteja >>>>> > presente nesse e-mail, a empresa n?o poder? aceitar a >>>>> responsabilidade por >>>>> > quaisquer perdas ou danos causados por esse e-mail ou por seus >>>>> anexos?.* >>>>> > >>>>> > >>>>> > >>>>> Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur >>>>> f?r die Verwertung durch den vorgesehenen Empf?nger bestimmt. >>>>> Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den >>>>> Absender bitte unverz?glich in Kenntnis und l?schen diese E Mail. >>>>> >>>>> Hinweise zum Datenschutz finden Sie hier< >>>>> https://www.datenschutz.schwarz>. >>>>> >>>>> >>>>> This e-mail may contain confidential content and is intended only for >>>>> the specified recipient/s. >>>>> If you are not the intended recipient, please inform the sender >>>>> immediately and delete this e-mail. >>>>> >>>>> Information on data protection can be found here< >>>>> https://www.datenschutz.schwarz>. >>>>> >>>>> >>> >>> *?Esta mensagem ? direcionada apenas para os endere?os constantes no >>> cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no >>> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa >>> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o >>> imediatamente anuladas e proibidas?.* >>> >>> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para >>> assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o >>> poder? aceitar a responsabilidade por quaisquer perdas ou danos causados >>> por esse e-mail ou por seus anexos?.* >>> >> > > *?Esta mensagem ? direcionada apenas para os endere?os constantes no > cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no > cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa > mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o > imediatamente anuladas e proibidas?.* > > *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para > assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o > poder? aceitar a responsabilidade por quaisquer perdas ou danos causados > por esse e-mail ou por seus anexos?.* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.acosta at luizalabs.com Mon Jul 17 16:12:49 2023 From: roberto.acosta at luizalabs.com (Roberto Bartzen Acosta) Date: Mon, 17 Jul 2023 13:12:49 -0300 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: Hi Rodolfo, Em seg., 17 de jul. de 2023 ?s 11:30, Rodolfo Alonso Hernandez < ralonsoh at redhat.com> escreveu: > Ok, I'll ask again for the tool/script/manual used to make the IC > connection. I will also ask again for the NB-IC database resources and the > NB resources needed, and how are related. > To be clear, this tool doesn't exist yet and we are working on the design of the software solution [1]. For now, the tests were performed using manual commands, such as those described in the RFE link and very similar to the case applied by the ovn-kube to use OVN-IC with OpenStack - Neutron [2]. The main idea is to develop an external layer interface with an API/SDK and an internal layer interface with the OVN-IC-NB global database and the OpenStack/AZ OVN-NB database [1]. [1] https://drive.google.com/file/d/1yx66YcxNMH_oB11--0-_0eZzmBJHu-Ox/view?usp=sharing [2] https://github.com/kubeovn/kube-ovn/blob/v1.11.0/docs/cluster-interconnection.md > If you want Neutron to handle these resources, then you don't need > anything special: you can add an external network, handled only by one > project. Inside this project you can create the needed resources: networks, > routers, etc. This project/user (controlled by the sysadmin), won't create > any VM on these networks/routers. Then you can use these resources to > connect the TS. > In my opinion the answer would be: yes and no, this approach would work for isolated cases, but think on a larger scale: how many external networks would need to be managed at the IP Fabric level? what is the service type? vlan (we would have segmentation limits) vxlan (we would have to extend the L2 domain across the entire IP fabric) virtual? . I don't know, there are many questions here. Felix can explain better and propose an RFE in the future. Even so, Neutron would need a filter to not delete the static routes learned by the router (the different proposals can be complementary...) About the proposed RFE, do you see any problem if Neutron deletes from the OVN only what it manages? > > On Mon, Jul 17, 2023 at 3:10?PM Roberto Bartzen Acosta < > roberto.acosta at luizalabs.com> wrote: > >> Hi Felix and Rodolfo, >> >> >> Em seg., 17 de jul. de 2023 ?s 05:39, Rodolfo Alonso Hernandez < >> ralonsoh at redhat.com> escreveu: >> >>> Hello Felix: >>> >>> That's the point: the launchpad bug describes the registers created by >>> the IC configuration but these registers do not have any identification >>> that informs about their relationship with the IC TS. >>> >>> Rather than modifying the Neutron resources (a network is a generic >>> resource that is used by multiple backends), I would try to create a NB-IC >>> resource map and the corresponding NB registers (NAT, LRP and LRSR). If you >>> can link a NB-IC TS with the corresponding NB resources, then the problem >>> is fixed because knowing them we can explicitly skip them from the DB sync >>> tool. >>> >>> If that is not the case, then please present an alternative with the >>> suggested idea of creating an external network type. >>> >>> BTW, I don't understand why you first say that "created by tools not >>> necessarily controlled by us we need to avoid changing them", but then you >>> are talking about refactoring these tools. Are we talking about the tools >>> that create these IC configurations? Can you share them? >>> >>> >> I think we are talking about two different cases. >> >> 1 - This proposed RFE intends to solve the problem of removing external >> resources and that is enough for the OVN-IC to work in generic scenarios: >> multi tenancy, single external provider, multiple external providers, etc. >> because the OVN-IC will not relate to Neutron's network topology or >> capabilities. >> >> 2- Felix' suggestion is related to integrating the TS to a network on >> Neutron (some kind of mapping), and in this case, we have a hybrid solution >> with the TS managed by an external tool, and the management of the NAT, LRP >> / TS link with the tenant being done directly on Neutron. For this >> proposal, I believe that the port filter would not be necessary because >> Neutron would know the resource mappings in the OVN-NB, but it seems to me >> a more complex solution. >> >> The question about case 2 is related to which level we are talking about >> modifying the generic network to map the TS. If it is a network provider >> level, the discussion is one - this case would solve the >> schedule/lrp-set-gateway-chassis problem as it would be a gateway port. >> but if it is a tenant' network level linked to a router / no gateway port >> (the discussion would be another) - does not solve the gateway port >> problem, but it seems closer to what the OVN-IC proposes to do at the >> network L3 interconnect level). >> >> BTW: in case the TS is connected to an external network, we would need an >> additional router for external/internet connectivity not related to the >> OVN-IC, or maybe create a router with multiple gateway ports (that doesn't >> exist yet)... >> >> Regards, >> Roberto >> >> >> >>> Regards. >>> >>> On Mon, Jul 17, 2023 at 8:43?AM Felix Huettner >>> wrote: >>> >>>> Hi everyone, >>>> >>>> On Thu, Jul 13, 2023 at 07:29:51PM -0300, Roberto Bartzen Acosta wrote: >>>> > Hi Rodolfo, >>>> > >>>> > Em qui., 13 de jul. de 2023 ?s 13:45, Rodolfo Alonso Hernandez < >>>> > ralonsoh at redhat.com> escreveu: >>>> > >>>> > > Hello Roberto: >>>> > > >>>> > > I think you have provided a very detailed explanation of the issue >>>> but you >>>> > > still didn't present any possible solution. I would ask you at >>>> least for a >>>> > > proposal and for specific details of what is failing in the OVN >>>> sync tool. >>>> > > >>>> > > For example, if you add a TS between two clouds, you'll need to >>>> create (1) >>>> > > routers (NAT registers), (2) router ports (LRP) and (3) some static >>>> routes >>>> > > (LRSR). All these elements are monitored by the DB sync tool and >>>> will fail >>>> > > because the counterparts in the Neutron DB don't exist. I guess >>>> that you >>>> > > have some script or tool or at least a document to manually add >>>> these >>>> > > resources; sharing it could help to find a solution. >>>> > > >>>> > > If these elements are manually created during the deployment phase, >>>> you >>>> > > can also control the information provided. I'm thinking about the >>>> > > "external_ids" field; we can push some defined constant that will >>>> make >>>> > > these registers transparent to the OVN DB sync tool. >>>> >>>> As these resources are also created by tools not necessarily controlled >>>> by us we need to avoid changing them. >>>> I would therefor propose to not add a "ignore" constant to the >>>> "external_ids", to to rather check for the existence of the neutron keys >>>> in "external_ids" (i think thats also what is described in the bug >>>> below). >>>> >>>> >> Thanks for the contribution Felix. The idea is not to add (or at least >> not expect it to be externally added) any identification in the >> external_ids, Neutron should only filter for DB consistency validation what >> it knows and has created by Neuron -> Neutron in the external ids key. >> >> >>> > > >>>> > > Please check this idea and if it is feasible. In that case, please >>>> add a >>>> > > topic to the Neutron drivers meeting agenda [1] to discuss it. >>>> > > >>>> > >>>> > That sounds good, thanks. >>>> > https://bugs.launchpad.net/neutron/+bug/2027742 >>>> >>>> From my perspective there is also an additional step after the above >>>> REF. Currently it just makes it possible to use ovn-ic without neutron >>>> killing it. >>>> However if we could get neutron to treat a Transit Switch as a external >>>> network we would skip the need to implement the NAT, LRP and LRSR logic >>>> in custom tooling and could use the existing neutron code. This would >>>> probably require creating a custom type of provider network and then >>>> specifying the transit switch name in the provider options. >>>> >>>> Is that something that from your perspective we should take a look at >>>> right away, or rather first focus und neutron not burning the transit >>>> switch. >>>> >>>> Regards >>>> Felix >>>> >>>> > >>>> > Regards, >>>> > Roberto >>>> > >>>> > >>>> > > >>>> > > Regards. >>>> > > >>>> > > [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers >>>> > > >>>> > > >>>> > > On Wed, Jul 12, 2023 at 2:12?PM Roberto Bartzen Acosta < >>>> > > roberto.acosta at luizalabs.com> wrote: >>>> > > >>>> > >> >>>> > >> >>>> > >> Em qua., 12 de jul. de 2023 ?s 09:05, Roberto Bartzen Acosta < >>>> > >> roberto.acosta at luizalabs.com> escreveu: >>>> > >> >>>> > >>> Hi Rodolfo, >>>> > >>> >>>> > >>> Thanks for the feedback. >>>> > >>> >>>> > >>> I liked the idea of having a different network type only to hold >>>> the >>>> > >>> Transit Switches and thus the LRP but it depends on some factors >>>> like >>>> > >>> multi-tenancy. >>>> > >>> From the OVN perspective, the TS is global and will only be >>>> linked to a >>>> > >>> tenant when it is plugged into the tenant router port. My concern >>>> here is >>>> > >>> that if Neutron manages the TS, it will need to assume the dynamic >>>> > >>> characteristics of this TS function, ovn-ic creates and removes >>>> the TS in >>>> > >>> NB_Global, in addition to plugging and removing ports in this >>>> switch >>>> > >>> according to the network topology. Additionally, Neutron would >>>> still need >>>> > >>> to handle the learned remote routes (which are listed as static >>>> routes from >>>> > >>> the tenant router perspective). >>>> > >>> >>>> > >> sorry: NB_Global -> Northbound Database. >>>> > >> >>>> > >> >>>> > >>> This is an open discussion, Felix can add ideas here. >>>> > >>> >>>> > >>> In general, it seems to me that we have no alternatives for this >>>> type of >>>> > >>> solution other than OVN-IC (note that ovn-bgp-agent does not >>>> learn remote >>>> > >>> routes at the SDN level / OVN). >>>> > >>> OpenStack seems to be designed to run like a "bubble" and this >>>> traffic >>>> > >>> from one VM to another always needs to be routed at the FIP level >>>> and >>>> > >>> external routing layers. >>>> > >>> >>>> > >>> Regards, >>>> > >>> Roberto >>>> > >>> >>>> > >>> Em qua., 12 de jul. de 2023 ?s 05:11, Rodolfo Alonso Hernandez < >>>> > >>> ralonsoh at redhat.com> escreveu: >>>> > >>> >>>> > >>>> Hello Roberto: >>>> > >>>> >>>> > >>>> We talked about this possible RFE during the PTG [1] but I don't >>>> > >>>> remember having any proposal. Actually what I remember (and was >>>> written in >>>> > >>>> the etherpad) is that you were going to present an RFE. Can you >>>> explain it >>>> > >>>> briefly? >>>> > >>>> >>>> > >>>> We also talked about the idea, proposed by Felix in the mailing >>>> list, >>>> > >>>> of having a different network type only to hold the Transit >>>> Switches and >>>> > >>>> thus the LRP. If you have a proposal, please present it. >>>> > >>>> >>>> > >>>> Regards. >>>> > >>>> >>>> > >>>> [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506 >>>> > >>>> >>>> > >>>> On Tue, Jul 11, 2023 at 10:50?PM Roberto Bartzen Acosta < >>>> > >>>> roberto.acosta at luizalabs.com> wrote: >>>> > >>>> >>>> > >>>>> Hello Neutron folks, >>>> > >>>>> >>>> > >>>>> Regarding the conversation started in March [1] about the use >>>> of OVN >>>> > >>>>> interconnect with Neutron, I would like to evolve the >>>> discussion in >>>> > >>>>> relation to resources allocated by OVN-IC and which are not >>>> managed by >>>> > >>>>> Neutron. In the March PTG [2], the problem related to the >>>> db_sync tool was >>>> > >>>>> presented, and a proposed solution in which Neutron does not >>>> manage these >>>> > >>>>> resources. >>>> > >>>>> >>>> > >>>>> After discussing some architectural designs with Felix/StackIT, >>>> it >>>> > >>>>> seems to make sense to come up with a proposal to change the >>>> mech_driver to >>>> > >>>>> validate the external_ids key and not remove resources present >>>> in the OVN >>>> > >>>>> backend without Neutron "signature". >>>> > >>>>> >>>> > >>>>> The discussion here is more complex than it seems, because >>>> > >>>>> architecturally Neutron does not allow us to interconnect >>>> workloads in >>>> > >>>>> multiple AZs (with different OpenStacks), but this could be a >>>> requirement >>>> > >>>>> for a high availability cloud solution (Cloud Region). >>>> Additionally, this >>>> > >>>>> OVN-IC solution allows interconnecting other cloud backends >>>> that use OVN, >>>> > >>>>> in the case of kubernetes with ovn-kube. >>>> > >>>>> >>>> > >>>>> We tested an OVN interconnect integrated with 3 OpenStack >>>> > >>>>> installations and it worked very well !!! considering direct L3 >>>> traffic at >>>> > >>>>> the router level between different network infrastructures. >>>> > >>>>> >>>> > >>>>> SYNC_REPAIR - problem >>>> > >>>>> >>>> > >>>>> * Static Routes (learned OVN-IC routes) >>>> > >>>>> * Router Port -> Transit Switches >>>> > >>>>> >>>> > >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>>> > >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>>> > >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>>> > >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router >>>> Port found in >>>> > >>>>> OVN but not in Neutron, port_id=rt2-admin-tenant1 >>>> > >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>>> > >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>>> > >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>>> > >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router >>>> > >>>>> 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes >>>> [{'destination': ' >>>> > >>>>> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': ' >>>> > >>>>> 10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but >>>> not in >>>> > >>>>> Neutron >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> Any suggestions on how to resolve this db_sync issue? since all >>>> other >>>> > >>>>> Neutron modules did not present any problem with OVN-IC (as far >>>> as I >>>> > >>>>> tested). >>>> > >>>>> I remember Terry was keen to suggest some things to help. I >>>> believe >>>> > >>>>> that before proposing some complex mechanism to solve this >>>> simple problem, >>>> > >>>>> I would like to hear the community opinion. In our use case, a >>>> bit change >>>> > >>>>> in db_sync with filter by neutron key in external_ids would be >>>> enough. >>>> > >>>>> >>>> > >>>>> Regards, >>>> > >>>>> Roberto >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> [1] >>>> > >>>>> >>>> https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html >>>> > >>>>> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> Additional logs: >>>> > >>>>> >>>> > >>>>> OpenStack 1 >>>> > >>>>> >>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>> ovn-nbctl >>>> > >>>>> lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 >>>> > >>>>> IPv4 Routes >>>> > >>>>> Route Table
: >>>> > >>>>> 20.0.1.0/24 169.254.200.2 dst-ip >>>> (learned) >>>> > >>>>> 20.0.2.0/24 169.254.200.3 dst-ip >>>> (learned) >>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>> > >>>>> >>>> > >>>>> IPv6 Routes >>>> > >>>>> Route Table
: >>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>> ovn-nbctl >>>> > >>>>> lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 >>>> > >>>>> IPv4 Routes >>>> > >>>>> Route Table
: >>>> > >>>>> 10.0.1.0/24 169.254.100.2 dst-ip >>>> (learned) >>>> > >>>>> 10.0.2.0/24 169.254.100.3 dst-ip >>>> (learned) >>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>> > >>>>> >>>> > >>>>> IPv6 Routes >>>> > >>>>> Route Table
: >>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> OpenStack 2 >>>> > >>>>> >>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# >>>> ovn-nbctl >>>> > >>>>> lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc >>>> > >>>>> IPv4 Routes >>>> > >>>>> Route Table
: >>>> > >>>>> 20.0.0.0/24 169.254.200.1 dst-ip >>>> (learned) >>>> > >>>>> 20.0.2.0/24 169.254.200.3 dst-ip >>>> (learned) >>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>> > >>>>> >>>> > >>>>> IPv6 Routes >>>> > >>>>> Route Table
: >>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# >>>> ovn-nbctl >>>> > >>>>> lr-route-list ce45f681-6454-43fe-974f-81344bb8113a >>>> > >>>>> IPv4 Routes >>>> > >>>>> Route Table
: >>>> > >>>>> 10.0.0.0/24 169.254.100.1 dst-ip >>>> (learned) >>>> > >>>>> 10.0.2.0/24 169.254.100.3 dst-ip >>>> (learned) >>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>> > >>>>> >>>> > >>>>> IPv6 Routes >>>> > >>>>> Route Table
: >>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> OpenStack 3 >>>> > >>>>> >>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >>>> ovn-nbctl >>>> > >>>>> lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 >>>> > >>>>> IPv4 Routes >>>> > >>>>> Route Table
: >>>> > >>>>> 20.0.0.0/24 169.254.200.1 dst-ip >>>> (learned) >>>> > >>>>> 20.0.1.0/24 169.254.200.2 dst-ip >>>> (learned) >>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>> > >>>>> >>>> > >>>>> IPv6 Routes >>>> > >>>>> Route Table
: >>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >>>> ovn-nbctl >>>> > >>>>> lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 >>>> > >>>>> IPv4 Routes >>>> > >>>>> Route Table
: >>>> > >>>>> 10.0.0.0/24 169.254.100.1 dst-ip >>>> (learned) >>>> > >>>>> 10.0.1.0/24 169.254.100.2 dst-ip >>>> (learned) >>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>> > >>>>> >>>> > >>>>> IPv6 Routes >>>> > >>>>> Route Table
: >>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> OVN-IC Global database >>>> > >>>>> >>>> > >>>>> root at ovn-global-db1:~# ovn-ic-sbctl show >>>> > >>>>> availability-zone osp1 >>>> > >>>>> gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 >>>> > >>>>> hostname: osp1-gwnode1 >>>> > >>>>> type: geneve >>>> > >>>>> ip: 192.168.200.28 >>>> > >>>>> port admin-rt1-tenant1 >>>> > >>>>> transit switch: admin-tenant1 >>>> > >>>>> address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 >>>> fe80::1/64"] >>>> > >>>>> port admin-rt1-tenant1_1 >>>> > >>>>> transit switch: admin-tenant1_1 >>>> > >>>>> address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] >>>> > >>>>> availability-zone osp2 >>>> > >>>>> gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 >>>> > >>>>> hostname: osp2-gwnode1 >>>> > >>>>> type: geneve >>>> > >>>>> ip: 192.168.200.128 >>>> > >>>>> port admin-rt2-tenant1 >>>> > >>>>> transit switch: admin-tenant1 >>>> > >>>>> address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 >>>> fe80::2/64"] >>>> > >>>>> port admin-rt2-tenant1_1 >>>> > >>>>> transit switch: admin-tenant1_1 >>>> > >>>>> address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] >>>> > >>>>> availability-zone osp3 >>>> > >>>>> gateway 97595af9-7896-40d0-a883-beadbff1aa5b >>>> > >>>>> hostname: osp3-gwnode1 >>>> > >>>>> type: geneve >>>> > >>>>> ip: 192.168.200.228 >>>> > >>>>> port admin-rt3-tenant1 >>>> > >>>>> transit switch: admin-tenant1 >>>> > >>>>> address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 >>>> fe80::3/64"] >>>> > >>>>> port admin-rt3-tenant1_1 >>>> > >>>>> transit switch: admin-tenant1_1 >>>> > >>>>> address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> *?Esta mensagem ? direcionada apenas para os endere?os >>>> constantes no >>>> > >>>>> cabe?alho inicial. Se voc? n?o est? listado nos endere?os >>>> constantes no >>>> > >>>>> cabe?alho, pedimos-lhe que desconsidere completamente o >>>> conte?do dessa >>>> > >>>>> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es >>>> citadas est?o >>>> > >>>>> imediatamente anuladas e proibidas?.* >>>> > >>>>> >>>> > >>>>> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis >>>> para >>>> > >>>>> assegurar que nenhum v?rus esteja presente nesse e-mail, a >>>> empresa n?o >>>> > >>>>> poder? aceitar a responsabilidade por quaisquer perdas ou danos >>>> causados >>>> > >>>>> por esse e-mail ou por seus anexos?.* >>>> > >>>>> >>>> > >>>> >>>> > >> >>>> > >> *?Esta mensagem ? direcionada apenas para os endere?os constantes >>>> no >>>> > >> cabe?alho inicial. Se voc? n?o est? listado nos endere?os >>>> constantes no >>>> > >> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do >>>> dessa >>>> > >> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es >>>> citadas est?o >>>> > >> imediatamente anuladas e proibidas?.* >>>> > >> >>>> > >> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis >>>> para >>>> > >> assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa >>>> n?o >>>> > >> poder? aceitar a responsabilidade por quaisquer perdas ou danos >>>> causados >>>> > >> por esse e-mail ou por seus anexos?.* >>>> > >> >>>> > > >>>> > >>>> > -- >>>> > >>>> > >>>> > >>>> > >>>> > _?Esta mensagem ? direcionada apenas para os endere?os constantes no >>>> > cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes >>>> no >>>> > cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa >>>> > mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas >>>> est?o >>>> > imediatamente anuladas e proibidas?._ >>>> > >>>> > >>>> > * **?Apesar do Magazine Luiza tomar >>>> > todas as precau??es razo?veis para assegurar que nenhum v?rus esteja >>>> > presente nesse e-mail, a empresa n?o poder? aceitar a >>>> responsabilidade por >>>> > quaisquer perdas ou danos causados por esse e-mail ou por seus >>>> anexos?.* >>>> > >>>> > >>>> > >>>> Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur >>>> f?r die Verwertung durch den vorgesehenen Empf?nger bestimmt. >>>> Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den >>>> Absender bitte unverz?glich in Kenntnis und l?schen diese E Mail. >>>> >>>> Hinweise zum Datenschutz finden Sie hier< >>>> https://www.datenschutz.schwarz>. >>>> >>>> >>>> This e-mail may contain confidential content and is intended only for >>>> the specified recipient/s. >>>> If you are not the intended recipient, please inform the sender >>>> immediately and delete this e-mail. >>>> >>>> Information on data protection can be found here< >>>> https://www.datenschutz.schwarz>. >>>> >>>> >> >> *?Esta mensagem ? direcionada apenas para os endere?os constantes no >> cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no >> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa >> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o >> imediatamente anuladas e proibidas?.* >> >> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para >> assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o >> poder? aceitar a responsabilidade por quaisquer perdas ou danos causados >> por esse e-mail ou por seus anexos?.* >> > -- _?Esta mensagem ? direcionada apenas para os endere?os constantes no cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o imediatamente anuladas e proibidas?._ *?**?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o poder? aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos?.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From pierre at stackhpc.com Mon Jul 17 20:10:30 2023 From: pierre at stackhpc.com (Pierre Riteau) Date: Mon, 17 Jul 2023 22:10:30 +0200 Subject: [cloudkitty] question to the scope_key in cloudkitty In-Reply-To: References: Message-ID: Hello Noah, Are you still interested in contributing your code change? It would be really useful when using metrics from different Prometheus exporters which may not be using the same labels. The contributor docs are here: https://docs.openstack.org/cloudkitty/latest/contributor/contributing.html Thanks, Pierre Riteau (priteau) On Mon, 20 Feb 2023 at 18:55, Rafael Weing?rtner < rafaelweingartner at gmail.com> wrote: > Hello Noah, > Even though we do not see a use case for the feature you described, you > seem to have a solid use case. Other approaches could have been used to > "rename"/"map" tenant_id to project_id, such as with Dynamic pollsters in > Ceilometer. However, the argument you present seems solid to me. > > I would say that if you open a patch, we would happily review it :). AS > long as things are backwards compatible, tested, and well documented, we > would for sure merge it. > > On Mon, Feb 20, 2023 at 10:39 AM Noah Elias Feldt > wrote: > >> Hello! I am currently trying to get CloudKitty to work with the >> Prometheus Collector and PyScripts. I noticed that the scope_key for >> Prometheus can only be changed once in the global configuration file. This >> was a problem for me because we need different scope_keys for different >> metrics. Many of the metrics still need the scope key "tenant_id" and some >> others "project_id". I have found in the documentation of CloudKitty no way >> to change this for each metric, so I have internally changed something in >> the source code and added the function to set the scope_key for each metric >> individually. I just wanted to ask if you would be interested if I submit >> the small change upstream? >> Thank you. Regards >> >> Noah Feldt >> >> > > -- > Rafael Weing?rtner > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.acosta at luizalabs.com Mon Jul 17 18:04:08 2023 From: roberto.acosta at luizalabs.com (Roberto Bartzen Acosta) Date: Mon, 17 Jul 2023 15:04:08 -0300 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: Em seg., 17 de jul. de 2023 ?s 13:33, Rodolfo Alonso Hernandez < ralonsoh at redhat.com> escreveu: > > In my opinion the answer would be: yes and no, this approach would work > for isolated cases, but think on a larger scale: how many external networks > would need to be managed at the IP Fabric level? > This is exactly what you are proposing in your previous mails: "Felix' > suggestion is related to integrating the TS to a network on Neutron (some > kind of mapping)". So at this point I'm curious about your proposal. And > about the static routes, if these routes are created from a Neutron defined > router, as I proposed, they won't be deleted. > The DB sync calls this method [1] to provide a list of the static routes related to some router (Neutron defined router), and then check if this list is consistent with the Neutron static route database [2]. This route will not exist in Neutron (it's a learned route), if this is not made flexible, the OVN-IC cannot be integrated with Neutron because those routes will be deleted. This behavior extends to any other resource dynamically learned by OVN / OVN-IC solution. [1] https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py#L344 [2] https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py#L348 > > > About the proposed RFE, do you see any problem if Neutron deletes from > the OVN only what it manages? > Yes, a huge one: OVN is a service for Neutron, same as RabbitMQ or MySQL. > These services are 100% controlled by Neutron because the resources can > clash. You can't manually create an IPAM register in the DB because that > will disable one IP address. The same for OVN: you can't manually create a > LRP in a Logical Switch that provides a Neutron network because this LRP > will have an IP address. If a new Neutron port is created and it uses the > same IP address of the existing LRP (that doesn't exist in Neutron), the > new port will be considered virtual and won't be able to be used in a VM > (this is an actual problem I found in a customer last week). So no, Neutron > should not allow other resources in OVN apart from what Neutron creates. > I understand and it makes sense. However, what would be the options? no other existing solution reinjects routes at the SDN level. The main reason to use OVN-IC is for the router to "learn" the remote routes, if that is not possible from a Neutron perspective and we have to create static route entries manually (via OpenStack) maybe this makes the OVN-IC solution unfeasible. > > On Mon, Jul 17, 2023 at 6:13?PM Roberto Bartzen Acosta < > roberto.acosta at luizalabs.com> wrote: > >> Hi Rodolfo, >> >> Em seg., 17 de jul. de 2023 ?s 11:30, Rodolfo Alonso Hernandez < >> ralonsoh at redhat.com> escreveu: >> >>> Ok, I'll ask again for the tool/script/manual used to make the IC >>> connection. I will also ask again for the NB-IC database resources and the >>> NB resources needed, and how are related. >>> >> >> To be clear, this tool doesn't exist yet and we are working on the design >> of the software solution [1]. For now, the tests were performed using >> manual commands, such as those described in the RFE link and very similar >> to the case applied by the ovn-kube to use OVN-IC with OpenStack - Neutron >> [2]. >> The main idea is to develop an external layer interface with an API/SDK >> and an internal layer interface with the OVN-IC-NB global database and the >> OpenStack/AZ OVN-NB database [1]. >> >> [1] >> https://drive.google.com/file/d/1yx66YcxNMH_oB11--0-_0eZzmBJHu-Ox/view?usp=sharing >> [2] >> https://github.com/kubeovn/kube-ovn/blob/v1.11.0/docs/cluster-interconnection.md >> >> >>> If you want Neutron to handle these resources, then you don't need >>> anything special: you can add an external network, handled only by one >>> project. Inside this project you can create the needed resources: networks, >>> routers, etc. This project/user (controlled by the sysadmin), won't create >>> any VM on these networks/routers. Then you can use these resources to >>> connect the TS. >>> >> >> In my opinion the answer would be: yes and no, this approach would work >> for isolated cases, but think on a larger scale: how many external networks >> would need to be managed at the IP Fabric level? what is the service type? >> vlan (we would have segmentation limits) vxlan (we would have to extend the >> L2 domain across the entire IP fabric) virtual? . I don't know, there are >> many questions here. Felix can explain better and propose an RFE in the >> future. Even so, Neutron would need a filter to not delete the static >> routes learned by the router (the different proposals can be >> complementary...) >> >> >> About the proposed RFE, do you see any problem if Neutron deletes from >> the OVN only what it manages? >> >> >>> >>> On Mon, Jul 17, 2023 at 3:10?PM Roberto Bartzen Acosta < >>> roberto.acosta at luizalabs.com> wrote: >>> >>>> Hi Felix and Rodolfo, >>>> >>>> >>>> Em seg., 17 de jul. de 2023 ?s 05:39, Rodolfo Alonso Hernandez < >>>> ralonsoh at redhat.com> escreveu: >>>> >>>>> Hello Felix: >>>>> >>>>> That's the point: the launchpad bug describes the registers created by >>>>> the IC configuration but these registers do not have any identification >>>>> that informs about their relationship with the IC TS. >>>>> >>>>> Rather than modifying the Neutron resources (a network is a generic >>>>> resource that is used by multiple backends), I would try to create a NB-IC >>>>> resource map and the corresponding NB registers (NAT, LRP and LRSR). If you >>>>> can link a NB-IC TS with the corresponding NB resources, then the problem >>>>> is fixed because knowing them we can explicitly skip them from the DB sync >>>>> tool. >>>>> >>>>> If that is not the case, then please present an alternative with the >>>>> suggested idea of creating an external network type. >>>>> >>>>> BTW, I don't understand why you first say that "created by tools not >>>>> necessarily controlled by us we need to avoid changing them", but then you >>>>> are talking about refactoring these tools. Are we talking about the tools >>>>> that create these IC configurations? Can you share them? >>>>> >>>>> >>>> I think we are talking about two different cases. >>>> >>>> 1 - This proposed RFE intends to solve the problem of removing external >>>> resources and that is enough for the OVN-IC to work in generic scenarios: >>>> multi tenancy, single external provider, multiple external providers, etc. >>>> because the OVN-IC will not relate to Neutron's network topology or >>>> capabilities. >>>> >>>> 2- Felix' suggestion is related to integrating the TS to a network on >>>> Neutron (some kind of mapping), and in this case, we have a hybrid solution >>>> with the TS managed by an external tool, and the management of the NAT, LRP >>>> / TS link with the tenant being done directly on Neutron. For this >>>> proposal, I believe that the port filter would not be necessary because >>>> Neutron would know the resource mappings in the OVN-NB, but it seems to me >>>> a more complex solution. >>>> >>>> The question about case 2 is related to which level we are talking >>>> about modifying the generic network to map the TS. If it is a network >>>> provider level, the discussion is one - this case would solve the >>>> schedule/lrp-set-gateway-chassis problem as it would be a gateway port. >>>> but if it is a tenant' network level linked to a router / no gateway >>>> port (the discussion would be another) - does not solve the gateway port >>>> problem, but it seems closer to what the OVN-IC proposes to do at the >>>> network L3 interconnect level). >>>> >>>> BTW: in case the TS is connected to an external network, we would need >>>> an additional router for external/internet connectivity not related to the >>>> OVN-IC, or maybe create a router with multiple gateway ports (that doesn't >>>> exist yet)... >>>> >>>> Regards, >>>> Roberto >>>> >>>> >>>> >>>>> Regards. >>>>> >>>>> On Mon, Jul 17, 2023 at 8:43?AM Felix Huettner >>>>> wrote: >>>>> >>>>>> Hi everyone, >>>>>> >>>>>> On Thu, Jul 13, 2023 at 07:29:51PM -0300, Roberto Bartzen Acosta >>>>>> wrote: >>>>>> > Hi Rodolfo, >>>>>> > >>>>>> > Em qui., 13 de jul. de 2023 ?s 13:45, Rodolfo Alonso Hernandez < >>>>>> > ralonsoh at redhat.com> escreveu: >>>>>> > >>>>>> > > Hello Roberto: >>>>>> > > >>>>>> > > I think you have provided a very detailed explanation of the >>>>>> issue but you >>>>>> > > still didn't present any possible solution. I would ask you at >>>>>> least for a >>>>>> > > proposal and for specific details of what is failing in the OVN >>>>>> sync tool. >>>>>> > > >>>>>> > > For example, if you add a TS between two clouds, you'll need to >>>>>> create (1) >>>>>> > > routers (NAT registers), (2) router ports (LRP) and (3) some >>>>>> static routes >>>>>> > > (LRSR). All these elements are monitored by the DB sync tool and >>>>>> will fail >>>>>> > > because the counterparts in the Neutron DB don't exist. I guess >>>>>> that you >>>>>> > > have some script or tool or at least a document to manually add >>>>>> these >>>>>> > > resources; sharing it could help to find a solution. >>>>>> > > >>>>>> > > If these elements are manually created during the deployment >>>>>> phase, you >>>>>> > > can also control the information provided. I'm thinking about the >>>>>> > > "external_ids" field; we can push some defined constant that will >>>>>> make >>>>>> > > these registers transparent to the OVN DB sync tool. >>>>>> >>>>>> As these resources are also created by tools not necessarily >>>>>> controlled >>>>>> by us we need to avoid changing them. >>>>>> I would therefor propose to not add a "ignore" constant to the >>>>>> "external_ids", to to rather check for the existence of the neutron >>>>>> keys >>>>>> in "external_ids" (i think thats also what is described in the bug >>>>>> below). >>>>>> >>>>>> >>>> Thanks for the contribution Felix. The idea is not to add (or at least >>>> not expect it to be externally added) any identification in the >>>> external_ids, Neutron should only filter for DB consistency validation what >>>> it knows and has created by Neuron -> Neutron in the external ids key. >>>> >>>> >>>>> > > >>>>>> > > Please check this idea and if it is feasible. In that case, >>>>>> please add a >>>>>> > > topic to the Neutron drivers meeting agenda [1] to discuss it. >>>>>> > > >>>>>> > >>>>>> > That sounds good, thanks. >>>>>> > https://bugs.launchpad.net/neutron/+bug/2027742 >>>>>> >>>>>> From my perspective there is also an additional step after the above >>>>>> REF. Currently it just makes it possible to use ovn-ic without neutron >>>>>> killing it. >>>>>> However if we could get neutron to treat a Transit Switch as a >>>>>> external >>>>>> network we would skip the need to implement the NAT, LRP and LRSR >>>>>> logic >>>>>> in custom tooling and could use the existing neutron code. This would >>>>>> probably require creating a custom type of provider network and then >>>>>> specifying the transit switch name in the provider options. >>>>>> >>>>>> Is that something that from your perspective we should take a look at >>>>>> right away, or rather first focus und neutron not burning the transit >>>>>> switch. >>>>>> >>>>>> Regards >>>>>> Felix >>>>>> >>>>>> > >>>>>> > Regards, >>>>>> > Roberto >>>>>> > >>>>>> > >>>>>> > > >>>>>> > > Regards. >>>>>> > > >>>>>> > > [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers >>>>>> > > >>>>>> > > >>>>>> > > On Wed, Jul 12, 2023 at 2:12?PM Roberto Bartzen Acosta < >>>>>> > > roberto.acosta at luizalabs.com> wrote: >>>>>> > > >>>>>> > >> >>>>>> > >> >>>>>> > >> Em qua., 12 de jul. de 2023 ?s 09:05, Roberto Bartzen Acosta < >>>>>> > >> roberto.acosta at luizalabs.com> escreveu: >>>>>> > >> >>>>>> > >>> Hi Rodolfo, >>>>>> > >>> >>>>>> > >>> Thanks for the feedback. >>>>>> > >>> >>>>>> > >>> I liked the idea of having a different network type only to >>>>>> hold the >>>>>> > >>> Transit Switches and thus the LRP but it depends on some >>>>>> factors like >>>>>> > >>> multi-tenancy. >>>>>> > >>> From the OVN perspective, the TS is global and will only be >>>>>> linked to a >>>>>> > >>> tenant when it is plugged into the tenant router port. My >>>>>> concern here is >>>>>> > >>> that if Neutron manages the TS, it will need to assume the >>>>>> dynamic >>>>>> > >>> characteristics of this TS function, ovn-ic creates and removes >>>>>> the TS in >>>>>> > >>> NB_Global, in addition to plugging and removing ports in this >>>>>> switch >>>>>> > >>> according to the network topology. Additionally, Neutron would >>>>>> still need >>>>>> > >>> to handle the learned remote routes (which are listed as static >>>>>> routes from >>>>>> > >>> the tenant router perspective). >>>>>> > >>> >>>>>> > >> sorry: NB_Global -> Northbound Database. >>>>>> > >> >>>>>> > >> >>>>>> > >>> This is an open discussion, Felix can add ideas here. >>>>>> > >>> >>>>>> > >>> In general, it seems to me that we have no alternatives for >>>>>> this type of >>>>>> > >>> solution other than OVN-IC (note that ovn-bgp-agent does not >>>>>> learn remote >>>>>> > >>> routes at the SDN level / OVN). >>>>>> > >>> OpenStack seems to be designed to run like a "bubble" and this >>>>>> traffic >>>>>> > >>> from one VM to another always needs to be routed at the FIP >>>>>> level and >>>>>> > >>> external routing layers. >>>>>> > >>> >>>>>> > >>> Regards, >>>>>> > >>> Roberto >>>>>> > >>> >>>>>> > >>> Em qua., 12 de jul. de 2023 ?s 05:11, Rodolfo Alonso Hernandez < >>>>>> > >>> ralonsoh at redhat.com> escreveu: >>>>>> > >>> >>>>>> > >>>> Hello Roberto: >>>>>> > >>>> >>>>>> > >>>> We talked about this possible RFE during the PTG [1] but I >>>>>> don't >>>>>> > >>>> remember having any proposal. Actually what I remember (and >>>>>> was written in >>>>>> > >>>> the etherpad) is that you were going to present an RFE. Can >>>>>> you explain it >>>>>> > >>>> briefly? >>>>>> > >>>> >>>>>> > >>>> We also talked about the idea, proposed by Felix in the >>>>>> mailing list, >>>>>> > >>>> of having a different network type only to hold the Transit >>>>>> Switches and >>>>>> > >>>> thus the LRP. If you have a proposal, please present it. >>>>>> > >>>> >>>>>> > >>>> Regards. >>>>>> > >>>> >>>>>> > >>>> [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506 >>>>>> > >>>> >>>>>> > >>>> On Tue, Jul 11, 2023 at 10:50?PM Roberto Bartzen Acosta < >>>>>> > >>>> roberto.acosta at luizalabs.com> wrote: >>>>>> > >>>> >>>>>> > >>>>> Hello Neutron folks, >>>>>> > >>>>> >>>>>> > >>>>> Regarding the conversation started in March [1] about the use >>>>>> of OVN >>>>>> > >>>>> interconnect with Neutron, I would like to evolve the >>>>>> discussion in >>>>>> > >>>>> relation to resources allocated by OVN-IC and which are not >>>>>> managed by >>>>>> > >>>>> Neutron. In the March PTG [2], the problem related to the >>>>>> db_sync tool was >>>>>> > >>>>> presented, and a proposed solution in which Neutron does not >>>>>> manage these >>>>>> > >>>>> resources. >>>>>> > >>>>> >>>>>> > >>>>> After discussing some architectural designs with >>>>>> Felix/StackIT, it >>>>>> > >>>>> seems to make sense to come up with a proposal to change the >>>>>> mech_driver to >>>>>> > >>>>> validate the external_ids key and not remove resources >>>>>> present in the OVN >>>>>> > >>>>> backend without Neutron "signature". >>>>>> > >>>>> >>>>>> > >>>>> The discussion here is more complex than it seems, because >>>>>> > >>>>> architecturally Neutron does not allow us to interconnect >>>>>> workloads in >>>>>> > >>>>> multiple AZs (with different OpenStacks), but this could be a >>>>>> requirement >>>>>> > >>>>> for a high availability cloud solution (Cloud Region). >>>>>> Additionally, this >>>>>> > >>>>> OVN-IC solution allows interconnecting other cloud backends >>>>>> that use OVN, >>>>>> > >>>>> in the case of kubernetes with ovn-kube. >>>>>> > >>>>> >>>>>> > >>>>> We tested an OVN interconnect integrated with 3 OpenStack >>>>>> > >>>>> installations and it worked very well !!! considering direct >>>>>> L3 traffic at >>>>>> > >>>>> the router level between different network infrastructures. >>>>>> > >>>>> >>>>>> > >>>>> SYNC_REPAIR - problem >>>>>> > >>>>> >>>>>> > >>>>> * Static Routes (learned OVN-IC routes) >>>>>> > >>>>> * Router Port -> Transit Switches >>>>>> > >>>>> >>>>>> > >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>>>>> > >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>>>>> > >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>>>>> > >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router >>>>>> Port found in >>>>>> > >>>>> OVN but not in Neutron, port_id=rt2-admin-tenant1 >>>>>> > >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae >>>>>> > >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING >>>>>> > >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync >>>>>> > >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router >>>>>> > >>>>> 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes >>>>>> [{'destination': ' >>>>>> > >>>>> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': ' >>>>>> > >>>>> 10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but >>>>>> not in >>>>>> > >>>>> Neutron >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> Any suggestions on how to resolve this db_sync issue? since >>>>>> all other >>>>>> > >>>>> Neutron modules did not present any problem with OVN-IC (as >>>>>> far as I >>>>>> > >>>>> tested). >>>>>> > >>>>> I remember Terry was keen to suggest some things to help. I >>>>>> believe >>>>>> > >>>>> that before proposing some complex mechanism to solve this >>>>>> simple problem, >>>>>> > >>>>> I would like to hear the community opinion. In our use case, >>>>>> a bit change >>>>>> > >>>>> in db_sync with filter by neutron key in external_ids would >>>>>> be enough. >>>>>> > >>>>> >>>>>> > >>>>> Regards, >>>>>> > >>>>> Roberto >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> [1] >>>>>> > >>>>> >>>>>> https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html >>>>>> > >>>>> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> Additional logs: >>>>>> > >>>>> >>>>>> > >>>>> OpenStack 1 >>>>>> > >>>>> >>>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>>>> ovn-nbctl >>>>>> > >>>>> lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 >>>>>> > >>>>> IPv4 Routes >>>>>> > >>>>> Route Table
: >>>>>> > >>>>> 20.0.1.0/24 169.254.200.2 dst-ip >>>>>> (learned) >>>>>> > >>>>> 20.0.2.0/24 169.254.200.3 dst-ip >>>>>> (learned) >>>>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>>> > >>>>> >>>>>> > >>>>> IPv6 Routes >>>>>> > >>>>> Route Table
: >>>>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>>>> ovn-nbctl >>>>>> > >>>>> lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 >>>>>> > >>>>> IPv4 Routes >>>>>> > >>>>> Route Table
: >>>>>> > >>>>> 10.0.1.0/24 169.254.100.2 dst-ip >>>>>> (learned) >>>>>> > >>>>> 10.0.2.0/24 169.254.100.3 dst-ip >>>>>> (learned) >>>>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>>> > >>>>> >>>>>> > >>>>> IPv6 Routes >>>>>> > >>>>> Route Table
: >>>>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~# >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> OpenStack 2 >>>>>> > >>>>> >>>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# >>>>>> ovn-nbctl >>>>>> > >>>>> lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc >>>>>> > >>>>> IPv4 Routes >>>>>> > >>>>> Route Table
: >>>>>> > >>>>> 20.0.0.0/24 169.254.200.1 dst-ip >>>>>> (learned) >>>>>> > >>>>> 20.0.2.0/24 169.254.200.3 dst-ip >>>>>> (learned) >>>>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>>> > >>>>> >>>>>> > >>>>> IPv6 Routes >>>>>> > >>>>> Route Table
: >>>>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~# >>>>>> ovn-nbctl >>>>>> > >>>>> lr-route-list ce45f681-6454-43fe-974f-81344bb8113a >>>>>> > >>>>> IPv4 Routes >>>>>> > >>>>> Route Table
: >>>>>> > >>>>> 10.0.0.0/24 169.254.100.1 dst-ip >>>>>> (learned) >>>>>> > >>>>> 10.0.2.0/24 169.254.100.3 dst-ip >>>>>> (learned) >>>>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>>> > >>>>> >>>>>> > >>>>> IPv6 Routes >>>>>> > >>>>> Route Table
: >>>>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> OpenStack 3 >>>>>> > >>>>> >>>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >>>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >>>>>> ovn-nbctl >>>>>> > >>>>> lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 >>>>>> > >>>>> IPv4 Routes >>>>>> > >>>>> Route Table
: >>>>>> > >>>>> 20.0.0.0/24 169.254.200.1 dst-ip >>>>>> (learned) >>>>>> > >>>>> 20.0.1.0/24 169.254.200.2 dst-ip >>>>>> (learned) >>>>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>>> > >>>>> >>>>>> > >>>>> IPv6 Routes >>>>>> > >>>>> Route Table
: >>>>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~# >>>>>> ovn-nbctl >>>>>> > >>>>> lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 >>>>>> > >>>>> IPv4 Routes >>>>>> > >>>>> Route Table
: >>>>>> > >>>>> 10.0.0.0/24 169.254.100.1 dst-ip >>>>>> (learned) >>>>>> > >>>>> 10.0.1.0/24 169.254.100.2 dst-ip >>>>>> (learned) >>>>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip >>>>>> > >>>>> >>>>>> > >>>>> IPv6 Routes >>>>>> > >>>>> Route Table
: >>>>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> OVN-IC Global database >>>>>> > >>>>> >>>>>> > >>>>> root at ovn-global-db1:~# ovn-ic-sbctl show >>>>>> > >>>>> availability-zone osp1 >>>>>> > >>>>> gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 >>>>>> > >>>>> hostname: osp1-gwnode1 >>>>>> > >>>>> type: geneve >>>>>> > >>>>> ip: 192.168.200.28 >>>>>> > >>>>> port admin-rt1-tenant1 >>>>>> > >>>>> transit switch: admin-tenant1 >>>>>> > >>>>> address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 >>>>>> fe80::1/64"] >>>>>> > >>>>> port admin-rt1-tenant1_1 >>>>>> > >>>>> transit switch: admin-tenant1_1 >>>>>> > >>>>> address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] >>>>>> > >>>>> availability-zone osp2 >>>>>> > >>>>> gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 >>>>>> > >>>>> hostname: osp2-gwnode1 >>>>>> > >>>>> type: geneve >>>>>> > >>>>> ip: 192.168.200.128 >>>>>> > >>>>> port admin-rt2-tenant1 >>>>>> > >>>>> transit switch: admin-tenant1 >>>>>> > >>>>> address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 >>>>>> fe80::2/64"] >>>>>> > >>>>> port admin-rt2-tenant1_1 >>>>>> > >>>>> transit switch: admin-tenant1_1 >>>>>> > >>>>> address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] >>>>>> > >>>>> availability-zone osp3 >>>>>> > >>>>> gateway 97595af9-7896-40d0-a883-beadbff1aa5b >>>>>> > >>>>> hostname: osp3-gwnode1 >>>>>> > >>>>> type: geneve >>>>>> > >>>>> ip: 192.168.200.228 >>>>>> > >>>>> port admin-rt3-tenant1 >>>>>> > >>>>> transit switch: admin-tenant1 >>>>>> > >>>>> address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 >>>>>> fe80::3/64"] >>>>>> > >>>>> port admin-rt3-tenant1_1 >>>>>> > >>>>> transit switch: admin-tenant1_1 >>>>>> > >>>>> address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> >>>>>> > >>>>> *?Esta mensagem ? direcionada apenas para os endere?os >>>>>> constantes no >>>>>> > >>>>> cabe?alho inicial. Se voc? n?o est? listado nos endere?os >>>>>> constantes no >>>>>> > >>>>> cabe?alho, pedimos-lhe que desconsidere completamente o >>>>>> conte?do dessa >>>>>> > >>>>> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es >>>>>> citadas est?o >>>>>> > >>>>> imediatamente anuladas e proibidas?.* >>>>>> > >>>>> >>>>>> > >>>>> *?Apesar do Magazine Luiza tomar todas as precau??es >>>>>> razo?veis para >>>>>> > >>>>> assegurar que nenhum v?rus esteja presente nesse e-mail, a >>>>>> empresa n?o >>>>>> > >>>>> poder? aceitar a responsabilidade por quaisquer perdas ou >>>>>> danos causados >>>>>> > >>>>> por esse e-mail ou por seus anexos?.* >>>>>> > >>>>> >>>>>> > >>>> >>>>>> > >> >>>>>> > >> *?Esta mensagem ? direcionada apenas para os endere?os >>>>>> constantes no >>>>>> > >> cabe?alho inicial. Se voc? n?o est? listado nos endere?os >>>>>> constantes no >>>>>> > >> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do >>>>>> dessa >>>>>> > >> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es >>>>>> citadas est?o >>>>>> > >> imediatamente anuladas e proibidas?.* >>>>>> > >> >>>>>> > >> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis >>>>>> para >>>>>> > >> assegurar que nenhum v?rus esteja presente nesse e-mail, a >>>>>> empresa n?o >>>>>> > >> poder? aceitar a responsabilidade por quaisquer perdas ou danos >>>>>> causados >>>>>> > >> por esse e-mail ou por seus anexos?.* >>>>>> > >> >>>>>> > > >>>>>> > >>>>>> > -- >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > _?Esta mensagem ? direcionada apenas para os endere?os constantes no >>>>>> > cabe?alho inicial. Se voc? n?o est? listado nos endere?os >>>>>> constantes no >>>>>> > cabe?alho, pedimos-lhe que desconsidere completamente o conte?do >>>>>> dessa >>>>>> > mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es >>>>>> citadas est?o >>>>>> > imediatamente anuladas e proibidas?._ >>>>>> > >>>>>> > >>>>>> > * **?Apesar do Magazine Luiza tomar >>>>>> > todas as precau??es razo?veis para assegurar que nenhum v?rus esteja >>>>>> > presente nesse e-mail, a empresa n?o poder? aceitar a >>>>>> responsabilidade por >>>>>> > quaisquer perdas ou danos causados por esse e-mail ou por seus >>>>>> anexos?.* >>>>>> > >>>>>> > >>>>>> > >>>>>> Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur >>>>>> f?r die Verwertung durch den vorgesehenen Empf?nger bestimmt. >>>>>> Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den >>>>>> Absender bitte unverz?glich in Kenntnis und l?schen diese E Mail. >>>>>> >>>>>> Hinweise zum Datenschutz finden Sie hier< >>>>>> https://www.datenschutz.schwarz>. >>>>>> >>>>>> >>>>>> This e-mail may contain confidential content and is intended only for >>>>>> the specified recipient/s. >>>>>> If you are not the intended recipient, please inform the sender >>>>>> immediately and delete this e-mail. >>>>>> >>>>>> Information on data protection can be found here< >>>>>> https://www.datenschutz.schwarz>. >>>>>> >>>>>> >>>> >>>> *?Esta mensagem ? direcionada apenas para os endere?os constantes no >>>> cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no >>>> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa >>>> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o >>>> imediatamente anuladas e proibidas?.* >>>> >>>> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para >>>> assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o >>>> poder? aceitar a responsabilidade por quaisquer perdas ou danos causados >>>> por esse e-mail ou por seus anexos?.* >>>> >>> >> >> *?Esta mensagem ? direcionada apenas para os endere?os constantes no >> cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no >> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa >> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o >> imediatamente anuladas e proibidas?.* >> >> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para >> assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o >> poder? aceitar a responsabilidade por quaisquer perdas ou danos causados >> por esse e-mail ou por seus anexos?.* >> > -- _?Esta mensagem ? direcionada apenas para os endere?os constantes no cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o imediatamente anuladas e proibidas?._ *?**?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o poder? aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos?.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From nguyenhuukhoinw at gmail.com Tue Jul 18 05:07:12 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Tue, 18 Jul 2023 12:07:12 +0700 Subject: [openstack][largescale-sig] Openstack multi region deployment Message-ID: Hello guys, I am going to deploy openstack multi regions and I know that keystone replication is the most challenging. I plan to set up 2 regions which use centralize galera cluster(3 nodes). and one standby edge galera cluster(3 nodes) When region 1 is node available, I will map region 2 to use standby edge galera cluster. I hope you give me some experience and advice with multi regions. Thank you very much. -------------- next part -------------- An HTML attachment was scrubbed... URL: From felix.huettner at mail.schwarz Tue Jul 18 07:34:35 2023 From: felix.huettner at mail.schwarz (Felix Huettner) Date: Tue, 18 Jul 2023 09:34:35 +0200 Subject: [openstack][largescale-sig] Openstack multi region deployment In-Reply-To: References: Message-ID: Hi, i think you have two options here: 1. you could span a single galera cluster over all of your regions. this might have some latency issues, but if your are not too write heavy that might be fine. I know some companies use that setup. 2. you use some kind of multiple galera clusters with replication. But i have not yet heard of anybody using this setup. An alternative might be to have separate keystone setups with separate databases. This would probably reduce the error potential, but might not fit your usecase. Thanks Felix On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote: > Hello guys, > > I am going to deploy openstack multi regions and I know that keystone > replication is the most challenging. > > I plan to set up 2 regions which use centralize galera cluster(3 nodes). > and one standby edge galera cluster(3 nodes) > > When region 1 is node available, I will map region 2 to use standby edge > galera cluster. > > I hope you give me some experience and advice with multi regions. > > Thank you very much. Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r die Verwertung durch den vorgesehenen Empf?nger bestimmt. Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender bitte unverz?glich in Kenntnis und l?schen diese E Mail. Hinweise zum Datenschutz finden Sie hier. This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail. Information on data protection can be found here. From nguyenhuukhoinw at gmail.com Tue Jul 18 08:36:11 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Tue, 18 Jul 2023 15:36:11 +0700 Subject: [openstack][largescale-sig] Openstack multi region deployment In-Reply-To: References: Message-ID: Hi. Thank you for your reply. The first one has a problem because each region is too soft. If a member is down, so this region is gone. It is so challenge with us. Nguyen Huu Khoi On Tue, Jul 18, 2023 at 2:34?PM Felix Huettner wrote: > Hi, > > i think you have two options here: > 1. you could span a single galera cluster over all of your regions. > this might have some latency issues, but if your are not too write > heavy that might be fine. I know some companies use that setup. > 2. you use some kind of multiple galera clusters with replication. > But i have not yet heard of anybody using this setup. > > An alternative might be to have separate keystone setups with separate > databases. This would probably reduce the error potential, but might not > fit your usecase. > > Thanks > Felix > > > On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote: > > Hello guys, > > > > I am going to deploy openstack multi regions and I know that keystone > > replication is the most challenging. > > > > I plan to set up 2 regions which use centralize galera cluster(3 nodes). > > and one standby edge galera cluster(3 nodes) > > > > When region 1 is node available, I will map region 2 to use standby edge > > galera cluster. > > > > I hope you give me some experience and advice with multi regions. > > > > Thank you very much. > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r > die Verwertung durch den vorgesehenen Empf?nger bestimmt. > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender > bitte unverz?glich in Kenntnis und l?schen diese E Mail. > > Hinweise zum Datenschutz finden Sie hier. > > > This e-mail may contain confidential content and is intended only for the > specified recipient/s. > If you are not the intended recipient, please inform the sender > immediately and delete this e-mail. > > Information on data protection can be found here< > https://www.datenschutz.schwarz>. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralonsoh at redhat.com Tue Jul 18 11:23:27 2023 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Tue, 18 Jul 2023 13:23:27 +0200 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: Ok, this is being tortuous. First of all: define a strategy. If you are going to create the resources in Neutron, define how. I've provided a way to do this, find a formal strategy to ground it. Second: (again) try to find a connection between resources, if you are going to use the strategy of creating the resources in Neutron. The "Logical_Router_Static_Route" belongs to a router univocally. If that router has been created by OpenStack, then you can modify the DB sync method to consider learned routes too. In order to do this, you'll need a set of resources that are going to be needed in Neutron, the OVN counterparts and other resources (like "Logical_Router_Static_Route") that will be added and will be present in OVN and not in Neutron DB. Also you'll need to know how to relate all of them. -------------- next part -------------- An HTML attachment was scrubbed... URL: From felix.huettner at mail.schwarz Tue Jul 18 12:39:57 2023 From: felix.huettner at mail.schwarz (Felix Huettner) Date: Tue, 18 Jul 2023 14:39:57 +0200 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: Hi everyone, we will validate in the next weeks the idea i'll describe below. If it sounds feasible to us i would write up a spec for this for later implementation. Setup: 1. Two neutron setups with OVN 2. A OVN-IC setup not managed by any tooling (used manually) Target setup for a individual project +-------------------------------------+ | Transit Switch (ts1) | | 172.24.0.0/24 | +-------------------------------------+ | | +----------------+ +----------------+ | 172.24.0.10 | | 172.24.0.11 | | Router | | Router | | 192.168.0.1 | | 192.168.1.1 | +----------------+ +----------------+ | | +----------------+ +----------------+ | Network | | Network | | 192.168.0.0/24 | | 192.168.1.0/24 | +----------------+ +----------------+ | | +----------------+ +----------------+ | VM | | VM | | 192.168.0.10 | | 192.168.1.10 | +----------------+ +----------------+ Main creation workflow of a interconnected network: 1. Create a transit switch in the OVN-IC-NB 2. In each neutron: a. Create a new provider network that maps to the transit switch. E.g. using `--provider-network-type ovn-ic --provider-phyiscal-network ts1` b. Create the internal network and router as always and attach the router to the internal network and the provider network Changes needed in Neutron (from my perspective): 1. Neutron must not touch the resources created by OVN-IC. This might be done using two options: a. Ignore all resources where `external_ids` does not contain any key starting with `neutron`. b. Ignore specific resources as defined by individual rules: I. ignore all Logical_Router that contain `other_config:interconn-ts` II. ignore all Logical_Router_Static_Route that contain `external_ids:ic-learned-route` 2. Neutron must allow OVN-IC transit switches to be treated as external networks. AFAIK this should work in the same way as creating any kind of OVN network (so Logical_Switch), but just importing an existing Logical_Switch. >From my perspective with these two changes we should be good to interconnect two neutron setups with probably minimal changes needed. This is because OVN-IC would only touch the transit switch (which looks in the Northbound like a regular Logical_Switch). All other resources would still be managed by neutron and work as they do now. I will get back to this discussion and (if appropriate) provide a spec or this idea, once we had time to validate this on our side (which will take some weeks). Thanks Felix On Tue, Jul 18, 2023 at 01:23:27PM +0200, Rodolfo Alonso Hernandez wrote: > Ok, this is being tortuous. First of all: define a strategy. If you are > going to create the resources in Neutron, define how. I've provided a way > to do this, find a formal strategy to ground it. > > Second: (again) try to find a connection between resources, if you are > going to use the strategy of creating the resources in Neutron. The > "Logical_Router_Static_Route" belongs to a router univocally. If that > router has been created by OpenStack, then you can modify the DB sync > method to consider learned routes too. > > In order to do this, you'll need a set of resources that are going to be > needed in Neutron, the OVN counterparts and other resources (like > "Logical_Router_Static_Route") that will be added and will be present in > OVN and not in Neutron DB. Also you'll need to know how to relate all of > them. Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r die Verwertung durch den vorgesehenen Empf?nger bestimmt. Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender bitte unverz?glich in Kenntnis und l?schen diese E Mail. Hinweise zum Datenschutz finden Sie hier. This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail. Information on data protection can be found here. From murilo at evocorp.com.br Tue Jul 18 12:44:40 2023 From: murilo at evocorp.com.br (Murilo Morais) Date: Tue, 18 Jul 2023 09:44:40 -0300 Subject: [OSA][NEUTRON] Failed to bind port XXX on host YYY for vnic_type normal using segments... In-Reply-To: References: Message-ID: Hi Dmitriy, thanks for answering! I really didn't send any details of my setup, apologies for that. I'm using OVS with the following configuration: provider_networks: - network: container_bridge: br-provider container_type: veth type: vlan range: "100:200" net_name: vlan group_binds: - neutron_openvswitch_agent - network: container_bridge: br-provider container_type: veth type: flat net_name: flat group_binds: - neutron_openvswitch_agent neutron_plugin_type: ml2.ovs neutron_ml2_drivers_type: "flat,vlan" neutron_plugin_base: - router - metering root at dcn2-utility-container-c45f8b09:/# openstack network agent list +--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+ | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | +--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+ | 9a4625ef-2988-4b96-a927-30a9bb0244a4 | Metadata agent | dcn2 | None | :-) | UP | neutron-metadata-agent | | a222c0ae-c2e5-44cc-b478-ca8176daad19 | Metering agent | dcn2 | None | :-) | UP | neutron-metering-agent | | c6be1985-a67e-4099-a1d6-fa517810e138 | L3 agent | dcn2 | nova | :-) | UP | neutron-l3-agent | | da97e2a6-535b-4f1e-828d-9b2fbb3e036b | DHCP agent | dcn2 | nova | :-) | UP | neutron-dhcp-agent | +--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+ root at dcn2-utility-container-c45f8b09:/# openstack compute service list +--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+ | ID | Binary | Host | Zone | Status | State | Updated At | +--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+ | a0524af5-bc88-4793-aee2-c2c87cd0e8cc | nova-conductor | dcn2-nova-api-container-aac2b913 | internal | enabled | up | 2023-07-18T12:32:26.000000 | | c860f25e-bd30-49f9-8289-076b230bbc2d | nova-scheduler | dcn2-nova-api-container-aac2b913 | internal | enabled | up | 2023-07-18T12:32:27.000000 | | 6457e0a1-b075-4999-8855-0f36e2e3a95a | nova-compute | dcn2 | nova | enabled | up | 2023-07-18T12:32:27.000000 | +--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+ The br-provider bridge exists and is UP. -------------- next part -------------- An HTML attachment was scrubbed... URL: From kkloppenborg at rwts.com.au Tue Jul 18 13:10:00 2023 From: kkloppenborg at rwts.com.au (Karl Kloppenborg) Date: Tue, 18 Jul 2023 13:10:00 +0000 Subject: [openstack][largescale-sig] Openstack multi region deployment In-Reply-To: References: Message-ID: Hi Nguy, We?ve deployed a large multi-region openstack deployment. As a rule of thumb we?ve got a ?keystone? region which is as best we can highly available and very redundant. We then have all other regions talk back to this region, we just usually call it ?keystone? or ?core? and it?s hidden from the UI from users. We then just run a large well kept Galara cluster to support it. --Karl. From: openstack-discuss-request at lists.openstack.org Date: Tuesday, 18 July 2023 at 9:25 pm To: openstack-discuss at lists.openstack.org Subject: openstack-discuss Digest, Vol 57, Issue 55 Send openstack-discuss mailing list submissions to openstack-discuss at lists.openstack.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss or, via email, send a message with subject or body 'help' to openstack-discuss-request at lists.openstack.org You can reach the person managing the list at openstack-discuss-owner at lists.openstack.org When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..." Today's Topics: 1. [openstack][largescale-sig] Openstack multi region deployment (Nguy?n H?u Kh?i) 2. Re: [openstack][largescale-sig] Openstack multi region deployment (Felix Huettner) 3. Re: [openstack][largescale-sig] Openstack multi region deployment (Nguy?n H?u Kh?i) 4. Re: [neutron] unmanaged router resources - OVN interconnect (Rodolfo Alonso Hernandez) ---------------------------------------------------------------------- Message: 1 Date: Tue, 18 Jul 2023 12:07:12 +0700 From: Nguy?n H?u Kh?i To: OpenStack Discuss Subject: [openstack][largescale-sig] Openstack multi region deployment Message-ID: Content-Type: text/plain; charset="utf-8" Hello guys, I am going to deploy openstack multi regions and I know that keystone replication is the most challenging. I plan to set up 2 regions which use centralize galera cluster(3 nodes). and one standby edge galera cluster(3 nodes) When region 1 is node available, I will map region 2 to use standby edge galera cluster. I hope you give me some experience and advice with multi regions. Thank you very much. -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 2 Date: Tue, 18 Jul 2023 09:34:35 +0200 From: Felix Huettner To: Nguy?n H?u Kh?i Cc: OpenStack Discuss Subject: Re: [openstack][largescale-sig] Openstack multi region deployment Message-ID: Content-Type: text/plain; charset=utf-8 Hi, i think you have two options here: 1. you could span a single galera cluster over all of your regions. this might have some latency issues, but if your are not too write heavy that might be fine. I know some companies use that setup. 2. you use some kind of multiple galera clusters with replication. But i have not yet heard of anybody using this setup. An alternative might be to have separate keystone setups with separate databases. This would probably reduce the error potential, but might not fit your usecase. Thanks Felix On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote: > Hello guys, > > I am going to deploy openstack multi regions and I know that keystone > replication is the most challenging. > > I plan to set up 2 regions which use centralize galera cluster(3 nodes). > and one standby edge galera cluster(3 nodes) > > When region 1 is node available, I will map region 2 to use standby edge > galera cluster. > > I hope you give me some experience and advice with multi regions. > > Thank you very much. Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r die Verwertung durch den vorgesehenen Empf?nger bestimmt. Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender bitte unverz?glich in Kenntnis und l?schen diese E Mail. Hinweise zum Datenschutz finden Sie hier. This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail. Information on data protection can be found here. ------------------------------ Message: 3 Date: Tue, 18 Jul 2023 15:36:11 +0700 From: Nguy?n H?u Kh?i To: Nguy?n H?u Kh?i , OpenStack Discuss Subject: Re: [openstack][largescale-sig] Openstack multi region deployment Message-ID: Content-Type: text/plain; charset="utf-8" Hi. Thank you for your reply. The first one has a problem because each region is too soft. If a member is down, so this region is gone. It is so challenge with us. Nguyen Huu Khoi On Tue, Jul 18, 2023 at 2:34?PM Felix Huettner wrote: > Hi, > > i think you have two options here: > 1. you could span a single galera cluster over all of your regions. > this might have some latency issues, but if your are not too write > heavy that might be fine. I know some companies use that setup. > 2. you use some kind of multiple galera clusters with replication. > But i have not yet heard of anybody using this setup. > > An alternative might be to have separate keystone setups with separate > databases. This would probably reduce the error potential, but might not > fit your usecase. > > Thanks > Felix > > > On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote: > > Hello guys, > > > > I am going to deploy openstack multi regions and I know that keystone > > replication is the most challenging. > > > > I plan to set up 2 regions which use centralize galera cluster(3 nodes). > > and one standby edge galera cluster(3 nodes) > > > > When region 1 is node available, I will map region 2 to use standby edge > > galera cluster. > > > > I hope you give me some experience and advice with multi regions. > > > > Thank you very much. > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r > die Verwertung durch den vorgesehenen Empf?nger bestimmt. > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender > bitte unverz?glich in Kenntnis und l?schen diese E Mail. > > Hinweise zum Datenschutz finden Sie hier. > > > This e-mail may contain confidential content and is intended only for the > specified recipient/s. > If you are not the intended recipient, please inform the sender > immediately and delete this e-mail. > > Information on data protection can be found here< > https://www.datenschutz.schwarz>. > -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 4 Date: Tue, 18 Jul 2023 13:23:27 +0200 From: Rodolfo Alonso Hernandez To: Roberto Bartzen Acosta Cc: openstack-discuss , Terry Wilson , Tiago Pires Subject: Re: [neutron] unmanaged router resources - OVN interconnect Message-ID: Content-Type: text/plain; charset="utf-8" Ok, this is being tortuous. First of all: define a strategy. If you are going to create the resources in Neutron, define how. I've provided a way to do this, find a formal strategy to ground it. Second: (again) try to find a connection between resources, if you are going to use the strategy of creating the resources in Neutron. The "Logical_Router_Static_Route" belongs to a router univocally. If that router has been created by OpenStack, then you can modify the DB sync method to consider learned routes too. In order to do this, you'll need a set of resources that are going to be needed in Neutron, the OVN counterparts and other resources (like "Logical_Router_Static_Route") that will be added and will be present in OVN and not in Neutron DB. Also you'll need to know how to relate all of them. -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Subject: Digest Footer _______________________________________________ openstack-discuss mailing list openstack-discuss at lists.openstack.org ------------------------------ End of openstack-discuss Digest, Vol 57, Issue 55 ************************************************* -------------- next part -------------- An HTML attachment was scrubbed... URL: From murilo at evocorp.com.br Tue Jul 18 13:14:06 2023 From: murilo at evocorp.com.br (Murilo Morais) Date: Tue, 18 Jul 2023 10:14:06 -0300 Subject: [OSA][NEUTRON] Failed to bind port XXX on host YYY for vnic_type normal using segments... In-Reply-To: References: Message-ID: I noticed that the "neutron-openvswitch-agent" process constantly goes up to 100% in CPU usage. This is normal? Em ter., 18 de jul. de 2023 ?s 09:44, Murilo Morais escreveu: > Hi Dmitriy, thanks for answering! > > I really didn't send any details of my setup, apologies for that. > > I'm using OVS with the following configuration: > > provider_networks: > - network: > container_bridge: br-provider > container_type: veth > type: vlan > range: "100:200" > net_name: vlan > group_binds: > - neutron_openvswitch_agent > > - network: > container_bridge: br-provider > container_type: veth > type: flat > net_name: flat > group_binds: > - neutron_openvswitch_agent > > > neutron_plugin_type: ml2.ovs > neutron_ml2_drivers_type: "flat,vlan" > neutron_plugin_base: > - router > - metering > > > root at dcn2-utility-container-c45f8b09:/# openstack network agent list > > +--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+ > | ID | Agent Type | Host | > Availability Zone | Alive | State | Binary | > > +--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+ > | 9a4625ef-2988-4b96-a927-30a9bb0244a4 | Metadata agent | dcn2 | None > | :-) | UP | neutron-metadata-agent | > | a222c0ae-c2e5-44cc-b478-ca8176daad19 | Metering agent | dcn2 | None > | :-) | UP | neutron-metering-agent | > | c6be1985-a67e-4099-a1d6-fa517810e138 | L3 agent | dcn2 | nova > | :-) | UP | neutron-l3-agent | > | da97e2a6-535b-4f1e-828d-9b2fbb3e036b | DHCP agent | dcn2 | nova > | :-) | UP | neutron-dhcp-agent | > > +--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+ > > > root at dcn2-utility-container-c45f8b09:/# openstack compute service list > > +--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+ > | ID | Binary | Host > | Zone | Status | State | Updated At | > > +--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+ > | a0524af5-bc88-4793-aee2-c2c87cd0e8cc | nova-conductor | > dcn2-nova-api-container-aac2b913 | internal | enabled | up | > 2023-07-18T12:32:26.000000 | > | c860f25e-bd30-49f9-8289-076b230bbc2d | nova-scheduler | > dcn2-nova-api-container-aac2b913 | internal | enabled | up | > 2023-07-18T12:32:27.000000 | > | 6457e0a1-b075-4999-8855-0f36e2e3a95a | nova-compute | dcn2 > | nova | enabled | up | 2023-07-18T12:32:27.000000 | > > +--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+ > > > The br-provider bridge exists and is UP. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.acosta at luizalabs.com Tue Jul 18 13:20:12 2023 From: roberto.acosta at luizalabs.com (Roberto Bartzen Acosta) Date: Tue, 18 Jul 2023 10:20:12 -0300 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: Hi Rodolfo, I agree with you, my initial idea with this thread was exactly what we are doing, debating problems and possible solutions for interconnecting workloads using unmanaged dynamic router resources. Thank you for being engaged with this! I'm sorry if the strategy wasn't presented clearly, let me talk about it then. Imagine we have a tenant on OpenStack, and this client has all the resources created by OpenStack itself 1 - Tenant OpenStack project 2 - Tenant self-service network 3 - Tenant self-service subnet - e.g. 10.0.0.0/24 4 - Security groups 5 - Maybe FIPs 6 - Tenant Logical Router 7 - Tenant Logical Router Ports: 1. external network (provider/default gw) ; 2. internal network (self-service subnet) 8 - Tenant VMs - some VMs on the self-service subnet 10.0.0.0/24 Now that same tenant expects to connect these VMs with another deployment in the same region - High Availability. Option 1 - Floating IPs -> it's not a good option Option 2 - OVN-IC - L3 routing between self-service networks :) What's the strategy? connect the Tenant Logical Router with the TS of the OVN-IC. Creation steps: 1. Create a transit switch in the OVN-IC-NB 2. In each OpenStack / Neutron: a. Create a new Logical_Router_Port (externally managed) on the Tenant Logical Router and link this port with the TS. What is the point here, as Felix comments: even with the creation of the special network type, it will still be necessary to filter learned resources. " a. Ignore all resources where `external_ids` does not contain any key starting with `neutron`." So, from my perspective: Neutron changes for the [RFE] unmanaged dynamic router resources - OVN b. Ignore specific resources as defined by individual rules: I. ignore all Logical_Router_Ports where `external_ids` does not contain any key starting with `neutron`. II. ignore all Logical_Router_Static_Route where `external_ids` does not contain any key starting with `neutron`. We have already tested this and it is enough for interconnect to work in multi-tenant scenarios, and with the router connected to a generic external network provider (where all other tenants are connected). That is, the Logical Router was created by Neutron, so it can decide to learn dynamic routes and ignore external interconnect Logical_Router_Ports. BTW: Felix, the transit switch will learn remote Logical_Switch_Ports ports automatically via OVN-NB, in this case these ports were not created by Neutron and would need more special treatment to skip these resources. Note: about creating a new external provider network type, I already expressed my arguments about the multi tenant case and the need to create more resources to have this together with the default gw. If it were possible to have this in the same Logical_router tenant, I mean, use two external networks on the same Router: here we would have a lot of gain in my perspective Regards, Roberto Em ter., 18 de jul. de 2023 ?s 09:40, Felix Huettner escreveu: > Hi everyone, > > we will validate in the next weeks the idea i'll describe below. If it > sounds feasible to us i would write up a spec for this for later > implementation. > > Setup: > 1. Two neutron setups with OVN > 2. A OVN-IC setup not managed by any tooling (used manually) > > Target setup for a individual project > > +-------------------------------------+ > | Transit Switch (ts1) | > | 172.24.0.0/24 | > +-------------------------------------+ > | | > +----------------+ +----------------+ > | 172.24.0.10 | | 172.24.0.11 | > | Router | | Router | > | 192.168.0.1 | | 192.168.1.1 | > +----------------+ +----------------+ > | | > +----------------+ +----------------+ > | Network | | Network | > | 192.168.0.0/24 | | 192.168.1.0/24 | > +----------------+ +----------------+ > | | > +----------------+ +----------------+ > | VM | | VM | > | 192.168.0.10 | | 192.168.1.10 | > +----------------+ +----------------+ > > Main creation workflow of a interconnected network: > 1. Create a transit switch in the OVN-IC-NB > 2. In each neutron: > a. Create a new provider network that maps to the transit switch. > E.g. using `--provider-network-type ovn-ic > --provider-phyiscal-network ts1` > b. Create the internal network and router as always and attach the > router to the internal network and the provider network > > Changes needed in Neutron (from my perspective): > 1. Neutron must not touch the resources created by OVN-IC. This might be > done using two options: > a. Ignore all resources where `external_ids` does not contain any key > starting with `neutron`. > b. Ignore specific resources as defined by individual rules: > I. ignore all Logical_Router that contain > `other_config:interconn-ts` > II. ignore all Logical_Router_Static_Route that contain > `external_ids:ic-learned-route` > 2. Neutron must allow OVN-IC transit switches to be treated as external > networks. AFAIK this should work in the same way as creating any kind > of OVN network (so Logical_Switch), but just importing an existing > Logical_Switch. > > From my perspective with these two changes we should be good to > interconnect two neutron setups with probably minimal changes needed. > This is because OVN-IC would only touch the transit switch (which looks > in the Northbound like a regular Logical_Switch). All other resources > would still be managed by neutron and work as they do now. > > I will get back to this discussion and (if appropriate) provide a spec > or this idea, once we had time to validate this on our side (which will > take some weeks). > > Thanks > Felix > > On Tue, Jul 18, 2023 at 01:23:27PM +0200, Rodolfo Alonso Hernandez wrote: > > Ok, this is being tortuous. First of all: define a strategy. If you are > > going to create the resources in Neutron, define how. I've provided a way > > to do this, find a formal strategy to ground it. > > > > Second: (again) try to find a connection between resources, if you are > > going to use the strategy of creating the resources in Neutron. The > > "Logical_Router_Static_Route" belongs to a router univocally. If that > > router has been created by OpenStack, then you can modify the DB sync > > method to consider learned routes too. > > > > In order to do this, you'll need a set of resources that are going to be > > needed in Neutron, the OVN counterparts and other resources (like > > "Logical_Router_Static_Route") that will be added and will be present in > > OVN and not in Neutron DB. Also you'll need to know how to relate all of > > them. > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r > die Verwertung durch den vorgesehenen Empf?nger bestimmt. > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender > bitte unverz?glich in Kenntnis und l?schen diese E Mail. > > Hinweise zum Datenschutz finden Sie hier. > > > This e-mail may contain confidential content and is intended only for the > specified recipient/s. > If you are not the intended recipient, please inform the sender > immediately and delete this e-mail. > > Information on data protection can be found here< > https://www.datenschutz.schwarz>. > -- _?Esta mensagem ? direcionada apenas para os endere?os constantes no cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o imediatamente anuladas e proibidas?._ *?**?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o poder? aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos?.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew at giassa.net Tue Jul 18 03:21:08 2023 From: matthew at giassa.net (Matthew Giassa) Date: Mon, 17 Jul 2023 20:21:08 -0700 Subject: [ops] Built-in rolling-update/instance refresh capabilities Message-ID: Good day, I was originally discussing this on the IRC channel, #openstack, and was advised to reach out to this mailing list. In short, does OpenStack (i.e. via Heat, Senlin, Masakari, or some other feature/plugin), have the ability to perform a rolling upgrade operation on a group of VMs (i.e. similar to the "rolling update" feature in K8S, or the "instance refresh" feature in AWS auto-scaling groups / ASGs). I've reviewed the available documentation, but all documents regarding "rolling upgrades" are with respect to a rolling upgrade of the OpenStack/IaaS stack itself (i.e. operator-level functionality) as opposed to groups of VMs managed by OpenStack (i.e. user-level functionality). I'd ideally like to use built-in functionality to handle this operation, rather than writing a service from scratch. I've also looked into tools such as Spinnaker.io and Harness.io, but could not find any FOSS or COTS tools for implementing rolling upgrades on OpenStack out-of-the-box (though if there are any options I've overlooked, I'm all ears). Thank you. -- ============================================================ Matthew Giassa e-mail: matthew at giassa.net website: www.giassa.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.acosta at luizalabs.com Tue Jul 18 13:51:47 2023 From: roberto.acosta at luizalabs.com (Roberto Bartzen Acosta) Date: Tue, 18 Jul 2023 10:51:47 -0300 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: Would it be possible to implement multiple external network ports on the same router? It is not the same case of multi-homing [1], in the case of OVN-IC there are no symmetric internet routes.... [1] https://review.opendev.org/c/openstack/neutron/+/874199 Basically an external port for ovn-ic doesn't have SNAT! Could this be done Rodolfo? What would be the effort/implications? If possible, we can integrate everything we are discussing in terms of solutions into a single reference design. Em ter., 18 de jul. de 2023 ?s 10:20, Roberto Bartzen Acosta < roberto.acosta at luizalabs.com> escreveu: > Hi Rodolfo, > > I agree with you, my initial idea with this thread was exactly what we are > doing, debating problems and possible solutions for interconnecting > workloads using unmanaged dynamic router resources. Thank you for being > engaged with this! > I'm sorry if the strategy wasn't presented clearly, let me talk about it > then. > > Imagine we have a tenant on OpenStack, and this client has all the > resources created by OpenStack itself > 1 - Tenant OpenStack project > 2 - Tenant self-service network > 3 - Tenant self-service subnet - e.g. 10.0.0.0/24 > 4 - Security groups > 5 - Maybe FIPs > 6 - Tenant Logical Router > 7 - Tenant Logical Router Ports: 1. external network (provider/default > gw) ; 2. internal network (self-service subnet) > 8 - Tenant VMs - some VMs on the self-service subnet 10.0.0.0/24 > > Now that same tenant expects to connect these VMs with another deployment > in the same region - High Availability. > > Option 1 - Floating IPs -> it's not a good option > Option 2 - OVN-IC - L3 routing between self-service networks :) > > What's the strategy? connect the Tenant Logical Router with the TS of the > OVN-IC. > > Creation steps: > 1. Create a transit switch in the OVN-IC-NB > 2. In each OpenStack / Neutron: > a. Create a new Logical_Router_Port (externally managed) on the Tenant > Logical Router and link this port with the TS. > > What is the point here, as Felix comments: even with the creation of the > special network type, it will still be necessary to filter learned > resources. > " a. Ignore all resources where `external_ids` does not contain any key > starting with `neutron`." > > So, from my perspective: > Neutron changes for the [RFE] unmanaged dynamic router resources - OVN > b. Ignore specific resources as defined by individual rules: > I. ignore all Logical_Router_Ports where `external_ids` does not > contain any key > starting with `neutron`. > II. ignore all Logical_Router_Static_Route where `external_ids` does > not contain any key > starting with `neutron`. > > > We have already tested this and it is enough for interconnect to work in > multi-tenant scenarios, and with the router connected to a generic external > network provider (where all other tenants are connected). > > That is, the Logical Router was created by Neutron, so it can decide to > learn dynamic routes and ignore external interconnect Logical_Router_Ports. > > BTW: Felix, the transit switch will learn remote Logical_Switch_Ports > ports automatically via OVN-NB, in this case these ports were not created > by Neutron and would need more special treatment to skip these resources. > > Note: about creating a new external provider network type, I already > expressed my arguments about the multi tenant case and the need to create > more resources to have this together with the default gw. If it were > possible to have this in the same Logical_router tenant, I mean, use two > external networks on the same Router: here we would have a lot of gain in > my perspective > > Regards, > Roberto > > > > Em ter., 18 de jul. de 2023 ?s 09:40, Felix Huettner > escreveu: > >> Hi everyone, >> >> we will validate in the next weeks the idea i'll describe below. If it >> sounds feasible to us i would write up a spec for this for later >> implementation. >> >> Setup: >> 1. Two neutron setups with OVN >> 2. A OVN-IC setup not managed by any tooling (used manually) >> >> Target setup for a individual project >> >> +-------------------------------------+ >> | Transit Switch (ts1) | >> | 172.24.0.0/24 | >> +-------------------------------------+ >> | | >> +----------------+ +----------------+ >> | 172.24.0.10 | | 172.24.0.11 | >> | Router | | Router | >> | 192.168.0.1 | | 192.168.1.1 | >> +----------------+ +----------------+ >> | | >> +----------------+ +----------------+ >> | Network | | Network | >> | 192.168.0.0/24 | | 192.168.1.0/24 | >> +----------------+ +----------------+ >> | | >> +----------------+ +----------------+ >> | VM | | VM | >> | 192.168.0.10 | | 192.168.1.10 | >> +----------------+ +----------------+ >> >> Main creation workflow of a interconnected network: >> 1. Create a transit switch in the OVN-IC-NB >> 2. In each neutron: >> a. Create a new provider network that maps to the transit switch. >> E.g. using `--provider-network-type ovn-ic >> --provider-phyiscal-network ts1` >> b. Create the internal network and router as always and attach the >> router to the internal network and the provider network >> >> Changes needed in Neutron (from my perspective): >> 1. Neutron must not touch the resources created by OVN-IC. This might be >> done using two options: >> a. Ignore all resources where `external_ids` does not contain any key >> starting with `neutron`. >> b. Ignore specific resources as defined by individual rules: >> I. ignore all Logical_Router that contain >> `other_config:interconn-ts` >> II. ignore all Logical_Router_Static_Route that contain >> `external_ids:ic-learned-route` >> 2. Neutron must allow OVN-IC transit switches to be treated as external >> networks. AFAIK this should work in the same way as creating any kind >> of OVN network (so Logical_Switch), but just importing an existing >> Logical_Switch. >> >> From my perspective with these two changes we should be good to >> interconnect two neutron setups with probably minimal changes needed. >> This is because OVN-IC would only touch the transit switch (which looks >> in the Northbound like a regular Logical_Switch). All other resources >> would still be managed by neutron and work as they do now. >> >> I will get back to this discussion and (if appropriate) provide a spec >> or this idea, once we had time to validate this on our side (which will >> take some weeks). >> >> Thanks >> Felix >> >> On Tue, Jul 18, 2023 at 01:23:27PM +0200, Rodolfo Alonso Hernandez wrote: >> > Ok, this is being tortuous. First of all: define a strategy. If you are >> > going to create the resources in Neutron, define how. I've provided a >> way >> > to do this, find a formal strategy to ground it. >> > >> > Second: (again) try to find a connection between resources, if you are >> > going to use the strategy of creating the resources in Neutron. The >> > "Logical_Router_Static_Route" belongs to a router univocally. If that >> > router has been created by OpenStack, then you can modify the DB sync >> > method to consider learned routes too. >> > >> > In order to do this, you'll need a set of resources that are going to be >> > needed in Neutron, the OVN counterparts and other resources (like >> > "Logical_Router_Static_Route") that will be added and will be present in >> > OVN and not in Neutron DB. Also you'll need to know how to relate all of >> > them. >> Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r >> die Verwertung durch den vorgesehenen Empf?nger bestimmt. >> Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender >> bitte unverz?glich in Kenntnis und l?schen diese E Mail. >> >> Hinweise zum Datenschutz finden Sie hier> >. >> >> >> This e-mail may contain confidential content and is intended only for the >> specified recipient/s. >> If you are not the intended recipient, please inform the sender >> immediately and delete this e-mail. >> >> Information on data protection can be found here< >> https://www.datenschutz.schwarz>. >> > -- _?Esta mensagem ? direcionada apenas para os endere?os constantes no cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o imediatamente anuladas e proibidas?._ *?**?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o poder? aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos?.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From knikolla at bu.edu Tue Jul 18 14:03:09 2023 From: knikolla at bu.edu (Nikolla, Kristi) Date: Tue, 18 Jul 2023 14:03:09 +0000 Subject: [tc] Technical Committee next weekly meeting today on July 18, 2023 Message-ID: Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held Today on Tuesday, July 18, 2023, at 1800 UTC on #openstack-tc on OFTC IRC. Please find below the agenda for today's meeting: * Roll call * Follow up on past action items * knikolla will amend proposal for Unmaintained branches. * Extended Maintenance * https://etherpad.opendev.org/p/openstack-exteneded-maintenance * Release notes guidelines for SLURP/NON-SLURP cadence * Gate health check * Open Discussion and Reviews * https://review.opendev.org/q/projects:openstack/governance+is:open Thank you, Kristi Nikolla -------------- next part -------------- An HTML attachment was scrubbed... URL: From james.page at canonical.com Tue Jul 18 14:15:05 2023 From: james.page at canonical.com (James Page) Date: Tue, 18 Jul 2023 15:15:05 +0100 Subject: [charms][sunbeam] Proposing Guillaume Boutry for charms-core Message-ID: Hi All Guillaume has been working on the Sunbeam Charms for the last few months and has proven an able reviewer and contributor to the charms; we need to dis-entangle the sunbeam project charms from the main openstack charms set but until that work is completed I'd like to propose Guillaume as a member of the charms-core team. Cheers James -------------- next part -------------- An HTML attachment was scrubbed... URL: From knikolla at bu.edu Tue Jul 18 14:24:28 2023 From: knikolla at bu.edu (Nikolla, Kristi) Date: Tue, 18 Jul 2023 14:24:28 +0000 Subject: [all][tc] Proposals for reforming extended maintenance In-Reply-To: References: Message-ID: Based on discussion from the OpenStack Technical Committee on July 11 and the feedback received on the proposed patch, I have submitted a newer version of the proposal at https://review.opendev.org/c/openstack/governance/+/888771 From: Jay Faulkner Date: Tuesday, July 11, 2023 at 7:06 AM To: Nikolla, Kristi Cc: openstack-discuss Subject: Re: [all][tc] Proposals for reforming extended maintenance There is interesting and useful discussion in the governance patches posted by Kristi. I strongly recommend folks take a look and participate. The more independent opinions we get, the better off the end result will be. Thanks, Jay Faulkner TC Member On Fri, Jul 7, 2023 at 4:25?PM Nikolla, Kristi > wrote: Hi all, There have been quite a few discussions related extended maintenance in the last few months[0][1]. I have made a first-go at trying to write a policy for extended maintenance moving forward which tries to preserve the value being brought by extended maintenance while strengthening the requirements for keeping branches under that phase. [2] This was based on multiple discussions that I was present at and feedback that I have received from various teams and community members with which I talked, and is my interpretation of that feedback into something that looks like policy. I have also written two opposing policy proposals about what do to with the current branches under extended maintenance. One which attempts to apply the new requirements defined in the first proposal almost immediately[3], and one that aims for a more gradual EOL[4]. Please provide feedback on the proposals and help shape up the future of extended maintenance. Thank you, Kristi Nikolla [0]. https://lists.openstack.org/pipermail/openstack-discuss/2023-June/033980.html [1]. https://etherpad.opendev.org/p/vancouver-2023-em [2]. https://review.opendev.org/c/openstack/governance/+/887966 [3]. https://review.opendev.org/c/openstack/governance/+/887968 [4]. https://review.opendev.org/c/openstack/governance/+/887969 -------------- next part -------------- An HTML attachment was scrubbed... URL: From noonedeadpunk at gmail.com Tue Jul 18 17:11:56 2023 From: noonedeadpunk at gmail.com (Dmitriy Rabotyagov) Date: Tue, 18 Jul 2023 19:11:56 +0200 Subject: [OSA][NEUTRON] Failed to bind port XXX on host YYY for vnic_type normal using segments... In-Reply-To: References: Message-ID: Hey, Thanks for providing the output. Regarding neutron-openvswitch-agent utilization - that might be a side-effect of neutron-server failures. Regarding neutron-server failures, reading stack trace again, it seems that neutron-server is failing to reach the placement service: HTTPConnectionPool(host='172.29.236.2', port=8780) Failed to establish a new connection: [Errno 32] EPIPE I assume that 172.29.236.2 is your internal VIP that's on haproxy? Are you able to reach the IP and connect to the port 8780 from inside of the neutron-server container? You can try leveraging telnet or simple curl from inside of the neutron-server container to verify that connectivity is fine. Also good to ensure, that haproxy does have placement backends UP, you can verify that with running `echo "show stat" | nc -U /run/haproxy.stat | grep placement` ??, 18 ???. 2023??. ? 15:14, Murilo Morais : > > I noticed that the "neutron-openvswitch-agent" process constantly goes up to 100% in CPU usage. This is normal? > > Em ter., 18 de jul. de 2023 ?s 09:44, Murilo Morais escreveu: >> >> Hi Dmitriy, thanks for answering! >> >> I really didn't send any details of my setup, apologies for that. >> >> I'm using OVS with the following configuration: >> >> provider_networks: >> - network: >> container_bridge: br-provider >> container_type: veth >> type: vlan >> range: "100:200" >> net_name: vlan >> group_binds: >> - neutron_openvswitch_agent >> >> - network: >> container_bridge: br-provider >> container_type: veth >> type: flat >> net_name: flat >> group_binds: >> - neutron_openvswitch_agent >> >> >> neutron_plugin_type: ml2.ovs >> neutron_ml2_drivers_type: "flat,vlan" >> neutron_plugin_base: >> - router >> - metering >> >> >> root at dcn2-utility-container-c45f8b09:/# openstack network agent list >> +--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+ >> | ID | Agent Type | Host | Availability Zone | Alive | State | Binary | >> +--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+ >> | 9a4625ef-2988-4b96-a927-30a9bb0244a4 | Metadata agent | dcn2 | None | :-) | UP | neutron-metadata-agent | >> | a222c0ae-c2e5-44cc-b478-ca8176daad19 | Metering agent | dcn2 | None | :-) | UP | neutron-metering-agent | >> | c6be1985-a67e-4099-a1d6-fa517810e138 | L3 agent | dcn2 | nova | :-) | UP | neutron-l3-agent | >> | da97e2a6-535b-4f1e-828d-9b2fbb3e036b | DHCP agent | dcn2 | nova | :-) | UP | neutron-dhcp-agent | >> +--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+ >> >> >> root at dcn2-utility-container-c45f8b09:/# openstack compute service list >> +--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+ >> | ID | Binary | Host | Zone | Status | State | Updated At | >> +--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+ >> | a0524af5-bc88-4793-aee2-c2c87cd0e8cc | nova-conductor | dcn2-nova-api-container-aac2b913 | internal | enabled | up | 2023-07-18T12:32:26.000000 | >> | c860f25e-bd30-49f9-8289-076b230bbc2d | nova-scheduler | dcn2-nova-api-container-aac2b913 | internal | enabled | up | 2023-07-18T12:32:27.000000 | >> | 6457e0a1-b075-4999-8855-0f36e2e3a95a | nova-compute | dcn2 | nova | enabled | up | 2023-07-18T12:32:27.000000 | >> +--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+ >> >> >> The br-provider bridge exists and is UP. From nguyenhuukhoinw at gmail.com Wed Jul 19 07:03:35 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Wed, 19 Jul 2023 14:03:35 +0700 Subject: [openstack][largescale-sig] Openstack multi region deployment In-Reply-To: References: Message-ID: Hello, thank you very much. But can I ask how we process if 1 region at ASIA and 2 regions in the USA? Database latency will be our problem. Nguyen Huu Khoi On Tue, Jul 18, 2023 at 8:21?PM Karl Kloppenborg wrote: > Hi Nguy, > > > > We?ve deployed a large multi-region openstack deployment. > > As a rule of thumb we?ve got a ?keystone? region which is as best we can > highly available and very redundant. > > > > We then have all other regions talk back to this region, we just usually > call it ?keystone? or ?core? and it?s hidden from the UI from users. > > > > We then just run a large well kept Galara cluster to support it. > > > > --Karl. > > > > *From: *openstack-discuss-request at lists.openstack.org < > openstack-discuss-request at lists.openstack.org> > *Date: *Tuesday, 18 July 2023 at 9:25 pm > *To: *openstack-discuss at lists.openstack.org < > openstack-discuss at lists.openstack.org> > *Subject: *openstack-discuss Digest, Vol 57, Issue 55 > > Send openstack-discuss mailing list submissions to > openstack-discuss at lists.openstack.org > > To subscribe or unsubscribe via the World Wide Web, visit > > https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss > > or, via email, send a message with subject or body 'help' to > openstack-discuss-request at lists.openstack.org > > You can reach the person managing the list at > openstack-discuss-owner at lists.openstack.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of openstack-discuss digest..." > > > Today's Topics: > > 1. [openstack][largescale-sig] Openstack multi region deployment > (Nguy?n H?u Kh?i) > 2. Re: [openstack][largescale-sig] Openstack multi region > deployment (Felix Huettner) > 3. Re: [openstack][largescale-sig] Openstack multi region > deployment (Nguy?n H?u Kh?i) > 4. Re: [neutron] unmanaged router resources - OVN interconnect > (Rodolfo Alonso Hernandez) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 18 Jul 2023 12:07:12 +0700 > From: Nguy?n H?u Kh?i > To: OpenStack Discuss > Subject: [openstack][largescale-sig] Openstack multi region deployment > Message-ID: > < > CABAODReJ6QW8A4OENEjmhFCiM-15B0qc2La_aMr1EKfaENq9iw at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hello guys, > > I am going to deploy openstack multi regions and I know that keystone > replication is the most challenging. > > I plan to set up 2 regions which use centralize galera cluster(3 nodes). > and one standby edge galera cluster(3 nodes) > > When region 1 is node available, I will map region 2 to use standby edge > galera cluster. > > I hope you give me some experience and advice with multi regions. > > Thank you very much. > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/c95d3675/attachment-0001.htm > > > > ------------------------------ > > Message: 2 > Date: Tue, 18 Jul 2023 09:34:35 +0200 > From: Felix Huettner > To: Nguy?n H?u Kh?i > Cc: OpenStack Discuss > Subject: Re: [openstack][largescale-sig] Openstack multi region > deployment > Message-ID: > Content-Type: text/plain; charset=utf-8 > > Hi, > > i think you have two options here: > 1. you could span a single galera cluster over all of your regions. > this might have some latency issues, but if your are not too write > heavy that might be fine. I know some companies use that setup. > 2. you use some kind of multiple galera clusters with replication. > But i have not yet heard of anybody using this setup. > > An alternative might be to have separate keystone setups with separate > databases. This would probably reduce the error potential, but might not > fit your usecase. > > Thanks > Felix > > > On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote: > > Hello guys, > > > > I am going to deploy openstack multi regions and I know that keystone > > replication is the most challenging. > > > > I plan to set up 2 regions which use centralize galera cluster(3 nodes). > > and one standby edge galera cluster(3 nodes) > > > > When region 1 is node available, I will map region 2 to use standby edge > > galera cluster. > > > > I hope you give me some experience and advice with multi regions. > > > > Thank you very much. > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r > die Verwertung durch den vorgesehenen Empf?nger bestimmt. > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender > bitte unverz?glich in Kenntnis und l?schen diese E Mail. > > Hinweise zum Datenschutz finden Sie hier. > > > This e-mail may contain confidential content and is intended only for the > specified recipient/s. > If you are not the intended recipient, please inform the sender > immediately and delete this e-mail. > > Information on data protection can be found here< > https://www.datenschutz.schwarz>. > > > > ------------------------------ > > Message: 3 > Date: Tue, 18 Jul 2023 15:36:11 +0700 > From: Nguy?n H?u Kh?i > To: Nguy?n H?u Kh?i , OpenStack Discuss > > Subject: Re: [openstack][largescale-sig] Openstack multi region > deployment > Message-ID: > CGBW1_bRkLQJAxLZxAx8V4qvbdBmTUQBUm2SRsow at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi. > Thank you for your reply. > > The first one has a problem because each region is too soft. If a member is > down, so this region is gone. > > It is so challenge with us. > > > Nguyen Huu Khoi > > > On Tue, Jul 18, 2023 at 2:34?PM Felix Huettner > > wrote: > > > Hi, > > > > i think you have two options here: > > 1. you could span a single galera cluster over all of your regions. > > this might have some latency issues, but if your are not too write > > heavy that might be fine. I know some companies use that setup. > > 2. you use some kind of multiple galera clusters with replication. > > But i have not yet heard of anybody using this setup. > > > > An alternative might be to have separate keystone setups with separate > > databases. This would probably reduce the error potential, but might not > > fit your usecase. > > > > Thanks > > Felix > > > > > > On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote: > > > Hello guys, > > > > > > I am going to deploy openstack multi regions and I know that keystone > > > replication is the most challenging. > > > > > > I plan to set up 2 regions which use centralize galera cluster(3 > nodes). > > > and one standby edge galera cluster(3 nodes) > > > > > > When region 1 is node available, I will map region 2 to use standby > edge > > > galera cluster. > > > > > > I hope you give me some experience and advice with multi regions. > > > > > > Thank you very much. > > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r > > die Verwertung durch den vorgesehenen Empf?nger bestimmt. > > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender > > bitte unverz?glich in Kenntnis und l?schen diese E Mail. > > > > Hinweise zum Datenschutz finden Sie hier >. > > > > > > This e-mail may contain confidential content and is intended only for the > > specified recipient/s. > > If you are not the intended recipient, please inform the sender > > immediately and delete this e-mail. > > > > Information on data protection can be found here< > > https://www.datenschutz.schwarz>. > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/749440e3/attachment-0001.htm > > > > ------------------------------ > > Message: 4 > Date: Tue, 18 Jul 2023 13:23:27 +0200 > From: Rodolfo Alonso Hernandez > To: Roberto Bartzen Acosta > Cc: openstack-discuss , Terry > Wilson , Tiago Pires < > tiago.pires at luizalabs.com> > Subject: Re: [neutron] unmanaged router resources - OVN interconnect > Message-ID: > < > CAECr9X7U7YsGBv9ajcmeOCxfdD+YLar8QyPwYBN0qaP10CzTug at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Ok, this is being tortuous. First of all: define a strategy. If you are > going to create the resources in Neutron, define how. I've provided a way > to do this, find a formal strategy to ground it. > > Second: (again) try to find a connection between resources, if you are > going to use the strategy of creating the resources in Neutron. The > "Logical_Router_Static_Route" belongs to a router univocally. If that > router has been created by OpenStack, then you can modify the DB sync > method to consider learned routes too. > > In order to do this, you'll need a set of resources that are going to be > needed in Neutron, the OVN counterparts and other resources (like > "Logical_Router_Static_Route") that will be added and will be present in > OVN and not in Neutron DB. Also you'll need to know how to relate all of > them. > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/90712e47/attachment.htm > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > openstack-discuss mailing list > openstack-discuss at lists.openstack.org > > > ------------------------------ > > End of openstack-discuss Digest, Vol 57, Issue 55 > ************************************************* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eblock at nde.ag Wed Jul 19 07:39:46 2023 From: eblock at nde.ag (Eugen Block) Date: Wed, 19 Jul 2023 07:39:46 +0000 Subject: Ussuri: how to delete lbaas loadbalancer left over? In-Reply-To: <1a342d65-191b-c367-9872-1d72f83479f6@ijclab.in2p3.fr> References: <20230707164654.Horde.sUTt6M8NRpBGR8HwFy7KwGv@webmail.nde.ag> <1893244f3d8.27f3.86e99a005b57e9644b7bb922d8d7f665@ijclab.in2p3.fr> <20230707220512.Horde.10Q-iIqvZct1kSTQs9Gd4-u@webmail.nde.ag> <20230707221544.Horde.tGBcMHNkuxKWyNahfmPmJ9V@webmail.nde.ag> <1a342d65-191b-c367-9872-1d72f83479f6@ijclab.in2p3.fr> Message-ID: <20230719073946.Horde.fWBdNQBJmzBBB-urC9pGkgE@webmail.nde.ag> Hi, it sounds promising, I hope you get rid of the LBs in a clean manner. I just thought it might be the cleanest way to first migrate and then delete them properly without leaving any orphans in the database. In case you succeed it would be of great value to have your solution posted here, thanks! Zitat von Michel Jouvin : > Hi Eugen, > > I found some time to look at my issue in the light of your advices! > In my research, I found the presentation > https://opendev.org/openstack/neutron-lbaas which is both synthetic > and clear, with a list of the migration options offered (at the time > of the presentation). This includes the DB migration tool, > nlbaas2octavia.py. Following your broken link, I manage to find it, > it is still there but hidden. You need to checkout the repo > https://opendev.org/openstack/neutron-lbaas and go back a couple of > revisions (it is explained in the README). It is Python2 but at > least it is a starting point, either installing a Python2 venv or > migrating the script which is not very long and seems to have very > few dependcies, apart from oslo. > > I'm afraid that the neutron-lbaas-proxy is no longer an option as, > as far as I understood, it relies on some LBAAS code still being > present in the Neutron server and this part of the Neutron code has > been completely removed I think in Ussuri release (it's the reason > of my problems in fact, too bad, it was a matter of a few days, just > missed the very old announcement that it would be the case). > > If I succeed to clean the things (again I don't really want to > migrate the existing LBAAS, I just want to delete them), I'll report > in this thread in case it is useful to somebody else... > > Best regards, > > Michel > > Le 08/07/2023 ? 00:15, Eugen Block a ?crit?: >> Unfortunately, the link to the migration tool doesn?t work: >> >> https://wiki.openstack.org/wiki/Neutron/LBaaS/Deprecation >> >> But maybe it can get you in the right direction, >> neutron-lbaas-proxy seems to be the keyword. But as I already >> mentioned, I don?t have experience with this path. >> >> Zitat von Eugen Block : >> >>> Hi, >>> >>> I mean the latter. Once you have Octavia installed you can create >>> new LBs, but as I understand it you won?t be able to delete the >>> legacy LBs. Did the neutron config change when you upgraded to >>> Ussuri? I wonder if there?s just some config missing to be able to >>> delete the old LBs, I don?t have a clue tbh. Maybe someone else >>> has some more experience and will chime in. >>> >>> Zitat von Michel Jouvin : >>> >>>> Ho Eug?ne, >>>> >>>> Thanks for your answer. Do you mean that after installing octavia >>>> (it is planned) we'll have again the ability to delete the >>>> remaining LBAAS instances? Or just that octavia is the LBAAS >>>> replacement in terms of functionalities? >>>> >>>> Best regards, >>>> >>>> Michel >>>> Sent from my mobile >>>> Le 7 juillet 2023 18:52:30 Eugen Block a ?crit : >>>> >>>>> Hi, >>>>> >>>>> neutron lbaas was deprecated in Queens so you may have to migrate the >>>>> existing LBs to octavia. I have never done that but I remember reading >>>>> through the SUSE Docs when one of our customers had to decide whether >>>>> they wanted to upgrade or reinstall with a newer openstack release. >>>>> They decided to do the latter, so we set up octavia from scratch and >>>>> didn't have to migrate anything. There's also a video I've never >>>>> watched [2], maybe that helps. I can't really tell if a migration is >>>>> possible to work around your issue but I thought I'd share anyway. >>>>> >>>>> Regards, >>>>> Eugen >>>>> >>>>> [1] >>>>> https://documentation.suse.com/soc/9/single-html/suse-openstack-cloud-crowbar-deployment/#sec-depl-ostack-octavia-migrate-users [2] >>>>> https://www.youtube.com/watch?v=jj4KMJPA0Pk >>>>> >>>>> Zitat von Michel Jouvin : >>>>> >>>>>> Hi, >>>>>> >>>>>> We had a few Magnum (K8s) clusters created a couple of years ago >>>>>> (with Rocky and Stein versions) and forgotten. We started to delete >>>>>> them this spring when we where running Train Neutron service. >>>>>> Basically we managed to do this with the following sequence: >>>>>> >>>>>> - openstack coe cluster delete xxx and waiting for DELETE_FAILED >>>>>> - Use openstack coe cluster show / openstack stack resource list -n2 >>>>>> to identify the neutron entry causing the error and pick the >>>>>> corresponding resource ID >>>>>> - Find the ports associated with the router with openstack port list >>>>>> --router previously_found_id >>>>>> - Use the port subnet to find the port corresponding lbaas load >>>>>> balancer ID, use the neutron CLI to delete the load balancer >>>>>> (deleting one by one all the dependencies preventing the load >>>>>> balancer removal) >>>>>> - Rerun openstack coe cluster delete >>>>>> >>>>>> For some reasons, we didn't cleanup all the abandoned clusters and >>>>>> upgraded Neutron to Ussuri. Unfortunately, since then our previous >>>>>> process is no longer working as it seems that the Neutron server >>>>>> doesn't know anymore anything about the LBAAS load balancers (and >>>>>> "neutron lbaas-loadbalancer list" returns nothing). In the neutron >>>>>> server, any attempt to delete the subnet attached to the load >>>>>> balancer (or to list them with Neutron CLI) results in the following >>>>>> errors in Neutron server.log : >>>>>> >>>>>> ------ >>>>>> >>>>>> 2023-07-07 16:27:31.139 14962 WARNING >>>>>> neutron.pecan_wsgi.controllers.root >>>>>> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >>>>>> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >>>>>> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >>>>>> 1367c9a4d5da4b229c35789c271dc7aa] No controller found for: lbaas - >>>>>> returning response code 404: pecan.routing.PecanNotFound >>>>>> 2023-07-07 16:27:31.140 14962 INFO >>>>>> neutron.pecan_wsgi.hooks.translation >>>>>> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >>>>>> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >>>>>> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >>>>>> 1367c9a4d5da4b229c35789c271dc7aa] GET failed (client error): The >>>>>> resource could not be found. >>>>>> 2023-07-07 16:27:31.141 14962 INFO neutron.wsgi >>>>>> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >>>>>> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >>>>>> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >>>>>> 1367c9a4d5da4b229c35789c271dc7aa] 157.136.249.153 "GET >>>>>> /v2.0/lbaas/loadbalancers?name=kube_service_964f7e76-d2d5-4126-ab11-cd689f6dd9f9_runnerdeploy-wm9sm-5h52l_hello-node-x-default-x-runnerdeploy-wm9sm-5h52l HTTP/1.1" status: 404? len: 304 >>>>>> time: >>>>>> 0.0052643 >>>>>> ------ >>>>>> >>>>>> Any suggestion to workaround this problem and be able to >>>>>> successfully delete our old Magnum clusters? >>>>>> >>>>>> Thanks in advance for any help. Best regards, >>>>>> >>>>>> Michel >> >> >> >> From ralonsoh at redhat.com Wed Jul 19 07:57:08 2023 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Wed, 19 Jul 2023 09:57:08 +0200 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: Before that you need to explain why you need multiple external network ports and why cannot be solved with the current architecture Neutron is providing. On Tue, Jul 18, 2023 at 3:52?PM Roberto Bartzen Acosta < roberto.acosta at luizalabs.com> wrote: > > Would it be possible to implement multiple external network ports on the > same router? > > It is not the same case of multi-homing [1], in the case of OVN-IC there > are no symmetric internet routes.... > [1] https://review.opendev.org/c/openstack/neutron/+/874199 > > Basically an external port for ovn-ic doesn't have SNAT! > > Could this be done Rodolfo? What would be the effort/implications? > > If possible, we can integrate everything we are discussing in terms of > solutions into a single reference design. > > > Em ter., 18 de jul. de 2023 ?s 10:20, Roberto Bartzen Acosta < > roberto.acosta at luizalabs.com> escreveu: > >> Hi Rodolfo, >> >> I agree with you, my initial idea with this thread was exactly what we >> are doing, debating problems and possible solutions for interconnecting >> workloads using unmanaged dynamic router resources. Thank you for being >> engaged with this! >> I'm sorry if the strategy wasn't presented clearly, let me talk about it >> then. >> >> Imagine we have a tenant on OpenStack, and this client has all the >> resources created by OpenStack itself >> 1 - Tenant OpenStack project >> 2 - Tenant self-service network >> 3 - Tenant self-service subnet - e.g. 10.0.0.0/24 >> 4 - Security groups >> 5 - Maybe FIPs >> 6 - Tenant Logical Router >> 7 - Tenant Logical Router Ports: 1. external network (provider/default >> gw) ; 2. internal network (self-service subnet) >> 8 - Tenant VMs - some VMs on the self-service subnet 10.0.0.0/24 >> >> Now that same tenant expects to connect these VMs with another deployment >> in the same region - High Availability. >> >> Option 1 - Floating IPs -> it's not a good option >> Option 2 - OVN-IC - L3 routing between self-service networks :) >> >> What's the strategy? connect the Tenant Logical Router with the TS of the >> OVN-IC. >> >> Creation steps: >> 1. Create a transit switch in the OVN-IC-NB >> 2. In each OpenStack / Neutron: >> a. Create a new Logical_Router_Port (externally managed) on the Tenant >> Logical Router and link this port with the TS. >> >> What is the point here, as Felix comments: even with the creation of the >> special network type, it will still be necessary to filter learned >> resources. >> " a. Ignore all resources where `external_ids` does not contain any key >> starting with `neutron`." >> >> So, from my perspective: >> Neutron changes for the [RFE] unmanaged dynamic router resources - OVN >> b. Ignore specific resources as defined by individual rules: >> I. ignore all Logical_Router_Ports where `external_ids` does not >> contain any key >> starting with `neutron`. >> II. ignore all Logical_Router_Static_Route where `external_ids` >> does not contain any key >> starting with `neutron`. >> >> >> We have already tested this and it is enough for interconnect to work in >> multi-tenant scenarios, and with the router connected to a generic external >> network provider (where all other tenants are connected). >> >> That is, the Logical Router was created by Neutron, so it can decide to >> learn dynamic routes and ignore external interconnect Logical_Router_Ports. >> >> BTW: Felix, the transit switch will learn remote Logical_Switch_Ports >> ports automatically via OVN-NB, in this case these ports were not created >> by Neutron and would need more special treatment to skip these resources. >> >> Note: about creating a new external provider network type, I already >> expressed my arguments about the multi tenant case and the need to create >> more resources to have this together with the default gw. If it were >> possible to have this in the same Logical_router tenant, I mean, use two >> external networks on the same Router: here we would have a lot of gain in >> my perspective >> >> Regards, >> Roberto >> >> >> >> Em ter., 18 de jul. de 2023 ?s 09:40, Felix Huettner >> escreveu: >> >>> Hi everyone, >>> >>> we will validate in the next weeks the idea i'll describe below. If it >>> sounds feasible to us i would write up a spec for this for later >>> implementation. >>> >>> Setup: >>> 1. Two neutron setups with OVN >>> 2. A OVN-IC setup not managed by any tooling (used manually) >>> >>> Target setup for a individual project >>> >>> +-------------------------------------+ >>> | Transit Switch (ts1) | >>> | 172.24.0.0/24 | >>> +-------------------------------------+ >>> | | >>> +----------------+ +----------------+ >>> | 172.24.0.10 | | 172.24.0.11 | >>> | Router | | Router | >>> | 192.168.0.1 | | 192.168.1.1 | >>> +----------------+ +----------------+ >>> | | >>> +----------------+ +----------------+ >>> | Network | | Network | >>> | 192.168.0.0/24 | | 192.168.1.0/24 | >>> +----------------+ +----------------+ >>> | | >>> +----------------+ +----------------+ >>> | VM | | VM | >>> | 192.168.0.10 | | 192.168.1.10 | >>> +----------------+ +----------------+ >>> >>> Main creation workflow of a interconnected network: >>> 1. Create a transit switch in the OVN-IC-NB >>> 2. In each neutron: >>> a. Create a new provider network that maps to the transit switch. >>> E.g. using `--provider-network-type ovn-ic >>> --provider-phyiscal-network ts1` >>> b. Create the internal network and router as always and attach the >>> router to the internal network and the provider network >>> >>> Changes needed in Neutron (from my perspective): >>> 1. Neutron must not touch the resources created by OVN-IC. This might be >>> done using two options: >>> a. Ignore all resources where `external_ids` does not contain any key >>> starting with `neutron`. >>> b. Ignore specific resources as defined by individual rules: >>> I. ignore all Logical_Router that contain >>> `other_config:interconn-ts` >>> II. ignore all Logical_Router_Static_Route that contain >>> `external_ids:ic-learned-route` >>> 2. Neutron must allow OVN-IC transit switches to be treated as external >>> networks. AFAIK this should work in the same way as creating any kind >>> of OVN network (so Logical_Switch), but just importing an existing >>> Logical_Switch. >>> >>> From my perspective with these two changes we should be good to >>> interconnect two neutron setups with probably minimal changes needed. >>> This is because OVN-IC would only touch the transit switch (which looks >>> in the Northbound like a regular Logical_Switch). All other resources >>> would still be managed by neutron and work as they do now. >>> >>> I will get back to this discussion and (if appropriate) provide a spec >>> or this idea, once we had time to validate this on our side (which will >>> take some weeks). >>> >>> Thanks >>> Felix >>> >>> On Tue, Jul 18, 2023 at 01:23:27PM +0200, Rodolfo Alonso Hernandez wrote: >>> > Ok, this is being tortuous. First of all: define a strategy. If you are >>> > going to create the resources in Neutron, define how. I've provided a >>> way >>> > to do this, find a formal strategy to ground it. >>> > >>> > Second: (again) try to find a connection between resources, if you are >>> > going to use the strategy of creating the resources in Neutron. The >>> > "Logical_Router_Static_Route" belongs to a router univocally. If that >>> > router has been created by OpenStack, then you can modify the DB sync >>> > method to consider learned routes too. >>> > >>> > In order to do this, you'll need a set of resources that are going to >>> be >>> > needed in Neutron, the OVN counterparts and other resources (like >>> > "Logical_Router_Static_Route") that will be added and will be present >>> in >>> > OVN and not in Neutron DB. Also you'll need to know how to relate all >>> of >>> > them. >>> Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r >>> die Verwertung durch den vorgesehenen Empf?nger bestimmt. >>> Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den >>> Absender bitte unverz?glich in Kenntnis und l?schen diese E Mail. >>> >>> Hinweise zum Datenschutz finden Sie hier>> >. >>> >>> >>> This e-mail may contain confidential content and is intended only for >>> the specified recipient/s. >>> If you are not the intended recipient, please inform the sender >>> immediately and delete this e-mail. >>> >>> Information on data protection can be found here< >>> https://www.datenschutz.schwarz>. >>> >> > > *?Esta mensagem ? direcionada apenas para os endere?os constantes no > cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no > cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa > mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o > imediatamente anuladas e proibidas?.* > > *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para > assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o > poder? aceitar a responsabilidade por quaisquer perdas ou danos causados > por esse e-mail ou por seus anexos?.* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From senrique at redhat.com Wed Jul 19 11:25:23 2023 From: senrique at redhat.com (Sofia Enriquez) Date: Wed, 19 Jul 2023 12:25:23 +0100 Subject: Cinder Bug Report 2023-07-19 Message-ID: Hello Argonauts, Cinder Bug Meeting Etherpad Medium - nvmeof connector _get_host_uuid incompatible when cinder runs in lxd . - *Status*: This looks related to bug 2026257 . Low - Volume to Image upload is very slow due to use of tpool.Proxy() . - *Status*: Fix proposed to master . Cheers, -- Sof?a Enriquez she/her Software Engineer Red Hat PnT IRC: @enriquetaso @RedHat Red Hat Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From roberto.acosta at luizalabs.com Wed Jul 19 14:31:06 2023 From: roberto.acosta at luizalabs.com (Roberto Bartzen Acosta) Date: Wed, 19 Jul 2023 11:31:06 -0300 Subject: [neutron] unmanaged router resources - OVN interconnect In-Reply-To: References: Message-ID: Em qua., 19 de jul. de 2023 ?s 04:57, Rodolfo Alonso Hernandez < ralonsoh at redhat.com> escreveu: > Before that you need to explain why you need multiple external network > ports and why cannot be solved with the current architecture Neutron is > providing. > I and maybe others need this because Neutron's networking backend doesn't have infinite resources. Although the architecture allows for adding a new Logical_Router and configuring static routes, etc... This means consuming more resources on the SDN backend. So the problem here would be scalability: each tenant would need 2 routers to do the work of only 1. In addition to the set of additional LRP's, LSP's, Static Routes that the backend would need to configure. For a few tenants this would be fine, but when we talk about thousands, this question is very relevant. ML2/OVN, for example, limits backend resources according to the OVN version (each one has its scale limitations), in Ussuri's OVN 20.03 (we can't create more than 600 routers and 400k openflow flows without triggering config timeouts - ovn-controller/openvswitch recompute snowball)... in Yoga's OVN 22.03 the numbers are a little better, but we also have limits... Dedicating more resources to doing this would be a waste of infrastructure. > > On Tue, Jul 18, 2023 at 3:52?PM Roberto Bartzen Acosta < > roberto.acosta at luizalabs.com> wrote: > >> >> Would it be possible to implement multiple external network ports on the >> same router? >> >> It is not the same case of multi-homing [1], in the case of OVN-IC there >> are no symmetric internet routes.... >> [1] https://review.opendev.org/c/openstack/neutron/+/874199 >> >> Basically an external port for ovn-ic doesn't have SNAT! >> >> Could this be done Rodolfo? What would be the effort/implications? >> >> If possible, we can integrate everything we are discussing in terms of >> solutions into a single reference design. >> >> >> Em ter., 18 de jul. de 2023 ?s 10:20, Roberto Bartzen Acosta < >> roberto.acosta at luizalabs.com> escreveu: >> >>> Hi Rodolfo, >>> >>> I agree with you, my initial idea with this thread was exactly what we >>> are doing, debating problems and possible solutions for interconnecting >>> workloads using unmanaged dynamic router resources. Thank you for being >>> engaged with this! >>> I'm sorry if the strategy wasn't presented clearly, let me talk about it >>> then. >>> >>> Imagine we have a tenant on OpenStack, and this client has all the >>> resources created by OpenStack itself >>> 1 - Tenant OpenStack project >>> 2 - Tenant self-service network >>> 3 - Tenant self-service subnet - e.g. 10.0.0.0/24 >>> 4 - Security groups >>> 5 - Maybe FIPs >>> 6 - Tenant Logical Router >>> 7 - Tenant Logical Router Ports: 1. external network (provider/default >>> gw) ; 2. internal network (self-service subnet) >>> 8 - Tenant VMs - some VMs on the self-service subnet 10.0.0.0/24 >>> >>> Now that same tenant expects to connect these VMs with another >>> deployment in the same region - High Availability. >>> >>> Option 1 - Floating IPs -> it's not a good option >>> Option 2 - OVN-IC - L3 routing between self-service networks :) >>> >>> What's the strategy? connect the Tenant Logical Router with the TS of >>> the OVN-IC. >>> >>> Creation steps: >>> 1. Create a transit switch in the OVN-IC-NB >>> 2. In each OpenStack / Neutron: >>> a. Create a new Logical_Router_Port (externally managed) on the >>> Tenant Logical Router and link this port with the TS. >>> >>> What is the point here, as Felix comments: even with the creation of the >>> special network type, it will still be necessary to filter learned >>> resources. >>> " a. Ignore all resources where `external_ids` does not contain any key >>> starting with `neutron`." >>> >>> So, from my perspective: >>> Neutron changes for the [RFE] unmanaged dynamic router resources - OVN >>> b. Ignore specific resources as defined by individual rules: >>> I. ignore all Logical_Router_Ports where `external_ids` does not >>> contain any key >>> starting with `neutron`. >>> II. ignore all Logical_Router_Static_Route where `external_ids` >>> does not contain any key >>> starting with `neutron`. >>> >>> >>> We have already tested this and it is enough for interconnect to work in >>> multi-tenant scenarios, and with the router connected to a generic external >>> network provider (where all other tenants are connected). >>> >>> That is, the Logical Router was created by Neutron, so it can decide to >>> learn dynamic routes and ignore external interconnect Logical_Router_Ports. >>> >>> BTW: Felix, the transit switch will learn remote Logical_Switch_Ports >>> ports automatically via OVN-NB, in this case these ports were not created >>> by Neutron and would need more special treatment to skip these resources. >>> >>> Note: about creating a new external provider network type, I already >>> expressed my arguments about the multi tenant case and the need to create >>> more resources to have this together with the default gw. If it were >>> possible to have this in the same Logical_router tenant, I mean, use two >>> external networks on the same Router: here we would have a lot of gain in >>> my perspective >>> >>> Regards, >>> Roberto >>> >>> >>> >>> Em ter., 18 de jul. de 2023 ?s 09:40, Felix Huettner >>> escreveu: >>> >>>> Hi everyone, >>>> >>>> we will validate in the next weeks the idea i'll describe below. If it >>>> sounds feasible to us i would write up a spec for this for later >>>> implementation. >>>> >>>> Setup: >>>> 1. Two neutron setups with OVN >>>> 2. A OVN-IC setup not managed by any tooling (used manually) >>>> >>>> Target setup for a individual project >>>> >>>> +-------------------------------------+ >>>> | Transit Switch (ts1) | >>>> | 172.24.0.0/24 | >>>> +-------------------------------------+ >>>> | | >>>> +----------------+ +----------------+ >>>> | 172.24.0.10 | | 172.24.0.11 | >>>> | Router | | Router | >>>> | 192.168.0.1 | | 192.168.1.1 | >>>> +----------------+ +----------------+ >>>> | | >>>> +----------------+ +----------------+ >>>> | Network | | Network | >>>> | 192.168.0.0/24 | | 192.168.1.0/24 | >>>> +----------------+ +----------------+ >>>> | | >>>> +----------------+ +----------------+ >>>> | VM | | VM | >>>> | 192.168.0.10 | | 192.168.1.10 | >>>> +----------------+ +----------------+ >>>> >>>> Main creation workflow of a interconnected network: >>>> 1. Create a transit switch in the OVN-IC-NB >>>> 2. In each neutron: >>>> a. Create a new provider network that maps to the transit switch. >>>> E.g. using `--provider-network-type ovn-ic >>>> --provider-phyiscal-network ts1` >>>> b. Create the internal network and router as always and attach the >>>> router to the internal network and the provider network >>>> >>>> Changes needed in Neutron (from my perspective): >>>> 1. Neutron must not touch the resources created by OVN-IC. This might be >>>> done using two options: >>>> a. Ignore all resources where `external_ids` does not contain any key >>>> starting with `neutron`. >>>> b. Ignore specific resources as defined by individual rules: >>>> I. ignore all Logical_Router that contain >>>> `other_config:interconn-ts` >>>> II. ignore all Logical_Router_Static_Route that contain >>>> `external_ids:ic-learned-route` >>>> 2. Neutron must allow OVN-IC transit switches to be treated as external >>>> networks. AFAIK this should work in the same way as creating any kind >>>> of OVN network (so Logical_Switch), but just importing an existing >>>> Logical_Switch. >>>> >>>> From my perspective with these two changes we should be good to >>>> interconnect two neutron setups with probably minimal changes needed. >>>> This is because OVN-IC would only touch the transit switch (which looks >>>> in the Northbound like a regular Logical_Switch). All other resources >>>> would still be managed by neutron and work as they do now. >>>> >>>> I will get back to this discussion and (if appropriate) provide a spec >>>> or this idea, once we had time to validate this on our side (which will >>>> take some weeks). >>>> >>>> Thanks >>>> Felix >>>> >>>> On Tue, Jul 18, 2023 at 01:23:27PM +0200, Rodolfo Alonso Hernandez >>>> wrote: >>>> > Ok, this is being tortuous. First of all: define a strategy. If you >>>> are >>>> > going to create the resources in Neutron, define how. I've provided a >>>> way >>>> > to do this, find a formal strategy to ground it. >>>> > >>>> > Second: (again) try to find a connection between resources, if you are >>>> > going to use the strategy of creating the resources in Neutron. The >>>> > "Logical_Router_Static_Route" belongs to a router univocally. If that >>>> > router has been created by OpenStack, then you can modify the DB sync >>>> > method to consider learned routes too. >>>> > >>>> > In order to do this, you'll need a set of resources that are going to >>>> be >>>> > needed in Neutron, the OVN counterparts and other resources (like >>>> > "Logical_Router_Static_Route") that will be added and will be present >>>> in >>>> > OVN and not in Neutron DB. Also you'll need to know how to relate all >>>> of >>>> > them. >>>> Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur >>>> f?r die Verwertung durch den vorgesehenen Empf?nger bestimmt. >>>> Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den >>>> Absender bitte unverz?glich in Kenntnis und l?schen diese E Mail. >>>> >>>> Hinweise zum Datenschutz finden Sie hier< >>>> https://www.datenschutz.schwarz>. >>>> >>>> >>>> This e-mail may contain confidential content and is intended only for >>>> the specified recipient/s. >>>> If you are not the intended recipient, please inform the sender >>>> immediately and delete this e-mail. >>>> >>>> Information on data protection can be found here< >>>> https://www.datenschutz.schwarz>. >>>> >>> >> >> *?Esta mensagem ? direcionada apenas para os endere?os constantes no >> cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no >> cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa >> mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o >> imediatamente anuladas e proibidas?.* >> >> *?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para >> assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o >> poder? aceitar a responsabilidade por quaisquer perdas ou danos causados >> por esse e-mail ou por seus anexos?.* >> > -- _?Esta mensagem ? direcionada apenas para os endere?os constantes no cabe?alho inicial. Se voc? n?o est? listado nos endere?os constantes no cabe?alho, pedimos-lhe que desconsidere completamente o conte?do dessa mensagem e cuja c?pia, encaminhamento e/ou execu??o das a??es citadas est?o imediatamente anuladas e proibidas?._ *?**?Apesar do Magazine Luiza tomar todas as precau??es razo?veis para assegurar que nenhum v?rus esteja presente nesse e-mail, a empresa n?o poder? aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos?.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ozzzo at yahoo.com Wed Jul 19 14:55:03 2023 From: ozzzo at yahoo.com (Albert Braden) Date: Wed, 19 Jul 2023 14:55:03 +0000 (UTC) Subject: LDAP failover fails References: <41076567.2372818.1689778503721.ref@mail.yahoo.com> Message-ID: <41076567.2372818.1689778503721@mail.yahoo.com> We are experiencing the LDAP failover issue described in [1]. Redhat?s solution is to not bother fixing the bug, and to tell customers to put the LDAP server behind a load-balancer. According to Redhat, that is not a good solution for FreeIPA, as explained in [2] and further elucidated in the blog post [3] that it references. I see that the community has a bug open for this [4] and the bug is being worked on here [5] but there has been no activity since 10/22. What is the status of this bugfix? Does it just need someone to review and merge it, or is there more work to be done? How are other FreeIPA users working around this problem? [1] https://bugzilla.redhat.com/show_bug.cgi?id=2024602#c3 [2] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/load-balancing [3] http://ssimo.org/blog/id_019.html [4] https://bugs.launchpad.net/keystone/+bug/1953622 [5] https://review.opendev.org/c/openstack/keystone/+/821086 From kieske at osism.tech Wed Jul 19 15:37:21 2023 From: kieske at osism.tech (Sven Kieske) Date: Wed, 19 Jul 2023 17:37:21 +0200 Subject: [keystone] LDAP failover fails In-Reply-To: <41076567.2372818.1689778503721@mail.yahoo.com> References: <41076567.2372818.1689778503721.ref@mail.yahoo.com> <41076567.2372818.1689778503721@mail.yahoo.com> Message-ID: Hi, I noticed that https://review.opendev.org/c/openstack/keystone/+/860118 is also linked from your bugzilla link. I wasn't aware of the work in https://review.opendev.org/c/openstack/keystone/+/821086 I'm currently trying to fix the ldap breakage in keystone. during the last keystone reviewathons it became clear that it would be better to solve this stuff in the ldappool library itself. regarding the overall project status I guess it's fair to say that ldap support ist pretty dormant right now. This is my first dive into the keystone codebase, so I guess it's save to say that additional people interested in ldap would be more than welcome. But I guess the core keystone team can say more about this. Having said all this, I guess this explains the general status of ldap related patches in keystone. HTH & kind regards Am Mittwoch, dem 19.07.2023 um 14:55 +0000 schrieb Albert Braden: > We are experiencing the LDAP failover issue described in [1]. > Redhat?s solution is to not bother fixing the bug, and to tell > customers to put the LDAP server behind a load-balancer. According to > Redhat, that is not a good solution for FreeIPA, as explained in [2] > and further elucidated in the blog post [3] that it references. I see > that the community has a bug open for this [4] and the bug is being > worked on here [5] but there has been no activity since 10/22. > > What is the status of this bugfix? Does it just need someone to > review and merge it, or is there more work to be done? How are other > FreeIPA users working around this problem? > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=2024602#c3 > [2] > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/load-balancing > [3] http://ssimo.org/blog/id_019.html > [4] https://bugs.launchpad.net/keystone/+bug/1953622 > [5] https://review.opendev.org/c/openstack/keystone/+/821086 > -- Sven Kieske Senior Cloud Engineer Mail: kieske at osism.tech Web: https://osism.tech OSISM GmbH Teckstra?e 62 / 70190 Stuttgart / Deutschland Gesch?ftsf?hrer: Christian Berendt Unternehmenssitz: Stuttgart Amtsgericht: Stuttgart, HRB 756139 From kristin at openinfra.dev Wed Jul 19 19:13:33 2023 From: kristin at openinfra.dev (Kristin Barrientos) Date: Wed, 19 Jul 2023 14:13:33 -0500 Subject: Announcement: OpenStack "C" Release Name Message-ID: <3F7034D9-7F0A-4CD4-82B3-EFC2BFD51E23@openinfra.dev> Hi Everyone, Happy 13th birthday to OpenStack! We're proud to announce that the name chosen for the ?C? release is? Caracal! Thank you to everyone who voted. Kristin Barrientos Marketing Coordinator OpenInfra Foundation -------------- next part -------------- An HTML attachment was scrubbed... URL: From mnaser at vexxhost.com Thu Jul 20 05:18:45 2023 From: mnaser at vexxhost.com (Mohammed Naser) Date: Thu, 20 Jul 2023 05:18:45 +0000 Subject: [neutron][vpnaas][ovn] Pushing VPNaaS support through OVN Message-ID: Hi all! One of the biggest show stoppers for us for OVN is VPNaaS but I think it?s time that we?d like to be involved in getting it over the line, I want to thank Bodo who has shown a lot of patience in getting the code necessary, and it seems that the code works for them in production as well. I?d like to ask if possible for folks to review these two outstanding changes: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/847007 https://review.opendev.org/c/openstack/neutron-vpnaas/+/765353 I know Slawek had some concerns, so hoping that raising via the email list if Bodo can address those and over the next bit of time we can get this landed in, since it even has actual proper testing both in CI and with user feedback. Thanks! Mohammed -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Thu Jul 20 06:28:54 2023 From: skaplons at redhat.com (Slawek Kaplonski) Date: Thu, 20 Jul 2023 08:28:54 +0200 Subject: [neutron][vpnaas][ovn] Pushing VPNaaS support through OVN In-Reply-To: References: Message-ID: <2101585.0t5x7J7qO2@p1> Hi, Dnia czwartek, 20 lipca 2023 07:18:45 CEST Mohammed Naser pisze: > Hi all! > > One of the biggest show stoppers for us for OVN is VPNaaS but I think it?s time that we?d like to be involved in getting it over the line, I want to thank Bodo who has shown a lot of patience in getting the code necessary, and it seems that the code works for them in production as well. > > I?d like to ask if possible for folks to review these two outstanding changes: > > https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/847007 > https://review.opendev.org/c/openstack/neutron-vpnaas/+/765353 > > I know Slawek had some concerns, so hoping that raising via the email list if Bodo can address those and over the next bit of time we can get this landed in, since it even has actual proper testing both in CI and with user feedback. > > Thanks! > Mohammed > Thx for raising this up. I just added those patches to my review list but I don't think I will be able to do it this week. So probably next week I will look at them. -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From noonedeadpunk at gmail.com Thu Jul 20 06:40:05 2023 From: noonedeadpunk at gmail.com (Dmitriy Rabotyagov) Date: Thu, 20 Jul 2023 08:40:05 +0200 Subject: [neutron][vpnaas][ovn] Pushing VPNaaS support through OVN In-Reply-To: <2101585.0t5x7J7qO2@p1> References: <2101585.0t5x7J7qO2@p1> Message-ID: I wanted to test and review these patches as well, but this has not left a backlog yet. But wanted to say that we are also interested in this and will be able to help out, but only after summer. On Thu, Jul 20, 2023, 08:34 Slawek Kaplonski wrote: > Hi, > > Dnia czwartek, 20 lipca 2023 07:18:45 CEST Mohammed Naser pisze: > > Hi all! > > > > One of the biggest show stoppers for us for OVN is VPNaaS but I think > it?s time that we?d like to be involved in getting it over the line, I want > to thank Bodo who has shown a lot of patience in getting the code > necessary, and it seems that the code works for them in production as well. > > > > I?d like to ask if possible for folks to review these two outstanding > changes: > > > > https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/847007 > > https://review.opendev.org/c/openstack/neutron-vpnaas/+/765353 > > > > I know Slawek had some concerns, so hoping that raising via the email > list if Bodo can address those and over the next bit of time we can get > this landed in, since it even has actual proper testing both in CI and with > user feedback. > > > > Thanks! > > Mohammed > > > > Thx for raising this up. I just added those patches to my review list but > I don't think I will be able to do it this week. So probably next week I > will look at them. > > -- > Slawek Kaplonski > Principal Software Engineer > Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian.rohmann at inovex.de Thu Jul 20 10:11:51 2023 From: christian.rohmann at inovex.de (Christian Rohmann) Date: Thu, 20 Jul 2023 12:11:51 +0200 Subject: [all] [oslo.messaging] Interest in collaboration on a NATS driver In-Reply-To: <12EB84AA-A70A-4246-ABEC-4A4BC9E7B682@binero.com> References: <52AA12A0-AE67-4EF7-B924-DE1F2873B909@binero.com> <0E347D5A-967D-42E3-AC00-56034907053B@binero.com> <8F7F790F-0F5C-4257-845A-1BCE671AF844@binero.com> <3b2f711e-a2f1-71da-92ba-d36c229420a2@inovex.de> <12EB84AA-A70A-4246-ABEC-4A4BC9E7B682@binero.com> Message-ID: Hey Tobias, On 15/07/2023 16:27, Tobias Urdin wrote: > The effort is mostly now to get/find a supported third-party Python library that does not > use asyncio for usage in a oslo.messaging driver. > > The investigation into introducing asyncio support in the ecosystem (oslo and then all > separate projects) was a too massive effort. > > Since then I?ve opened an issue [1] on the nats.py GitHub and some proof-of-concept > patches [2] on introducing synchronous support in nats.py that we could then use in a driver. > > The oslo.messaging driver patch [3] has Devstack passing when using [2] but it?s also > more of a proof-of-concept patch that would need more work in itself. > > Unfortunately I didn?t get any feedback from the company maintaining the nats.py library > on supported developer time implemeting the sync client feature, I assume (my view) that > they didn?t like that we cannot guarantee that NATS would become a core component (or used, or the primary bus) > in the ecosystem (since we don?t, and don?t want to for that matter, force a backend/driver on operators). > > Any help is of course appreciated, right now this is not a top priority and I don?t spend any full-time > effort into it. > > Best regards > Tobias > > [1] https://github.com/nats-io/nats.py/issues/462 > [2] https://github.com/tobias-urdin/nats.py/commits/sync-client > [3] https://review.opendev.org/c/openstack/oslo.messaging/+/878588 > Thanks for the rapid response and full overview of your NATS endeavors! I shall subscribe to your kindly linked issues. There currently also is a larger number of fixes for oslo.messaging via RabbitMQ incoming: a shift to quorum queues, fixed threading for heartbeats, adjusted timers and timeouts, .... Maybe they relieve a bit of the perceived pain with (oslo.messaging via) RabbitMQ. To me it's not about having even more choices in messaging drivers (or coordination backends for that matter). All of them working slightly different, having different features and them each having +50 options to fiddle with all sorts of timers or adjust other settings.? I'd rather have a single solution that is broadly used by most installs and that, proper setup, configuration and scaling provided, "just works". Even though there is an abstraction with oslo.(messaging|coordination), there always remain implications for the OpenStack components depending on which backend / driver one uses. Anyways, thanks again for your response! Regards Christian From rdhasman at redhat.com Thu Jul 20 17:13:20 2023 From: rdhasman at redhat.com (Rajat Dhasmana) Date: Thu, 20 Jul 2023 22:43:20 +0530 Subject: [cinder] festival of feature reviews 21st July, 2023 Message-ID: Hello Argonauts, We will be having our monthly festival of reviews today i.e. 21st July (Friday) from 1400-1600 UTC. Following are some additional details: Date: 21st July, 2023 Time: 1400-1600 UTC Meeting link: will be shared in #openstack-cinder around 1400 UTC etherpad: https://etherpad.opendev.org/p/cinder-festival-of-reviews Attached is the invite if you want to add it to your calendar. Thanks Rajat Dhasmana -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/calendar Size: 916 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cinder-festival-of-reviews (3).ics Type: text/calendar Size: 937 bytes Desc: not available URL: From ozzzo at yahoo.com Thu Jul 20 19:14:23 2023 From: ozzzo at yahoo.com (Albert Braden) Date: Thu, 20 Jul 2023 19:14:23 +0000 (UTC) Subject: [keystone] LDAP failover fails In-Reply-To: References: <41076567.2372818.1689778503721.ref@mail.yahoo.com> <41076567.2372818.1689778503721@mail.yahoo.com> Message-ID: <1876779442.3052584.1689880463257@mail.yahoo.com> When you say "solve this stuff in the ldappool library" are you talking about moving the broken server to the end of the pool instead of removing it? Since the first server in the URL is always used, and failover doesn't seem to work, it seems like moving the broken URL to the end of the list would be a good solution, and that would eliminate the problem of having to add it back after it starts working again. I'm looking at the ldappool code here: https://opendev.org/openstack/ldappool/src/branch/master/ldappool/__init__.py So far it's not obvious to me how the pool is being assembled. What is the relationship between the LDAP URLs in the Keystone config, and the connections in the pool? What would have to change, to allow a failing URL to be treated as if it were not the first one in the list? On Wednesday, July 19, 2023 at 11:46:25 AM EDT, Sven Kieske wrote: Hi, I noticed that https://review.opendev.org/c/openstack/keystone/+/860118 is also linked from your bugzilla link. I wasn't aware of the work in https://review.opendev.org/c/openstack/keystone/+/821086 I'm currently trying to fix the ldap breakage in keystone. during the last keystone reviewathons it became clear that it would be better to solve this stuff in the ldappool library itself. regarding the overall project status I guess it's fair to say that ldap support ist pretty dormant right now. This is my first dive into the keystone codebase, so I guess it's save to say that additional people interested in ldap would be more than welcome. But I guess the core keystone team can say more about this. Having said all this, I guess this explains the general status of ldap related patches in keystone. HTH & kind regards Am Mittwoch, dem 19.07.2023 um 14:55 +0000 schrieb Albert Braden: > We are experiencing the LDAP failover issue described in [1]. > Redhat?s solution is to not bother fixing the bug, and to tell > customers to put the LDAP server behind a load-balancer. According to > Redhat, that is not a good solution for FreeIPA, as explained in [2] > and further elucidated in the blog post [3] that it references. I see > that the community has a bug open for this [4] and the bug is being > worked on here [5] but there has been no activity since 10/22. > > What is the status of this bugfix? Does it just need someone to > review and merge it, or is there more work to be done? How are other > FreeIPA users working around this problem? > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=2024602#c3 > [2] > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/load-balancing > [3] http://ssimo.org/blog/id_019.html > [4] https://bugs.launchpad.net/keystone/+bug/1953622 > [5] https://review.opendev.org/c/openstack/keystone/+/821086 > -- Sven Kieske Senior Cloud Engineer Mail: kieske at osism.tech Web: https://osism.tech OSISM GmbH Teckstra?e 62 / 70190 Stuttgart / Deutschland Gesch?ftsf?hrer: Christian Berendt Unternehmenssitz: Stuttgart Amtsgericht: Stuttgart, HRB 756139 -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristin at openinfra.dev Thu Jul 20 21:15:04 2023 From: kristin at openinfra.dev (Kristin Barrientos) Date: Thu, 20 Jul 2023 16:15:04 -0500 Subject: Superuser / OpenInfra Live Opportunities? Message-ID: <940B1F14-42B3-44AD-B7D3-F46B8E0E2234@openinfra.dev> Hi Everyone, OpenInfra Live is looking for ideas and participants to join a panel to discuss those topics. That being said? do you have a topic in mind that you would like to see on OpenInfra Live? We want to hear from you! Some ideas can include: Data Sovereignty Multi-Region Deployment Kolla-Ansible AI/ML And many more! If you would like to participate or have an idea, you can contact me via email (kristin at openinfra.dev) or reply here, and we can work on getting something started. If you would rather write a Superuser article, feel free to email me an abstract and we can go from there. I look forward to hearing everyone?s ideas! Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation -------------- next part -------------- An HTML attachment was scrubbed... URL: From murilo at evocorp.com.br Thu Jul 20 23:00:32 2023 From: murilo at evocorp.com.br (Murilo Morais) Date: Thu, 20 Jul 2023 20:00:32 -0300 Subject: [OSA][NEUTRON] Failed to bind port XXX on host YYY for vnic_type normal using segments... In-Reply-To: References: Message-ID: Dmitry, good night! I decided to start from scratch and remove things that I won't be using (at least not now). When removing the flat network the errors are gone and the VMs start. I just have another question related to Neutron, but I'll start another thread. Thanks a lot for the help! Em ter., 18 de jul. de 2023 ?s 14:12, Dmitriy Rabotyagov < noonedeadpunk at gmail.com> escreveu: > Hey, > > Thanks for providing the output. > Regarding neutron-openvswitch-agent utilization - that might be a > side-effect of neutron-server failures. > > Regarding neutron-server failures, reading stack trace again, it seems > that neutron-server is failing to reach the placement service: > HTTPConnectionPool(host='172.29.236.2', port=8780) Failed to > establish a new connection: [Errno 32] EPIPE > > I assume that 172.29.236.2 is your internal VIP that's on haproxy? Are > you able to reach the IP and connect to the port 8780 from inside of > the neutron-server container? You can try leveraging telnet or simple > curl from inside of the neutron-server container to verify that > connectivity is fine. > > Also good to ensure, that haproxy does have placement backends UP, you > can verify that with running `echo "show stat" | nc -U > /run/haproxy.stat | grep placement` > > ??, 18 ???. 2023??. ? 15:14, Murilo Morais : > > > > I noticed that the "neutron-openvswitch-agent" process constantly goes > up to 100% in CPU usage. This is normal? > > > > Em ter., 18 de jul. de 2023 ?s 09:44, Murilo Morais < > murilo at evocorp.com.br> escreveu: > >> > >> Hi Dmitriy, thanks for answering! > >> > >> I really didn't send any details of my setup, apologies for that. > >> > >> I'm using OVS with the following configuration: > >> > >> provider_networks: > >> - network: > >> container_bridge: br-provider > >> container_type: veth > >> type: vlan > >> range: "100:200" > >> net_name: vlan > >> group_binds: > >> - neutron_openvswitch_agent > >> > >> - network: > >> container_bridge: br-provider > >> container_type: veth > >> type: flat > >> net_name: flat > >> group_binds: > >> - neutron_openvswitch_agent > >> > >> > >> neutron_plugin_type: ml2.ovs > >> neutron_ml2_drivers_type: "flat,vlan" > >> neutron_plugin_base: > >> - router > >> - metering > >> > >> > >> root at dcn2-utility-container-c45f8b09:/# openstack network agent list > >> > +--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+ > >> | ID | Agent Type | Host | > Availability Zone | Alive | State | Binary | > >> > +--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+ > >> | 9a4625ef-2988-4b96-a927-30a9bb0244a4 | Metadata agent | dcn2 | None > | :-) | UP | neutron-metadata-agent | > >> | a222c0ae-c2e5-44cc-b478-ca8176daad19 | Metering agent | dcn2 | None > | :-) | UP | neutron-metering-agent | > >> | c6be1985-a67e-4099-a1d6-fa517810e138 | L3 agent | dcn2 | nova > | :-) | UP | neutron-l3-agent | > >> | da97e2a6-535b-4f1e-828d-9b2fbb3e036b | DHCP agent | dcn2 | nova > | :-) | UP | neutron-dhcp-agent | > >> > +--------------------------------------+----------------+------+-------------------+-------+-------+------------------------+ > >> > >> > >> root at dcn2-utility-container-c45f8b09:/# openstack compute service list > >> > +--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+ > >> | ID | Binary | Host > | Zone | Status | State | Updated At > | > >> > +--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+ > >> | a0524af5-bc88-4793-aee2-c2c87cd0e8cc | nova-conductor | > dcn2-nova-api-container-aac2b913 | internal | enabled | up | > 2023-07-18T12:32:26.000000 | > >> | c860f25e-bd30-49f9-8289-076b230bbc2d | nova-scheduler | > dcn2-nova-api-container-aac2b913 | internal | enabled | up | > 2023-07-18T12:32:27.000000 | > >> | 6457e0a1-b075-4999-8855-0f36e2e3a95a | nova-compute | dcn2 > | nova | enabled | up | > 2023-07-18T12:32:27.000000 | > >> > +--------------------------------------+----------------+----------------------------------+----------+---------+-------+----------------------------+ > >> > >> > >> The br-provider bridge exists and is UP. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From murilo at evocorp.com.br Thu Jul 20 23:17:48 2023 From: murilo at evocorp.com.br (Murilo Morais) Date: Thu, 20 Jul 2023 20:17:48 -0300 Subject: [OSA][NEUTRON] DHCP port IP Message-ID: Good evening everyone! I'm using OVS + VLAN Provider Guys, is there any way to remove IP from DHCP (network:dhcp)? My gateway is external, I need the VMs to obtain the IPs directly on the interface without using SNAT/DNAT, but when enabling DHCP on the subnet, the DHCP interface consumes an IP (the first one available). Is there a way to remove this blessed IP? If not, what other path can I take? I'm thinking of other alternatives: 1. Use static IP 2. Use a DHCP Server on the Gateway (which makes me have to perform some manual operations to assign the correct IP) What would be more viable? Thanks in advance! -------------- next part -------------- An HTML attachment was scrubbed... URL: From ihrachys at redhat.com Fri Jul 21 01:19:25 2023 From: ihrachys at redhat.com (Ihar Hrachyshka) Date: Thu, 20 Jul 2023 21:19:25 -0400 Subject: [OSA][NEUTRON] DHCP port IP In-Reply-To: References: Message-ID: On Thu, Jul 20, 2023 at 7:18?PM Murilo Morais wrote: > Good evening everyone! > > I'm using OVS + VLAN Provider > > Guys, is there any way to remove IP from DHCP (network:dhcp)? > > You can create a subnet with enable_dhcp=False, then no DHCP port is allocated. Then let Neutron allocate addresses from the subnet range for your VMs (just make sure that the addresses are not reused outside of Neutron). My gateway is external, I need the VMs to obtain the IPs directly on the > interface without using SNAT/DNAT, but when enabling DHCP on the subnet, > the DHCP interface consumes an IP (the first one available). > > Is there a way to remove this blessed IP? If not, what other path can I > take? > > I'm thinking of other alternatives: > 1. Use static IP > 2. Use a DHCP Server on the Gateway (which makes me have to perform some > manual operations to assign the correct IP) > > What would be more viable? > > Thanks in advance! > -------------- next part -------------- An HTML attachment was scrubbed... URL: From noonedeadpunk at gmail.com Fri Jul 21 02:08:14 2023 From: noonedeadpunk at gmail.com (Dmitriy Rabotyagov) Date: Fri, 21 Jul 2023 04:08:14 +0200 Subject: [OSA][NEUTRON] DHCP port IP In-Reply-To: References: Message-ID: If the thing that concerns you is that DHCP consumes gateway IP, you should define your gateway when creating the subnet with --gateway option [1]. Also I *think* that if you define --allocation-pool while creating subnet, DHCP server will obtain IP within this pool, but I'm not 100% sure now, as I had such setup years ago. If you are disabling DHCP, then ensure your VMs will use config-drives for metadata instead of fetching it through the network. For that you will need to supply an option while creating the VM [1] https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/subnet.html#cmdoption-openstack-subnet-create-gateway On Fri, Jul 21, 2023, 03:24 Ihar Hrachyshka wrote: > On Thu, Jul 20, 2023 at 7:18?PM Murilo Morais > wrote: > >> Good evening everyone! >> >> I'm using OVS + VLAN Provider >> >> Guys, is there any way to remove IP from DHCP (network:dhcp)? >> >> > You can create a subnet with enable_dhcp=False, then no DHCP port is > allocated. Then let Neutron allocate addresses from the subnet range for > your VMs (just make sure that the addresses are not reused outside of > Neutron). > > My gateway is external, I need the VMs to obtain the IPs directly on the >> interface without using SNAT/DNAT, but when enabling DHCP on the subnet, >> the DHCP interface consumes an IP (the first one available). >> >> Is there a way to remove this blessed IP? If not, what other path can I >> take? >> >> I'm thinking of other alternatives: >> 1. Use static IP >> 2. Use a DHCP Server on the Gateway (which makes me have to perform some >> manual operations to assign the correct IP) >> >> What would be more viable? >> >> Thanks in advance! >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michel.jouvin at ijclab.in2p3.fr Fri Jul 21 05:45:00 2023 From: michel.jouvin at ijclab.in2p3.fr (Michel Jouvin) Date: Fri, 21 Jul 2023 07:45:00 +0200 Subject: Nova Ussuri: DB online migration cannot migrate (some) populate_user_id Message-ID: Hi, We upgraded one of our clouds to Ussuri (we are late!) a couple of months ago. After the upgrade 'nova-manage db online_data_migration' was run. fill_virtual_interface_list and populate_user_id where non zero and after running the command fill_virtual_interface_list was 0 but populate_user_id stayed 24. Looking in nova-manage.log, I found the follwoing message (twice every time the command is run): Some instance mappings were not migratable. This may be transient due to in-flight instance builds, or could be due to stale data that will be cleaned up after running "nova-manage db archive_deleted_rows --purge". I ran the suggest command but the result is that fill_virtual_interface_list is non zero again (15) and there is no change to populate_user_i. If I run again online_data_migration, it clears again? fill_virtual_interface_list... and so on. After 2 months the situation remains the same, no increase in populate_user_id but no possibility to fix them. We were wondering if it could have been caused by the fact that we had not yet upgraded all the compute nodes when we first ran the online_data_migration. We are planning to upgrade to Victoria and we'd like to fix the situation before as we understand that online data migration not done can be blocking for the next upgrade. We don't have any clue on the reason this happened. It is a test cloud with a very low usage. On our production cloud we have a somewhat similar problem but populate_user_id=1 and running the purge doesn't increase fill_virtual_interface_list. Thanks in advance for any hint! Best regards, Michel From zigo at debian.org Fri Jul 21 07:00:58 2023 From: zigo at debian.org (Thomas Goirand) Date: Fri, 21 Jul 2023 09:00:58 +0200 Subject: [all] [oslo.messaging] Interest in collaboration on a NATS driver In-Reply-To: <12EB84AA-A70A-4246-ABEC-4A4BC9E7B682@binero.com> References: <52AA12A0-AE67-4EF7-B924-DE1F2873B909@binero.com> <0E347D5A-967D-42E3-AC00-56034907053B@binero.com> <8F7F790F-0F5C-4257-845A-1BCE671AF844@binero.com> <3b2f711e-a2f1-71da-92ba-d36c229420a2@inovex.de> <12EB84AA-A70A-4246-ABEC-4A4BC9E7B682@binero.com> Message-ID: <74d9d34b-7048-e0d5-14f4-7a5361afbdb1@debian.org> On 7/15/23 16:27, Tobias Urdin wrote: > The effort is mostly now to get/find a supported third-party Python library that does not > use asyncio for usage in a oslo.messaging driver. Sorry to ask, but what's the problem with asyncio? Do we want something *not* async? If yes, why? > Unfortunately I didn?t get any feedback from the company maintaining the nats.py library > on supported developer time implemeting the sync client feature, I assume (my view) that > they didn?t like that we cannot guarantee that NATS would become a core component (or used, or the primary bus) > in the ecosystem (since we don?t, and don?t want to for that matter, force a backend/driver on operators). > > Any help is of course appreciated, right now this is not a top priority and I don?t spend any full-time > effort into it. Would it help if I was packaging NATS in Debian (and therefore, as a consequence, in Ubuntu)? By reading the go.mod, it feels like it has a reasonable amount of dependencies (though I didn't check for the indirect ones), so it feels like doable... Cheers, Thomas Goirand (zigo) From ralonsoh at redhat.com Fri Jul 21 08:13:32 2023 From: ralonsoh at redhat.com (Rodolfo Alonso Hernandez) Date: Fri, 21 Jul 2023 10:13:32 +0200 Subject: [neutron] Next Neutron meetings schedule Message-ID: Hello Neutrinos: Today we have the Neutron drivers meeting at 1400UTC. This is the agenda: https://wiki.openstack.org/wiki/Meetings/NeutronDrivers. We have one topic: https://bugs.launchpad.net/neutron/+bug/2027742. Next Tuesday we have cancelled the Neutron team meeting (I'll be on PTO). But remember that the Neutron CI meeting will take place at 1500UTC. Regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From radoslaw.piliszek at gmail.com Fri Jul 21 09:53:19 2023 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Fri, 21 Jul 2023 11:53:19 +0200 Subject: Superuser / OpenInfra Live Opportunities? In-Reply-To: <940B1F14-42B3-44AD-B7D3-F46B8E0E2234@openinfra.dev> References: <940B1F14-42B3-44AD-B7D3-F46B8E0E2234@openinfra.dev> Message-ID: Dear Kristin, I am open for AI/ML panels/discussions, as well as multi- and hybrid-cloud. Kind regards, Radek On Thu, 20 Jul 2023 at 23:17, Kristin Barrientos wrote: > > Hi Everyone, > > OpenInfra Live is looking for ideas and participants to join a panel to discuss those topics. > > That being said? do you have a topic in mind that you would like to see on OpenInfra Live? We want to hear from you! > > Some ideas can include: > > Data Sovereignty > Multi-Region Deployment > Kolla-Ansible > AI/ML > And many more! > > > If you would like to participate or have an idea, you can contact me via email (kristin at openinfra.dev) or reply here, and we can work on getting something started. > > If you would rather write a Superuser article, feel free to email me an abstract and we can go from there. > > I look forward to hearing everyone?s ideas! > > Thanks, > > Kristin Barrientos > Marketing Coordinator > OpenInfra Foundation > > > > From radoslaw.piliszek at gmail.com Fri Jul 21 09:59:00 2023 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Fri, 21 Jul 2023 11:59:00 +0200 Subject: [all] [oslo.messaging] Interest in collaboration on a NATS driver In-Reply-To: <74d9d34b-7048-e0d5-14f4-7a5361afbdb1@debian.org> References: <52AA12A0-AE67-4EF7-B924-DE1F2873B909@binero.com> <0E347D5A-967D-42E3-AC00-56034907053B@binero.com> <8F7F790F-0F5C-4257-845A-1BCE671AF844@binero.com> <3b2f711e-a2f1-71da-92ba-d36c229420a2@inovex.de> <12EB84AA-A70A-4246-ABEC-4A4BC9E7B682@binero.com> <74d9d34b-7048-e0d5-14f4-7a5361afbdb1@debian.org> Message-ID: On Fri, 21 Jul 2023 at 09:03, Thomas Goirand wrote: > > On 7/15/23 16:27, Tobias Urdin wrote: > > The effort is mostly now to get/find a supported third-party Python library that does not > > use asyncio for usage in a oslo.messaging driver. > > Sorry to ask, but what's the problem with asyncio? Do we want something > *not* async? If yes, why? Large swaths of OpenStack code are not ready for async. We want to take baby steps and get to some better place with tangible benefits. > > Unfortunately I didn?t get any feedback from the company maintaining the nats.py library > > on supported developer time implemeting the sync client feature, I assume (my view) that > > they didn?t like that we cannot guarantee that NATS would become a core component (or used, or the primary bus) > > in the ecosystem (since we don?t, and don?t want to for that matter, force a backend/driver on operators). > > > > Any help is of course appreciated, right now this is not a top priority and I don?t spend any full-time > > effort into it. > > Would it help if I was packaging NATS in Debian (and therefore, as a > consequence, in Ubuntu)? By reading the go.mod, it feels like it has a > reasonable amount of dependencies (though I didn't check for the > indirect ones), so it feels like doable... NATS's preferred way of delivery is a single (!) binary offered by the upstream. I guess you could help considerably with minimum effort by repackaging it into a deb so that users going for apt hunting get the upstream NATS. Then it would only boil down to finally packaging the Python library that works well with both OpenStack and NATS but we don't have it yet. Kind regards, Radek -yoctozepto From smooney at redhat.com Fri Jul 21 11:32:01 2023 From: smooney at redhat.com (smooney at redhat.com) Date: Fri, 21 Jul 2023 12:32:01 +0100 Subject: [all] [oslo.messaging] Interest in collaboration on a NATS driver In-Reply-To: References: <52AA12A0-AE67-4EF7-B924-DE1F2873B909@binero.com> <0E347D5A-967D-42E3-AC00-56034907053B@binero.com> <8F7F790F-0F5C-4257-845A-1BCE671AF844@binero.com> <3b2f711e-a2f1-71da-92ba-d36c229420a2@inovex.de> <12EB84AA-A70A-4246-ABEC-4A4BC9E7B682@binero.com> <74d9d34b-7048-e0d5-14f4-7a5361afbdb1@debian.org> Message-ID: On Fri, 2023-07-21 at 11:59 +0200, Rados?aw Piliszek wrote: > On Fri, 21 Jul 2023 at 09:03, Thomas Goirand wrote: > > > > On 7/15/23 16:27, Tobias Urdin wrote: > > > The effort is mostly now to get/find a supported third-party Python library that does not > > > use asyncio for usage in a oslo.messaging driver. > > > > Sorry to ask, but what's the problem with asyncio? Do we want something > > *not* async? If yes, why? > > Large swaths of OpenStack code are not ready for async. > We want to take baby steps and get to some better place with tangible benefits. that not entirly correct. much of openstack is already async but we made a decision ot use eventlet as our async model many many years ago after we ported form twisted. The problem is you cannot really mix eventlet code and asyncio safely in the python process. That means you cannot have the async nats lib used in nova,swift,cinder,neutron, glance and progrably other projects today. i proposed one option a long time ago which is to take a privsep like approch where we spawn the oslo messaging driver into a sperate process and comunicate with it over a unix socket. that would allow the oslo.messaging drvier to run with asynio and all the existing project to consume it unmodifed. a unix socket is just one way to hanel that of couse you coudl use any other ipc liek shared memory but the core logic for that exist in oslo.privsep in the form of the chanels it provids so we have prior art to work form. but thats the main problem. most of openstack uses eventlets for implict async. eventlets and explict async are not safe to mix in the same process. and that is the bocker to the current python nats driver as far as i am aware. > > > > Unfortunately I didn?t get any feedback from the company maintaining the nats.py library > > > on supported developer time implemeting the sync client feature, I assume (my view) that > > > they didn?t like that we cannot guarantee that NATS would become a core component (or used, or the primary bus) > > > in the ecosystem (since we don?t, and don?t want to for that matter, force a backend/driver on operators). > > > > > > Any help is of course appreciated, right now this is not a top priority and I don?t spend any full-time > > > effort into it. > > > > Would it help if I was packaging NATS in Debian (and therefore, as a > > consequence, in Ubuntu)? By reading the go.mod, it feels like it has a > > reasonable amount of dependencies (though I didn't check for the > > indirect ones), so it feels like doable... > > NATS's preferred way of delivery is a single (!) binary offered by the upstream. > I guess you could help considerably with minimum effort by repackaging > it into a deb so that users going for apt hunting get the upstream > NATS. > Then it would only boil down to finally packaging the Python library > that works well with both OpenStack and NATS but we don't have it yet. > > Kind regards, > Radek > -yoctozepto > From radoslaw.piliszek at gmail.com Fri Jul 21 11:42:53 2023 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Fri, 21 Jul 2023 13:42:53 +0200 Subject: [all] [oslo.messaging] Interest in collaboration on a NATS driver In-Reply-To: References: <52AA12A0-AE67-4EF7-B924-DE1F2873B909@binero.com> <0E347D5A-967D-42E3-AC00-56034907053B@binero.com> <8F7F790F-0F5C-4257-845A-1BCE671AF844@binero.com> <3b2f711e-a2f1-71da-92ba-d36c229420a2@inovex.de> <12EB84AA-A70A-4246-ABEC-4A4BC9E7B682@binero.com> <74d9d34b-7048-e0d5-14f4-7a5361afbdb1@debian.org> Message-ID: Thanks, Sean. Your summary is more elaborate. And I meant asyncio*, not async programming. Sorry for any confusion. On Fri, 21 Jul 2023, 13:32 , wrote: > On Fri, 2023-07-21 at 11:59 +0200, Rados?aw Piliszek wrote: > > On Fri, 21 Jul 2023 at 09:03, Thomas Goirand wrote: > > > > > > On 7/15/23 16:27, Tobias Urdin wrote: > > > > The effort is mostly now to get/find a supported third-party Python > library that does not > > > > use asyncio for usage in a oslo.messaging driver. > > > > > > Sorry to ask, but what's the problem with asyncio? Do we want something > > > *not* async? If yes, why? > > > > Large swaths of OpenStack code are not ready for async. > > We want to take baby steps and get to some better place with tangible > benefits. > that not entirly correct. > much of openstack is already async but we made a decision ot use eventlet > as our async model many many > years ago after we ported form twisted. > > The problem is you cannot really mix eventlet code and asyncio safely in > the python process. > That means you cannot have the async nats lib used in > nova,swift,cinder,neutron, glance and progrably other > projects today. > > i proposed one option a long time ago which is to take a privsep like > approch where we spawn the oslo messaging driver > into a sperate process and comunicate with it over a unix socket. > > that would allow the oslo.messaging drvier to run with asynio and all the > existing project to consume it unmodifed. > a unix socket is just one way to hanel that of couse you coudl use any > other ipc liek shared memory but the > core logic for that exist in oslo.privsep in the form of the chanels it > provids so we have prior art to work form. > > but thats the main problem. most of openstack uses eventlets for implict > async. > eventlets and explict async are not safe to mix in the same process. > and that is the bocker to the current python nats driver as far as i am > aware. > > > > > > Unfortunately I didn?t get any feedback from the company maintaining > the nats.py library > > > > on supported developer time implemeting the sync client feature, I > assume (my view) that > > > > they didn?t like that we cannot guarantee that NATS would become a > core component (or used, or the primary bus) > > > > in the ecosystem (since we don?t, and don?t want to for that matter, > force a backend/driver on operators). > > > > > > > > Any help is of course appreciated, right now this is not a top > priority and I don?t spend any full-time > > > > effort into it. > > > > > > Would it help if I was packaging NATS in Debian (and therefore, as a > > > consequence, in Ubuntu)? By reading the go.mod, it feels like it has a > > > reasonable amount of dependencies (though I didn't check for the > > > indirect ones), so it feels like doable... > > > > NATS's preferred way of delivery is a single (!) binary offered by the > upstream. > > I guess you could help considerably with minimum effort by repackaging > > it into a deb so that users going for apt hunting get the upstream > > NATS. > > Then it would only boil down to finally packaging the Python library > > that works well with both OpenStack and NATS but we don't have it yet. > > > > Kind regards, > > Radek > > -yoctozepto > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zigo at debian.org Fri Jul 21 13:35:03 2023 From: zigo at debian.org (Thomas Goirand) Date: Fri, 21 Jul 2023 15:35:03 +0200 Subject: [all] [oslo.messaging] Interest in collaboration on a NATS driver In-Reply-To: References: <52AA12A0-AE67-4EF7-B924-DE1F2873B909@binero.com> <0E347D5A-967D-42E3-AC00-56034907053B@binero.com> <8F7F790F-0F5C-4257-845A-1BCE671AF844@binero.com> <3b2f711e-a2f1-71da-92ba-d36c229420a2@inovex.de> <12EB84AA-A70A-4246-ABEC-4A4BC9E7B682@binero.com> <74d9d34b-7048-e0d5-14f4-7a5361afbdb1@debian.org> Message-ID: <1ca9e974-a6a2-7b08-b129-416fe00172f5@debian.org> On 7/21/23 11:59, Rados?aw Piliszek wrote: > On Fri, 21 Jul 2023 at 09:03, Thomas Goirand wrote: >>> Unfortunately I didn?t get any feedback from the company maintaining the nats.py library >>> on supported developer time implemeting the sync client feature, I assume (my view) that >>> they didn?t like that we cannot guarantee that NATS would become a core component (or used, or the primary bus) >>> in the ecosystem (since we don?t, and don?t want to for that matter, force a backend/driver on operators). >>> >>> Any help is of course appreciated, right now this is not a top priority and I don?t spend any full-time >>> effort into it. >> >> Would it help if I was packaging NATS in Debian (and therefore, as a >> consequence, in Ubuntu)? By reading the go.mod, it feels like it has a >> reasonable amount of dependencies (though I didn't check for the >> indirect ones), so it feels like doable... > > NATS's preferred way of delivery is a single (!) binary offered by the upstream. > I guess you could help considerably with minimum effort by repackaging > it into a deb so that users going for apt hunting get the upstream > NATS. No way... :) If I'm to do something, it wont be ugly bianry-only packaging. I was more thinking about doing a *real* packaging work, meaning packaging any missing Go dependency in Debian, and building from source. Otherwise, what value would this bring? > Then it would only boil down to finally packaging the Python library > that works well with both OpenStack and NATS but we don't have it yet. Considering the 400+ Python package I maintain for OpenStack in Debian, that'd be trivial 15 to 30 minutes work for me. :) Packaging a Go app the correct way, without breaking anything else in the whole Go stack in Debian is a way harder (ie: if something needs updating, then all reverse dependencies must be checked and possibly fixed...). Cheers, Thomas Goirand (zigo) From zigo at debian.org Fri Jul 21 13:40:05 2023 From: zigo at debian.org (Thomas Goirand) Date: Fri, 21 Jul 2023 15:40:05 +0200 Subject: [all] [oslo.messaging] Interest in collaboration on a NATS driver In-Reply-To: References: <52AA12A0-AE67-4EF7-B924-DE1F2873B909@binero.com> <0E347D5A-967D-42E3-AC00-56034907053B@binero.com> <8F7F790F-0F5C-4257-845A-1BCE671AF844@binero.com> <3b2f711e-a2f1-71da-92ba-d36c229420a2@inovex.de> <12EB84AA-A70A-4246-ABEC-4A4BC9E7B682@binero.com> <74d9d34b-7048-e0d5-14f4-7a5361afbdb1@debian.org> Message-ID: <24028e18-cddd-283a-a58f-62c25b374c66@debian.org> On 7/21/23 13:32, smooney at redhat.com wrote: > On Fri, 2023-07-21 at 11:59 +0200, Rados?aw Piliszek wrote: >> On Fri, 21 Jul 2023 at 09:03, Thomas Goirand wrote: >>> >>> On 7/15/23 16:27, Tobias Urdin wrote: >>>> The effort is mostly now to get/find a supported third-party Python library that does not >>>> use asyncio for usage in a oslo.messaging driver. >>> >>> Sorry to ask, but what's the problem with asyncio? Do we want something >>> *not* async? If yes, why? >> >> Large swaths of OpenStack code are not ready for async. >> We want to take baby steps and get to some better place with tangible benefits. > that not entirly correct. > much of openstack is already async but we made a decision ot use eventlet as our async model many many > years ago after we ported form twisted. > > The problem is you cannot really mix eventlet code and asyncio safely in the python process. > That means you cannot have the async nats lib used in nova,swift,cinder,neutron, glance and progrably other > projects today. > > i proposed one option a long time ago which is to take a privsep like approch where we spawn the oslo messaging driver > into a sperate process and comunicate with it over a unix socket. IRRC, back in 2014, at the Hongkong summit, there was a large consensus that Eventlet was a wrong choice. I can still remember Jay Pipes shouting "die Eventlet, die...". Fast forward nearly 10 years later, nothing has moved in that direction, and we're still stuck with that horrible hack. Why don't we just get rid of Eventlet and switch to asyncio instead? That'd be a *HUGE* improvement in terms of ... well many things ... less bugs, more maintainability, less breakage on each minor Python release, etc. That'd be a lot of work, I'm sure... At the same time, everyone would be happy to finally get rid of it. Cheers, Thomas Goirand (zigo) From zigo at debian.org Fri Jul 21 13:42:46 2023 From: zigo at debian.org (Thomas Goirand) Date: Fri, 21 Jul 2023 15:42:46 +0200 Subject: [all] [oslo.messaging] Interest in collaboration on a NATS driver In-Reply-To: References: <52AA12A0-AE67-4EF7-B924-DE1F2873B909@binero.com> <0E347D5A-967D-42E3-AC00-56034907053B@binero.com> <8F7F790F-0F5C-4257-845A-1BCE671AF844@binero.com> <3E16378D-DCFE-4096-B02F-E91DABE710EA@binero.com> Message-ID: <84384f9f-4977-99fd-b45a-4a4a9f8a2ce4@debian.org> On 12/8/22 22:55, Rados?aw Piliszek wrote: > Hi! > > I just wanted to thank Tobias for all his work in this topic. He uses > the plural form of verbs but I believe we could switch most to > singular and it would be true nonetheless. > Thank you again! > > Kind regards, > Radek > > -yoctozepto +1 Thank you Tobias! Thomas From smooney at redhat.com Fri Jul 21 14:16:38 2023 From: smooney at redhat.com (smooney at redhat.com) Date: Fri, 21 Jul 2023 15:16:38 +0100 Subject: [all] [oslo.messaging] Interest in collaboration on a NATS driver In-Reply-To: <24028e18-cddd-283a-a58f-62c25b374c66@debian.org> References: <52AA12A0-AE67-4EF7-B924-DE1F2873B909@binero.com> <0E347D5A-967D-42E3-AC00-56034907053B@binero.com> <8F7F790F-0F5C-4257-845A-1BCE671AF844@binero.com> <3b2f711e-a2f1-71da-92ba-d36c229420a2@inovex.de> <12EB84AA-A70A-4246-ABEC-4A4BC9E7B682@binero.com> <74d9d34b-7048-e0d5-14f4-7a5361afbdb1@debian.org> <24028e18-cddd-283a-a58f-62c25b374c66@debian.org> Message-ID: <77f01f4962ae48f0f4246df3e4e7b866b4a7bf0b.camel@redhat.com> On Fri, 2023-07-21 at 15:40 +0200, Thomas Goirand wrote: > On 7/21/23 13:32, smooney at redhat.com?wrote: > > On Fri, 2023-07-21 at 11:59 +0200, Rados?aw Piliszek wrote: > > > On Fri, 21 Jul 2023 at 09:03, Thomas Goirand wrote: > > > > > > > > On 7/15/23 16:27, Tobias Urdin wrote: > > > > > The effort is mostly now to get/find a supported third-party Python library that does not > > > > > use asyncio for usage in a oslo.messaging driver. > > > > > > > > Sorry to ask, but what's the problem with asyncio? Do we want something > > > > *not* async? If yes, why? > > > > > > Large swaths of OpenStack code are not ready for async. > > > We want to take baby steps and get to some better place with tangible benefits. > > that not entirly correct. > > much of openstack is already async but we made a decision ot use eventlet as our async model many many > > years ago after we ported form twisted. > > > > The problem is you cannot really mix eventlet code and asyncio safely in the python process. > > That means you cannot have the async nats lib used in nova,swift,cinder,neutron, glance and progrably other > > projects today. > > > > i proposed one option a long time ago which is to take a privsep like approch where we spawn the oslo messaging > > driver > > into a sperate process and comunicate with it over a unix socket. > > IRRC, back in 2014, at the Hongkong summit, there was a large consensus > that Eventlet was a wrong choice. I can still remember Jay Pipes > shouting "die Eventlet, die...". Fast forward nearly 10 years later, > nothing has moved in that direction, and we're still stuck with that > horrible hack. Why don't we just get rid of Eventlet and switch to > asyncio instead? That'd be a *HUGE* improvement in terms of ... well > many things ... less bugs, more maintainability, less breakage on each > minor Python release, etc. That'd be a lot of work, I'm sure... At the > same time, everyone would be happy to finally get rid of it. we have consider that in the past and the reality is it would be a massive rewrite we considerd doing this when we were dicussiong moving to python3 and at various points since the problem is two fold. 1 we dont have contibutors able to work on this currently. 2 we dont have a good solution for how to do this incrementally. in nova at least we did breicly consider tyring to add asyncio console-script entries points so that we couuld have an eventlet version and non eventlet version of each service to act as a bridge its somethign we could discuss again but makign such a change without introducing bugs or performance regressions would require a lot of careful engenieing work. i would love to regarin the ablity ot run nova under a debugger and drop the dep on eventlet but i dont want to do that at the expense of stablity or perfromace for our customer. some services liek the api and schduler would be relitivly simple to port. the compute agent and conductor on the other hand rely on the implict acync behavior to fucntion at all. nova-comptue with libvirt as at lest 3 while true loops to compunticate with rabbit, libvirt and privsep to name just one of the the issues tha towuld need to be resolved. > > Cheers, > > Thomas Goirand (zigo) > > From radoslaw.piliszek at gmail.com Fri Jul 21 17:48:29 2023 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Fri, 21 Jul 2023 19:48:29 +0200 Subject: [all] [oslo.messaging] Interest in collaboration on a NATS driver In-Reply-To: <1ca9e974-a6a2-7b08-b129-416fe00172f5@debian.org> References: <52AA12A0-AE67-4EF7-B924-DE1F2873B909@binero.com> <0E347D5A-967D-42E3-AC00-56034907053B@binero.com> <8F7F790F-0F5C-4257-845A-1BCE671AF844@binero.com> <3b2f711e-a2f1-71da-92ba-d36c229420a2@inovex.de> <12EB84AA-A70A-4246-ABEC-4A4BC9E7B682@binero.com> <74d9d34b-7048-e0d5-14f4-7a5361afbdb1@debian.org> <1ca9e974-a6a2-7b08-b129-416fe00172f5@debian.org> Message-ID: On Fri, 21 Jul 2023 at 15:37, Thomas Goirand wrote: > > NATS's preferred way of delivery is a single (!) binary offered by the upstream. > > I guess you could help considerably with minimum effort by repackaging > > it into a deb so that users going for apt hunting get the upstream > > NATS. > > No way... :) > > If I'm to do something, it wont be ugly bianry-only packaging. I was > more thinking about doing a *real* packaging work, meaning packaging any > missing Go dependency in Debian, and building from source. Otherwise, > what value would this bring? apt-get install support with automatic signature validation, version vetting with testing against your OpenStack stack. Just less work for you. In the end you can create two packages: one directly from upstream and one built yourself - and then compare the two. So many possibilities! > > Then it would only boil down to finally packaging the Python library > > that works well with both OpenStack and NATS but we don't have it yet. > > Considering the 400+ Python package I maintain for OpenStack in Debian, > that'd be trivial 15 to 30 minutes work for me. :) That's lovely to hear (read)! > Packaging a Go app > the correct way, without breaking anything else in the whole Go stack in > Debian is a way harder (ie: if something needs updating, then all > reverse dependencies must be checked and possibly fixed...). The point with Go is that it has no dynamic linking... and this is *on purpose*. What you would be doing is repeating the compilation to produce the single static binary and package it. The upstream does this well - no reason to repeat unless building for fancy architectures. This also helps with ensuring everyone is bug-bug compatible if they run the same version - no need to check anything else. :D From fungi at yuggoth.org Fri Jul 21 18:12:30 2023 From: fungi at yuggoth.org (Jeremy Stanley) Date: Fri, 21 Jul 2023 18:12:30 +0000 Subject: [all] [oslo.messaging] Interest in collaboration on a NATS driver In-Reply-To: References: <3b2f711e-a2f1-71da-92ba-d36c229420a2@inovex.de> <12EB84AA-A70A-4246-ABEC-4A4BC9E7B682@binero.com> <74d9d34b-7048-e0d5-14f4-7a5361afbdb1@debian.org> <1ca9e974-a6a2-7b08-b129-416fe00172f5@debian.org> Message-ID: <20230721181229.zgt3bem2755vblht@yuggoth.org> On 2023-07-21 19:48:29 +0200 (+0200), Rados?aw Piliszek wrote: [...] > The point with Go is that it has no dynamic linking... and this is > *on purpose*. What you would be doing is repeating the compilation > to produce the single static binary and package it. The upstream > does this well - no reason to repeat unless building for fancy > architectures. This also helps with ensuring everyone is bug-bug > compatible if they run the same version - no need to check > anything else. :D And while that concept certainly has its merits, it runs counter to the guiding principles of some software distributions which want to guarantee rebuildability with only tools and components present within that distribution, the ability to backport critical fixes to otherwise frozen contemporary versions of the software which may not fit with the upstream maintainers' release and support policies, the ability to prove reproducibility of builds, to be able to confirm that the binaries shipped were actually built from the source code upstream claimed, and so on. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From tobias.urdin at binero.com Fri Jul 21 21:29:37 2023 From: tobias.urdin at binero.com (Tobias Urdin) Date: Fri, 21 Jul 2023 21:29:37 +0000 Subject: [all] [oslo.messaging] Interest in collaboration on a NATS driver In-Reply-To: <77f01f4962ae48f0f4246df3e4e7b866b4a7bf0b.camel@redhat.com> References: <52AA12A0-AE67-4EF7-B924-DE1F2873B909@binero.com> <0E347D5A-967D-42E3-AC00-56034907053B@binero.com> <8F7F790F-0F5C-4257-845A-1BCE671AF844@binero.com> <3b2f711e-a2f1-71da-92ba-d36c229420a2@inovex.de> <12EB84AA-A70A-4246-ABEC-4A4BC9E7B682@binero.com> <74d9d34b-7048-e0d5-14f4-7a5361afbdb1@debian.org> <24028e18-cddd-283a-a58f-62c25b374c66@debian.org> <77f01f4962ae48f0f4246df3e4e7b866b4a7bf0b.camel@redhat.com> Message-ID: <7E3FEEC5-A9E6-4AF3-93C5-9AD330C2F741@binero.com> Hello, And just to iterate on this, just look at the code duplication (double the maintenance for an async and sync with eventlet code path). review.opendev.org [favicon.ico] We?d need to duplicate oslo.service, oslo.messaging and all other common libraries being called in a asyncio code path, and not only that we?d need to consolidate everyone on a Pecan (REST API library) replacement that would support asyncio (a new oslo library?) and just imagine that migration and consolidation effort for multiple projects. Even then we?d also need to port every single project to using it without regressions in performance. It?s a crazy massive effort, but I fully agree that eventlet should have been gone long ago. Best regards Tobias On 21 Jul 2023, at 16:16, smooney at redhat.com wrote: On Fri, 2023-07-21 at 15:40 +0200, Thomas Goirand wrote: On 7/21/23 13:32, smooney at redhat.com wrote: On Fri, 2023-07-21 at 11:59 +0200, Rados?aw Piliszek wrote: On Fri, 21 Jul 2023 at 09:03, Thomas Goirand wrote: On 7/15/23 16:27, Tobias Urdin wrote: The effort is mostly now to get/find a supported third-party Python library that does not use asyncio for usage in a oslo.messaging driver. Sorry to ask, but what's the problem with asyncio? Do we want something *not* async? If yes, why? Large swaths of OpenStack code are not ready for async. We want to take baby steps and get to some better place with tangible benefits. that not entirly correct. much of openstack is already async but we made a decision ot use eventlet as our async model many many years ago after we ported form twisted. The problem is you cannot really mix eventlet code and asyncio safely in the python process. That means you cannot have the async nats lib used in nova,swift,cinder,neutron, glance and progrably other projects today. i proposed one option a long time ago which is to take a privsep like approch where we spawn the oslo messaging driver into a sperate process and comunicate with it over a unix socket. IRRC, back in 2014, at the Hongkong summit, there was a large consensus that Eventlet was a wrong choice. I can still remember Jay Pipes shouting "die Eventlet, die...". Fast forward nearly 10 years later, nothing has moved in that direction, and we're still stuck with that horrible hack. Why don't we just get rid of Eventlet and switch to asyncio instead? That'd be a *HUGE* improvement in terms of ... well many things ... less bugs, more maintainability, less breakage on each minor Python release, etc. That'd be a lot of work, I'm sure... At the same time, everyone would be happy to finally get rid of it. we have consider that in the past and the reality is it would be a massive rewrite we considerd doing this when we were dicussiong moving to python3 and at various points since the problem is two fold. 1 we dont have contibutors able to work on this currently. 2 we dont have a good solution for how to do this incrementally. in nova at least we did breicly consider tyring to add asyncio console-script entries points so that we couuld have an eventlet version and non eventlet version of each service to act as a bridge its somethign we could discuss again but makign such a change without introducing bugs or performance regressions would require a lot of careful engenieing work. i would love to regarin the ablity ot run nova under a debugger and drop the dep on eventlet but i dont want to do that at the expense of stablity or perfromace for our customer. some services liek the api and schduler would be relitivly simple to port. the compute agent and conductor on the other hand rely on the implict acync behavior to fucntion at all. nova-comptue with libvirt as at lest 3 while true loops to compunticate with rabbit, libvirt and privsep to name just one of the the issues tha towuld need to be resolved. Cheers, Thomas Goirand (zigo) -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: favicon.ico Type: image/vnd.microsoft.icon Size: 5430 bytes Desc: favicon.ico URL: From radoslaw.piliszek at gmail.com Sat Jul 22 07:22:44 2023 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Sat, 22 Jul 2023 09:22:44 +0200 Subject: [all] [oslo.messaging] Interest in collaboration on a NATS driver In-Reply-To: <7E3FEEC5-A9E6-4AF3-93C5-9AD330C2F741@binero.com> References: <52AA12A0-AE67-4EF7-B924-DE1F2873B909@binero.com> <0E347D5A-967D-42E3-AC00-56034907053B@binero.com> <8F7F790F-0F5C-4257-845A-1BCE671AF844@binero.com> <3b2f711e-a2f1-71da-92ba-d36c229420a2@inovex.de> <12EB84AA-A70A-4246-ABEC-4A4BC9E7B682@binero.com> <74d9d34b-7048-e0d5-14f4-7a5361afbdb1@debian.org> <24028e18-cddd-283a-a58f-62c25b374c66@debian.org> <77f01f4962ae48f0f4246df3e4e7b866b4a7bf0b.camel@redhat.com> <7E3FEEC5-A9E6-4AF3-93C5-9AD330C2F741@binero.com> Message-ID: To add insult to injury, the services that would benefit the most (think nova) would require the most work. And, since the goal is to have NATS support and not remove eventlet (because there is like no force to do it now), it seems best to focus on porting NATS support to eventlet. On Fri, 21 Jul 2023 at 23:31, Tobias Urdin wrote: > Hello, > > And just to iterate on this, just look at the code duplication (double the > maintenance for an async and sync with eventlet code path). > review.opendev.org > > > > We?d need to duplicate oslo.service, oslo.messaging and all other common > libraries being called in a asyncio code path, and not only > that we?d need to consolidate everyone on a Pecan (REST API library) > replacement that would support asyncio (a new oslo library?) and > just imagine that migration and consolidation effort for multiple projects. > > Even then we?d also need to port every single project to using it without > regressions in performance. > > It?s a crazy massive effort, but I fully agree that eventlet should have > been gone long ago. > > Best regards > Tobias > > On 21 Jul 2023, at 16:16, smooney at redhat.com wrote: > > On Fri, 2023-07-21 at 15:40 +0200, Thomas Goirand wrote: > > On 7/21/23 13:32, smooney at redhat.com wrote: > > On Fri, 2023-07-21 at 11:59 +0200, Rados?aw Piliszek wrote: > > On Fri, 21 Jul 2023 at 09:03, Thomas Goirand wrote: > > > On 7/15/23 16:27, Tobias Urdin wrote: > > The effort is mostly now to get/find a supported third-party Python > library that does not > use asyncio for usage in a oslo.messaging driver. > > > Sorry to ask, but what's the problem with asyncio? Do we want something > *not* async? If yes, why? > > > Large swaths of OpenStack code are not ready for async. > We want to take baby steps and get to some better place with tangible > benefits. > > that not entirly correct. > much of openstack is already async but we made a decision ot use eventlet > as our async model many many > years ago after we ported form twisted. > > The problem is you cannot really mix eventlet code and asyncio safely in > the python process. > That means you cannot have the async nats lib used in > nova,swift,cinder,neutron, glance and progrably other > projects today. > > i proposed one option a long time ago which is to take a privsep like > approch where we spawn the oslo messaging > driver > into a sperate process and comunicate with it over a unix socket. > > > IRRC, back in 2014, at the Hongkong summit, there was a large consensus > that Eventlet was a wrong choice. I can still remember Jay Pipes > shouting "die Eventlet, die...". Fast forward nearly 10 years later, > nothing has moved in that direction, and we're still stuck with that > horrible hack. Why don't we just get rid of Eventlet and switch to > asyncio instead? That'd be a *HUGE* improvement in terms of ... well > many things ... less bugs, more maintainability, less breakage on each > minor Python release, etc. That'd be a lot of work, I'm sure... At the > same time, everyone would be happy to finally get rid of it. > > we have consider that in the past and the reality is it would be a massive > rewrite > we considerd doing this when we were dicussiong moving to python3 and at > various points since > the problem is two fold. 1 we dont have contibutors able to work on this > currently. 2 we dont have > a good solution for how to do this incrementally. > > in nova at least we did breicly consider tyring to add asyncio > console-script entries points > so that we couuld have an eventlet version and non eventlet version of > each service to act as a bridge > its somethign we could discuss again but makign such a change without > introducing bugs or performance regressions > would require a lot of careful engenieing work. > > i would love to regarin the ablity ot run nova under a debugger and drop > the dep on eventlet but i dont want > to do that at the expense of stablity or perfromace for our customer. > > some services liek the api and schduler would be relitivly simple to port. > the compute agent and conductor on the other > hand rely on the implict acync behavior to fucntion at all. nova-comptue > with libvirt as at lest 3 while true loops > to compunticate with rabbit, libvirt and privsep to name just one of the > the issues tha towuld need to be resolved. > > > Cheers, > > Thomas Goirand (zigo) > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From michel.jouvin at ijclab.in2p3.fr Sat Jul 22 09:11:11 2023 From: michel.jouvin at ijclab.in2p3.fr (Michel Jouvin) Date: Sat, 22 Jul 2023 11:11:11 +0200 Subject: Ussuri: how to delete lbaas loadbalancer left over? - SOLVED In-Reply-To: <1a342d65-191b-c367-9872-1d72f83479f6@ijclab.in2p3.fr> References: <20230707164654.Horde.sUTt6M8NRpBGR8HwFy7KwGv@webmail.nde.ag> <1893244f3d8.27f3.86e99a005b57e9644b7bb922d8d7f665@ijclab.in2p3.fr> <20230707220512.Horde.10Q-iIqvZct1kSTQs9Gd4-u@webmail.nde.ag> <20230707221544.Horde.tGBcMHNkuxKWyNahfmPmJ9V@webmail.nde.ag> <1a342d65-191b-c367-9872-1d72f83479f6@ijclab.in2p3.fr> Message-ID: <0ee75b98-af96-09c5-d30a-dfd9aef4a508@ijclab.in2p3.fr> Hi, As promised, the conclusion of the story! After installing Octavia in our cloud, we ran the nlbaas2octavia.py tool as suggested by Eugen. It worked like a charm, without any modification. Once the LBs had been converted to Octavia LB, we just had to fix the provider attribute in the DB, changing it from "haproxy" (which is not a provider in Octavia) to "amphora". And then the "openstack coe cluster delete" worked just fine removing these broken LBs as part of the cluster deletion. Cheers, Michel Le 14/07/2023 ? 23:24, Michel Jouvin a ?crit?: > Hi Eugen, > > I found some time to look at my issue in the light of your advices! In > my research, I found the presentation > https://opendev.org/openstack/neutron-lbaas which is both synthetic > and clear, with a list of the migration options offered (at the time > of the presentation). This includes the DB migration tool, > nlbaas2octavia.py. Following your broken link, I manage to find it, it > is still there but hidden. You need to checkout the repo > https://opendev.org/openstack/neutron-lbaas and go back a couple of > revisions (it is explained in the README). It is Python2 but at least > it is a starting point, either installing a Python2 venv or migrating > the script which is not very long and seems to have very few > dependcies, apart from oslo. > > I'm afraid that the neutron-lbaas-proxy is no longer an option as, as > far as I understood, it relies on some LBAAS code still being present > in the Neutron server and this part of the Neutron code has been > completely removed I think in Ussuri release (it's the reason of my > problems in fact, too bad, it was a matter of a few days, just missed > the very old announcement that it would be the case). > > If I succeed to clean the things (again I don't really want to migrate > the existing LBAAS, I just want to delete them), I'll report in this > thread in case it is useful to somebody else... > > Best regards, > > Michel > > Le 08/07/2023 ? 00:15, Eugen Block a ?crit?: >> Unfortunately, the link to the migration tool doesn?t work: >> >> https://wiki.openstack.org/wiki/Neutron/LBaaS/Deprecation >> >> But maybe it can get you in the right direction, neutron-lbaas-proxy >> seems to be the keyword. But as I already mentioned, I don?t have >> experience with this path. >> >> Zitat von Eugen Block : >> >>> Hi, >>> >>> I mean the latter. Once you have Octavia installed you can create >>> new LBs, but as I understand it you won?t be able to delete the >>> legacy LBs. Did the neutron config change when you upgraded to >>> Ussuri? I wonder if there?s just some config missing to be able to >>> delete the old LBs, I don?t have a clue tbh. Maybe someone else has >>> some more experience and will chime in. >>> >>> Zitat von Michel Jouvin : >>> >>>> Ho Eug?ne, >>>> >>>> Thanks for your answer. Do you mean that after installing octavia >>>> (it is planned) we'll have again the ability to delete the >>>> remaining LBAAS instances? Or just that octavia is the LBAAS >>>> replacement in terms of functionalities? >>>> >>>> Best regards, >>>> >>>> Michel >>>> Sent from my mobile >>>> Le 7 juillet 2023 18:52:30 Eugen Block a ?crit : >>>> >>>>> Hi, >>>>> >>>>> neutron lbaas was deprecated in Queens so you may have to migrate the >>>>> existing LBs to octavia. I have never done that but I remember >>>>> reading >>>>> through the SUSE Docs when one of our customers had to decide whether >>>>> they wanted to upgrade or reinstall with a newer openstack release. >>>>> They decided to do the latter, so we set up octavia from scratch and >>>>> didn't have to migrate anything. There's also a video I've never >>>>> watched [2], maybe that helps. I can't really tell if a migration is >>>>> possible to work around your issue but I thought I'd share anyway. >>>>> >>>>> Regards, >>>>> Eugen >>>>> >>>>> [1] >>>>> https://documentation.suse.com/soc/9/single-html/suse-openstack-cloud-crowbar-deployment/#sec-depl-ostack-octavia-migrate-users >>>>> >>>>> [2] https://www.youtube.com/watch?v=jj4KMJPA0Pk >>>>> >>>>> Zitat von Michel Jouvin : >>>>> >>>>>> Hi, >>>>>> >>>>>> We had a few Magnum (K8s) clusters created a couple of years ago >>>>>> (with Rocky and Stein versions) and forgotten. We started to delete >>>>>> them this spring when we where running Train Neutron service. >>>>>> Basically we managed to do this with the following sequence: >>>>>> >>>>>> - openstack coe cluster delete xxx and waiting for DELETE_FAILED >>>>>> - Use openstack coe cluster show / openstack stack resource list -n2 >>>>>> to identify the neutron entry causing the error and pick the >>>>>> corresponding resource ID >>>>>> - Find the ports associated with the router with openstack port list >>>>>> --router previously_found_id >>>>>> - Use the port subnet to find the port corresponding lbaas load >>>>>> balancer ID, use the neutron CLI to delete the load balancer >>>>>> (deleting one by one all the dependencies preventing the load >>>>>> balancer removal) >>>>>> - Rerun openstack coe cluster delete >>>>>> >>>>>> For some reasons, we didn't cleanup all the abandoned clusters and >>>>>> upgraded Neutron to Ussuri. Unfortunately, since then our previous >>>>>> process is no longer working as it seems that the Neutron server >>>>>> doesn't know anymore anything about the LBAAS load balancers (and >>>>>> "neutron lbaas-loadbalancer list" returns nothing). In the neutron >>>>>> server, any attempt to delete the subnet attached to the load >>>>>> balancer (or to list them with Neutron CLI) results in the following >>>>>> errors in Neutron server.log : >>>>>> >>>>>> ------ >>>>>> >>>>>> 2023-07-07 16:27:31.139 14962 WARNING >>>>>> neutron.pecan_wsgi.controllers.root >>>>>> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >>>>>> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >>>>>> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >>>>>> 1367c9a4d5da4b229c35789c271dc7aa] No controller found for: lbaas - >>>>>> returning response code 404: pecan.routing.PecanNotFound >>>>>> 2023-07-07 16:27:31.140 14962 INFO >>>>>> neutron.pecan_wsgi.hooks.translation >>>>>> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >>>>>> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >>>>>> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >>>>>> 1367c9a4d5da4b229c35789c271dc7aa] GET failed (client error): The >>>>>> resource could not be found. >>>>>> 2023-07-07 16:27:31.141 14962 INFO neutron.wsgi >>>>>> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >>>>>> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >>>>>> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >>>>>> 1367c9a4d5da4b229c35789c271dc7aa] 157.136.249.153 "GET >>>>>> /v2.0/lbaas/loadbalancers?name=kube_service_964f7e76-d2d5-4126-ab11-cd689f6dd9f9_runnerdeploy-wm9sm-5h52l_hello-node-x-default-x-runnerdeploy-wm9sm-5h52l >>>>>> HTTP/1.1" status: 404? len: 304 time: >>>>>> 0.0052643 >>>>>> ------ >>>>>> >>>>>> Any suggestion to workaround this problem and be able to >>>>>> successfully delete our old Magnum clusters? >>>>>> >>>>>> Thanks in advance for any help. Best regards, >>>>>> >>>>>> Michel >> >> >> >> From zigo at debian.org Sat Jul 22 11:10:49 2023 From: zigo at debian.org (Thomas Goirand) Date: Sat, 22 Jul 2023 13:10:49 +0200 Subject: [all] [oslo.messaging] Interest in collaboration on a NATS driver In-Reply-To: <77f01f4962ae48f0f4246df3e4e7b866b4a7bf0b.camel@redhat.com> References: <52AA12A0-AE67-4EF7-B924-DE1F2873B909@binero.com> <0E347D5A-967D-42E3-AC00-56034907053B@binero.com> <8F7F790F-0F5C-4257-845A-1BCE671AF844@binero.com> <3b2f711e-a2f1-71da-92ba-d36c229420a2@inovex.de> <12EB84AA-A70A-4246-ABEC-4A4BC9E7B682@binero.com> <74d9d34b-7048-e0d5-14f4-7a5361afbdb1@debian.org> <24028e18-cddd-283a-a58f-62c25b374c66@debian.org> <77f01f4962ae48f0f4246df3e4e7b866b4a7bf0b.camel@redhat.com> Message-ID: On 7/21/23 16:16, smooney at redhat.com wrote: > we have consider that in the past and the reality is it would be a massive rewrite > we considerd doing this when we were dicussiong moving to python3 and at various points since > the problem is two fold. 1 we dont have contibutors able to work on this currently. 2 we dont have > a good solution for how to do this incrementally. > > in nova at least we did breicly consider tyring to add asyncio console-script entries points > so that we couuld have an eventlet version and non eventlet version of each service to act as a bridge > its somethign we could discuss again but makign such a change without introducing bugs or performance regressions > would require a lot of careful engenieing work. > > i would love to regarin the ablity ot run nova under a debugger and drop the dep on eventlet but i dont want > to do that at the expense of stablity or perfromace for our customer. > > some services liek the api and schduler would be relitivly simple to port. the compute agent and conductor on the other > hand rely on the implict acync behavior to fucntion at all. nova-comptue with libvirt as at lest 3 while true loops > to compunticate with rabbit, libvirt and privsep to name just one of the the issues tha towuld need to be resolved. THere's probably another route. What if we were to somehow fork Eventlet, make it stop doing monkey patching, and reimplement its calls using asyncio? Do you think such approach would work? If so, it would have the huge advantage to implement the changes for all projects at once. Your thoughts? Cheers, Thomas Goirand (zigo) From radoslaw.piliszek at gmail.com Sat Jul 22 11:21:04 2023 From: radoslaw.piliszek at gmail.com (=?UTF-8?Q?Rados=C5=82aw_Piliszek?=) Date: Sat, 22 Jul 2023 13:21:04 +0200 Subject: [all] [oslo.messaging] Interest in collaboration on a NATS driver In-Reply-To: References: <52AA12A0-AE67-4EF7-B924-DE1F2873B909@binero.com> <0E347D5A-967D-42E3-AC00-56034907053B@binero.com> <8F7F790F-0F5C-4257-845A-1BCE671AF844@binero.com> <3b2f711e-a2f1-71da-92ba-d36c229420a2@inovex.de> <12EB84AA-A70A-4246-ABEC-4A4BC9E7B682@binero.com> <74d9d34b-7048-e0d5-14f4-7a5361afbdb1@debian.org> <24028e18-cddd-283a-a58f-62c25b374c66@debian.org> <77f01f4962ae48f0f4246df3e4e7b866b4a7bf0b.camel@redhat.com> Message-ID: On Sat, 22 Jul 2023 at 13:12, Thomas Goirand wrote: > > On 7/21/23 16:16, smooney at redhat.com wrote: > > we have consider that in the past and the reality is it would be a massive rewrite > > we considerd doing this when we were dicussiong moving to python3 and at various points since > > the problem is two fold. 1 we dont have contibutors able to work on this currently. 2 we dont have > > a good solution for how to do this incrementally. > > > > in nova at least we did breicly consider tyring to add asyncio console-script entries points > > so that we couuld have an eventlet version and non eventlet version of each service to act as a bridge > > its somethign we could discuss again but makign such a change without introducing bugs or performance regressions > > would require a lot of careful engenieing work. > > > > i would love to regarin the ablity ot run nova under a debugger and drop the dep on eventlet but i dont want > > to do that at the expense of stablity or perfromace for our customer. > > > > some services liek the api and schduler would be relitivly simple to port. the compute agent and conductor on the other > > hand rely on the implict acync behavior to fucntion at all. nova-comptue with libvirt as at lest 3 while true loops > > to compunticate with rabbit, libvirt and privsep to name just one of the the issues tha towuld need to be resolved. > > THere's probably another route. What if we were to somehow fork > Eventlet, make it stop doing monkey patching, and reimplement its calls > using asyncio? Do you think such approach would work? If so, it would > have the huge advantage to implement the changes for all projects at > once. Your thoughts? zigo, the problem is that nova et al are relying on this monkey patching to work. So we are back to "rewrite it all so that it works in asyncio". From zigo at debian.org Sat Jul 22 12:04:49 2023 From: zigo at debian.org (Thomas Goirand) Date: Sat, 22 Jul 2023 14:04:49 +0200 Subject: [all] [oslo.messaging] Interest in collaboration on a NATS driver In-Reply-To: References: <52AA12A0-AE67-4EF7-B924-DE1F2873B909@binero.com> <0E347D5A-967D-42E3-AC00-56034907053B@binero.com> <8F7F790F-0F5C-4257-845A-1BCE671AF844@binero.com> <3b2f711e-a2f1-71da-92ba-d36c229420a2@inovex.de> <12EB84AA-A70A-4246-ABEC-4A4BC9E7B682@binero.com> <74d9d34b-7048-e0d5-14f4-7a5361afbdb1@debian.org> <1ca9e974-a6a2-7b08-b129-416fe00172f5@debian.org> Message-ID: <6e0363eb-04f4-b1e6-9c5d-07ddfc9c55f0@debian.org> On 7/21/23 19:48, Rados?aw Piliszek wrote: > The point with Go is that it has no dynamic linking... I would say: s/The point with/The main defect of/ > What you would be doing is repeating the compilation to produce the > single static binary and package it. All what Jeremy replied, plus: - having a version in Debian stable that we may *really* rely on, with bugs reported and fixed for months before the distro is released as stable. Not just the latest upstream release with the latest not-yet discovered bugs. - A real auditable software supply chain, not just "curl | sudo bash" from the internet. > The upstream does this well - no reason to repeat unless building for > fancy architectures. yeah, there's that too! Having support for: - ppc64el - arm64 - risc-v - ... (you name it...) ... That's one of the nice things with Debian! :) > This also helps with ensuring everyone is bug-bug compatible if they > run the same version - no need to check anything else. :D What helps is having everyone on the version from the distro, not everyone running the latest version from when they did their deployments (understand: everyone running something different). Of course, YMMV. But these are the reasons I only deploy packaged software, and packaged the way we do in Debian. I understand why others prefer some other path, and deploy from upstream, it's not just what I like to do, and what I recommend against. Cheers, Thomas Goirand (zigo) From danish52.jmi at gmail.com Sat Jul 22 23:21:52 2023 From: danish52.jmi at gmail.com (Danish Khan) Date: Sun, 23 Jul 2023 04:51:52 +0530 Subject: [openstack-ansible][wallaby deployment getting failed] error:- Service user token configuration is required for all Nova Message-ID: Dear Team, I am trying to deploy openstack-ansible wallaby but it is getting failed with error: service user token configuration is required for all Nova services. For more details see the following: https://docs.openstack.org/latest/nova/admin/configuration/service-user-token.html But the mentioned webpage is not available. This is fixed in yoga but wallay is still failing. I tried to copy few variables from Yoga but that is working for me. Can someone please help me on this ? where I need to make some changes to deploy openstack-ansible wallaby? Thanks in advance :) Regards, Danish From noonedeadpunk at gmail.com Sun Jul 23 04:41:41 2023 From: noonedeadpunk at gmail.com (Dmitriy Rabotyagov) Date: Sun, 23 Jul 2023 06:41:41 +0200 Subject: [openstack-ansible][wallaby deployment getting failed] error:- Service user token configuration is required for all Nova In-Reply-To: References: Message-ID: Hey Danish, This failure is related to the security vulnerability [1]. There are several things to mention with this regards: 1. Cinder has not backported fix to Wallaby, due to its complexity, so vulnerability is still not fully covered there. 2. Nova has backported the fix, which is raising the error you see 3. In OpenStack-Ansible we also have not backported work that is required to support service tokens to Wallaby, as there was huge amount of changes that are required for this to fix, while vulnerability is not fixed in services themselves. 4. There is huge ongoing discussion in Technical Commetee on what to do with releases in Extended Maintenance and if we should End Of Life them or not due, which was raised by this vulnerability. Keeping all that in mind, you still should be able to deploy OpenStack-Ansible. And there are several ways of doing that. 1. Deploy vulnerable version of services. So I would try using 23.4.3 instead of stable/wallaby or 23.4.4 or wallaby-em. You can also override nova SHA to install not patched version by providing `nova_git_install_branch: a9e81626c5e9dac897759c5f66c7ae1b4efa3c6d` to user-variables 2. Apply manual config overrides for nova and cinder services to comply with new requirements for this vulnerability. So you need smth like that: nova_nova_conf_overrides: keystone_authtoken: service_token_roles_required: True service_token_roles: admin service_user: send_service_user_token: True region_name: "{{ nova_service_region }}" auth_type: password username: "{{ nova_service_user_name }}" password: "{{ nova_service_password }}" project_name: "{{ nova_service_project_name }}" user_domain_id: "{{ nova_service_user_domain_id }}" project_domain_id: "{{ nova_service_project_domain_id }}" auth_url: "{{ keystone_service_adminurl }}" insecure: "{{ keystone_service_adminuri_insecure | bool }}" [1] https://security.openstack.org/ossa/OSSA-2023-003.html On Sun, Jul 23, 2023, 01:25 Danish Khan wrote: > Dear Team, > > I am trying to deploy openstack-ansible wallaby but it is getting > failed with error: > > service user token configuration is required for all Nova services. > For more details see the following: > > https://docs.openstack.org/latest/nova/admin/configuration/service-user-token.html > > But the mentioned webpage is not available. > > This is fixed in yoga but wallay is still failing. > > I tried to copy few variables from Yoga but that is working for me. > > Can someone please help me on this ? where I need to make some changes > to deploy openstack-ansible wallaby? > > Thanks in advance :) > > Regards, > Danish > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arnaud.morin at gmail.com Sun Jul 23 12:56:20 2023 From: arnaud.morin at gmail.com (Arnaud Morin) Date: Sun, 23 Jul 2023 12:56:20 +0000 Subject: [openstack][largescale-sig] Openstack multi region deployment In-Reply-To: References: Message-ID: We have this model also with only one keystone. We have multiple galera clusters synchronized together. Only one cluster is used for write requests (located in one region), others are read only / cache. Most of the calls done to our keystone are "read" or token validation requests, and this works fine with a read galera cluster / cache. I know that we also have a custom way to invalidate cache across regions, but I dont remember the details, I can ask the team. Anyway, this is do-able :) I imagine it also depends on the usage you have, if you create a lot of users/projects/assignments, then it may be harder to achieve. Cheers, Arnaud On 19.07.23 - 14:03, Nguy?n H?u Kh?i wrote: > Hello, thank you very much. > > But can I ask how we process if 1 region at ASIA and 2 regions in the USA? > > Database latency will be our problem. > > Nguyen Huu Khoi > > > On Tue, Jul 18, 2023 at 8:21?PM Karl Kloppenborg > wrote: > > > Hi Nguy, > > > > > > > > We?ve deployed a large multi-region openstack deployment. > > > > As a rule of thumb we?ve got a ?keystone? region which is as best we can > > highly available and very redundant. > > > > > > > > We then have all other regions talk back to this region, we just usually > > call it ?keystone? or ?core? and it?s hidden from the UI from users. > > > > > > > > We then just run a large well kept Galara cluster to support it. > > > > > > > > --Karl. > > > > > > > > *From: *openstack-discuss-request at lists.openstack.org < > > openstack-discuss-request at lists.openstack.org> > > *Date: *Tuesday, 18 July 2023 at 9:25 pm > > *To: *openstack-discuss at lists.openstack.org < > > openstack-discuss at lists.openstack.org> > > *Subject: *openstack-discuss Digest, Vol 57, Issue 55 > > > > Send openstack-discuss mailing list submissions to > > openstack-discuss at lists.openstack.org > > > > To subscribe or unsubscribe via the World Wide Web, visit > > > > https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss > > > > or, via email, send a message with subject or body 'help' to > > openstack-discuss-request at lists.openstack.org > > > > You can reach the person managing the list at > > openstack-discuss-owner at lists.openstack.org > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of openstack-discuss digest..." > > > > > > Today's Topics: > > > > 1. [openstack][largescale-sig] Openstack multi region deployment > > (Nguy?n H?u Kh?i) > > 2. Re: [openstack][largescale-sig] Openstack multi region > > deployment (Felix Huettner) > > 3. Re: [openstack][largescale-sig] Openstack multi region > > deployment (Nguy?n H?u Kh?i) > > 4. Re: [neutron] unmanaged router resources - OVN interconnect > > (Rodolfo Alonso Hernandez) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Tue, 18 Jul 2023 12:07:12 +0700 > > From: Nguy?n H?u Kh?i > > To: OpenStack Discuss > > Subject: [openstack][largescale-sig] Openstack multi region deployment > > Message-ID: > > < > > CABAODReJ6QW8A4OENEjmhFCiM-15B0qc2La_aMr1EKfaENq9iw at mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > Hello guys, > > > > I am going to deploy openstack multi regions and I know that keystone > > replication is the most challenging. > > > > I plan to set up 2 regions which use centralize galera cluster(3 nodes). > > and one standby edge galera cluster(3 nodes) > > > > When region 1 is node available, I will map region 2 to use standby edge > > galera cluster. > > > > I hope you give me some experience and advice with multi regions. > > > > Thank you very much. > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: < > > https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/c95d3675/attachment-0001.htm > > > > > > > ------------------------------ > > > > Message: 2 > > Date: Tue, 18 Jul 2023 09:34:35 +0200 > > From: Felix Huettner > > To: Nguy?n H?u Kh?i > > Cc: OpenStack Discuss > > Subject: Re: [openstack][largescale-sig] Openstack multi region > > deployment > > Message-ID: > > Content-Type: text/plain; charset=utf-8 > > > > Hi, > > > > i think you have two options here: > > 1. you could span a single galera cluster over all of your regions. > > this might have some latency issues, but if your are not too write > > heavy that might be fine. I know some companies use that setup. > > 2. you use some kind of multiple galera clusters with replication. > > But i have not yet heard of anybody using this setup. > > > > An alternative might be to have separate keystone setups with separate > > databases. This would probably reduce the error potential, but might not > > fit your usecase. > > > > Thanks > > Felix > > > > > > On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote: > > > Hello guys, > > > > > > I am going to deploy openstack multi regions and I know that keystone > > > replication is the most challenging. > > > > > > I plan to set up 2 regions which use centralize galera cluster(3 nodes). > > > and one standby edge galera cluster(3 nodes) > > > > > > When region 1 is node available, I will map region 2 to use standby edge > > > galera cluster. > > > > > > I hope you give me some experience and advice with multi regions. > > > > > > Thank you very much. > > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r > > die Verwertung durch den vorgesehenen Empf?nger bestimmt. > > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender > > bitte unverz?glich in Kenntnis und l?schen diese E Mail. > > > > Hinweise zum Datenschutz finden Sie hier. > > > > > > This e-mail may contain confidential content and is intended only for the > > specified recipient/s. > > If you are not the intended recipient, please inform the sender > > immediately and delete this e-mail. > > > > Information on data protection can be found here< > > https://www.datenschutz.schwarz>. > > > > > > > > ------------------------------ > > > > Message: 3 > > Date: Tue, 18 Jul 2023 15:36:11 +0700 > > From: Nguy?n H?u Kh?i > > To: Nguy?n H?u Kh?i , OpenStack Discuss > > > > Subject: Re: [openstack][largescale-sig] Openstack multi region > > deployment > > Message-ID: > > > CGBW1_bRkLQJAxLZxAx8V4qvbdBmTUQBUm2SRsow at mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > Hi. > > Thank you for your reply. > > > > The first one has a problem because each region is too soft. If a member is > > down, so this region is gone. > > > > It is so challenge with us. > > > > > > Nguyen Huu Khoi > > > > > > On Tue, Jul 18, 2023 at 2:34?PM Felix Huettner > > > > wrote: > > > > > Hi, > > > > > > i think you have two options here: > > > 1. you could span a single galera cluster over all of your regions. > > > this might have some latency issues, but if your are not too write > > > heavy that might be fine. I know some companies use that setup. > > > 2. you use some kind of multiple galera clusters with replication. > > > But i have not yet heard of anybody using this setup. > > > > > > An alternative might be to have separate keystone setups with separate > > > databases. This would probably reduce the error potential, but might not > > > fit your usecase. > > > > > > Thanks > > > Felix > > > > > > > > > On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote: > > > > Hello guys, > > > > > > > > I am going to deploy openstack multi regions and I know that keystone > > > > replication is the most challenging. > > > > > > > > I plan to set up 2 regions which use centralize galera cluster(3 > > nodes). > > > > and one standby edge galera cluster(3 nodes) > > > > > > > > When region 1 is node available, I will map region 2 to use standby > > edge > > > > galera cluster. > > > > > > > > I hope you give me some experience and advice with multi regions. > > > > > > > > Thank you very much. > > > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r > > > die Verwertung durch den vorgesehenen Empf?nger bestimmt. > > > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender > > > bitte unverz?glich in Kenntnis und l?schen diese E Mail. > > > > > > Hinweise zum Datenschutz finden Sie hier > >. > > > > > > > > > This e-mail may contain confidential content and is intended only for the > > > specified recipient/s. > > > If you are not the intended recipient, please inform the sender > > > immediately and delete this e-mail. > > > > > > Information on data protection can be found here< > > > https://www.datenschutz.schwarz>. > > > > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: < > > https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/749440e3/attachment-0001.htm > > > > > > > ------------------------------ > > > > Message: 4 > > Date: Tue, 18 Jul 2023 13:23:27 +0200 > > From: Rodolfo Alonso Hernandez > > To: Roberto Bartzen Acosta > > Cc: openstack-discuss , Terry > > Wilson , Tiago Pires < > > tiago.pires at luizalabs.com> > > Subject: Re: [neutron] unmanaged router resources - OVN interconnect > > Message-ID: > > < > > CAECr9X7U7YsGBv9ajcmeOCxfdD+YLar8QyPwYBN0qaP10CzTug at mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > Ok, this is being tortuous. First of all: define a strategy. If you are > > going to create the resources in Neutron, define how. I've provided a way > > to do this, find a formal strategy to ground it. > > > > Second: (again) try to find a connection between resources, if you are > > going to use the strategy of creating the resources in Neutron. The > > "Logical_Router_Static_Route" belongs to a router univocally. If that > > router has been created by OpenStack, then you can modify the DB sync > > method to consider learned routes too. > > > > In order to do this, you'll need a set of resources that are going to be > > needed in Neutron, the OVN counterparts and other resources (like > > "Logical_Router_Static_Route") that will be added and will be present in > > OVN and not in Neutron DB. Also you'll need to know how to relate all of > > them. > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: < > > https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/90712e47/attachment.htm > > > > > > > ------------------------------ > > > > Subject: Digest Footer > > > > _______________________________________________ > > openstack-discuss mailing list > > openstack-discuss at lists.openstack.org > > > > > > ------------------------------ > > > > End of openstack-discuss Digest, Vol 57, Issue 55 > > ************************************************* > > From arnaud.morin at gmail.com Sun Jul 23 13:03:09 2023 From: arnaud.morin at gmail.com (Arnaud Morin) Date: Sun, 23 Jul 2023 13:03:09 +0000 Subject: [olso.messaging] RabbitMQ and streams Message-ID: Hey all, Is there any chance that someone already worked on integrating RabbitMQ streams (see [1]) to replace fanouts in oslo rabbitmq driver? We are wondering if this would help lowering the rabbit usage, and maybe increase the rabbit stability? Our deployment is relying a lot on fanouts to delivers messages to computes (mostly neutron, e.g. secgroup update / remote cache population), this is leading to a massive number of messages beeing delivered by seconds. Cheers, Arnaud [1] https://www.rabbitmq.com/streams.html From kennelson11 at gmail.com Mon Jul 24 04:35:43 2023 From: kennelson11 at gmail.com (Kendall Nelson) Date: Sun, 23 Jul 2023 23:35:43 -0500 Subject: vPTG October 2023 Team Signup Kickoff Message-ID: Hi Everyone, Hopefully, you have seen by now that the next OpenInfra vPTG[1] which will take place October 23-27, 2023 virtually and registration is open[2]! *To sign up your team, you must complete the survey[3] by August 20th at 7:00 UTC.* We NEED accurate contact information (email and IRC nickname ideally) for the moderator of your team?s sessions. This is because the survey information will be used to organize the schedule sign ups which will be done via the PTGBot. If you are not on IRC, please get setup[4] on the OFTC network and join #openinfra-events. You are also encouraged to familiarize yourself with the PTGBot documentation[5] as well. If you have any questions, please reach out! Information about signing up for time will be sent to moderators late August after the interest survey deadline on August 20. -Kendall (diablo_rojo) [1] PTG Site: https://openinfra.dev/ptg/ [2] PTG Registration: http://ptg2023.openinfra.dev/ [3] Team Survey: https://openinfrafoundation.formstack.com/forms/oct2023_ptg_team_signup [4] Setup IRC: https://docs.openstack.org/contributors/common/irc.html [5] PTGBot README: https://opendev.org/openstack/ptgbot/src/branch/master/README.rst -------------- next part -------------- An HTML attachment was scrubbed... URL: From eblock at nde.ag Mon Jul 24 07:27:56 2023 From: eblock at nde.ag (Eugen Block) Date: Mon, 24 Jul 2023 07:27:56 +0000 Subject: Ussuri: how to delete lbaas loadbalancer left over? - SOLVED In-Reply-To: <0ee75b98-af96-09c5-d30a-dfd9aef4a508@ijclab.in2p3.fr> References: <20230707164654.Horde.sUTt6M8NRpBGR8HwFy7KwGv@webmail.nde.ag> <1893244f3d8.27f3.86e99a005b57e9644b7bb922d8d7f665@ijclab.in2p3.fr> <20230707220512.Horde.10Q-iIqvZct1kSTQs9Gd4-u@webmail.nde.ag> <20230707221544.Horde.tGBcMHNkuxKWyNahfmPmJ9V@webmail.nde.ag> <1a342d65-191b-c367-9872-1d72f83479f6@ijclab.in2p3.fr> <0ee75b98-af96-09c5-d30a-dfd9aef4a508@ijclab.in2p3.fr> Message-ID: <20230724072756.Horde.e4x2v9lD1pwO0xnZMd3WL96@webmail.nde.ag> Awesome, that's great news! Thanks for sharing! Zitat von Michel Jouvin : > Hi, > > As promised, the conclusion of the story! After installing Octavia > in our cloud, we ran the nlbaas2octavia.py tool as suggested by > Eugen. It worked like a charm, without any modification. Once the > LBs had been converted to Octavia LB, we just had to fix the > provider attribute in the DB, changing it from "haproxy" (which is > not a provider in Octavia) to "amphora". And then the "openstack coe > cluster delete" worked just fine removing these broken LBs as part > of the cluster deletion. > > Cheers, > > Michel > > Le 14/07/2023 ? 23:24, Michel Jouvin a ?crit?: >> Hi Eugen, >> >> I found some time to look at my issue in the light of your advices! >> In my research, I found the presentation >> https://opendev.org/openstack/neutron-lbaas which is both synthetic >> and clear, with a list of the migration options offered (at the >> time of the presentation). This includes the DB migration tool, >> nlbaas2octavia.py. Following your broken link, I manage to find it, >> it is still there but hidden. You need to checkout the repo >> https://opendev.org/openstack/neutron-lbaas and go back a couple of >> revisions (it is explained in the README). It is Python2 but at >> least it is a starting point, either installing a Python2 venv or >> migrating the script which is not very long and seems to have very >> few dependcies, apart from oslo. >> >> I'm afraid that the neutron-lbaas-proxy is no longer an option as, >> as far as I understood, it relies on some LBAAS code still being >> present in the Neutron server and this part of the Neutron code has >> been completely removed I think in Ussuri release (it's the reason >> of my problems in fact, too bad, it was a matter of a few days, >> just missed the very old announcement that it would be the case). >> >> If I succeed to clean the things (again I don't really want to >> migrate the existing LBAAS, I just want to delete them), I'll >> report in this thread in case it is useful to somebody else... >> >> Best regards, >> >> Michel >> >> Le 08/07/2023 ? 00:15, Eugen Block a ?crit?: >>> Unfortunately, the link to the migration tool doesn?t work: >>> >>> https://wiki.openstack.org/wiki/Neutron/LBaaS/Deprecation >>> >>> But maybe it can get you in the right direction, >>> neutron-lbaas-proxy seems to be the keyword. But as I already >>> mentioned, I don?t have experience with this path. >>> >>> Zitat von Eugen Block : >>> >>>> Hi, >>>> >>>> I mean the latter. Once you have Octavia installed you can create >>>> new LBs, but as I understand it you won?t be able to delete the >>>> legacy LBs. Did the neutron config change when you upgraded to >>>> Ussuri? I wonder if there?s just some config missing to be able >>>> to delete the old LBs, I don?t have a clue tbh. Maybe someone >>>> else has some more experience and will chime in. >>>> >>>> Zitat von Michel Jouvin : >>>> >>>>> Ho Eug?ne, >>>>> >>>>> Thanks for your answer. Do you mean that after installing >>>>> octavia (it is planned) we'll have again the ability to delete >>>>> the remaining LBAAS instances? Or just that octavia is the LBAAS >>>>> replacement in terms of functionalities? >>>>> >>>>> Best regards, >>>>> >>>>> Michel >>>>> Sent from my mobile >>>>> Le 7 juillet 2023 18:52:30 Eugen Block a ?crit : >>>>> >>>>>> Hi, >>>>>> >>>>>> neutron lbaas was deprecated in Queens so you may have to migrate the >>>>>> existing LBs to octavia. I have never done that but I remember reading >>>>>> through the SUSE Docs when one of our customers had to decide whether >>>>>> they wanted to upgrade or reinstall with a newer openstack release. >>>>>> They decided to do the latter, so we set up octavia from scratch and >>>>>> didn't have to migrate anything. There's also a video I've never >>>>>> watched [2], maybe that helps. I can't really tell if a migration is >>>>>> possible to work around your issue but I thought I'd share anyway. >>>>>> >>>>>> Regards, >>>>>> Eugen >>>>>> >>>>>> [1] >>>>>> https://documentation.suse.com/soc/9/single-html/suse-openstack-cloud-crowbar-deployment/#sec-depl-ostack-octavia-migrate-users [2] >>>>>> https://www.youtube.com/watch?v=jj4KMJPA0Pk >>>>>> >>>>>> Zitat von Michel Jouvin : >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> We had a few Magnum (K8s) clusters created a couple of years ago >>>>>>> (with Rocky and Stein versions) and forgotten. We started to delete >>>>>>> them this spring when we where running Train Neutron service. >>>>>>> Basically we managed to do this with the following sequence: >>>>>>> >>>>>>> - openstack coe cluster delete xxx and waiting for DELETE_FAILED >>>>>>> - Use openstack coe cluster show / openstack stack resource list -n2 >>>>>>> to identify the neutron entry causing the error and pick the >>>>>>> corresponding resource ID >>>>>>> - Find the ports associated with the router with openstack port list >>>>>>> --router previously_found_id >>>>>>> - Use the port subnet to find the port corresponding lbaas load >>>>>>> balancer ID, use the neutron CLI to delete the load balancer >>>>>>> (deleting one by one all the dependencies preventing the load >>>>>>> balancer removal) >>>>>>> - Rerun openstack coe cluster delete >>>>>>> >>>>>>> For some reasons, we didn't cleanup all the abandoned clusters and >>>>>>> upgraded Neutron to Ussuri. Unfortunately, since then our previous >>>>>>> process is no longer working as it seems that the Neutron server >>>>>>> doesn't know anymore anything about the LBAAS load balancers (and >>>>>>> "neutron lbaas-loadbalancer list" returns nothing). In the neutron >>>>>>> server, any attempt to delete the subnet attached to the load >>>>>>> balancer (or to list them with Neutron CLI) results in the following >>>>>>> errors in Neutron server.log : >>>>>>> >>>>>>> ------ >>>>>>> >>>>>>> 2023-07-07 16:27:31.139 14962 WARNING >>>>>>> neutron.pecan_wsgi.controllers.root >>>>>>> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >>>>>>> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >>>>>>> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >>>>>>> 1367c9a4d5da4b229c35789c271dc7aa] No controller found for: lbaas - >>>>>>> returning response code 404: pecan.routing.PecanNotFound >>>>>>> 2023-07-07 16:27:31.140 14962 INFO >>>>>>> neutron.pecan_wsgi.hooks.translation >>>>>>> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >>>>>>> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >>>>>>> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >>>>>>> 1367c9a4d5da4b229c35789c271dc7aa] GET failed (client error): The >>>>>>> resource could not be found. >>>>>>> 2023-07-07 16:27:31.141 14962 INFO neutron.wsgi >>>>>>> [req-71e712fc-d8a7-4815-90b3-b406c10e0caa >>>>>>> a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c >>>>>>> 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa >>>>>>> 1367c9a4d5da4b229c35789c271dc7aa] 157.136.249.153 "GET >>>>>>> /v2.0/lbaas/loadbalancers?name=kube_service_964f7e76-d2d5-4126-ab11-cd689f6dd9f9_runnerdeploy-wm9sm-5h52l_hello-node-x-default-x-runnerdeploy-wm9sm-5h52l HTTP/1.1" status: 404? len: 304 >>>>>>> time: >>>>>>> 0.0052643 >>>>>>> ------ >>>>>>> >>>>>>> Any suggestion to workaround this problem and be able to >>>>>>> successfully delete our old Magnum clusters? >>>>>>> >>>>>>> Thanks in advance for any help. Best regards, >>>>>>> >>>>>>> Michel >>> >>> >>> >>> From jim2k7 at gmail.com Mon Jul 24 11:44:33 2023 From: jim2k7 at gmail.com (Jaime Ibar) Date: Mon, 24 Jul 2023 12:44:33 +0100 Subject: [zun] container run database insert error Message-ID: Hi all, I have installed and configured zun for container support, all went good and I can list the available services with > openstack appcontainer service list however when I try to launch a container with > openstack appcontainer run --name container --net network=$NET_ID cirros ping 8.8.8.8 as stated in the documentation, I get the following error in zun-api oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`zun`.`container_actions`, CONSTRAINT `container_actions_ibfk_1` FOREIGN KEY (`container_uuid`) REFERENCES `container` (`uuid`))') I have followed this installation guide for yoga release(the one I'm running at the moment) https://docs.openstack.org/zun/yoga/install/index.html. Did anyone see this error before? TIA -- Jaime -------------- next part -------------- An HTML attachment was scrubbed... URL: From kkloppenborg at rwts.com.au Mon Jul 24 03:46:10 2023 From: kkloppenborg at rwts.com.au (Karl Kloppenborg) Date: Mon, 24 Jul 2023 03:46:10 +0000 Subject: [openstack][largescale-sig] Openstack multi region deployment In-Reply-To: References: Message-ID: Apologies I?ve been off sick. However yes, this is the way we do it as well. I would say this is also the most sane way to deal with this. Thanks, Karl. From: Arnaud Morin Date: Sunday, 23 July 2023 at 10:56 pm To: Nguy?n H?u Kh?i Cc: Karl Kloppenborg , OpenStack Discuss Subject: Re: [openstack][largescale-sig] Openstack multi region deployment We have this model also with only one keystone. We have multiple galera clusters synchronized together. Only one cluster is used for write requests (located in one region), others are read only / cache. Most of the calls done to our keystone are "read" or token validation requests, and this works fine with a read galera cluster / cache. I know that we also have a custom way to invalidate cache across regions, but I dont remember the details, I can ask the team. Anyway, this is do-able :) I imagine it also depends on the usage you have, if you create a lot of users/projects/assignments, then it may be harder to achieve. Cheers, Arnaud On 19.07.23 - 14:03, Nguy?n H?u Kh?i wrote: > Hello, thank you very much. > > But can I ask how we process if 1 region at ASIA and 2 regions in the USA? > > Database latency will be our problem. > > Nguyen Huu Khoi > > > On Tue, Jul 18, 2023 at 8:21?PM Karl Kloppenborg > wrote: > > > Hi Nguy, > > > > > > > > We?ve deployed a large multi-region openstack deployment. > > > > As a rule of thumb we?ve got a ?keystone? region which is as best we can > > highly available and very redundant. > > > > > > > > We then have all other regions talk back to this region, we just usually > > call it ?keystone? or ?core? and it?s hidden from the UI from users. > > > > > > > > We then just run a large well kept Galara cluster to support it. > > > > > > > > --Karl. > > > > > > > > *From: *openstack-discuss-request at lists.openstack.org < > > openstack-discuss-request at lists.openstack.org> > > *Date: *Tuesday, 18 July 2023 at 9:25 pm > > *To: *openstack-discuss at lists.openstack.org < > > openstack-discuss at lists.openstack.org> > > *Subject: *openstack-discuss Digest, Vol 57, Issue 55 > > > > Send openstack-discuss mailing list submissions to > > openstack-discuss at lists.openstack.org > > > > To subscribe or unsubscribe via the World Wide Web, visit > > > > https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss > > > > or, via email, send a message with subject or body 'help' to > > openstack-discuss-request at lists.openstack.org > > > > You can reach the person managing the list at > > openstack-discuss-owner at lists.openstack.org > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of openstack-discuss digest..." > > > > > > Today's Topics: > > > > 1. [openstack][largescale-sig] Openstack multi region deployment > > (Nguy?n H?u Kh?i) > > 2. Re: [openstack][largescale-sig] Openstack multi region > > deployment (Felix Huettner) > > 3. Re: [openstack][largescale-sig] Openstack multi region > > deployment (Nguy?n H?u Kh?i) > > 4. Re: [neutron] unmanaged router resources - OVN interconnect > > (Rodolfo Alonso Hernandez) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Tue, 18 Jul 2023 12:07:12 +0700 > > From: Nguy?n H?u Kh?i > > To: OpenStack Discuss > > Subject: [openstack][largescale-sig] Openstack multi region deployment > > Message-ID: > > < > > CABAODReJ6QW8A4OENEjmhFCiM-15B0qc2La_aMr1EKfaENq9iw at mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > Hello guys, > > > > I am going to deploy openstack multi regions and I know that keystone > > replication is the most challenging. > > > > I plan to set up 2 regions which use centralize galera cluster(3 nodes). > > and one standby edge galera cluster(3 nodes) > > > > When region 1 is node available, I will map region 2 to use standby edge > > galera cluster. > > > > I hope you give me some experience and advice with multi regions. > > > > Thank you very much. > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: < > > https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/c95d3675/attachment-0001.htm > > > > > > > ------------------------------ > > > > Message: 2 > > Date: Tue, 18 Jul 2023 09:34:35 +0200 > > From: Felix Huettner > > To: Nguy?n H?u Kh?i > > Cc: OpenStack Discuss > > Subject: Re: [openstack][largescale-sig] Openstack multi region > > deployment > > Message-ID: > > Content-Type: text/plain; charset=utf-8 > > > > Hi, > > > > i think you have two options here: > > 1. you could span a single galera cluster over all of your regions. > > this might have some latency issues, but if your are not too write > > heavy that might be fine. I know some companies use that setup. > > 2. you use some kind of multiple galera clusters with replication. > > But i have not yet heard of anybody using this setup. > > > > An alternative might be to have separate keystone setups with separate > > databases. This would probably reduce the error potential, but might not > > fit your usecase. > > > > Thanks > > Felix > > > > > > On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote: > > > Hello guys, > > > > > > I am going to deploy openstack multi regions and I know that keystone > > > replication is the most challenging. > > > > > > I plan to set up 2 regions which use centralize galera cluster(3 nodes). > > > and one standby edge galera cluster(3 nodes) > > > > > > When region 1 is node available, I will map region 2 to use standby edge > > > galera cluster. > > > > > > I hope you give me some experience and advice with multi regions. > > > > > > Thank you very much. > > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r > > die Verwertung durch den vorgesehenen Empf?nger bestimmt. > > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender > > bitte unverz?glich in Kenntnis und l?schen diese E Mail. > > > > Hinweise zum Datenschutz finden Sie hier. > > > > > > This e-mail may contain confidential content and is intended only for the > > specified recipient/s. > > If you are not the intended recipient, please inform the sender > > immediately and delete this e-mail. > > > > Information on data protection can be found here< > > https://www.datenschutz.schwarz>. > > > > > > > > ------------------------------ > > > > Message: 3 > > Date: Tue, 18 Jul 2023 15:36:11 +0700 > > From: Nguy?n H?u Kh?i > > To: Nguy?n H?u Kh?i , OpenStack Discuss > > > > Subject: Re: [openstack][largescale-sig] Openstack multi region > > deployment > > Message-ID: > > > CGBW1_bRkLQJAxLZxAx8V4qvbdBmTUQBUm2SRsow at mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > Hi. > > Thank you for your reply. > > > > The first one has a problem because each region is too soft. If a member is > > down, so this region is gone. > > > > It is so challenge with us. > > > > > > Nguyen Huu Khoi > > > > > > On Tue, Jul 18, 2023 at 2:34?PM Felix Huettner > > > > wrote: > > > > > Hi, > > > > > > i think you have two options here: > > > 1. you could span a single galera cluster over all of your regions. > > > this might have some latency issues, but if your are not too write > > > heavy that might be fine. I know some companies use that setup. > > > 2. you use some kind of multiple galera clusters with replication. > > > But i have not yet heard of anybody using this setup. > > > > > > An alternative might be to have separate keystone setups with separate > > > databases. This would probably reduce the error potential, but might not > > > fit your usecase. > > > > > > Thanks > > > Felix > > > > > > > > > On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote: > > > > Hello guys, > > > > > > > > I am going to deploy openstack multi regions and I know that keystone > > > > replication is the most challenging. > > > > > > > > I plan to set up 2 regions which use centralize galera cluster(3 > > nodes). > > > > and one standby edge galera cluster(3 nodes) > > > > > > > > When region 1 is node available, I will map region 2 to use standby > > edge > > > > galera cluster. > > > > > > > > I hope you give me some experience and advice with multi regions. > > > > > > > > Thank you very much. > > > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r > > > die Verwertung durch den vorgesehenen Empf?nger bestimmt. > > > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender > > > bitte unverz?glich in Kenntnis und l?schen diese E Mail. > > > > > > Hinweise zum Datenschutz finden Sie hier > >. > > > > > > > > > This e-mail may contain confidential content and is intended only for the > > > specified recipient/s. > > > If you are not the intended recipient, please inform the sender > > > immediately and delete this e-mail. > > > > > > Information on data protection can be found here< > > > https://www.datenschutz.schwarz>. > > > > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: < > > https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/749440e3/attachment-0001.htm > > > > > > > ------------------------------ > > > > Message: 4 > > Date: Tue, 18 Jul 2023 13:23:27 +0200 > > From: Rodolfo Alonso Hernandez > > To: Roberto Bartzen Acosta > > Cc: openstack-discuss , Terry > > Wilson , Tiago Pires < > > tiago.pires at luizalabs.com> > > Subject: Re: [neutron] unmanaged router resources - OVN interconnect > > Message-ID: > > < > > CAECr9X7U7YsGBv9ajcmeOCxfdD+YLar8QyPwYBN0qaP10CzTug at mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > Ok, this is being tortuous. First of all: define a strategy. If you are > > going to create the resources in Neutron, define how. I've provided a way > > to do this, find a formal strategy to ground it. > > > > Second: (again) try to find a connection between resources, if you are > > going to use the strategy of creating the resources in Neutron. The > > "Logical_Router_Static_Route" belongs to a router univocally. If that > > router has been created by OpenStack, then you can modify the DB sync > > method to consider learned routes too. > > > > In order to do this, you'll need a set of resources that are going to be > > needed in Neutron, the OVN counterparts and other resources (like > > "Logical_Router_Static_Route") that will be added and will be present in > > OVN and not in Neutron DB. Also you'll need to know how to relate all of > > them. > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: < > > https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/90712e47/attachment.htm > > > > > > > ------------------------------ > > > > Subject: Digest Footer > > > > _______________________________________________ > > openstack-discuss mailing list > > openstack-discuss at lists.openstack.org > > > > > > ------------------------------ > > > > End of openstack-discuss Digest, Vol 57, Issue 55 > > ************************************************* > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From senrique at redhat.com Mon Jul 24 07:00:00 2023 From: senrique at redhat.com (Sofia Enriquez) Date: Mon, 24 Jul 2023 08:00:00 +0100 Subject: [cinder] Resignation from Cinder core & Bug Deputy Message-ID: Hello Argonauts, I will be transitioning from my current role at Red Hat, and my availability for patch making and reviewing will be limited after July. Considering these circumstances, I believe it is appropriate for me to resign as a Cinder core reviewer and as Cinder Bug Deputy. Feel free to connect with me on LinkedIn and/or follow on Twitter / Github . ? I look forward to staying in touch and crossing paths again in the future. [image: denver_ptg.jpeg] Wishing you all the best, and thank you!? Cheers, Sofia -- Sof?a Enriquez she/her Software Engineer Red Hat PnT IRC: @enriquetaso @RedHat Red Hat Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: denver_ptg.jpeg Type: image/jpeg Size: 88520 bytes Desc: not available URL: From helena at openinfra.dev Mon Jul 24 13:20:20 2023 From: helena at openinfra.dev (Helena Spease) Date: Mon, 24 Jul 2023 15:20:20 +0200 Subject: Take the OpenStack User Survey! Message-ID: <2AECC3B6-F500-42B5-9A49-11166B0850C0@openinfra.dev> Hello everyone, It?s User Survey season! The 2023 OpenStack User Survey is closing August 23, 2023. That?s just under a month to submit your anonymized feedback to the upstream contributors. https://www.openstack.org/user-survey/survey-2023/landing We have posts from the OpenInfra Foundation social media channels that you can re-share to your networks to encourage them to participate in the User Survey as well. Mastodon: https://social.openinfra.dev/@OpenStack/110768926571772510 Twitter: https://twitter.com/OpenStack/status/1683441577648181248 LinkedIn: https://www.linkedin.com/posts/openstack_take-the-2023-openstack-user-survey-the-activity-7089207280319033344-zXd8 Evaluating another OpenInfra project? We have User Surveys for them as well! Kata Containers: https://openinfrafoundation.formstack.com/forms/kata_containers_user_survey OpenStack: https://www.openstack.org/user-survey/survey-2023/landing StarlingX: https://openinfrafoundation.formstack.com/forms/starlingx_user_survey Zuul: https://openinfrafoundation.formstack.com/forms/zuul_user_survey We really appreciate your participation in the User Survey and are really excited to share with you all the cool data we collect! Cheers, Helena helena at openinfra.dev Marketing & Community Coordinator The OpenInfra Foundation -------------- next part -------------- An HTML attachment was scrubbed... URL: From knikolla at bu.edu Mon Jul 24 18:47:24 2023 From: knikolla at bu.edu (Nikolla, Kristi) Date: Mon, 24 Jul 2023 18:47:24 +0000 Subject: [tc] Technical Committee next weekly meeting today on July 25, 2023 Message-ID: <84765F39-25B5-46CE-AA50-753C48FAD707@bu.edu> Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held on Tuesday, July 25, 2023 at 1800 UTC on #openstack-tc on OFTC IRC. Please find below the agenda for the meeting ? Roll call ? Follow up on past action items ? knikolla will amend proposal for Unmaintained branches. ? tc-members to review https://review.opendev.org/c/openstack/project-team-guide/+/843457 ? Extended Maintenance ? https://etherpad.opendev.org/p/openstack-unmaintained-workflow ? https://etherpad.opendev.org/p/openstack-exteneded-maintenance ? Release notes guidelines for SLURP/NON-SLURP cadence ? Gate health check ? Open Discussion and Reviews ? https://review.opendev.org/q/projects:openstack/governance+is:open Thank you, Kristi Nikolla From jungleboyj at gmail.com Mon Jul 24 17:22:15 2023 From: jungleboyj at gmail.com (Jay Bryant) Date: Mon, 24 Jul 2023 12:22:15 -0500 Subject: [cinder] Resignation from Cinder core & Bug Deputy In-Reply-To: References: Message-ID: Sofia, So sorry to hear you are moving on.? Thank you for all your contributions over the past years! Jay On 7/24/2023 2:00 AM, Sofia Enriquez wrote: > Hello Argonauts, > > I will be transitioning from my current role at Red Hat, and my > availability for patch making and reviewing will be limited after > July. Considering these circumstances, I believe it is appropriate for > me to resign as a Cinder core reviewer and as Cinder Bug Deputy. > > Feel free to connect with me on LinkedIn > and/or follow on Twitter > / Github > .?? I look forward to staying in > touch and crossing paths again in the future. > > denver_ptg.jpeg > > Wishing you all the best, and thank you!? > > Cheers, > Sofia > -- > > Sof?a Enriquez > > she/her > > Software Engineer > > Red Hat PnT > > IRC: @enriquetaso > > @RedHat Red Hat > Red Hat > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: denver_ptg.jpeg Type: image/jpeg Size: 88520 bytes Desc: not available URL: From alsotoes at gmail.com Mon Jul 24 17:52:42 2023 From: alsotoes at gmail.com (Alvaro Soto) Date: Mon, 24 Jul 2023 11:52:42 -0600 Subject: [cinder] Resignation from Cinder core & Bug Deputy In-Reply-To: References: Message-ID: =D On Mon, Jul 24, 2023 at 10:28?AM Sofia Enriquez wrote: > Hello Argonauts, > > I will be transitioning from my current role at Red Hat, and my > availability for patch making and reviewing will be limited after July. > Considering these circumstances, I believe it is appropriate for me to > resign as a Cinder core reviewer and as Cinder Bug Deputy. > > Feel free to connect with me on LinkedIn > and/or follow on Twitter > / Github > . ? I look forward to staying in touch > and crossing paths again in the future. > > [image: denver_ptg.jpeg] > > Wishing you all the best, and thank you!? > > Cheers, > Sofia > -- > > Sof?a Enriquez > > she/her > > Software Engineer > > Red Hat PnT > > IRC: @enriquetaso > @RedHat Red Hat > Red Hat > > > > -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: denver_ptg.jpeg Type: image/jpeg Size: 88520 bytes Desc: not available URL: From nguyenhuukhoinw at gmail.com Mon Jul 24 23:15:30 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Tue, 25 Jul 2023 06:15:30 +0700 Subject: [openstack][largescale-sig] Openstack multi region deployment In-Reply-To: References: Message-ID: Hello Arnaud, Thank you for sharing. But how do you deal with db latency, example one region is in Asia and one is in America. Nguyen Huu Khoi On Sun, Jul 23, 2023 at 7:56?PM Arnaud Morin wrote: > We have this model also with only one keystone. > We have multiple galera clusters synchronized together. > Only one cluster is used for write requests (located in one region), > others are read only / cache. > Most of the calls done to our keystone are "read" or token validation > requests, and this works fine with a read galera cluster / cache. > > I know that we also have a custom way to invalidate cache across > regions, but I dont remember the details, I can ask the team. > > Anyway, this is do-able :) > > I imagine it also depends on the usage you have, if you create a lot of > users/projects/assignments, then it may be harder to achieve. > > Cheers, > Arnaud > > On 19.07.23 - 14:03, Nguy?n H?u Kh?i wrote: > > Hello, thank you very much. > > > > But can I ask how we process if 1 region at ASIA and 2 regions in the > USA? > > > > Database latency will be our problem. > > > > Nguyen Huu Khoi > > > > > > On Tue, Jul 18, 2023 at 8:21?PM Karl Kloppenborg < > kkloppenborg at rwts.com.au> > > wrote: > > > > > Hi Nguy, > > > > > > > > > > > > We?ve deployed a large multi-region openstack deployment. > > > > > > As a rule of thumb we?ve got a ?keystone? region which is as best we > can > > > highly available and very redundant. > > > > > > > > > > > > We then have all other regions talk back to this region, we just > usually > > > call it ?keystone? or ?core? and it?s hidden from the UI from users. > > > > > > > > > > > > We then just run a large well kept Galara cluster to support it. > > > > > > > > > > > > --Karl. > > > > > > > > > > > > *From: *openstack-discuss-request at lists.openstack.org < > > > openstack-discuss-request at lists.openstack.org> > > > *Date: *Tuesday, 18 July 2023 at 9:25 pm > > > *To: *openstack-discuss at lists.openstack.org < > > > openstack-discuss at lists.openstack.org> > > > *Subject: *openstack-discuss Digest, Vol 57, Issue 55 > > > > > > Send openstack-discuss mailing list submissions to > > > openstack-discuss at lists.openstack.org > > > > > > To subscribe or unsubscribe via the World Wide Web, visit > > > > > > https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss > > > > > > or, via email, send a message with subject or body 'help' to > > > openstack-discuss-request at lists.openstack.org > > > > > > You can reach the person managing the list at > > > openstack-discuss-owner at lists.openstack.org > > > > > > When replying, please edit your Subject line so it is more specific > > > than "Re: Contents of openstack-discuss digest..." > > > > > > > > > Today's Topics: > > > > > > 1. [openstack][largescale-sig] Openstack multi region deployment > > > (Nguy?n H?u Kh?i) > > > 2. Re: [openstack][largescale-sig] Openstack multi region > > > deployment (Felix Huettner) > > > 3. Re: [openstack][largescale-sig] Openstack multi region > > > deployment (Nguy?n H?u Kh?i) > > > 4. Re: [neutron] unmanaged router resources - OVN interconnect > > > (Rodolfo Alonso Hernandez) > > > > > > > > > ---------------------------------------------------------------------- > > > > > > Message: 1 > > > Date: Tue, 18 Jul 2023 12:07:12 +0700 > > > From: Nguy?n H?u Kh?i > > > To: OpenStack Discuss > > > Subject: [openstack][largescale-sig] Openstack multi region deployment > > > Message-ID: > > > < > > > CABAODReJ6QW8A4OENEjmhFCiM-15B0qc2La_aMr1EKfaENq9iw at mail.gmail.com> > > > Content-Type: text/plain; charset="utf-8" > > > > > > Hello guys, > > > > > > I am going to deploy openstack multi regions and I know that keystone > > > replication is the most challenging. > > > > > > I plan to set up 2 regions which use centralize galera cluster(3 > nodes). > > > and one standby edge galera cluster(3 nodes) > > > > > > When region 1 is node available, I will map region 2 to use standby > edge > > > galera cluster. > > > > > > I hope you give me some experience and advice with multi regions. > > > > > > Thank you very much. > > > -------------- next part -------------- > > > An HTML attachment was scrubbed... > > > URL: < > > > > https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/c95d3675/attachment-0001.htm > > > > > > > > > > ------------------------------ > > > > > > Message: 2 > > > Date: Tue, 18 Jul 2023 09:34:35 +0200 > > > From: Felix Huettner > > > To: Nguy?n H?u Kh?i > > > Cc: OpenStack Discuss > > > Subject: Re: [openstack][largescale-sig] Openstack multi region > > > deployment > > > Message-ID: > > > Content-Type: text/plain; charset=utf-8 > > > > > > Hi, > > > > > > i think you have two options here: > > > 1. you could span a single galera cluster over all of your regions. > > > this might have some latency issues, but if your are not too write > > > heavy that might be fine. I know some companies use that setup. > > > 2. you use some kind of multiple galera clusters with replication. > > > But i have not yet heard of anybody using this setup. > > > > > > An alternative might be to have separate keystone setups with separate > > > databases. This would probably reduce the error potential, but might > not > > > fit your usecase. > > > > > > Thanks > > > Felix > > > > > > > > > On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote: > > > > Hello guys, > > > > > > > > I am going to deploy openstack multi regions and I know that keystone > > > > replication is the most challenging. > > > > > > > > I plan to set up 2 regions which use centralize galera cluster(3 > nodes). > > > > and one standby edge galera cluster(3 nodes) > > > > > > > > When region 1 is node available, I will map region 2 to use standby > edge > > > > galera cluster. > > > > > > > > I hope you give me some experience and advice with multi regions. > > > > > > > > Thank you very much. > > > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur > f?r > > > die Verwertung durch den vorgesehenen Empf?nger bestimmt. > > > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den > Absender > > > bitte unverz?glich in Kenntnis und l?schen diese E Mail. > > > > > > Hinweise zum Datenschutz finden Sie hier< > https://www.datenschutz.schwarz>. > > > > > > > > > This e-mail may contain confidential content and is intended only for > the > > > specified recipient/s. > > > If you are not the intended recipient, please inform the sender > > > immediately and delete this e-mail. > > > > > > Information on data protection can be found here< > > > https://www.datenschutz.schwarz>. > > > > > > > > > > > > ------------------------------ > > > > > > Message: 3 > > > Date: Tue, 18 Jul 2023 15:36:11 +0700 > > > From: Nguy?n H?u Kh?i > > > To: Nguy?n H?u Kh?i , OpenStack Discuss > > > > > > Subject: Re: [openstack][largescale-sig] Openstack multi region > > > deployment > > > Message-ID: > > > > > CGBW1_bRkLQJAxLZxAx8V4qvbdBmTUQBUm2SRsow at mail.gmail.com> > > > Content-Type: text/plain; charset="utf-8" > > > > > > Hi. > > > Thank you for your reply. > > > > > > The first one has a problem because each region is too soft. If a > member is > > > down, so this region is gone. > > > > > > It is so challenge with us. > > > > > > > > > Nguyen Huu Khoi > > > > > > > > > On Tue, Jul 18, 2023 at 2:34?PM Felix Huettner > > > > > > > wrote: > > > > > > > Hi, > > > > > > > > i think you have two options here: > > > > 1. you could span a single galera cluster over all of your regions. > > > > this might have some latency issues, but if your are not too write > > > > heavy that might be fine. I know some companies use that setup. > > > > 2. you use some kind of multiple galera clusters with replication. > > > > But i have not yet heard of anybody using this setup. > > > > > > > > An alternative might be to have separate keystone setups with > separate > > > > databases. This would probably reduce the error potential, but might > not > > > > fit your usecase. > > > > > > > > Thanks > > > > Felix > > > > > > > > > > > > On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote: > > > > > Hello guys, > > > > > > > > > > I am going to deploy openstack multi regions and I know that > keystone > > > > > replication is the most challenging. > > > > > > > > > > I plan to set up 2 regions which use centralize galera cluster(3 > > > nodes). > > > > > and one standby edge galera cluster(3 nodes) > > > > > > > > > > When region 1 is node available, I will map region 2 to use standby > > > edge > > > > > galera cluster. > > > > > > > > > > I hope you give me some experience and advice with multi regions. > > > > > > > > > > Thank you very much. > > > > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur > f?r > > > > die Verwertung durch den vorgesehenen Empf?nger bestimmt. > > > > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den > Absender > > > > bitte unverz?glich in Kenntnis und l?schen diese E Mail. > > > > > > > > Hinweise zum Datenschutz finden Sie hier< > https://www.datenschutz.schwarz > > > >. > > > > > > > > > > > > This e-mail may contain confidential content and is intended only > for the > > > > specified recipient/s. > > > > If you are not the intended recipient, please inform the sender > > > > immediately and delete this e-mail. > > > > > > > > Information on data protection can be found here< > > > > https://www.datenschutz.schwarz>. > > > > > > > -------------- next part -------------- > > > An HTML attachment was scrubbed... > > > URL: < > > > > https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/749440e3/attachment-0001.htm > > > > > > > > > > ------------------------------ > > > > > > Message: 4 > > > Date: Tue, 18 Jul 2023 13:23:27 +0200 > > > From: Rodolfo Alonso Hernandez > > > To: Roberto Bartzen Acosta > > > Cc: openstack-discuss , Terry > > > Wilson , Tiago Pires < > > > tiago.pires at luizalabs.com> > > > Subject: Re: [neutron] unmanaged router resources - OVN interconnect > > > Message-ID: > > > < > > > CAECr9X7U7YsGBv9ajcmeOCxfdD+YLar8QyPwYBN0qaP10CzTug at mail.gmail.com> > > > Content-Type: text/plain; charset="utf-8" > > > > > > Ok, this is being tortuous. First of all: define a strategy. If you are > > > going to create the resources in Neutron, define how. I've provided a > way > > > to do this, find a formal strategy to ground it. > > > > > > Second: (again) try to find a connection between resources, if you are > > > going to use the strategy of creating the resources in Neutron. The > > > "Logical_Router_Static_Route" belongs to a router univocally. If that > > > router has been created by OpenStack, then you can modify the DB sync > > > method to consider learned routes too. > > > > > > In order to do this, you'll need a set of resources that are going to > be > > > needed in Neutron, the OVN counterparts and other resources (like > > > "Logical_Router_Static_Route") that will be added and will be present > in > > > OVN and not in Neutron DB. Also you'll need to know how to relate all > of > > > them. > > > -------------- next part -------------- > > > An HTML attachment was scrubbed... > > > URL: < > > > > https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/90712e47/attachment.htm > > > > > > > > > > ------------------------------ > > > > > > Subject: Digest Footer > > > > > > _______________________________________________ > > > openstack-discuss mailing list > > > openstack-discuss at lists.openstack.org > > > > > > > > > ------------------------------ > > > > > > End of openstack-discuss Digest, Vol 57, Issue 55 > > > ************************************************* > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nguyenhuukhoinw at gmail.com Mon Jul 24 23:17:20 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Tue, 25 Jul 2023 06:17:20 +0700 Subject: [openstack][largescale-sig] Openstack multi region deployment In-Reply-To: References: Message-ID: Hello Karl, How are you? Thank you for your response.. Hope you are ok. Nguyen Huu Khoi On Mon, Jul 24, 2023 at 10:46?AM Karl Kloppenborg wrote: > Apologies I?ve been off sick. > > > > However yes, this is the way we do it as well. > > I would say this is also the most sane way to deal with this. > > > > Thanks, > Karl. > > > > *From: *Arnaud Morin > *Date: *Sunday, 23 July 2023 at 10:56 pm > *To: *Nguy?n H?u Kh?i > *Cc: *Karl Kloppenborg , OpenStack Discuss < > openstack-discuss at lists.openstack.org> > *Subject: *Re: [openstack][largescale-sig] Openstack multi region > deployment > > We have this model also with only one keystone. > We have multiple galera clusters synchronized together. > Only one cluster is used for write requests (located in one region), > others are read only / cache. > Most of the calls done to our keystone are "read" or token validation > requests, and this works fine with a read galera cluster / cache. > > I know that we also have a custom way to invalidate cache across > regions, but I dont remember the details, I can ask the team. > > Anyway, this is do-able :) > > I imagine it also depends on the usage you have, if you create a lot of > users/projects/assignments, then it may be harder to achieve. > > Cheers, > Arnaud > > On 19.07.23 - 14:03, Nguy?n H?u Kh?i wrote: > > Hello, thank you very much. > > > > But can I ask how we process if 1 region at ASIA and 2 regions in the > USA? > > > > Database latency will be our problem. > > > > Nguyen Huu Khoi > > > > > > On Tue, Jul 18, 2023 at 8:21?PM Karl Kloppenborg < > kkloppenborg at rwts.com.au> > > wrote: > > > > > Hi Nguy, > > > > > > > > > > > > We?ve deployed a large multi-region openstack deployment. > > > > > > As a rule of thumb we?ve got a ?keystone? region which is as best we > can > > > highly available and very redundant. > > > > > > > > > > > > We then have all other regions talk back to this region, we just > usually > > > call it ?keystone? or ?core? and it?s hidden from the UI from users. > > > > > > > > > > > > We then just run a large well kept Galara cluster to support it. > > > > > > > > > > > > --Karl. > > > > > > > > > > > > *From: *openstack-discuss-request at lists.openstack.org < > > > openstack-discuss-request at lists.openstack.org> > > > *Date: *Tuesday, 18 July 2023 at 9:25 pm > > > *To: *openstack-discuss at lists.openstack.org < > > > openstack-discuss at lists.openstack.org> > > > *Subject: *openstack-discuss Digest, Vol 57, Issue 55 > > > > > > Send openstack-discuss mailing list submissions to > > > openstack-discuss at lists.openstack.org > > > > > > To subscribe or unsubscribe via the World Wide Web, visit > > > > > > https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss > > > > > > or, via email, send a message with subject or body 'help' to > > > openstack-discuss-request at lists.openstack.org > > > > > > You can reach the person managing the list at > > > openstack-discuss-owner at lists.openstack.org > > > > > > When replying, please edit your Subject line so it is more specific > > > than "Re: Contents of openstack-discuss digest..." > > > > > > > > > Today's Topics: > > > > > > 1. [openstack][largescale-sig] Openstack multi region deployment > > > (Nguy?n H?u Kh?i) > > > 2. Re: [openstack][largescale-sig] Openstack multi region > > > deployment (Felix Huettner) > > > 3. Re: [openstack][largescale-sig] Openstack multi region > > > deployment (Nguy?n H?u Kh?i) > > > 4. Re: [neutron] unmanaged router resources - OVN interconnect > > > (Rodolfo Alonso Hernandez) > > > > > > > > > ---------------------------------------------------------------------- > > > > > > Message: 1 > > > Date: Tue, 18 Jul 2023 12:07:12 +0700 > > > From: Nguy?n H?u Kh?i > > > To: OpenStack Discuss > > > Subject: [openstack][largescale-sig] Openstack multi region deployment > > > Message-ID: > > > < > > > CABAODReJ6QW8A4OENEjmhFCiM-15B0qc2La_aMr1EKfaENq9iw at mail.gmail.com> > > > Content-Type: text/plain; charset="utf-8" > > > > > > Hello guys, > > > > > > I am going to deploy openstack multi regions and I know that keystone > > > replication is the most challenging. > > > > > > I plan to set up 2 regions which use centralize galera cluster(3 > nodes). > > > and one standby edge galera cluster(3 nodes) > > > > > > When region 1 is node available, I will map region 2 to use standby > edge > > > galera cluster. > > > > > > I hope you give me some experience and advice with multi regions. > > > > > > Thank you very much. > > > -------------- next part -------------- > > > An HTML attachment was scrubbed... > > > URL: < > > > > https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/c95d3675/attachment-0001.htm > > > > > > > > > > ------------------------------ > > > > > > Message: 2 > > > Date: Tue, 18 Jul 2023 09:34:35 +0200 > > > From: Felix Huettner > > > To: Nguy?n H?u Kh?i > > > Cc: OpenStack Discuss > > > Subject: Re: [openstack][largescale-sig] Openstack multi region > > > deployment > > > Message-ID: > > > Content-Type: text/plain; charset=utf-8 > > > > > > Hi, > > > > > > i think you have two options here: > > > 1. you could span a single galera cluster over all of your regions. > > > this might have some latency issues, but if your are not too write > > > heavy that might be fine. I know some companies use that setup. > > > 2. you use some kind of multiple galera clusters with replication. > > > But i have not yet heard of anybody using this setup. > > > > > > An alternative might be to have separate keystone setups with separate > > > databases. This would probably reduce the error potential, but might > not > > > fit your usecase. > > > > > > Thanks > > > Felix > > > > > > > > > On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote: > > > > Hello guys, > > > > > > > > I am going to deploy openstack multi regions and I know that keystone > > > > replication is the most challenging. > > > > > > > > I plan to set up 2 regions which use centralize galera cluster(3 > nodes). > > > > and one standby edge galera cluster(3 nodes) > > > > > > > > When region 1 is node available, I will map region 2 to use standby > edge > > > > galera cluster. > > > > > > > > I hope you give me some experience and advice with multi regions. > > > > > > > > Thank you very much. > > > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur > f?r > > > die Verwertung durch den vorgesehenen Empf?nger bestimmt. > > > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den > Absender > > > bitte unverz?glich in Kenntnis und l?schen diese E Mail. > > > > > > Hinweise zum Datenschutz finden Sie hier< > https://www.datenschutz.schwarz>. > > > > > > > > > This e-mail may contain confidential content and is intended only for > the > > > specified recipient/s. > > > If you are not the intended recipient, please inform the sender > > > immediately and delete this e-mail. > > > > > > Information on data protection can be found here< > > > https://www.datenschutz.schwarz>. > > > > > > > > > > > > ------------------------------ > > > > > > Message: 3 > > > Date: Tue, 18 Jul 2023 15:36:11 +0700 > > > From: Nguy?n H?u Kh?i > > > To: Nguy?n H?u Kh?i , OpenStack Discuss > > > > > > Subject: Re: [openstack][largescale-sig] Openstack multi region > > > deployment > > > Message-ID: > > > > > CGBW1_bRkLQJAxLZxAx8V4qvbdBmTUQBUm2SRsow at mail.gmail.com> > > > Content-Type: text/plain; charset="utf-8" > > > > > > Hi. > > > Thank you for your reply. > > > > > > The first one has a problem because each region is too soft. If a > member is > > > down, so this region is gone. > > > > > > It is so challenge with us. > > > > > > > > > Nguyen Huu Khoi > > > > > > > > > On Tue, Jul 18, 2023 at 2:34?PM Felix Huettner > > > > > > > wrote: > > > > > > > Hi, > > > > > > > > i think you have two options here: > > > > 1. you could span a single galera cluster over all of your regions. > > > > this might have some latency issues, but if your are not too write > > > > heavy that might be fine. I know some companies use that setup. > > > > 2. you use some kind of multiple galera clusters with replication. > > > > But i have not yet heard of anybody using this setup. > > > > > > > > An alternative might be to have separate keystone setups with > separate > > > > databases. This would probably reduce the error potential, but might > not > > > > fit your usecase. > > > > > > > > Thanks > > > > Felix > > > > > > > > > > > > On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote: > > > > > Hello guys, > > > > > > > > > > I am going to deploy openstack multi regions and I know that > keystone > > > > > replication is the most challenging. > > > > > > > > > > I plan to set up 2 regions which use centralize galera cluster(3 > > > nodes). > > > > > and one standby edge galera cluster(3 nodes) > > > > > > > > > > When region 1 is node available, I will map region 2 to use standby > > > edge > > > > > galera cluster. > > > > > > > > > > I hope you give me some experience and advice with multi regions. > > > > > > > > > > Thank you very much. > > > > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur > f?r > > > > die Verwertung durch den vorgesehenen Empf?nger bestimmt. > > > > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den > Absender > > > > bitte unverz?glich in Kenntnis und l?schen diese E Mail. > > > > > > > > Hinweise zum Datenschutz finden Sie hier< > https://www.datenschutz.schwarz > > > >. > > > > > > > > > > > > This e-mail may contain confidential content and is intended only > for the > > > > specified recipient/s. > > > > If you are not the intended recipient, please inform the sender > > > > immediately and delete this e-mail. > > > > > > > > Information on data protection can be found here< > > > > https://www.datenschutz.schwarz>. > > > > > > > -------------- next part -------------- > > > An HTML attachment was scrubbed... > > > URL: < > > > > https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/749440e3/attachment-0001.htm > > > > > > > > > > ------------------------------ > > > > > > Message: 4 > > > Date: Tue, 18 Jul 2023 13:23:27 +0200 > > > From: Rodolfo Alonso Hernandez > > > To: Roberto Bartzen Acosta > > > Cc: openstack-discuss , Terry > > > Wilson , Tiago Pires < > > > tiago.pires at luizalabs.com> > > > Subject: Re: [neutron] unmanaged router resources - OVN interconnect > > > Message-ID: > > > < > > > CAECr9X7U7YsGBv9ajcmeOCxfdD+YLar8QyPwYBN0qaP10CzTug at mail.gmail.com> > > > Content-Type: text/plain; charset="utf-8" > > > > > > Ok, this is being tortuous. First of all: define a strategy. If you are > > > going to create the resources in Neutron, define how. I've provided a > way > > > to do this, find a formal strategy to ground it. > > > > > > Second: (again) try to find a connection between resources, if you are > > > going to use the strategy of creating the resources in Neutron. The > > > "Logical_Router_Static_Route" belongs to a router univocally. If that > > > router has been created by OpenStack, then you can modify the DB sync > > > method to consider learned routes too. > > > > > > In order to do this, you'll need a set of resources that are going to > be > > > needed in Neutron, the OVN counterparts and other resources (like > > > "Logical_Router_Static_Route") that will be added and will be present > in > > > OVN and not in Neutron DB. Also you'll need to know how to relate all > of > > > them. > > > -------------- next part -------------- > > > An HTML attachment was scrubbed... > > > URL: < > > > > https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/90712e47/attachment.htm > > > > > > > > > > ------------------------------ > > > > > > Subject: Digest Footer > > > > > > _______________________________________________ > > > openstack-discuss mailing list > > > openstack-discuss at lists.openstack.org > > > > > > > > > ------------------------------ > > > > > > End of openstack-discuss Digest, Vol 57, Issue 55 > > > ************************************************* > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kkloppenborg at rwts.com.au Mon Jul 24 23:30:12 2023 From: kkloppenborg at rwts.com.au (Karl Kloppenborg) Date: Tue, 25 Jul 2023 09:30:12 +1000 Subject: [openstack][largescale-sig] Openstack multi region deployment In-Reply-To: References: Message-ID: Good morning Nguy?n. Latency won?t be too much of an issue because if you have read replicas in each region, you can route your read requests to the read slaves and only have the writes pushed to the master. There?s a few different options in terms of how you route, we use proxysql.com for our systems. I?d say about 90% of traffic in your keystone environment will be read traffic, 10% write. So I really would not be too concerned about your latency. Also, keystone isn?t a very heavy use database. TXN replication or WAL latency shouldn?t be too much of a problem between the continents. The fact of the matter is, the way you?re architecting this will always need to have compromises. You?re basically hitting the issues described in CAP theorem. (Read more here: https://en.m.wikipedia.org/wiki/CAP_theorem) You?ll notice that most AWS and GCP, Azure etc whenever IAM or Keystone equivalents are written to, I.e a token made, permissions updated, there?s a delay in the response, this is usually artificial and designed to induce enough time into the request so that reader slaves have received the latest WAL replications. / or db version equivalent. The only alternative to this is if you can deploy something like cockroachDB or Yugabyte. However, this will be fraught with heavy and costly complexity. Thanks, Karl. On Tue, 25 Jul 2023 at 09:17, Nguy?n H?u Kh?i wrote: > Hello Karl, > How are you? Thank you for your response.. Hope you are ok. > Nguyen Huu Khoi > > > On Mon, Jul 24, 2023 at 10:46?AM Karl Kloppenborg < > kkloppenborg at rwts.com.au> wrote: > >> Apologies I?ve been off sick. >> >> >> >> However yes, this is the way we do it as well. >> >> I would say this is also the most sane way to deal with this. >> >> >> >> Thanks, >> Karl. >> >> >> >> *From: *Arnaud Morin >> *Date: *Sunday, 23 July 2023 at 10:56 pm >> *To: *Nguy?n H?u Kh?i >> *Cc: *Karl Kloppenborg , OpenStack Discuss < >> openstack-discuss at lists.openstack.org> >> *Subject: *Re: [openstack][largescale-sig] Openstack multi region >> deployment >> >> We have this model also with only one keystone. >> We have multiple galera clusters synchronized together. >> Only one cluster is used for write requests (located in one region), >> others are read only / cache. >> Most of the calls done to our keystone are "read" or token validation >> requests, and this works fine with a read galera cluster / cache. >> >> I know that we also have a custom way to invalidate cache across >> regions, but I dont remember the details, I can ask the team. >> >> Anyway, this is do-able :) >> >> I imagine it also depends on the usage you have, if you create a lot of >> users/projects/assignments, then it may be harder to achieve. >> >> Cheers, >> Arnaud >> >> On 19.07.23 - 14:03, Nguy?n H?u Kh?i wrote: >> > Hello, thank you very much. >> > >> > But can I ask how we process if 1 region at ASIA and 2 regions in the >> USA? >> > >> > Database latency will be our problem. >> > >> > Nguyen Huu Khoi >> > >> > >> > On Tue, Jul 18, 2023 at 8:21?PM Karl Kloppenborg < >> kkloppenborg at rwts.com.au> >> > wrote: >> > >> > > Hi Nguy, >> > > >> > > >> > > >> > > We?ve deployed a large multi-region openstack deployment. >> > > >> > > As a rule of thumb we?ve got a ?keystone? region which is as best we >> can >> > > highly available and very redundant. >> > > >> > > >> > > >> > > We then have all other regions talk back to this region, we just >> usually >> > > call it ?keystone? or ?core? and it?s hidden from the UI from users. >> > > >> > > >> > > >> > > We then just run a large well kept Galara cluster to support it. >> > > >> > > >> > > >> > > --Karl. >> > > >> > > >> > > >> > > *From: *openstack-discuss-request at lists.openstack.org < >> > > openstack-discuss-request at lists.openstack.org> >> > > *Date: *Tuesday, 18 July 2023 at 9:25 pm >> > > *To: *openstack-discuss at lists.openstack.org < >> > > openstack-discuss at lists.openstack.org> >> > > *Subject: *openstack-discuss Digest, Vol 57, Issue 55 >> > > >> > > Send openstack-discuss mailing list submissions to >> > > openstack-discuss at lists.openstack.org >> > > >> > > To subscribe or unsubscribe via the World Wide Web, visit >> > > >> > > >> https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss >> > > >> > > or, via email, send a message with subject or body 'help' to >> > > openstack-discuss-request at lists.openstack.org >> > > >> > > You can reach the person managing the list at >> > > openstack-discuss-owner at lists.openstack.org >> > > >> > > When replying, please edit your Subject line so it is more specific >> > > than "Re: Contents of openstack-discuss digest..." >> > > >> > > >> > > Today's Topics: >> > > >> > > 1. [openstack][largescale-sig] Openstack multi region deployment >> > > (Nguy?n H?u Kh?i) >> > > 2. Re: [openstack][largescale-sig] Openstack multi region >> > > deployment (Felix Huettner) >> > > 3. Re: [openstack][largescale-sig] Openstack multi region >> > > deployment (Nguy?n H?u Kh?i) >> > > 4. Re: [neutron] unmanaged router resources - OVN interconnect >> > > (Rodolfo Alonso Hernandez) >> > > >> > > >> > > ---------------------------------------------------------------------- >> > > >> > > Message: 1 >> > > Date: Tue, 18 Jul 2023 12:07:12 +0700 >> > > From: Nguy?n H?u Kh?i >> > > To: OpenStack Discuss >> > > Subject: [openstack][largescale-sig] Openstack multi region deployment >> > > Message-ID: >> > > < >> > > CABAODReJ6QW8A4OENEjmhFCiM-15B0qc2La_aMr1EKfaENq9iw at mail.gmail.com> >> > > Content-Type: text/plain; charset="utf-8" >> > > >> > > Hello guys, >> > > >> > > I am going to deploy openstack multi regions and I know that keystone >> > > replication is the most challenging. >> > > >> > > I plan to set up 2 regions which use centralize galera cluster(3 >> nodes). >> > > and one standby edge galera cluster(3 nodes) >> > > >> > > When region 1 is node available, I will map region 2 to use standby >> edge >> > > galera cluster. >> > > >> > > I hope you give me some experience and advice with multi regions. >> > > >> > > Thank you very much. >> > > -------------- next part -------------- >> > > An HTML attachment was scrubbed... >> > > URL: < >> > > >> https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/c95d3675/attachment-0001.htm >> > > > >> > > >> > > ------------------------------ >> > > >> > > Message: 2 >> > > Date: Tue, 18 Jul 2023 09:34:35 +0200 >> > > From: Felix Huettner >> > > To: Nguy?n H?u Kh?i >> > > Cc: OpenStack Discuss >> > > Subject: Re: [openstack][largescale-sig] Openstack multi region >> > > deployment >> > > Message-ID: >> > > Content-Type: text/plain; charset=utf-8 >> > > >> > > Hi, >> > > >> > > i think you have two options here: >> > > 1. you could span a single galera cluster over all of your regions. >> > > this might have some latency issues, but if your are not too write >> > > heavy that might be fine. I know some companies use that setup. >> > > 2. you use some kind of multiple galera clusters with replication. >> > > But i have not yet heard of anybody using this setup. >> > > >> > > An alternative might be to have separate keystone setups with separate >> > > databases. This would probably reduce the error potential, but might >> not >> > > fit your usecase. >> > > >> > > Thanks >> > > Felix >> > > >> > > >> > > On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote: >> > > > Hello guys, >> > > > >> > > > I am going to deploy openstack multi regions and I know that >> keystone >> > > > replication is the most challenging. >> > > > >> > > > I plan to set up 2 regions which use centralize galera cluster(3 >> nodes). >> > > > and one standby edge galera cluster(3 nodes) >> > > > >> > > > When region 1 is node available, I will map region 2 to use standby >> edge >> > > > galera cluster. >> > > > >> > > > I hope you give me some experience and advice with multi regions. >> > > > >> > > > Thank you very much. >> > > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur >> f?r >> > > die Verwertung durch den vorgesehenen Empf?nger bestimmt. >> > > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den >> Absender >> > > bitte unverz?glich in Kenntnis und l?schen diese E Mail. >> > > >> > > Hinweise zum Datenschutz finden Sie hier< >> https://www.datenschutz.schwarz>. >> > > >> > > >> > > This e-mail may contain confidential content and is intended only for >> the >> > > specified recipient/s. >> > > If you are not the intended recipient, please inform the sender >> > > immediately and delete this e-mail. >> > > >> > > Information on data protection can be found here< >> > > https://www.datenschutz.schwarz>. >> > > >> > > >> > > >> > > ------------------------------ >> > > >> > > Message: 3 >> > > Date: Tue, 18 Jul 2023 15:36:11 +0700 >> > > From: Nguy?n H?u Kh?i >> > > To: Nguy?n H?u Kh?i , OpenStack Discuss >> > > >> > > Subject: Re: [openstack][largescale-sig] Openstack multi region >> > > deployment >> > > Message-ID: >> > > > > > CGBW1_bRkLQJAxLZxAx8V4qvbdBmTUQBUm2SRsow at mail.gmail.com> >> > > Content-Type: text/plain; charset="utf-8" >> > > >> > > Hi. >> > > Thank you for your reply. >> > > >> > > The first one has a problem because each region is too soft. If a >> member is >> > > down, so this region is gone. >> > > >> > > It is so challenge with us. >> > > >> > > >> > > Nguyen Huu Khoi >> > > >> > > >> > > On Tue, Jul 18, 2023 at 2:34?PM Felix Huettner >> > > > > >> > > wrote: >> > > >> > > > Hi, >> > > > >> > > > i think you have two options here: >> > > > 1. you could span a single galera cluster over all of your regions. >> > > > this might have some latency issues, but if your are not too >> write >> > > > heavy that might be fine. I know some companies use that setup. >> > > > 2. you use some kind of multiple galera clusters with replication. >> > > > But i have not yet heard of anybody using this setup. >> > > > >> > > > An alternative might be to have separate keystone setups with >> separate >> > > > databases. This would probably reduce the error potential, but >> might not >> > > > fit your usecase. >> > > > >> > > > Thanks >> > > > Felix >> > > > >> > > > >> > > > On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote: >> > > > > Hello guys, >> > > > > >> > > > > I am going to deploy openstack multi regions and I know that >> keystone >> > > > > replication is the most challenging. >> > > > > >> > > > > I plan to set up 2 regions which use centralize galera cluster(3 >> > > nodes). >> > > > > and one standby edge galera cluster(3 nodes) >> > > > > >> > > > > When region 1 is node available, I will map region 2 to use >> standby >> > > edge >> > > > > galera cluster. >> > > > > >> > > > > I hope you give me some experience and advice with multi regions. >> > > > > >> > > > > Thank you very much. >> > > > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist >> nur f?r >> > > > die Verwertung durch den vorgesehenen Empf?nger bestimmt. >> > > > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den >> Absender >> > > > bitte unverz?glich in Kenntnis und l?schen diese E Mail. >> > > > >> > > > Hinweise zum Datenschutz finden Sie hier< >> https://www.datenschutz.schwarz >> > > >. >> > > > >> > > > >> > > > This e-mail may contain confidential content and is intended only >> for the >> > > > specified recipient/s. >> > > > If you are not the intended recipient, please inform the sender >> > > > immediately and delete this e-mail. >> > > > >> > > > Information on data protection can be found here< >> > > > https://www.datenschutz.schwarz>. >> > > > >> > > -------------- next part -------------- >> > > An HTML attachment was scrubbed... >> > > URL: < >> > > >> https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/749440e3/attachment-0001.htm >> > > > >> > > >> > > ------------------------------ >> > > >> > > Message: 4 >> > > Date: Tue, 18 Jul 2023 13:23:27 +0200 >> > > From: Rodolfo Alonso Hernandez >> > > To: Roberto Bartzen Acosta >> > > Cc: openstack-discuss , Terry >> > > Wilson , Tiago Pires < >> > > tiago.pires at luizalabs.com> >> > > Subject: Re: [neutron] unmanaged router resources - OVN interconnect >> > > Message-ID: >> > > < >> > > CAECr9X7U7YsGBv9ajcmeOCxfdD+YLar8QyPwYBN0qaP10CzTug at mail.gmail.com> >> > > Content-Type: text/plain; charset="utf-8" >> > > >> > > Ok, this is being tortuous. First of all: define a strategy. If you >> are >> > > going to create the resources in Neutron, define how. I've provided a >> way >> > > to do this, find a formal strategy to ground it. >> > > >> > > Second: (again) try to find a connection between resources, if you are >> > > going to use the strategy of creating the resources in Neutron. The >> > > "Logical_Router_Static_Route" belongs to a router univocally. If that >> > > router has been created by OpenStack, then you can modify the DB sync >> > > method to consider learned routes too. >> > > >> > > In order to do this, you'll need a set of resources that are going to >> be >> > > needed in Neutron, the OVN counterparts and other resources (like >> > > "Logical_Router_Static_Route") that will be added and will be present >> in >> > > OVN and not in Neutron DB. Also you'll need to know how to relate all >> of >> > > them. >> > > -------------- next part -------------- >> > > An HTML attachment was scrubbed... >> > > URL: < >> > > >> https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/90712e47/attachment.htm >> > > > >> > > >> > > ------------------------------ >> > > >> > > Subject: Digest Footer >> > > >> > > _______________________________________________ >> > > openstack-discuss mailing list >> > > openstack-discuss at lists.openstack.org >> > > >> > > >> > > ------------------------------ >> > > >> > > End of openstack-discuss Digest, Vol 57, Issue 55 >> > > ************************************************* >> > > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nguyenhuukhoinw at gmail.com Mon Jul 24 23:55:16 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Tue, 25 Jul 2023 06:55:16 +0700 Subject: [openstack][largescale-sig] Openstack multi region deployment In-Reply-To: References: Message-ID: Thanks for your explanation., Very details. This time, I have not much knowledge, But I will keep build and sharing with people as the same way you help me. -------------- next part -------------- An HTML attachment was scrubbed... URL: From melwittt at gmail.com Mon Jul 24 23:57:29 2023 From: melwittt at gmail.com (melanie witt) Date: Mon, 24 Jul 2023 16:57:29 -0700 Subject: Nova Ussuri: DB online migration cannot migrate (some) populate_user_id In-Reply-To: References: Message-ID: <283172c5-64b2-b8e8-de0a-01f2ae4c4538@gmail.com> On 07/20/23 22:45, Michel Jouvin wrote: > Hi, > > We upgraded one of our clouds to Ussuri (we are late!) a couple of > months ago. After the upgrade 'nova-manage db online_data_migration' was > run. fill_virtual_interface_list and populate_user_id where non zero and > after running the command fill_virtual_interface_list was 0 but > populate_user_id stayed 24. Looking in nova-manage.log, I found the > follwoing message (twice every time the command is run): > > Some instance mappings were not migratable. This may be transient due to > in-flight instance builds, or could be due to stale data that will be > cleaned up after running "nova-manage db archive_deleted_rows --purge". > > I ran the suggest command but the result is that > fill_virtual_interface_list is non zero again (15) and there is no > change to populate_user_i. If I run again online_data_migration, it > clears again? fill_virtual_interface_list... and so on. > > After 2 months the situation remains the same, no increase in > populate_user_id but no possibility to fix them. We were wondering if it > could have been caused by the fact that we had not yet upgraded all the > compute nodes when we first ran the online_data_migration. We are > planning to upgrade to Victoria and we'd like to fix the situation > before as we understand that online data migration not done can be > blocking for the next upgrade. > > We don't have any clue on the reason this happened. It is a test cloud > with a very low usage. On our production cloud we have a somewhat > similar problem but populate_user_id=1 and running the purge doesn't > increase fill_virtual_interface_list. I'm not sure whether it's related but your post reminded me of this past bug: https://bugs.launchpad.net/nova/+bug/1824435 so it might be worth checking whether you have a duplicate 'security_groups' table record in your database. If so, there is a workaround described in comment #28 on that bug report. -melwitt From ozzzo at yahoo.com Tue Jul 25 04:51:20 2023 From: ozzzo at yahoo.com (Albert Braden) Date: Tue, 25 Jul 2023 04:51:20 +0000 (UTC) Subject: [keystone] LDAP failover fails In-Reply-To: <1876779442.3052584.1689880463257@mail.yahoo.com> References: <41076567.2372818.1689778503721.ref@mail.yahoo.com> <41076567.2372818.1689778503721@mail.yahoo.com> <1876779442.3052584.1689880463257@mail.yahoo.com> Message-ID: <1878443503.4796653.1690260680861@mail.yahoo.com> Does anyone on the keystone team want to comment on this? On Thursday, July 20, 2023 at 03:14:23 PM EDT, Albert Braden wrote: When you say "solve this stuff in the ldappool library" are you talking about moving the broken server to the end of the pool instead of removing it? Since the first server in the URL is always used, and failover doesn't seem to work, it seems like moving the broken URL to the end of the list would be a good solution, and that would eliminate the problem of having to add it back after it starts working again. I'm looking at the ldappool code here: https://opendev.org/openstack/ldappool/src/branch/master/ldappool/__init__.py So far it's not obvious to me how the pool is being assembled. What is the relationship between the LDAP URLs in the Keystone config, and the connections in the pool? What would have to change, to allow a failing URL to be treated as if it were not the first one in the list? On Wednesday, July 19, 2023 at 11:46:25 AM EDT, Sven Kieske wrote: Hi, I noticed that https://review.opendev.org/c/openstack/keystone/+/860118 is also linked from your bugzilla link. I wasn't aware of the work in https://review.opendev.org/c/openstack/keystone/+/821086 I'm currently trying to fix the ldap breakage in keystone. during the last keystone reviewathons it became clear that it would be better to solve this stuff in the ldappool library itself. regarding the overall project status I guess it's fair to say that ldap support ist pretty dormant right now. This is my first dive into the keystone codebase, so I guess it's save to say that additional people interested in ldap would be more than welcome. But I guess the core keystone team can say more about this. Having said all this, I guess this explains the general status of ldap related patches in keystone. HTH & kind regards Am Mittwoch, dem 19.07.2023 um 14:55 +0000 schrieb Albert Braden: > We are experiencing the LDAP failover issue described in [1]. > Redhat?s solution is to not bother fixing the bug, and to tell > customers to put the LDAP server behind a load-balancer. According to > Redhat, that is not a good solution for FreeIPA, as explained in [2] > and further elucidated in the blog post [3] that it references. I see > that the community has a bug open for this [4] and the bug is being > worked on here [5] but there has been no activity since 10/22. > > What is the status of this bugfix? Does it just need someone to > review and merge it, or is there more work to be done? How are other > FreeIPA users working around this problem? > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=2024602#c3 > [2] > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/load-balancing > [3] http://ssimo.org/blog/id_019.html > [4] https://bugs.launchpad.net/keystone/+bug/1953622 > [5] https://review.opendev.org/c/openstack/keystone/+/821086 > -- Sven Kieske Senior Cloud Engineer Mail: kieske at osism.tech Web: https://osism.tech OSISM GmbH Teckstra?e 62 / 70190 Stuttgart / Deutschland Gesch?ftsf?hrer: Christian Berendt Unternehmenssitz: Stuttgart Amtsgericht: Stuttgart, HRB 756139 -------------- next part -------------- An HTML attachment was scrubbed... URL: From fkr at osb-alliance.com Tue Jul 25 09:10:17 2023 From: fkr at osb-alliance.com (Felix Kronlage-Dammers) Date: Tue, 25 Jul 2023 11:10:17 +0200 Subject: Info Public Cloud SIG for openstack-discuss Message-ID: Hi, in the Public Cloud SIG we discussed on how to get more OpenStack Public Cloud Operators back into the Public Cloud SIG and concluded that we?ll have a lightning talk once per month as part of our Public Cloud SIG meeting. This means that once a month we will exchange the IRC-session for a videocall session (in the same timeslot). To make it easy for everyone we will paste the link at the time of the meeting onto #openstack-operators as well as it being included in the reminder mail sent to openstack-discuss@ beforehand. We will be starting with that after the current vacation period in September. We?ll be reaching out to the OpenInfra community as well as the various OpenStack based public clouds out there to get people involved :) Due to vacation the irc meeting on August 2nd (next week) will not be happening, since both Tobias and me are on vacation. felix (in the name of Tobias and myself ;) -- Felix Kronlage-Dammers Product Owner IaaS & Operations Sovereign Cloud Stack Sovereign Cloud Stack ? standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband f?r digitale Souver?nit?t e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr at osb-alliance.com From senrique at redhat.com Tue Jul 25 10:16:05 2023 From: senrique at redhat.com (Sofia Enriquez) Date: Tue, 25 Jul 2023 11:16:05 +0100 Subject: =?UTF-8?Q?=5Ball=5D=5Boutreachy=5D_=E2=98=95_Looking_for_at_least_two_volunt?= =?UTF-8?Q?eers_for_a_virtual_coffee_chat?= Message-ID: Hello, We're currently seeking two volunteers to engage in casual chats with our Outreachy interns, @Desire Barine and @toheeb.olawale.to23 at gmail.com Asking to network with someone can be intimidating. An informal chat, on the other hand, is a friendly conversation with another person about their career or role as a free software contributor. This provides valuable insight for the intern, allowing them to learn about a professional's work, career journey, and networking experiences within the free software community. After the chat, the intern can request three additional people to speak with, thus expanding their network! Here's how to conduct an informal chat: - The intern will ask you questions about your career. - The chat can take place through a half-hour gmeet session (without video) or via IRC chat. - You can share tips on maximizing networking opportunities and offer advice to those interested in exploring career prospects in open source. If you are interested in participating, please contact @Rajat Dhasmana with the date and time you are available! ? The interns journey with Openstack has been featured on the Superuser blog: - Getting an Outreachy Internship with OpenStack - What is OpenStack? A Fresh Look from a Clear Mind Thank you for your consideration! -- Sof?a Enriquez she/her Software Engineer Red Hat PnT IRC: @enriquetaso @RedHat Red Hat Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From lokendrarathour at gmail.com Tue Jul 25 13:17:07 2023 From: lokendrarathour at gmail.com (Lokendra Rathour) Date: Tue, 25 Jul 2023 18:47:07 +0530 Subject: TripleO deployment over IPV6 In-Reply-To: References: Message-ID: Hi Takashi/ Team, Any inputs with respect to the same will be of great help. Thanks, Lokendra On Tue, Jul 25, 2023 at 4:29?PM Kushagr Gupta wrote: > Hi Team, > > We were trying to perform IPV6-based deployment of TripleO. > We are following the link: > > https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html/bare_metal_provisioning/sect-deploy#sect-example-templates > > > > As per our environment, we set the "ipv6_address_mode" to > "dhcpv6-stateful" and tried undercloud deployment. It was successful. > > > > But when we try node introspection, openstack is not providing an IP that > will be used for PXE booting the system. We have enabled PXE booting on the > IPV6 network. > On Dell machines, we get the following error: > "PXE-E16: No offer received." > > For your reference, the undercloud file which we used is attached. > We also tried to find the communication happening on the br-ctlplane > interface when the machine tries to boot over IPV6 PXE boot. I am attaching > those tcpdump records as well. We are unable to find the MAC address of our > machines. > > *Questions:* > 1. Is it even possible to perform tripleO deployment with IPV6? > 2. Do we need to do some extra settings, so that our baremetal machines > can be PXE booted? > > Thanks and Regards > -- ~ Lokendra skype: lokendrarathour -------------- next part -------------- An HTML attachment was scrubbed... URL: From juliaashleykreger at gmail.com Tue Jul 25 14:40:43 2023 From: juliaashleykreger at gmail.com (Julia Kreger) Date: Tue, 25 Jul 2023 07:40:43 -0700 Subject: TripleO deployment over IPV6 In-Reply-To: References: Message-ID: Greetings, I don't think I saw your original email. I suspect because you indicated there was an attachment, it might have been filtered for moderation by the mailing list. On Tue, Jul 25, 2023 at 6:24?AM Lokendra Rathour wrote: > Hi Takashi/ Team, > Any inputs with respect to the same will be of great help. > > Thanks, > Lokendra > > > On Tue, Jul 25, 2023 at 4:29?PM Kushagr Gupta < > kushagrguptasps.mun at gmail.com> wrote: > >> Hi Team, >> >> We were trying to perform IPV6-based deployment of TripleO. >> We are following the link: >> >> https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html/bare_metal_provisioning/sect-deploy#sect-example-templates >> >> >> > Try following the more recent documents, specifically OSP16.1 or 16.2. > As per our environment, we set the "ipv6_address_mode" to >> "dhcpv6-stateful" and tried undercloud deployment. It was successful. >> >> >> > Good, that is required for IPv6 PXE. > But when we try node introspection, openstack is not providing an IP that >> will be used for PXE booting the system. We have enabled PXE booting on the >> IPV6 network. >> On Dell machines, we get the following error: >> "PXE-E16: No offer received." >> > I suspect we will need to see the configuration. Paste.opendev.org? > For your reference, the undercloud file which we used is attached. >> We also tried to find the communication happening on the br-ctlplane >> interface when the machine tries to boot over IPV6 PXE boot. I am attaching >> those tcpdump records as well. We are unable to find the MAC address of our >> machines. >> > I suspect this might be in the territory of the hardware, while being set for IPv6 network booting, maybe the configuration needs to be checked/reasserted on the interface level. It sounds as if it is using a different interface. Not seeing the expected MAC addresses is rather suspect in this case, although PXE with v6 is a little different pattern wise. Have you tried mirroring the port your expecting to see traffic cross from the physical machine to see if it is seeing the announcements and issuing the DHCPv6 packets required? Additionally is this UEFI mode or Legacy BIOS mode? > *Questions:* >> 1. Is it even possible to perform tripleO deployment with IPV6? >> 2. Do we need to do some extra settings, so that our baremetal machines >> can be PXE booted? >> >> Thanks and Regards >> > > > -- > ~ Lokendra > skype: lokendrarathour > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hanoi952022 at gmail.com Tue Jul 25 02:37:34 2023 From: hanoi952022 at gmail.com (Ha Noi) Date: Tue, 25 Jul 2023 09:37:34 +0700 Subject: [openstack][nova] Resize nova instance with PCI devices to a flavor without PCI Devices Message-ID: Hello everyone, I would like to resize an instance with PCI devices (GPU Passthrough) to a flavor without PCI Devices, but nova-api return no valid host. Response: No valid host was found. No valid host found for resize (HTTP 400) (Request-ID: req-9d6cf3f9-8f6f-482f-b0b1-95ec286a0fc3) I checked nova-scheduler log: Filtering removed all hosts for the request with instance ID 'e7b607ef-6be5-4bbc-9849-e31d622ecdfe'. Filter results: ['AggregateInstanceExtraSpecsFilter: (start: 85, end: 44)', 'RetryFilter: (start: 44, end: 44)', 'AvailabilityZoneFilter: (start: 44, end: 29)', 'ComputeFilter: (start: 29, end: 29)', 'ImagePropertiesFilter: (start: 29, end: 29)', 'ServerGroupAntiAffinityFilter: (start: 29, end: 29)', 'ServerGroupAffinityFilter: (start: 29, end: 29)', 'PciPassthroughFilter: (start: 29, end: 0)'] Actually, new flavor is not contain PCI specs, But It cannot pass PCI Filter. Thanks & Regards -------------- next part -------------- An HTML attachment was scrubbed... URL: From raghavendra-uddhav.tilay at hpe.com Tue Jul 25 04:42:44 2023 From: raghavendra-uddhav.tilay at hpe.com (Tilay, Raghavendra) Date: Tue, 25 Jul 2023 04:42:44 +0000 Subject: [cinder] Resignation from Cinder core & Bug Deputy In-Reply-To: References: Message-ID: Thank you Sofia for all the help in reviewing patches. Wish you good luck. From: Sofia Enriquez Sent: Monday, July 24, 2023 12:30 PM To: openstack-discuss Subject: [cinder] Resignation from Cinder core & Bug Deputy Hello Argonauts, I will be transitioning from my current role at Red Hat, and my availability for patch making and reviewing will be limited after July. Considering these circumstances, I believe it is appropriate for me to resign as a Cinder core reviewer and as Cinder Bug Deputy. Feel free to connect with me on LinkedIn and/or follow on Twitter / Github. ? I look forward to staying in touch and crossing paths again in the future. [cid:image001.jpg at 01D9BEE0.67C18DB0] Wishing you all the best, and thank you!? Cheers, Sofia -- Sof?a Enriquez she/her Software Engineer Red Hat PnT IRC: @enriquetaso @RedHat Red Hat Red Hat [Image removed by sender.] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ~WRD0001.jpg Type: image/jpeg Size: 823 bytes Desc: ~WRD0001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 88520 bytes Desc: image001.jpg URL: From kushagrguptasps.mun at gmail.com Tue Jul 25 10:59:14 2023 From: kushagrguptasps.mun at gmail.com (Kushagr Gupta) Date: Tue, 25 Jul 2023 16:29:14 +0530 Subject: TripleO deployment over IPV6 Message-ID: Hi Team, We were trying to perform IPV6-based deployment of TripleO. We are following the link: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html/bare_metal_provisioning/sect-deploy#sect-example-templates As per our environment, we set the "ipv6_address_mode" to "dhcpv6-stateful" and tried undercloud deployment. It was successful. But when we try node introspection, openstack is not providing an IP that will be used for PXE booting the system. We have enabled PXE booting on the IPV6 network. On Dell machines, we get the following error: "PXE-E16: No offer received." For your reference, the undercloud file which we used is attached. We also tried to find the communication happening on the br-ctlplane interface when the machine tries to boot over IPV6 PXE boot. I am attaching those tcpdump records as well. We are unable to find the MAC address of our machines. *Questions:* 1. Is it even possible to perform tripleO deployment with IPV6? 2. Do we need to do some extra settings, so that our baremetal machines can be PXE booted? Thanks and Regards -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: undercloud.conf Type: application/octet-stream Size: 963 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: tcpdum.logs Type: application/octet-stream Size: 16779 bytes Desc: not available URL: From rdhasman at redhat.com Tue Jul 25 11:52:49 2023 From: rdhasman at redhat.com (Rajat Dhasmana) Date: Tue, 25 Jul 2023 17:22:49 +0530 Subject: [cinder] Resignation from Cinder core & Bug Deputy In-Reply-To: References: Message-ID: Thank you Sofia for all the contributions you have made to the Cinder community. You will be deeply missed. If you plan to return to the community, feel free to contact me (or the PTL at that time) and you will regain the core authorities. On Tue, Jul 25, 2023 at 2:20?AM Alvaro Soto wrote: > > > =D > > On Mon, Jul 24, 2023 at 10:28?AM Sofia Enriquez > wrote: > >> Hello Argonauts, >> >> I will be transitioning from my current role at Red Hat, and my >> availability for patch making and reviewing will be limited after July. >> Considering these circumstances, I believe it is appropriate for me to >> resign as a Cinder core reviewer and as Cinder Bug Deputy. >> >> Feel free to connect with me on LinkedIn >> and/or follow on Twitter >> / Github >> . ? I look forward to staying in touch >> and crossing paths again in the future. >> >> [image: denver_ptg.jpeg] >> >> Wishing you all the best, and thank you!? >> >> Cheers, >> Sofia >> -- >> >> Sof?a Enriquez >> >> she/her >> >> Software Engineer >> >> Red Hat PnT >> >> IRC: @enriquetaso >> @RedHat Red Hat >> Red Hat >> >> >> >> > > -- > > Alvaro Soto > > *Note: My work hours may not be your work hours. Please do not feel the > need to respond during a time that is not convenient for you.* > ---------------------------------------------------------- > Great people talk about ideas, > ordinary people talk about things, > small people talk... about other people. > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: denver_ptg.jpeg Type: image/jpeg Size: 88520 bytes Desc: not available URL: From tsrini.alr at gmail.com Tue Jul 25 14:23:23 2023 From: tsrini.alr at gmail.com (Srinivasan T) Date: Tue, 25 Jul 2023 19:53:23 +0530 Subject: Cinder API version Message-ID: Hi Team, We are using openstack4j version 3.6. The OpenStack supports both Cinder v2 and v3 endpoints. # openstack endpoint list | grep cinder | 0bff444d3bcf4179bea245d9c7af319c | RegionOne | cinder | volume | True | public | http://x.x.x.x:8776/v1/%(tenant_id)s | | 14079c14970f4e21b37ba659e6054ecf | RegionOne | cinderv2 | volumev2 | True | admin | http://x.x.x.x:8776/v2/%(tenant_id)s | | 1a2d43fde400446eaca463b6b408a513 | RegionOne | cinderv3 | volumev3 | True | internal | http://10.250.9.1:8776/v3/%(project_id)s | | 57f956ddeb4e4f1c8e352ec284f6f587 | RegionOne | cinderv2 | volumev2 | True | public | http://x.x.x.x:8776/v2/%(tenant_id)s | | 693f347bb31f4a739768c6b321bccd1c | RegionOne | cinder | volume | True | internal | http://10.250.9.1:8776/v1/%(tenant_id)s | | b51e10e2948e4bd2aabb3cf95a6cc3c7 | RegionOne | cinder | volume | True | admin | http://x.x.x.x:8776/v1/%(tenant_id)s | | bc8a7eb81a944972bd87b761c9557308 | RegionOne | cinderv2 | volumev2 | True | internal | http://10.250.9.1:8776/v2/%(tenant_id)s | | ecf4a7fb8eec4f978038f7a20ae931e5 | RegionOne | cinderv3 | volumev3 | True | public | http://x.x.x.x:8776/v3/%(project_id)s | | f680338073664c9398a8d4b573d516a1 | RegionOne | cinderv3 | volumev3 | True | admin | http://x.x.x.x:8776/v3/%(project_id)s | >From openstack4j, we are using Identity API v3 for authentication and trying to do cinder operations. All cinder operations are done via Cinder API v2. Any idea on how to make openstack4j to use v3 for Cinder APIs? Is this available in any latest openstack4j? Setting the OS_VOLUME_API_VERSION=3 doesn't help. Regards, Srini -------------- next part -------------- An HTML attachment was scrubbed... URL: From katonalala at gmail.com Tue Jul 25 15:45:32 2023 From: katonalala at gmail.com (Lajos Katona) Date: Tue, 25 Jul 2023 17:45:32 +0200 Subject: [neutron] Transition networking-bagpipe & networking-bgpvpn stable/victoria to EOL Message-ID: Hi Neutrinos (and everybody), The stable/victoria branch for networking-bagpipe and networking-bgpvpn is failing with tempest since long time. We tried to fix it (see [1] & [2]), and discussed it on our regular meeting (see [3]), but as the activity on this branch for these projects is low anyway I propose to EOL stable/victoria for networking-bagpipe and networking-bgpvpn. If you have any concerns, suggestions or questions, please let me know (via mail or IRC in the #openstack-neutron channel). Lajos (lajoskatona) [1]: https://review.opendev.org/c/openstack/networking-bgpvpn/+/888719 [2]: https://review.opendev.org/c/openstack/networking-bagpipe/+/862505 [3]: https://meetings.opendev.org/meetings/neutron_ci/2023/neutron_ci.2023-07-18-15.00.log.html#l-36 -------------- next part -------------- An HTML attachment was scrubbed... URL: From nguyenhuukhoinw at gmail.com Wed Jul 26 00:49:47 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Wed, 26 Jul 2023 07:49:47 +0700 Subject: [openstack][neutron[nova][kolla-ansible]instance cannot ping after live migrate Message-ID: Hello guys. I am using openstack yoga with kolla ansible. When I migrate: instance1 from host A to host B after that I cannot ping this instance(telnet also). I must restart neutron_openvswitch_agent or move this instance back to host B then this problem has gone. this is my settings: ----------------- neutron.conf ----------------- [nova] live_migration_events = True ------------------------------------------------ ----------------- nova.conf ----------------- [DEFAULT] vif_plugging_timeout = 600 vif_plugging_is_fatal = False debug = True [compute] live_migration_wait_for_vif_plug = True [workarounds] enable_qemu_monitor_announce_self = True ----------------- openvswitch_agent.ini----------------- [securitygroup] firewall_driver = openvswitch [ovs] openflow_processed_per_port = true I check nova, neutron, ops logs but they are ok. Thank you. Nguyen Huu Khoi -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdhasman at redhat.com Wed Jul 26 09:18:24 2023 From: rdhasman at redhat.com (Rajat Dhasmana) Date: Wed, 26 Jul 2023 14:48:24 +0530 Subject: [cinder] This week's meeting will be in video+IRC Message-ID: Hello Argonauts, As we keep last meeting of the month in video + IRC mode, this week's meeting (today) will be held in video + IRC mode with details as follows: Date: 26th July, 2023 Time: 1400 UTC Meeting link: https://meet.google.com/mmb-fhtd-sjm IRC Channel: #openstack-meeting-alt Make sure you're connected to both the gmeet meeting and IRC. Thanks Rajat Dhasmana -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Wed Jul 26 09:39:53 2023 From: smooney at redhat.com (smooney at redhat.com) Date: Wed, 26 Jul 2023 10:39:53 +0100 Subject: [openstack][neutron[nova][kolla-ansible]instance cannot ping after live migrate In-Reply-To: References: Message-ID: <75e2a0d49469e378a1e6fa2d4ebdb8490e48b418.camel@redhat.com> On Wed, 2023-07-26 at 07:49 +0700, Nguy?n H?u Kh?i wrote: > Hello guys. > > I am using openstack yoga with kolla ansible. without logs of some kind i dont think anyoen will be able to hlep you with this. you have one issue with the config which i noted inline but that should not break live migration. but it would allow it to proceed when otherwise it would have failed. and it woudl allow this issue to happen instead of the vm goign to error ro the migration being aborted in pre live migrate. > > When I migrate: > > instance1 from host A to host B after that I cannot ping this > instance(telnet also). I must restart neutron_openvswitch_agent or move > this instance back to host B? then this problem has gone. > > this is my settings: > > ----------------- neutron.conf ----------------- > [nova] > live_migration_events = True > ------------------------------------------------ > > ----------------- nova.conf ----------------- > [DEFAULT] > vif_plugging_timeout = 600 > vif_plugging_is_fatal = False you should never run with this set to false in production. it will break nova ability to detect if netroking is configured when booting or migrating a vm. we honestly should have remove this when we removed nova-networks > debug = True > > [compute] > live_migration_wait_for_vif_plug = True > > [workarounds] > enable_qemu_monitor_announce_self = True > > ----------------- openvswitch_agent.ini----------------- > [securitygroup] > firewall_driver = openvswitch > [ovs] > openflow_processed_per_port = true > > I check nova, neutron, ops logs but they are ok. > > Thank you. > > > Nguyen Huu Khoi From nguyenhuukhoinw at gmail.com Wed Jul 26 10:28:42 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Wed, 26 Jul 2023 17:28:42 +0700 Subject: [openstack][neutron[nova][kolla-ansible]instance cannot ping after live migrate In-Reply-To: <75e2a0d49469e378a1e6fa2d4ebdb8490e48b418.camel@redhat.com> References: <75e2a0d49469e378a1e6fa2d4ebdb8490e48b418.camel@redhat.com> Message-ID: Because I dont see any error logs. Althought, i set debug log to on. Your advices are very helpful to me. I will try to dig deeply. I am lost so some suggests are the best way for me to continue. :) On Wed, Jul 26, 2023, 4:39 PM wrote: > On Wed, 2023-07-26 at 07:49 +0700, Nguy?n H?u Kh?i wrote: > > Hello guys. > > > > I am using openstack yoga with kolla ansible. > without logs of some kind i dont think anyoen will be able to hlep you > with this. > you have one issue with the config which i noted inline but that should > not break live migration. > but it would allow it to proceed when otherwise it would have failed. > and it woudl allow this issue to happen instead of the vm goign to error > ro the migration > being aborted in pre live migrate. > > > > When I migrate: > > > > instance1 from host A to host B after that I cannot ping this > > instance(telnet also). I must restart neutron_openvswitch_agent or move > > this instance back to host B then this problem has gone. > > > > this is my settings: > > > > ----------------- neutron.conf ----------------- > > [nova] > > live_migration_events = True > > ------------------------------------------------ > > > > ----------------- nova.conf ----------------- > > [DEFAULT] > > vif_plugging_timeout = 600 > > vif_plugging_is_fatal = False > you should never run with this set to false in production. > it will break nova ability to detect if netroking is configured > when booting or migrating a vm. > we honestly should have remove this when we removed nova-networks > > debug = True > > > > [compute] > > live_migration_wait_for_vif_plug = True > > > > [workarounds] > > enable_qemu_monitor_announce_self = True > > > > ----------------- openvswitch_agent.ini----------------- > > [securitygroup] > > firewall_driver = openvswitch > > [ovs] > > openflow_processed_per_port = true > > > > I check nova, neutron, ops logs but they are ok. > > > > Thank you. > > > > > > Nguyen Huu Khoi > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosmaita.fossdev at gmail.com Wed Jul 26 12:04:50 2023 From: rosmaita.fossdev at gmail.com (Brian Rosmaita) Date: Wed, 26 Jul 2023 08:04:50 -0400 Subject: [cinder] Resignation from Cinder core & Bug Deputy In-Reply-To: References: Message-ID: Thanks for all your hard work, Sofia! You've been a great Cinder project team member, and are going to leave some big shoes to fill, with all your work as bug deputy and as our Outreachy coordinator. Good luck in your future endeavors! On 7/24/23 3:00 AM, Sofia Enriquez wrote: > Hello Argonauts, > > I will be transitioning from my current role at Red Hat, and my > availability for patch making and reviewing will be limited after July. > Considering these circumstances, I believe it is appropriate for me to > resign as a Cinder core reviewer and as Cinder Bug Deputy. > > Feel free to connect with me on LinkedIn > and/or follow on Twitter > / Github > .?? I look forward to staying in touch > and crossing paths again in the future. > > denver_ptg.jpeg > > Wishing you all the best, and thank you!? > > Cheers, > Sofia > -- > > Sof?a Enriquez > > she/her > > Software Engineer > > Red Hat PnT > > IRC: @enriquetaso > > @RedHat Red Hat > Red Hat > > > > From rosmaita.fossdev at gmail.com Wed Jul 26 12:51:01 2023 From: rosmaita.fossdev at gmail.com (Brian Rosmaita) Date: Wed, 26 Jul 2023 08:51:01 -0400 Subject: [cinder][all][tc] Cinder to EOL all EM branches In-Reply-To: References: <18927432cab.b35b5b43911163.6905573119740994317@ghanshyammann.com> <20230705201854.mcr67unio2s3uzly@yuggoth.org> <8F41F5B6-EFA9-4EC7-9109-7A276801D566@bu.edu> Message-ID: Here's an update on how this issue is shaping up. As a reminder, the issue is that the Cinder project team has proposed to EOL and delete all current EM branches (that is, stable/train through stable/xena) due to the complexity of coordinating a fix for CVE-2023-2088 across all affected projects in the EM branches. Given the badness of CVE-2023-2088, the team did not want any unfixed branches sitting around for people to use. In the meantime, there has been a Forum session at the recent OpenInfra Summit and ongoing discussion in the TC culminating in the following proposal that I believe is coming close to agreement: "Unmaintained status replaces Extended Maintenance" https://review.opendev.org/c/openstack/governance/+/888771 Here's my understanding of how the new Unmaintained status (and its transition plan) impacts the Cinder team's proposal to EOL all the EM branches. 1. stable/train and stable/ussuri can be tagged EOL and deleted. (For cinder, at least, stein and older are already EOL.) 2. stable/victoria, stable/wallaby, and stable/xena will immediately transition from EM to Unmaintained status. This means that the Cinder project team has *no obligations* with respect to maintaining these branches or their gates. (Technically, we never did, but now it will be completely clear.) The branches will be renamed to unmaintained/victoria, unmaintained/wallaby, and unmaintained/xena to make their Unmaintained status unmistakable. The maintenance of the Unmaintained branches will fall to a cinder-unmaintained-core team, whose exact composition is still under discussion on [0], but which will be completely separate from the cinder-core and cinder-stable-maint teams. The cinder-core and cinder-stable-maint members have no obligation to participate in cinder-unmaintained-core (though they can if they want to). 3. The existence and ultimate EOL and deletion of unmaintained/{victoria,wallaby,xena} will follow the Unmaintained branch policy. A sketch of what this means over the next few cycles is mapped out in this etherpad: https://etherpad.opendev.org/p/24gf87QcmV6xF4StbLQx This email you are reading right now only concerns the way the new Unmaintained status proposal affects the Cinder project's plans with respect to the current Extended Maintenance branches. I encourage everyone to read the full proposal [0] to understand how Unmaintained status will be applied across all projects going forward. My personal opinion is that Unmaintained status and its transition plan addresses the concerns of the Cinder project team with respect to the current EM branches; the Cinder team will decide the team's position on this issue at today's weekly meeting (1400 UTC) [1]. cheers, brian [0] https://review.opendev.org/c/openstack/governance/+/888771 [1] https://etherpad.opendev.org/p/cinder-bobcat-meetings On 7/6/23 2:01 PM, Brian Rosmaita wrote: > On 7/6/23 1:05 PM, Nikolla, Kristi wrote: > [snip] >> As a courtesy to allow the TC to provide a resolution with coordinated >> guidelines for EOL-ing branches across projects, we please ask that >> you wait until the end of July at the latest. > [snip] > > This seems reasonable to me.? The Cinder project has announced its > intentions, namely, that we will not fix the gates for any of the > branches currently in EM, and that the EOL tags will (eventually) be > made at the hashes indicated in the proposed release patches: > > https://review.opendev.org/q/topic:cinder-eol-june2023 > > If anyone in the wider OpenStack Community has a desire to have > backports merged into these branches, or to keep these branches open > longer, now would be a good time to step up and do all appropriate work > to make that happen. > > cheers, > brian > From geguileo at redhat.com Wed Jul 26 13:44:18 2023 From: geguileo at redhat.com (Gorka Eguileor) Date: Wed, 26 Jul 2023 15:44:18 +0200 Subject: [cinder] cinder-backup volume stuck in creating In-Reply-To: References: <20230201120237.5p35swkzyxsg2koj@localhost> Message-ID: <20230726134418.5dt5frgr5atbt7ie@localhost> On 15/07, Nguy?n H?u Kh?i wrote: > Hello guys, > I had the same problem, there is no more error log. > > I resolved by turning off cinder-api, cinder-scheduler and cinder-backup > then removing these queues, such as cinder, cinder-scheduler and > cinder-backup.. > > I enabled debug log on rabbitmq and cinder, however, I cannot see any > related logs, > > > 2023-07-13 20:57:22.903 666 INFO cinder.backup.manager > [req-019d2459-6f43-4839-8ad6-71c805dbd708 6371ebfe0fce499983b1f07e7c34bf6d > 183e47a6dd14489991db5c3cf4132a2a - - -] Create backup started, backup: > 2cebd275-6fc3-47ae-8c8d-3337f9227057 volume: > d8446fb4-032a-44a1-b47d-f922c4a0020e. > 2023-07-13 20:57:22.919 666 INFO cinder.backup.manager > [req-019d2459-6f43-4839-8ad6-71c805dbd708 6371ebfe0fce499983b1f07e7c34bf6d > 183e47a6dd14489991db5c3cf4132a2a - - -] Call Volume Manager to > get_backup_device for > Backup(availability_zone=None,container=None,created_at=2023-07-13T13:57:22Z,data_timestamp=2023-07-13T13:57:22Z,deleted=False,deleted_at=None,display_description='',display_name='ijoo',encryption_key_id=None,fail_reason=None,host='controller01',id=2cebd275-6fc3-47ae-8c8d-3337f9227057,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='183e47a6dd14489991db5c3cf4132a2a',restore_volume_id=None,service='cinder.backup.drivers.nfs.NFSBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='6371ebfe0fce499983b1f07e7c34bf6d',volume_id=d8446fb4-032a-44a1-b47d-f922c4a0020e) > > > It is hard to reproduce this problems. > > > Nguyen Huu Khoi Hi, Next time it happens you may want to explore the status of the RabbitMQ service and queues. I remember seeing at least once a RabbitMQ service going crazy and not delivering messages in some of its queues. Cheers, Gorka. > > > On Wed, Feb 1, 2023 at 7:08?PM Gorka Eguileor wrote: > > > On 27/01, Satish Patel wrote: > > > Thank you Jon/Sofia, > > > > > > Biggest issue is even if I turn on debugging, it's not producing enough > > > logs to see what is going on. See following output. > > > > > > https://paste.opendev.org/show/bh9OF9l2OrozrNMglv2Y/ > > > > Hi, > > > > I don't see the cinder-volume logs, which are necessary to debug this > > issue, because we can see in the backup logs that it is doing an RPC > > call to the cinder-volume service "Call Volume Manager to > > get_backup_device". > > > > What is the value of the "host" field of the source volume? > > > > Because if it's anything other than "kolla-infra-1.example.com at rbd-1", > > then the problem is that the cinder-volume service for that backend is > > currently down. > > > > Cheers, > > Gorka. > > > > > > > > On Fri, Jan 27, 2023 at 10:50 AM Jon Bernard > > wrote: > > > > > > > Without the logs themselves it's really hard to say. One way to > > proceed > > > > would be to file a bug [1] and the team can work with you there. You > > > > could also enable debugging (debug = True), reproduce the failure, and > > > > upload the relevant logs there as well. > > > > > > > > [1]: https://bugs.launchpad.net/cinder/+filebug > > > > > > > > -- > > > > Jon > > > > > > > > On Thu, Jan 26, 2023 at 2:20 PM Satish Patel > > wrote: > > > > > > > >> Folks, > > > >> > > > >> I have configured nova and cinder with ceph storage. VMs running on > > ceph > > > >> storage but now when i am trying to create a backup of cinder volume > > its > > > >> getting stuck on creating and doing nothing. Logs also do not give any > > > >> indication of bad. > > > >> > > > >> My cinder.conf > > > >> > > > >> [DEFAULT] > > > >> > > > >> enabled_backends = rbd-1 > > > >> backup_driver = cinder.backup.drivers.ceph.CephBackupDriver > > > >> backup_ceph_conf = /etc/ceph/ceph.conf > > > >> backup_ceph_user = cinder-backup > > > >> backup_ceph_chunk_size = 134217728 > > > >> backup_ceph_pool = backups > > > >> backup_ceph_stripe_unit = 0 > > > >> backup_ceph_stripe_count = 0 > > > >> restore_discard_excess_bytes = true > > > >> osapi_volume_listen = 10.73.0.181 > > > >> osapi_volume_listen_port = 8776 > > > >> > > > >> > > > >> Output of "openstack volume service list" showing cinder-backup > > service > > > >> is up but when i create a backup it's getting stuck in this stage and > > no > > > >> activity. I am not seeing anything getting transferred to the ceph > > backups > > > >> pool also. Any clue? or method to debug? > > > >> > > > >> # openstack volume backup list --all > > > >> > > > >> > > +--------------------------------------+------+-------------+----------+------+ > > > >> | ID | Name | Description | Status > > | > > > >> Size | > > > >> > > > >> > > +--------------------------------------+------+-------------+----------+------+ > > > >> | bc844d55-8c5a-4bd3-b0e9-7c4c780c95ad | foo1 | | > > creating | > > > >> 20 | > > > >> > > > >> > > +--------------------------------------+------+-------------+----------+------+ > > > >> > > > > > > > > > > From senrique at redhat.com Wed Jul 26 13:54:08 2023 From: senrique at redhat.com (Sofia Enriquez) Date: Wed, 26 Jul 2023 14:54:08 +0100 Subject: Cinder Bug Report 2023-07-26 Message-ID: Hello Argonauts, Cinder Bug Meeting Etherpad Medium - [Pure Storage] Replication failover issue. - *Status*: Fix merged on master but needs backports to stable/yoga. Invalid - [tempest] Container not deleted when swift is used as a back up driver . - *Status*: Marked as invalid because cinder has never deleted the container when deleting a backup. Cheers, -- Sof?a Enriquez she/her Software Engineer Red Hat PnT IRC: @enriquetaso @RedHat Red Hat Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: From ken at jots.org Wed Jul 26 15:05:40 2023 From: ken at jots.org (Ken D'Ambrosio) Date: Wed, 26 Jul 2023 11:05:40 -0400 Subject: Version support matrix? Message-ID: <5cf270bba18c04aa725a1476d838ce6d@jots.org> Hey, all. I'm trying to figure out which OpenStack versions support which Linux distros and versions, both guest-side and controller/compute side. Does such a document exist, or is there a generally good place to look for each version of OpenStack? Thanks! From nguyenhuukhoinw at gmail.com Wed Jul 26 15:26:12 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Wed, 26 Jul 2023 22:26:12 +0700 Subject: Version support matrix? In-Reply-To: <5cf270bba18c04aa725a1476d838ce6d@jots.org> References: <5cf270bba18c04aa725a1476d838ce6d@jots.org> Message-ID: Hello. May it help. https://releases.openstack.org/xena/ On Wed, Jul 26, 2023, 10:17 PM Ken D'Ambrosio wrote: > Hey, all. I'm trying to figure out which OpenStack versions support > which Linux distros and versions, both guest-side and controller/compute > side. Does such a document exist, or is there a generally good place to > look for each version of OpenStack? > > Thanks! > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jay at gr-oss.io Wed Jul 26 15:26:49 2023 From: jay at gr-oss.io (Jay Faulkner) Date: Wed, 26 Jul 2023 08:26:49 -0700 Subject: Version support matrix? In-Reply-To: <5cf270bba18c04aa725a1476d838ce6d@jots.org> References: <5cf270bba18c04aa725a1476d838ce6d@jots.org> Message-ID: Hey Ken, We publish a document each development cycle showing what we tested on for that cycle. https://governance.openstack.org/tc/reference/project-testing-interface.html is the one for the current cycle; the older ones are linked at the bottom. Thanks, Jay Faulkner On Wed, Jul 26, 2023 at 8:19?AM Ken D'Ambrosio wrote: > Hey, all. I'm trying to figure out which OpenStack versions support > which Linux distros and versions, both guest-side and controller/compute > side. Does such a document exist, or is there a generally good place to > look for each version of OpenStack? > > Thanks! > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Wed Jul 26 16:06:18 2023 From: smooney at redhat.com (smooney at redhat.com) Date: Wed, 26 Jul 2023 17:06:18 +0100 Subject: Version support matrix? In-Reply-To: References: <5cf270bba18c04aa725a1476d838ce6d@jots.org> Message-ID: <38d2458702cf28b2e6cf457f7fe79a64427f7b17.camel@redhat.com> On Wed, 2023-07-26 at 08:26 -0700, Jay Faulkner wrote: > Hey Ken, > > We publish a document each development cycle showing what we tested on for > that cycle. > https://governance.openstack.org/tc/reference/project-testing-interface.html > is the one for the current cycle; the older ones are linked at the bottom. they pti help but the testing runtimes is proably more useful https://github.com/openstack/governance/tree/master/reference/runtimes we have a file per release with what we tested with. That does not alwasys align with what a distro ships in its package manager but its ususally a good indication. > > Thanks, > Jay Faulkner > > On Wed, Jul 26, 2023 at 8:19?AM Ken D'Ambrosio wrote: > > > Hey, all.? I'm trying to figure out which OpenStack versions support > > which Linux distros and versions, both guest-side and controller/compute > > side.? Does such a document exist, or is there a generally good place to > > look for each version of OpenStack? > > > > Thanks! > > > > From sxmatch1986 at gmail.com Tue Jul 25 22:32:47 2023 From: sxmatch1986 at gmail.com (hao wang) Date: Tue, 25 Jul 2023 15:32:47 -0700 Subject: [cinder] Resignation from Cinder core & Bug Deputy In-Reply-To: References: Message-ID: Thank you Sofia for reviewing the patches and helping a lot! Best wishes to you. On Mon, Jul 24, 2023 at 9:24?AM Sofia Enriquez wrote: > Hello Argonauts, > > I will be transitioning from my current role at Red Hat, and my > availability for patch making and reviewing will be limited after July. > Considering these circumstances, I believe it is appropriate for me to > resign as a Cinder core reviewer and as Cinder Bug Deputy. > > Feel free to connect with me on LinkedIn > and/or follow on Twitter > / Github > . ? I look forward to staying in touch > and crossing paths again in the future. > > [image: denver_ptg.jpeg] > > Wishing you all the best, and thank you!? > > Cheers, > Sofia > -- > > Sof?a Enriquez > > she/her > > Software Engineer > > Red Hat PnT > > IRC: @enriquetaso > @RedHat Red Hat > Red Hat > > > > -- Best Wishes For You! -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: denver_ptg.jpeg Type: image/jpeg Size: 88520 bytes Desc: not available URL: From amoralej at redhat.com Wed Jul 26 16:41:05 2023 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Wed, 26 Jul 2023 18:41:05 +0200 Subject: [cinder][all][tc] Cinder to EOL all EM branches In-Reply-To: References: <18927432cab.b35b5b43911163.6905573119740994317@ghanshyammann.com> <20230705201854.mcr67unio2s3uzly@yuggoth.org> <8F41F5B6-EFA9-4EC7-9109-7A276801D566@bu.edu> Message-ID: On Wed, Jul 26, 2023 at 2:56?PM Brian Rosmaita wrote: > Here's an update on how this issue is shaping up. > > As a reminder, the issue is that the Cinder project team has proposed to > EOL and delete all current EM branches (that is, stable/train through > stable/xena) due to the complexity of coordinating a fix for > CVE-2023-2088 across all affected projects in the EM branches. Given > the badness of CVE-2023-2088, the team did not want any unfixed branches > sitting around for people to use. > > In the meantime, there has been a Forum session at the recent OpenInfra > Summit and ongoing discussion in the TC culminating in the following > proposal that I believe is coming close to agreement: > > "Unmaintained status replaces Extended Maintenance" > https://review.opendev.org/c/openstack/governance/+/888771 > > For clarity, will this new model affect all the OpenStack projects or is it an alternative to the current EM model based on each project election? Best regards, Alfredo > Here's my understanding of how the new Unmaintained status (and its > transition plan) impacts the Cinder team's proposal to EOL all the EM > branches. > > 1. stable/train and stable/ussuri can be tagged EOL and deleted. (For > cinder, at least, stein and older are already EOL.) > > 2. stable/victoria, stable/wallaby, and stable/xena will immediately > transition from EM to Unmaintained status. This means that the Cinder > project team has *no obligations* with respect to maintaining these > branches or their gates. (Technically, we never did, but now it will be > completely clear.) The branches will be renamed to > unmaintained/victoria, unmaintained/wallaby, and unmaintained/xena to > make their Unmaintained status unmistakable. > > The maintenance of the Unmaintained branches will fall to a > cinder-unmaintained-core team, whose exact composition is still under > discussion on [0], but which will be completely separate from the > cinder-core and cinder-stable-maint teams. The cinder-core and > cinder-stable-maint members have no obligation to participate in > cinder-unmaintained-core (though they can if they want to). > > 3. The existence and ultimate EOL and deletion of > unmaintained/{victoria,wallaby,xena} will follow the Unmaintained branch > policy. A sketch of what this means over the next few cycles is mapped > out in this etherpad: > https://etherpad.opendev.org/p/24gf87QcmV6xF4StbLQx > > This email you are reading right now only concerns the way the new > Unmaintained status proposal affects the Cinder project's plans with > respect to the current Extended Maintenance branches. I encourage > everyone to read the full proposal [0] to understand how Unmaintained > status will be applied across all projects going forward. > > My personal opinion is that Unmaintained status and its transition plan > addresses the concerns of the Cinder project team with respect to the > current EM branches; the Cinder team will decide the team's position on > this issue at today's weekly meeting (1400 UTC) [1]. > > > cheers, > brian > > > [0] https://review.opendev.org/c/openstack/governance/+/888771 > [1] https://etherpad.opendev.org/p/cinder-bobcat-meetings > > > On 7/6/23 2:01 PM, Brian Rosmaita wrote: > > On 7/6/23 1:05 PM, Nikolla, Kristi wrote: > > [snip] > >> As a courtesy to allow the TC to provide a resolution with coordinated > >> guidelines for EOL-ing branches across projects, we please ask that > >> you wait until the end of July at the latest. > > [snip] > > > > This seems reasonable to me. The Cinder project has announced its > > intentions, namely, that we will not fix the gates for any of the > > branches currently in EM, and that the EOL tags will (eventually) be > > made at the hashes indicated in the proposed release patches: > > > > https://review.opendev.org/q/topic:cinder-eol-june2023 > > > > If anyone in the wider OpenStack Community has a desire to have > > backports merged into these branches, or to keep these branches open > > longer, now would be a good time to step up and do all appropriate work > > to make that happen. > > > > cheers, > > brian > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Wed Jul 26 16:47:37 2023 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 26 Jul 2023 16:47:37 +0000 Subject: Version support matrix? In-Reply-To: <5cf270bba18c04aa725a1476d838ce6d@jots.org> References: <5cf270bba18c04aa725a1476d838ce6d@jots.org> Message-ID: <20230726164737.jt33poakabfyhpbb@yuggoth.org> On 2023-07-26 11:05:40 -0400 (-0400), Ken D'Ambrosio wrote: > Hey, all. I'm trying to figure out which OpenStack versions > support which Linux distros and versions, both guest-side and > controller/compute side. Does such a document exist, or is there a > generally good place to look for each version of OpenStack? For a bit of clarity, OpenStack avoids using the term "support" where possible, and tries to define what that word actually means in some contexts in order to correct expectations: https://governance.openstack.org/tc/resolutions/20170620-volunteer-support.html It sounds like you're more interested in what's tested, and for that others have already correctly pointed you to the per-release PTI documents which reflect the versions of GNU/Linux distributions and versions of Python, Node.js and Golang the software is expected to run on. There is no equivalent for "guest-side" versions of things, because OpenStack is largely agnostic to those sorts of payloads, and it comes down to what will or won't work with the local versions of things like the kernel on your compute nodes, the version of qemu you've got installed, libvirt versions, and so on. In general though, it should be safe to assume pretty much any GNU/Linux distributions can successfully run as guests as long as you have viable images for them and sufficient configuration. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From fungi at yuggoth.org Wed Jul 26 17:11:07 2023 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 26 Jul 2023 17:11:07 +0000 Subject: [cinder][all][tc] Cinder to EOL all EM branches In-Reply-To: References: <18927432cab.b35b5b43911163.6905573119740994317@ghanshyammann.com> <20230705201854.mcr67unio2s3uzly@yuggoth.org> <8F41F5B6-EFA9-4EC7-9109-7A276801D566@bu.edu> Message-ID: <20230726171106.5wqp4tmfpiz2vkvf@yuggoth.org> On 2023-07-26 18:41:05 +0200 (+0200), Alfredo Moralejo Alonso wrote: [...] > For clarity, will this new model affect all the OpenStack projects > or is it an alternative to the current EM model based on each > project election? [...] All OpenStack projects. The proposed idea is to get rid of the "Extended Maintenance" phase for stable/.* branches completely, so once normal maintenance ends for a branch it is renamed to unmaintained/.* to more clearly communicate the lack of official maintenance by its corresponding developer/reviewer team. Make sure you read the current text of the proposed resolution at https://review.opendev.org/888771 though. It does say "The phase of Extended Maintenance for a branch is renamed to Unmaintained." and makes no mention of introducing per-project schedules for the end of the "maintained" phase. Since end of "maintained" is project-wide per the chart at https://releases.openstack.org/ it stands to reason that projects will continue to coordinate the end date of that phase. For example, 2023.1 a.k.a. "Antelope" is expected to end its maintained phase around 2024-09-22, so if the current proposal were to be approved by the TC, that means that around 2024-09-22 the stable/2023.1 branches of all branched projects would be deleted and new unmaintained/2023.1 branches created from their prior state. Anyone pulling from the old stable/2023.1 branches would get an error from Git that the remote no longer exists, and they would need to start checking out unmaintained/2023.1 instead if they intended to continue using that source past the end of the maintained phase. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From fungi at yuggoth.org Wed Jul 26 17:18:29 2023 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 26 Jul 2023 17:18:29 +0000 Subject: [cinder][all][tc] Cinder to EOL all EM branches In-Reply-To: <20230726171106.5wqp4tmfpiz2vkvf@yuggoth.org> References: <18927432cab.b35b5b43911163.6905573119740994317@ghanshyammann.com> <20230705201854.mcr67unio2s3uzly@yuggoth.org> <8F41F5B6-EFA9-4EC7-9109-7A276801D566@bu.edu> <20230726171106.5wqp4tmfpiz2vkvf@yuggoth.org> Message-ID: <20230726171829.ipszzjuxci7j65og@yuggoth.org> On 2023-07-26 17:11:07 +0000 (+0000), Jeremy Stanley wrote: [...] > once normal maintenance ends for a branch it is renamed to > unmaintained/.* [...] Just to correct my own phrasing here, it's technically not "renaming" the branch since Git doesn't have the concept of remote branch rename tracking. As I state correctly later in my reply, it's actually deletion of the stable branch and creation of a new unmaintained branch with the same history. Git clients will see the old branch cease to exist on the remote, and the appearance of a new branch which they can choose to checkout/track/pull if they wish. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From gmann at ghanshyammann.com Wed Jul 26 17:29:37 2023 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Wed, 26 Jul 2023 10:29:37 -0700 Subject: [cinder][all][tc] Cinder to EOL all EM branches In-Reply-To: <20230726171106.5wqp4tmfpiz2vkvf@yuggoth.org> References: <18927432cab.b35b5b43911163.6905573119740994317@ghanshyammann.com> <20230705201854.mcr67unio2s3uzly@yuggoth.org> <8F41F5B6-EFA9-4EC7-9109-7A276801D566@bu.edu> <20230726171106.5wqp4tmfpiz2vkvf@yuggoth.org> Message-ID: <189933fc7cf.115af393f814968.6783883560861496433@ghanshyammann.com> ---- On Wed, 26 Jul 2023 10:11:07 -0700 Jeremy Stanley wrote --- > On 2023-07-26 18:41:05 +0200 (+0200), Alfredo Moralejo Alonso wrote: > [...] > > For clarity, will this new model affect all the OpenStack projects > > or is it an alternative to the current EM model based on each > > project election? > [...] > > All OpenStack projects. The proposed idea is to get rid of the > "Extended Maintenance" phase for stable/.* branches completely, so > once normal maintenance ends for a branch it is renamed to > unmaintained/.* to more clearly communicate the lack of official > maintenance by its corresponding developer/reviewer team. Make sure > you read the current text of the proposed resolution at > https://review.opendev.org/888771 though. It does say "The phase of > Extended Maintenance for a branch is renamed to Unmaintained." and > makes no mention of introducing per-project schedules for the end of > the "maintained" phase. > > Since end of "maintained" is project-wide per the chart at > https://releases.openstack.org/ it stands to reason that projects > will continue to coordinate the end date of that phase. For example, > 2023.1 a.k.a. "Antelope" is expected to end its maintained phase > around 2024-09-22, so if the current proposal were to be approved by > the TC, that means that around 2024-09-22 the stable/2023.1 branches > of all branched projects would be deleted and new > unmaintained/2023.1 branches created from their prior state. Anyone > pulling from the old stable/2023.1 branches would get an error from > Git that the remote no longer exists, and they would need to start > checking out unmaintained/2023.1 instead if they intended to > continue using that source past the end of the maintained phase. As Jeremy mentioned, it will apply to all the projects and replace the existing "Extended Maintenance" model. Once we merge the TC resolution, we will also update the project-team-guide document for details. Along with the "stable/2023.1 -> unmaintained/2023.1" transition (which is our SLURP release model), TC resolution proposes the plan for the existing EM branches also (currently, those are stable/rocky till stable/xena ). Considering they are 6-month upgradable releases only (pre-SLURP model releases), the current proposal for existing EM branches is to automatically move only the three latest EM branches to unmaintained, which will be: 1. stable/xena -> unmaintained/xena 2. stable/wallaby -> unmaintained/wallaby 3. stable/victoria -> unmaintained/victoria If the project team finds that there are no maintainers or interest in those three branches, it is possible to move them to EOL. This is the in-progress proposal, feel free to comment on Gerrit if you have any feedback or improvement points in this. -gmann > -- > Jeremy Stanley > From gmann at ghanshyammann.com Wed Jul 26 17:41:28 2023 From: gmann at ghanshyammann.com (Ghanshyam Mann) Date: Wed, 26 Jul 2023 10:41:28 -0700 Subject: [cinder][all][tc] Cinder to EOL all EM branches In-Reply-To: <189933fc7cf.115af393f814968.6783883560861496433@ghanshyammann.com> References: <18927432cab.b35b5b43911163.6905573119740994317@ghanshyammann.com> <20230705201854.mcr67unio2s3uzly@yuggoth.org> <8F41F5B6-EFA9-4EC7-9109-7A276801D566@bu.edu> <20230726171106.5wqp4tmfpiz2vkvf@yuggoth.org> <189933fc7cf.115af393f814968.6783883560861496433@ghanshyammann.com> Message-ID: <189934a9e00.e171748f815392.2885143495441213145@ghanshyammann.com> ---- On Wed, 26 Jul 2023 10:29:37 -0700 Ghanshyam Mann wrote --- > ---- On Wed, 26 Jul 2023 10:11:07 -0700 Jeremy Stanley wrote --- > > On 2023-07-26 18:41:05 +0200 (+0200), Alfredo Moralejo Alonso wrote: > > [...] > > > For clarity, will this new model affect all the OpenStack projects > > > or is it an alternative to the current EM model based on each > > > project election? > > [...] > > > > All OpenStack projects. The proposed idea is to get rid of the > > "Extended Maintenance" phase for stable/.* branches completely, so > > once normal maintenance ends for a branch it is renamed to > > unmaintained/.* to more clearly communicate the lack of official > > maintenance by its corresponding developer/reviewer team. Make sure > > you read the current text of the proposed resolution at > > https://review.opendev.org/888771 though. It does say "The phase of > > Extended Maintenance for a branch is renamed to Unmaintained." and > > makes no mention of introducing per-project schedules for the end of > > the "maintained" phase. > > > > Since end of "maintained" is project-wide per the chart at > > https://releases.openstack.org/ it stands to reason that projects > > will continue to coordinate the end date of that phase. For example, > > 2023.1 a.k.a. "Antelope" is expected to end its maintained phase > > around 2024-09-22, so if the current proposal were to be approved by > > the TC, that means that around 2024-09-22 the stable/2023.1 branches > > of all branched projects would be deleted and new > > unmaintained/2023.1 branches created from their prior state. Anyone > > pulling from the old stable/2023.1 branches would get an error from > > Git that the remote no longer exists, and they would need to start > > checking out unmaintained/2023.1 instead if they intended to > > continue using that source past the end of the maintained phase. > > As Jeremy mentioned, it will apply to all the projects and replace the existing > "Extended Maintenance" model. Once we merge the TC resolution, we will > also update the project-team-guide document for details. > > Along with the "stable/2023.1 -> unmaintained/2023.1" transition (which is > our SLURP release model), TC resolution proposes the plan for the existing EM > branches also (currently, those are stable/rocky till stable/xena ). Considering they > are 6-month upgradable releases only (pre-SLURP model releases), the current proposal > for existing EM branches is to automatically move only the three latest EM branches to > unmaintained, which will be: > > 1. stable/xena -> unmaintained/xena > 2. stable/wallaby -> unmaintained/wallaby > 3. stable/victoria -> unmaintained/victoria > > If the project team finds that there are no maintainers or interest in those three branches, it is > possible to move them to EOL. To clarify, after moving them to 'unmaintained' if there is no maintainers/interest, then the project team can propose moving them to EOL (but let's see how it goes and we should give enough time to communicate it to operators/users). -gmann > > This is the in-progress proposal, feel free to comment on Gerrit if you have any feedback or improvement > points in this. > > -gmann > > > -- > > Jeremy Stanley > > > > From amoralej at redhat.com Wed Jul 26 18:08:27 2023 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Wed, 26 Jul 2023 20:08:27 +0200 Subject: [cinder][all][tc] Cinder to EOL all EM branches In-Reply-To: <189934a9e00.e171748f815392.2885143495441213145@ghanshyammann.com> References: <18927432cab.b35b5b43911163.6905573119740994317@ghanshyammann.com> <20230705201854.mcr67unio2s3uzly@yuggoth.org> <8F41F5B6-EFA9-4EC7-9109-7A276801D566@bu.edu> <20230726171106.5wqp4tmfpiz2vkvf@yuggoth.org> <189933fc7cf.115af393f814968.6783883560861496433@ghanshyammann.com> <189934a9e00.e171748f815392.2885143495441213145@ghanshyammann.com> Message-ID: On Wed, Jul 26, 2023 at 7:46?PM Ghanshyam Mann wrote: > ---- On Wed, 26 Jul 2023 10:29:37 -0700 Ghanshyam Mann wrote --- > > ---- On Wed, 26 Jul 2023 10:11:07 -0700 Jeremy Stanley wrote --- > > > On 2023-07-26 18:41:05 +0200 (+0200), Alfredo Moralejo Alonso wrote: > > > [...] > > > > For clarity, will this new model affect all the OpenStack projects > > > > or is it an alternative to the current EM model based on each > > > > project election? > > > [...] > > > > > > All OpenStack projects. The proposed idea is to get rid of the > > > "Extended Maintenance" phase for stable/.* branches completely, so > > > once normal maintenance ends for a branch it is renamed to > > > unmaintained/.* to more clearly communicate the lack of official > > > maintenance by its corresponding developer/reviewer team. Make sure > > > you read the current text of the proposed resolution at > > > https://review.opendev.org/888771 though. It does say "The phase of > > > Extended Maintenance for a branch is renamed to Unmaintained." and > > > makes no mention of introducing per-project schedules for the end of > > > the "maintained" phase. > > > > > > Since end of "maintained" is project-wide per the chart at > > > https://releases.openstack.org/ it stands to reason that projects > > > will continue to coordinate the end date of that phase. For example, > > > 2023.1 a.k.a. "Antelope" is expected to end its maintained phase > > > around 2024-09-22, so if the current proposal were to be approved by > > > the TC, that means that around 2024-09-22 the stable/2023.1 branches > > > of all branched projects would be deleted and new > > > unmaintained/2023.1 branches created from their prior state. Anyone > > > pulling from the old stable/2023.1 branches would get an error from > > > Git that the remote no longer exists, and they would need to start > > > checking out unmaintained/2023.1 instead if they intended to > > > continue using that source past the end of the maintained phase. > > > > As Jeremy mentioned, it will apply to all the projects and replace the > existing > > "Extended Maintenance" model. Once we merge the TC resolution, we will > > also update the project-team-guide document for details. > > > > Along with the "stable/2023.1 -> unmaintained/2023.1" transition (which > is > > our SLURP release model), TC resolution proposes the plan for the > existing EM > > branches also (currently, those are stable/rocky till stable/xena ). > Considering they > > are 6-month upgradable releases only (pre-SLURP model releases), the > current proposal > > for existing EM branches is to automatically move only the three latest > EM branches to > > unmaintained, which will be: > > > > 1. stable/xena -> unmaintained/xena > > 2. stable/wallaby -> unmaintained/wallaby > > 3. stable/victoria -> unmaintained/victoria > > > > If the project team finds that there are no maintainers or interest in > those three branches, it is > > possible to move them to EOL. > > To clarify, after moving them to 'unmaintained' if there is no > maintainers/interest, then the project > team can propose moving them to EOL (but let's see how it goes and we > should give enough time to > communicate it to operators/users). > > -gmann > > Thanks both for the explanations! Alfredo > > > > This is the in-progress proposal, feel free to comment on Gerrit if you > have any feedback or improvement > > points in this. > > > > -gmann > > > > > -- > > > Jeremy Stanley > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Wed Jul 26 22:05:11 2023 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 26 Jul 2023 22:05:11 +0000 Subject: [tc] Reminder: 2023-08-09 OpenInfra Board Sync Message-ID: <20230726220510.nwe7csbv2krplgmc@yuggoth.org> The Open Infrastructure Foundation Board of Directors is endeavoring to engage in regular check-ins with official OpenInfra projects. The goal is for a loosely structured discussion one-hour in length, involving members of the board and the OpenStack TC, along with other interested community members. This is not intended to be a formal presentation, and no materials need to be prepared in advance. I've started an Etherpad where participants can brainstorm potential topics of conversation, time-permitting: https://etherpad.opendev.org/p/2023-08-board-openstack-sync As previously announced[*], we're planning to hold the next call at 20:00 UTC on Wednesday, August 9. A link to the conference bridge has been added to the pad and is also included in the schedule file attached to this message. [*] https://lists.openstack.org/pipermail/openstack-discuss/2023-May/033793.html -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: 2023-08-board-openstack-sync.ics Type: text/calendar Size: 845 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From fungi at yuggoth.org Wed Jul 26 22:21:49 2023 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 26 Jul 2023 22:21:49 +0000 Subject: [tc] Reminder: 2023-08-09 OpenInfra Board Sync In-Reply-To: <20230726220510.nwe7csbv2krplgmc@yuggoth.org> References: <20230726220510.nwe7csbv2krplgmc@yuggoth.org> Message-ID: <20230726222148.62gdpkp4dy2cbtif@yuggoth.org> On 2023-07-26 22:05:11 +0000 (+0000), Jeremy Stanley wrote: [...] > As previously announced[*], we're planning to hold the next call at > 20:00 UTC on Wednesday, August 9. [...] Oops! That's what I get for cutting and pasting earlier reminders. The scheduled time is 18:00 UTC, which is accurately reflected in the ICS file and on the agenda pad, just ignore what I said in the previous message. Sorry! -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From nguyenhuukhoinw at gmail.com Thu Jul 27 01:11:36 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Thu, 27 Jul 2023 08:11:36 +0700 Subject: [openstack][neutron[nova][kolla-ansible]instance cannot ping after live migrate In-Reply-To: References: <75e2a0d49469e378a1e6fa2d4ebdb8490e48b418.camel@redhat.com> Message-ID: Hello. When my instances was migrated to other computes. I check on dest host and I see that -A neutron-openvswi-i41ec1d15-e -d x.x.x.x(my instance ip)/32 -p udp -m udp --sport 67 --dport 68 -j RETURN missing and my instance cannot get IP. I must restart neutron_openvswitch_agent then this rule appears and I can touch the instance via network. I use openswitch and provider networks. This problem has happened this week. after the system was upgraded from xena to yoga and I enabled quorum queue. Nguyen Huu Khoi On Wed, Jul 26, 2023 at 5:28?PM Nguy?n H?u Kh?i wrote: > Because I dont see any error logs. Althought, i set debug log to on. > > Your advices are very helpful to me. I will try to dig deeply. I am lost > so some suggests are the best way for me to continue. :) > > On Wed, Jul 26, 2023, 4:39 PM wrote: > >> On Wed, 2023-07-26 at 07:49 +0700, Nguy?n H?u Kh?i wrote: >> > Hello guys. >> > >> > I am using openstack yoga with kolla ansible. >> without logs of some kind i dont think anyoen will be able to hlep you >> with this. >> you have one issue with the config which i noted inline but that should >> not break live migration. >> but it would allow it to proceed when otherwise it would have failed. >> and it woudl allow this issue to happen instead of the vm goign to error >> ro the migration >> being aborted in pre live migrate. >> > >> > When I migrate: >> > >> > instance1 from host A to host B after that I cannot ping this >> > instance(telnet also). I must restart neutron_openvswitch_agent or move >> > this instance back to host B then this problem has gone. >> > >> > this is my settings: >> > >> > ----------------- neutron.conf ----------------- >> > [nova] >> > live_migration_events = True >> > ------------------------------------------------ >> > >> > ----------------- nova.conf ----------------- >> > [DEFAULT] >> > vif_plugging_timeout = 600 >> > vif_plugging_is_fatal = False >> you should never run with this set to false in production. >> it will break nova ability to detect if netroking is configured >> when booting or migrating a vm. >> we honestly should have remove this when we removed nova-networks >> > debug = True >> > >> > [compute] >> > live_migration_wait_for_vif_plug = True >> > >> > [workarounds] >> > enable_qemu_monitor_announce_self = True >> > >> > ----------------- openvswitch_agent.ini----------------- >> > [securitygroup] >> > firewall_driver = openvswitch >> > [ovs] >> > openflow_processed_per_port = true >> > >> > I check nova, neutron, ops logs but they are ok. >> > >> > Thank you. >> > >> > >> > Nguyen Huu Khoi >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From pdeore at redhat.com Thu Jul 27 06:11:08 2023 From: pdeore at redhat.com (Pranali Deore) Date: Thu, 27 Jul 2023 11:41:08 +0530 Subject: [Glance] Weekly Meeting Cancelled Message-ID: Hi All, Canceling today's glance weekly meeting, I'm on sick leave today. Sorry for inconvenience! Thanks, Pranali D. -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Thu Jul 27 06:37:41 2023 From: skaplons at redhat.com (Slawek Kaplonski) Date: Thu, 27 Jul 2023 08:37:41 +0200 Subject: [tc] Reminder: 2023-08-09 OpenInfra Board Sync In-Reply-To: <20230726222148.62gdpkp4dy2cbtif@yuggoth.org> References: <20230726220510.nwe7csbv2krplgmc@yuggoth.org> <20230726222148.62gdpkp4dy2cbtif@yuggoth.org> Message-ID: <4336045.MCa49m1NnO@p1> Hi, Dnia czwartek, 27 lipca 2023 00:21:49 CEST Jeremy Stanley pisze: > On 2023-07-26 22:05:11 +0000 (+0000), Jeremy Stanley wrote: > [...] > > As previously announced[*], we're planning to hold the next call at > > 20:00 UTC on Wednesday, August 9. > [...] > > Oops! That's what I get for cutting and pasting earlier reminders. > The scheduled time is 18:00 UTC, which is accurately reflected in > the ICS file and on the agenda pad, just ignore what I said in the > previous message. Sorry! > -- > Jeremy Stanley > Thx. I just imported ICS file to my calendar and I was actually surprised with the time when it was added. I was even thinging for a second that it is some bug in my calendar application as it was added at 20:00 by my timezone (CEST) :) -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From alex.kavanagh at canonical.com Thu Jul 27 10:07:30 2023 From: alex.kavanagh at canonical.com (Alex Kavanagh) Date: Thu, 27 Jul 2023 11:07:30 +0100 Subject: Version support matrix? In-Reply-To: <5cf270bba18c04aa725a1476d838ce6d@jots.org> References: <5cf270bba18c04aa725a1476d838ce6d@jots.org> Message-ID: Just to add to what's already been said, from the perspective of Ubuntu and OpenStack, we have the following info. * https://ubuntu.com/about/release-cycle#openstack-release-cycle - maps OpenStack releases to Ubuntu releases and the cloud archive (LTS + OpenStack release). Note this is for packages as well, regardless of whether Charmed OpenStack is used. * https://openstack-ci-reports.ubuntu.com/reports/cloud-archive/index.html - the cloud archive; shows every version against every release pocket for distros and cloud archives. To clarify, we do a release of OpenStack packages each cycle and either carry them in the distro release (e.g. 22.04 (jammy) which carries Yoga). We also have the UCA (Ubuntu Cloud Archive) which carries the OpenStack releases that don't coincide with the Ubuntu LTS release, which for example supports 2023.1 on Ubuntu 22.04. https://docs.openstack.org/install-guide/environment-packages-ubuntu.html explains how it works. Cheers Alex. On Wed, 26 Jul 2023 at 16:13, Ken D'Ambrosio wrote: > Hey, all. I'm trying to figure out which OpenStack versions support > which Linux distros and versions, both guest-side and controller/compute > side. Does such a document exist, or is there a generally good place to > look for each version of OpenStack? > > Thanks! > > -- Alex Kavanagh OpenStack Engineering - Canonical Ltd -------------- next part -------------- An HTML attachment was scrubbed... URL: From fungi at yuggoth.org Thu Jul 27 12:06:57 2023 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 27 Jul 2023 12:06:57 +0000 Subject: Version support matrix? In-Reply-To: References: <5cf270bba18c04aa725a1476d838ce6d@jots.org> Message-ID: <20230727120657.7g7czk53jgumldgb@yuggoth.org> On 2023-07-27 11:07:30 +0100 (+0100), Alex Kavanagh wrote: > Just to add to what's already been said, from the perspective of > Ubuntu and OpenStack, we have the following info. [...] Yes, if taking downstream support into account, things do indeed get more confusing. This is mainly because upstream we test the version of OpenStack we're developing on the latest versions of distributions which are already available *at the time we start developing* (so basically at the time of our previous release). Conversely, a lot of downstream distributors test and package the latest available version of OpenStack on the version of their distribution they're developing (often whatever version is available at the time they freeze their packages in preparation, which can be months before they actually release their distribution). As a result, upstream testing tends to sometimes be on older distribution versions, while downstream packaging tends to be on newer distributions than what was tested upstream. The details will, of course, vary by distributor though. -- Jeremy Stanley -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: From elod.illes at est.tech Thu Jul 27 12:59:49 2023 From: elod.illes at est.tech (=?utf-8?B?RWzDtWQgSWxsw6lz?=) Date: Thu, 27 Jul 2023 12:59:49 +0000 Subject: Proposed 2024.1 "Caracal" release schedule Message-ID: Hi, Here is the proposed schedule for the 2024.1 "Caracal" release: https://review.opendev.org/c/openstack/releases/+/888942 For easier review, you can access the HTML rendering at: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_627/888942/2/check/openstack-tox-docs/627cdc4/docs/caracal/schedule.html It's a 26-week cycle with a final release date on April 3, 2024. It has a longer milestone-2 period to accomodate Thanksgiving and end-of-year holidays. Please review & comment! Thanks, El?d Ill?s irc: elodilles @ #openstack-release -------------- next part -------------- An HTML attachment was scrubbed... URL: From murilo at evocorp.com.br Thu Jul 27 13:12:48 2023 From: murilo at evocorp.com.br (Murilo Morais) Date: Thu, 27 Jul 2023 10:12:48 -0300 Subject: [OSA][NEUTRON] DHCP port IP In-Reply-To: References: Message-ID: Dmitiry, thank you very much! config-drive worked like a charm! Em qui., 20 de jul. de 2023 ?s 23:08, Dmitriy Rabotyagov < noonedeadpunk at gmail.com> escreveu: > If the thing that concerns you is that DHCP consumes gateway IP, you > should define your gateway when creating the subnet with --gateway option > [1]. > > Also I *think* that if you define --allocation-pool while creating subnet, > DHCP server will obtain IP within this pool, but I'm not 100% sure now, as > I had such setup years ago. > > If you are disabling DHCP, then ensure your VMs will use config-drives for > metadata instead of fetching it through the network. For that you will need > to supply an option while creating the VM > > [1] > https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/subnet.html#cmdoption-openstack-subnet-create-gateway > > On Fri, Jul 21, 2023, 03:24 Ihar Hrachyshka wrote: > >> On Thu, Jul 20, 2023 at 7:18?PM Murilo Morais >> wrote: >> >>> Good evening everyone! >>> >>> I'm using OVS + VLAN Provider >>> >>> Guys, is there any way to remove IP from DHCP (network:dhcp)? >>> >>> >> You can create a subnet with enable_dhcp=False, then no DHCP port is >> allocated. Then let Neutron allocate addresses from the subnet range for >> your VMs (just make sure that the addresses are not reused outside of >> Neutron). >> >> My gateway is external, I need the VMs to obtain the IPs directly on the >>> interface without using SNAT/DNAT, but when enabling DHCP on the subnet, >>> the DHCP interface consumes an IP (the first one available). >>> >>> Is there a way to remove this blessed IP? If not, what other path can I >>> take? >>> >>> I'm thinking of other alternatives: >>> 1. Use static IP >>> 2. Use a DHCP Server on the Gateway (which makes me have to perform some >>> manual operations to assign the correct IP) >>> >>> What would be more viable? >>> >>> Thanks in advance! >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From altidorjb at gmail.com Wed Jul 26 20:12:41 2023 From: altidorjb at gmail.com (Jean-Bernard Altidor) Date: Wed, 26 Jul 2023 16:12:41 -0400 Subject: Tap as a service on openstack with Juju and MaaS Message-ID: Hello, I've been struggling to configure tap as a service on openstack for a few months now. I can't find any guide or anything about it. Can you suggest a method of port mirroring? I've even via the ovs bridge but I really need a way to mirror data and send it to an instance for analysis as I'm working on an IDS. Thanks in advance for any ideas ! JB -------------- next part -------------- An HTML attachment was scrubbed... URL: From tsrini.alr at gmail.com Thu Jul 27 06:36:50 2023 From: tsrini.alr at gmail.com (Srinivasan T) Date: Thu, 27 Jul 2023 12:06:50 +0530 Subject: Cinder API version In-Reply-To: References: Message-ID: Hi Team, Could someone approve and help on this? Regards, Srini On Tue, Jul 25, 2023 at 7:53?PM Srinivasan T wrote: > Hi Team, > > We are using openstack4j version 3.6. > > The OpenStack supports both Cinder v2 and v3 endpoints. > > # openstack endpoint list | grep cinder > | 0bff444d3bcf4179bea245d9c7af319c | RegionOne | cinder | volume > | True | public | http://x.x.x.x:8776/v1/%(tenant_id)s | > | 14079c14970f4e21b37ba659e6054ecf | RegionOne | cinderv2 | volumev2 > | True | admin | http://x.x.x.x:8776/v2/%(tenant_id)s | > | 1a2d43fde400446eaca463b6b408a513 | RegionOne | cinderv3 | volumev3 > | True | internal | http://10.250.9.1:8776/v3/%(project_id)s | > | 57f956ddeb4e4f1c8e352ec284f6f587 | RegionOne | cinderv2 | volumev2 > | True | public | http://x.x.x.x:8776/v2/%(tenant_id)s | > | 693f347bb31f4a739768c6b321bccd1c | RegionOne | cinder | volume > | True | internal | http://10.250.9.1:8776/v1/%(tenant_id)s | > | b51e10e2948e4bd2aabb3cf95a6cc3c7 | RegionOne | cinder | volume > | True | admin | http://x.x.x.x:8776/v1/%(tenant_id)s | > | bc8a7eb81a944972bd87b761c9557308 | RegionOne | cinderv2 | volumev2 > | True | internal | http://10.250.9.1:8776/v2/%(tenant_id)s | > | ecf4a7fb8eec4f978038f7a20ae931e5 | RegionOne | cinderv3 | volumev3 > | True | public | http://x.x.x.x:8776/v3/%(project_id)s | > | f680338073664c9398a8d4b573d516a1 | RegionOne | cinderv3 | volumev3 > | True | admin | http://x.x.x.x:8776/v3/%(project_id)s | > > From openstack4j, we are using Identity API v3 for authentication and > trying to do cinder operations. All cinder operations are done via Cinder > API v2. Any idea on how to make openstack4j to use v3 for Cinder APIs? Is > this available in any latest openstack4j? > > Setting the OS_VOLUME_API_VERSION=3 doesn't help. > > Regards, > Srini > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kushagrguptasps.mun at gmail.com Thu Jul 27 11:41:01 2023 From: kushagrguptasps.mun at gmail.com (Kushagr Gupta) Date: Thu, 27 Jul 2023 17:11:01 +0530 Subject: TripleO deployment over IPV6 In-Reply-To: References: Message-ID: Hi Team,Julia, @Julia Thank you for the response. >>Additionally is this UEFI mode or Legacy BIOS mode? We are using UEFI. We tried introspection for two nodes from the undercloud node. Te following is our observation: 1. Undercloud is assigning an IPV6 IP to the node. The PXE boot Failed with the following error: PXE-E99: Unexpected network error. On further debuging this issue, We found that a container: ironic_pxe_tftp is in unhealthy state. But this container should be running. On checking the logs, we found the following: " [root at undercloud-ete log]# podman logs 356715c49cd4 dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help dnsmasq: bad command line options: try --help " We had previously raised this issue in 2022 as well: https://lists.openstack.org/pipermail/openstack-discuss/2022-June/029062.html For this a bug was raised which was fixed https://bugs.launchpad.net/tripleo/+bug/1978892 We cross-verified, and the changes are present on our undercloud. But still we are unable to start the ironic_pxe_tftp container. Could you please help us with this? Thanks and Regards Kushagra Gupta On Tue, Jul 25, 2023 at 8:10?PM Julia Kreger wrote: > Greetings, > > I don't think I saw your original email. I suspect because you indicated > there was an attachment, it might have been filtered for moderation by the > mailing list. > > > On Tue, Jul 25, 2023 at 6:24?AM Lokendra Rathour < > lokendrarathour at gmail.com> wrote: > >> Hi Takashi/ Team, >> Any inputs with respect to the same will be of great help. >> >> Thanks, >> Lokendra >> >> >> On Tue, Jul 25, 2023 at 4:29?PM Kushagr Gupta < >> kushagrguptasps.mun at gmail.com> wrote: >> >>> Hi Team, >>> >>> We were trying to perform IPV6-based deployment of TripleO. >>> We are following the link: >>> >>> https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html/bare_metal_provisioning/sect-deploy#sect-example-templates >>> >>> >>> >> > Try following the more recent documents, specifically OSP16.1 or 16.2. > >> As per our environment, we set the "ipv6_address_mode" to >>> "dhcpv6-stateful" and tried undercloud deployment. It was successful. >>> >>> >>> >> Good, that is required for IPv6 PXE. > >> But when we try node introspection, openstack is not providing an IP that >>> will be used for PXE booting the system. We have enabled PXE booting on the >>> IPV6 network. >>> On Dell machines, we get the following error: >>> "PXE-E16: No offer received." >>> >> > I suspect we will need to see the configuration. Paste.opendev.org? > >> For your reference, the undercloud file which we used is attached. >>> We also tried to find the communication happening on the br-ctlplane >>> interface when the machine tries to boot over IPV6 PXE boot. I am attaching >>> those tcpdump records as well. We are unable to find the MAC address of our >>> machines. >>> >> I suspect this might be in the territory of the hardware, while being set > for IPv6 network booting, maybe the configuration needs to be > checked/reasserted on the interface level. It sounds as if it is using a > different interface. > > Not seeing the expected MAC addresses is rather suspect in this case, > although PXE with v6 is a little different pattern wise. Have you tried > mirroring the port your expecting to see traffic cross from the physical > machine to see if it is seeing the announcements and issuing the DHCPv6 > packets required? > > Additionally is this UEFI mode or Legacy BIOS mode? > >> *Questions:* >>> 1. Is it even possible to perform tripleO deployment with IPV6? >>> 2. Do we need to do some extra settings, so that our baremetal machines >>> can be PXE booted? >>> >>> Thanks and Regards >>> >> >> >> -- >> ~ Lokendra >> skype: lokendrarathour >> >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From juliaashleykreger at gmail.com Thu Jul 27 13:29:28 2023 From: juliaashleykreger at gmail.com (Julia Kreger) Date: Thu, 27 Jul 2023 06:29:28 -0700 Subject: TripleO deployment over IPV6 In-Reply-To: References: Message-ID: On Thu, Jul 27, 2023 at 4:41?AM Kushagr Gupta wrote: > Hi Team,Julia, > > @Julia Thank you for the response. > >>Additionally is this UEFI mode or Legacy BIOS mode? > We are using UEFI. > > We tried introspection for two nodes from the undercloud node. Te > following is our observation: > 1. Undercloud is assigning an IPV6 IP to the node. > > The PXE boot Failed with the following error: > PXE-E99: Unexpected network error. > > I guess what is weird in this entire thing is it sounds like you're shifting over to what appears to be OPROM boot code in a network interface card, which might not support v6. Then again a port mirrored packet capture would be the needful item to troubleshoot further. > On further debuging this issue, We found that a container: ironic_pxe_tftp > is in unhealthy state. But this container should be running. > On checking the logs, we found the following: > " > [root at undercloud-ete log]# podman logs 356715c49cd4 > > dnsmasq: bad command line options: try --help > > dnsmasq: bad command line options: try --help > > dnsmasq: bad command line options: try --help > > Are you able to extract the exact command line which is being passed to the dnsmasq process for that container launch? I guess I'm worried if somehow dnsmasq changed or if an old version is somehow in the container image you're using. " > We had previously raised this issue in 2022 as well: > > https://lists.openstack.org/pipermail/openstack-discuss/2022-June/029062.html > > For this a bug was raised which was fixed > https://bugs.launchpad.net/tripleo/+bug/1978892 > > We cross-verified, and the changes are present on our undercloud. But > still we are unable to start the ironic_pxe_tftp container. > > Could you please help us with this? > > Thanks and Regards > Kushagra Gupta > > On Tue, Jul 25, 2023 at 8:10?PM Julia Kreger > wrote: > >> Greetings, >> >> I don't think I saw your original email. I suspect because you indicated >> there was an attachment, it might have been filtered for moderation by the >> mailing list. >> >> >> On Tue, Jul 25, 2023 at 6:24?AM Lokendra Rathour < >> lokendrarathour at gmail.com> wrote: >> >>> Hi Takashi/ Team, >>> Any inputs with respect to the same will be of great help. >>> >>> Thanks, >>> Lokendra >>> >>> >>> On Tue, Jul 25, 2023 at 4:29?PM Kushagr Gupta < >>> kushagrguptasps.mun at gmail.com> wrote: >>> >>>> Hi Team, >>>> >>>> We were trying to perform IPV6-based deployment of TripleO. >>>> We are following the link: >>>> >>>> https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html/bare_metal_provisioning/sect-deploy#sect-example-templates >>>> >>>> >>>> >>> >> Try following the more recent documents, specifically OSP16.1 or 16.2. >> >>> As per our environment, we set the "ipv6_address_mode" to >>>> "dhcpv6-stateful" and tried undercloud deployment. It was successful. >>>> >>>> >>>> >>> Good, that is required for IPv6 PXE. >> >>> But when we try node introspection, openstack is not providing an IP >>>> that will be used for PXE booting the system. We have enabled PXE booting >>>> on the IPV6 network. >>>> On Dell machines, we get the following error: >>>> "PXE-E16: No offer received." >>>> >>> >> I suspect we will need to see the configuration. Paste.opendev.org? >> >>> For your reference, the undercloud file which we used is attached. >>>> We also tried to find the communication happening on the br-ctlplane >>>> interface when the machine tries to boot over IPV6 PXE boot. I am attaching >>>> those tcpdump records as well. We are unable to find the MAC address of our >>>> machines. >>>> >>> I suspect this might be in the territory of the hardware, while being >> set for IPv6 network booting, maybe the configuration needs to be >> checked/reasserted on the interface level. It sounds as if it is using a >> different interface. >> >> Not seeing the expected MAC addresses is rather suspect in this case, >> although PXE with v6 is a little different pattern wise. Have you tried >> mirroring the port your expecting to see traffic cross from the physical >> machine to see if it is seeing the announcements and issuing the DHCPv6 >> packets required? >> >> Additionally is this UEFI mode or Legacy BIOS mode? >> >>> *Questions:* >>>> 1. Is it even possible to perform tripleO deployment with IPV6? >>>> 2. Do we need to do some extra settings, so that our baremetal machines >>>> can be PXE booted? >>>> >>>> Thanks and Regards >>>> >>> >>> >>> -- >>> ~ Lokendra >>> skype: lokendrarathour >>> >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From kkula at redhat.com Thu Jul 27 14:21:47 2023 From: kkula at redhat.com (Karolina Kula) Date: Thu, 27 Jul 2023 16:21:47 +0200 Subject: [requirements] Blockdiag replacement (not working with Pillow 10) Message-ID: Hello, There are projects[1] which are depending on python-sphinxcontrib-blockdiag[2], which depend on blockdiag [3]. Blockdiag project has no recent changes and seems to be unmaintained. Current version of blockdiag is not compatible with Pillow 10 [4] which is included in current in-flight requirements update [5]. I was trying to contact the maintainer, but with no success. What'd be the best way to proceed? Are there any replacements for this project? [1] https://codesearch.opendev.org/?q=blockdiag&i=nope&literal=nope&files=requirements.txt&excludeFiles=&repos= [2] https://github.com/blockdiag/sphinxcontrib-blockdiag [3] https://github.com/blockdiag/blockdiag [4] https://github.com/blockdiag/blockdiag/pull/174 [5] https://review.opendev.org/c/openstack/requirements/+/884564 Regards, Karolina Kula From elod.illes at ericsson.com Thu Jul 27 15:00:18 2023 From: elod.illes at ericsson.com (=?utf-8?B?RWzDtWQgSWxsw6lz?=) Date: Thu, 27 Jul 2023 15:00:18 +0000 Subject: [release] Release countdown for week R-9, Jul 31-Aug 04 Message-ID: General Information ------------------- The following cycle-with-intermediary deliverables have not done any intermediary release yet during this cycle. The cycle-with-rc release model is more suited for deliverables that plan to be released only once per cycle. As a result, we have detected that the following deliverables could be good candidates for a release model change: bifrost ironic-prometheus-exporter ironic-python-agent-builder ironic-ui networking-baremetal networking-generic-switch python-openstackclient swift PTLs and release liaisons for each of those deliverables should propose an intermediary release for their deliverables. If PTLs or release liaisons think that any of the deliverables should change to cycle-with-rc model then let us know and a patch needs to be created for the model change. We also published a proposed release schedule for the upcoming 2024.1 Caracal cycle. Please check out the separate thread: https://lists.openstack.org/pipermail/openstack-discuss/2023-July/034555.html Upcoming Deadlines & Dates -------------------------- Non-client library freeze: August 24th, 2023 (R-6 week) Client library freeze: August 31st, 2023 (R-5 week) Bobcat-3 milestone: August 31st, 2023 (R-5 week) Final 2023.2 Bobcat release: October 4th, 2023 El?d Ill?s irc: elodilles @ #openstack-release -------------- next part -------------- An HTML attachment was scrubbed... URL: From smooney at redhat.com Thu Jul 27 15:42:16 2023 From: smooney at redhat.com (smooney at redhat.com) Date: Thu, 27 Jul 2023 16:42:16 +0100 Subject: [requirements] Blockdiag replacement (not working with Pillow 10) In-Reply-To: References: Message-ID: we hit a similar issue with sphinxcontrib.seqdiag https://github.com/openstack/nova-specs/commit/00347ca9b7ea4e52b1833340c08869b72eedbb61 also caused by https://github.com/blockdiag/blockdiag we ended up rendering the images to SVGs statically and then removed the usage of sphinxcontrib.seqdiag and included the svg images directly instead. that is proably what will have to happen with https://github.com/blockdiag/sphinxcontrib-blockdiag too On Thu, 2023-07-27 at 16:21 +0200, Karolina Kula wrote: > Hello, > > There are projects[1] which are depending on > python-sphinxcontrib-blockdiag[2], which depend on blockdiag [3]. > Blockdiag project has no recent changes and seems to be unmaintained. > Current version of blockdiag is not compatible with Pillow 10 [4] > which is included in current in-flight requirements update [5]. > > I was trying to contact the maintainer, but with no success. > What'd be the best way to proceed? Are there any replacements for this project? > > [1] https://codesearch.opendev.org/?q=blockdiag&i=nope&literal=nope&files=requirements.txt&excludeFiles=&repos= > [2] https://github.com/blockdiag/sphinxcontrib-blockdiag > [3] https://github.com/blockdiag/blockdiag > [4] https://github.com/blockdiag/blockdiag/pull/174 > [5] https://review.opendev.org/c/openstack/requirements/+/884564 > > > Regards, > Karolina Kula > > From nguyenhuukhoinw at gmail.com Thu Jul 27 16:21:32 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Thu, 27 Jul 2023 23:21:32 +0700 Subject: [openstack][neutron[nova][kolla-ansible]instance cannot ping after live migrate In-Reply-To: References: <75e2a0d49469e378a1e6fa2d4ebdb8490e48b418.camel@redhat.com> Message-ID: Hello. I figured out that my rabbitmq queues are corrupt so neutron port cannot upgrade security rules. I need delete queues so I can migrate without problem. Thank you so much for replying to me. On Thu, Jul 27, 2023, 8:11 AM Nguy?n H?u Kh?i wrote: > Hello. > > When my instances was migrated to other computes. I check on dest host and > I see that > > -A neutron-openvswi-i41ec1d15-e -d x.x.x.x(my instance ip)/32 -p udp -m > udp --sport 67 --dport 68 -j RETURN missing and my instance cannot get IP. > I must restart neutron_openvswitch_agent then this rule appears and I can > touch the instance via network. > > I use openswitch and provider networks. This problem has happened this > week. after the system was upgraded from xena to yoga and I enabled quorum > queue. > > > > Nguyen Huu Khoi > > > On Wed, Jul 26, 2023 at 5:28?PM Nguy?n H?u Kh?i > wrote: > >> Because I dont see any error logs. Althought, i set debug log to on. >> >> Your advices are very helpful to me. I will try to dig deeply. I am lost >> so some suggests are the best way for me to continue. :) >> >> On Wed, Jul 26, 2023, 4:39 PM wrote: >> >>> On Wed, 2023-07-26 at 07:49 +0700, Nguy?n H?u Kh?i wrote: >>> > Hello guys. >>> > >>> > I am using openstack yoga with kolla ansible. >>> without logs of some kind i dont think anyoen will be able to hlep you >>> with this. >>> you have one issue with the config which i noted inline but that should >>> not break live migration. >>> but it would allow it to proceed when otherwise it would have failed. >>> and it woudl allow this issue to happen instead of the vm goign to error >>> ro the migration >>> being aborted in pre live migrate. >>> > >>> > When I migrate: >>> > >>> > instance1 from host A to host B after that I cannot ping this >>> > instance(telnet also). I must restart neutron_openvswitch_agent or move >>> > this instance back to host B then this problem has gone. >>> > >>> > this is my settings: >>> > >>> > ----------------- neutron.conf ----------------- >>> > [nova] >>> > live_migration_events = True >>> > ------------------------------------------------ >>> > >>> > ----------------- nova.conf ----------------- >>> > [DEFAULT] >>> > vif_plugging_timeout = 600 >>> > vif_plugging_is_fatal = False >>> you should never run with this set to false in production. >>> it will break nova ability to detect if netroking is configured >>> when booting or migrating a vm. >>> we honestly should have remove this when we removed nova-networks >>> > debug = True >>> > >>> > [compute] >>> > live_migration_wait_for_vif_plug = True >>> > >>> > [workarounds] >>> > enable_qemu_monitor_announce_self = True >>> > >>> > ----------------- openvswitch_agent.ini----------------- >>> > [securitygroup] >>> > firewall_driver = openvswitch >>> > [ovs] >>> > openflow_processed_per_port = true >>> > >>> > I check nova, neutron, ops logs but they are ok. >>> > >>> > Thank you. >>> > >>> > >>> > Nguyen Huu Khoi >>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From nguyenhuukhoinw at gmail.com Thu Jul 27 16:32:57 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Thu, 27 Jul 2023 23:32:57 +0700 Subject: [cinder] cinder-backup volume stuck in creating In-Reply-To: <20230726134418.5dt5frgr5atbt7ie@localhost> References: <20230201120237.5p35swkzyxsg2koj@localhost> <20230726134418.5dt5frgr5atbt7ie@localhost> Message-ID: Hello. I will debug as you suggested next time. Thank you much. On Wed, Jul 26, 2023, 8:44 PM Gorka Eguileor wrote: > On 15/07, Nguy?n H?u Kh?i wrote: > > Hello guys, > > I had the same problem, there is no more error log. > > > > I resolved by turning off cinder-api, cinder-scheduler and cinder-backup > > then removing these queues, such as cinder, cinder-scheduler and > > cinder-backup.. > > > > I enabled debug log on rabbitmq and cinder, however, I cannot see any > > related logs, > > > > > > 2023-07-13 20:57:22.903 666 INFO cinder.backup.manager > > [req-019d2459-6f43-4839-8ad6-71c805dbd708 > 6371ebfe0fce499983b1f07e7c34bf6d > > 183e47a6dd14489991db5c3cf4132a2a - - -] Create backup started, backup: > > 2cebd275-6fc3-47ae-8c8d-3337f9227057 volume: > > d8446fb4-032a-44a1-b47d-f922c4a0020e. > > 2023-07-13 20:57:22.919 666 INFO cinder.backup.manager > > [req-019d2459-6f43-4839-8ad6-71c805dbd708 > 6371ebfe0fce499983b1f07e7c34bf6d > > 183e47a6dd14489991db5c3cf4132a2a - - -] Call Volume Manager to > > get_backup_device for > > > Backup(availability_zone=None,container=None,created_at=2023-07-13T13:57:22Z,data_timestamp=2023-07-13T13:57:22Z,deleted=False,deleted_at=None,display_description='',display_name='ijoo',encryption_key_id=None,fail_reason=None,host='controller01',id=2cebd275-6fc3-47ae-8c8d-3337f9227057,metadata={},num_dependent_backups=0,object_count=0,parent=None,parent_id=None,project_id='183e47a6dd14489991db5c3cf4132a2a',restore_volume_id=None,service='cinder.backup.drivers.nfs.NFSBackupDriver',service_metadata=None,size=1,snapshot_id=None,status='creating',temp_snapshot_id=None,temp_volume_id=None,updated_at=None,user_id='6371ebfe0fce499983b1f07e7c34bf6d',volume_id=d8446fb4-032a-44a1-b47d-f922c4a0020e) > > > > > > It is hard to reproduce this problems. > > > > > > Nguyen Huu Khoi > > Hi, > > Next time it happens you may want to explore the status of the RabbitMQ > service and queues. > > I remember seeing at least once a RabbitMQ service going crazy and not > delivering messages in some of its queues. > > Cheers, > Gorka. > > > > > > > On Wed, Feb 1, 2023 at 7:08?PM Gorka Eguileor > wrote: > > > > > On 27/01, Satish Patel wrote: > > > > Thank you Jon/Sofia, > > > > > > > > Biggest issue is even if I turn on debugging, it's not producing > enough > > > > logs to see what is going on. See following output. > > > > > > > > https://paste.opendev.org/show/bh9OF9l2OrozrNMglv2Y/ > > > > > > Hi, > > > > > > I don't see the cinder-volume logs, which are necessary to debug this > > > issue, because we can see in the backup logs that it is doing an RPC > > > call to the cinder-volume service "Call Volume Manager to > > > get_backup_device". > > > > > > What is the value of the "host" field of the source volume? > > > > > > Because if it's anything other than "kolla-infra-1.example.com at rbd-1", > > > then the problem is that the cinder-volume service for that backend is > > > currently down. > > > > > > Cheers, > > > Gorka. > > > > > > > > > > > On Fri, Jan 27, 2023 at 10:50 AM Jon Bernard > > > wrote: > > > > > > > > > Without the logs themselves it's really hard to say. One way to > > > proceed > > > > > would be to file a bug [1] and the team can work with you there. > You > > > > > could also enable debugging (debug = True), reproduce the failure, > and > > > > > upload the relevant logs there as well. > > > > > > > > > > [1]: https://bugs.launchpad.net/cinder/+filebug > > > > > > > > > > -- > > > > > Jon > > > > > > > > > > On Thu, Jan 26, 2023 at 2:20 PM Satish Patel > > > > wrote: > > > > > > > > > >> Folks, > > > > >> > > > > >> I have configured nova and cinder with ceph storage. VMs running > on > > > ceph > > > > >> storage but now when i am trying to create a backup of cinder > volume > > > its > > > > >> getting stuck on creating and doing nothing. Logs also do not > give any > > > > >> indication of bad. > > > > >> > > > > >> My cinder.conf > > > > >> > > > > >> [DEFAULT] > > > > >> > > > > >> enabled_backends = rbd-1 > > > > >> backup_driver = cinder.backup.drivers.ceph.CephBackupDriver > > > > >> backup_ceph_conf = /etc/ceph/ceph.conf > > > > >> backup_ceph_user = cinder-backup > > > > >> backup_ceph_chunk_size = 134217728 > > > > >> backup_ceph_pool = backups > > > > >> backup_ceph_stripe_unit = 0 > > > > >> backup_ceph_stripe_count = 0 > > > > >> restore_discard_excess_bytes = true > > > > >> osapi_volume_listen = 10.73.0.181 > > > > >> osapi_volume_listen_port = 8776 > > > > >> > > > > >> > > > > >> Output of "openstack volume service list" showing cinder-backup > > > service > > > > >> is up but when i create a backup it's getting stuck in this stage > and > > > no > > > > >> activity. I am not seeing anything getting transferred to the ceph > > > backups > > > > >> pool also. Any clue? or method to debug? > > > > >> > > > > >> # openstack volume backup list --all > > > > >> > > > > >> > > > > +--------------------------------------+------+-------------+----------+------+ > > > > >> | ID | Name | Description | > Status > > > | > > > > >> Size | > > > > >> > > > > >> > > > > +--------------------------------------+------+-------------+----------+------+ > > > > >> | bc844d55-8c5a-4bd3-b0e9-7c4c780c95ad | foo1 | | > > > creating | > > > > >> 20 | > > > > >> > > > > >> > > > > +--------------------------------------+------+-------------+----------+------+ > > > > >> > > > > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aagautam at redhat.com Thu Jul 27 18:31:42 2023 From: aagautam at redhat.com (Aayushi Gautam) Date: Thu, 27 Jul 2023 14:31:42 -0400 Subject: [Rally] Message-ID: Hello, I am Aayushi , an intern with Redhat working on ESI ( Elastic secure Infracstructure) group. We were trying to use Rally to test the performance of our codebase. I have created a Plugin and task . And getting an error. The same bug is also asked on the bug page of Rally, but it was not answered. I have attached the code of the plugin and task and the error message. Looking forward to hearing from you. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- rally task start rally/esi_node_network_attach_task.yaml -------------------------------------------------------------------------------- Preparing input task -------------------------------------------------------------------------------- Task is: --- ScenarioPlugin.esi_node_network_attach: - args: node: name: "dell-2" network: name: "storage" runner: type: "constant" times: 2 concurrency: 1 Task syntax is correct :) Running Rally version 3.4.0 -------------------------------------------------------------------------------- Task 62bf294b-67f2-40c5-ab3d-8ee39f155f79: started -------------------------------------------------------------------------------- Running Task... This can take a while... To track task status use: rally task status or rally task detailed Using task: 62bf294b-67f2-40c5-ab3d-8ee39f155f79 2023-07-27 18:30:37.848 532740 INFO rally.task.engine [-] Task 62bf294b-67f2-40c5-ab3d-8ee39f155f79 | Starting: Task validation. 2023-07-27 18:30:38.030 532740 INFO rally.task.engine [-] Task 62bf294b-67f2-40c5-ab3d-8ee39f155f79 | Starting: Task validation of syntax. Input task is invalid! Subtask ScenarioPlugin.esi_node_network_attach[0] has wrong configuration Subtask configuration: {"version": 2, "title": "A cropped version of a bigger task.", "description": "Auto-generated task from a single workload", "subtasks": [{"title": "ScenarioPlugin.esi_node_network_attach", "description": "", "scenario": {"ScenarioPlugin.esi_node_network_attach": {"node": {"name": "dell-2"}, "network": {"name": "storage"}}}, "contexts": {}, "runner": {"constant": {"times": 2, "concurrency": 1}}, "hooks": [], "sla": {"failure_rate": {"max": 0}}}]} Reason(s): There is no Scenario plugin `ScenarioPlugin.esi_node_network_attach` in any platform. -------------- next part -------------- A non-text attachment was scrubbed... Name: esi_node_network_attach_plugin.py Type: text/x-python Size: 1277 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: esi_node_network_attach_task.yaml Type: application/x-yaml Size: 689 bytes Desc: not available URL: From alsotoes at gmail.com Thu Jul 27 19:04:39 2023 From: alsotoes at gmail.com (Alvaro Soto) Date: Thu, 27 Jul 2023 13:04:39 -0600 Subject: Cinder API version In-Reply-To: References: Message-ID: --- disclaimer: I don't use openstack4j --- Can you share your code? Are you running a URL resolver first and then trying to use storage services? Cheers! On Thu, Jul 27, 2023 at 7:31?AM Srinivasan T wrote: > Hi Team, > Could someone approve and help on this? > > Regards, > Srini > > On Tue, Jul 25, 2023 at 7:53?PM Srinivasan T wrote: > >> Hi Team, >> >> We are using openstack4j version 3.6. >> >> The OpenStack supports both Cinder v2 and v3 endpoints. >> >> # openstack endpoint list | grep cinder >> | 0bff444d3bcf4179bea245d9c7af319c | RegionOne | cinder | volume >> | True | public | http://x.x.x.x:8776/v1/%(tenant_id)s | >> | 14079c14970f4e21b37ba659e6054ecf | RegionOne | cinderv2 | volumev2 >> | True | admin | http://x.x.x.x:8776/v2/%(tenant_id)s | >> | 1a2d43fde400446eaca463b6b408a513 | RegionOne | cinderv3 | volumev3 >> | True | internal | http://10.250.9.1:8776/v3/%(project_id)s >> | >> | 57f956ddeb4e4f1c8e352ec284f6f587 | RegionOne | cinderv2 | volumev2 >> | True | public | http://x.x.x.x:8776/v2/%(tenant_id)s | >> | 693f347bb31f4a739768c6b321bccd1c | RegionOne | cinder | volume >> | True | internal | http://10.250.9.1:8776/v1/%(tenant_id)s >> | >> | b51e10e2948e4bd2aabb3cf95a6cc3c7 | RegionOne | cinder | volume >> | True | admin | http://x.x.x.x:8776/v1/%(tenant_id)s | >> | bc8a7eb81a944972bd87b761c9557308 | RegionOne | cinderv2 | volumev2 >> | True | internal | http://10.250.9.1:8776/v2/%(tenant_id)s >> | >> | ecf4a7fb8eec4f978038f7a20ae931e5 | RegionOne | cinderv3 | volumev3 >> | True | public | http://x.x.x.x:8776/v3/%(project_id)s | >> | f680338073664c9398a8d4b573d516a1 | RegionOne | cinderv3 | volumev3 >> | True | admin | http://x.x.x.x:8776/v3/%(project_id)s | >> >> From openstack4j, we are using Identity API v3 for authentication and >> trying to do cinder operations. All cinder operations are done via Cinder >> API v2. Any idea on how to make openstack4j to use v3 for Cinder APIs? Is >> this available in any latest openstack4j? >> >> Setting the OS_VOLUME_API_VERSION=3 doesn't help. >> >> Regards, >> Srini >> > -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people. -------------- next part -------------- An HTML attachment was scrubbed... URL: From johnsomor at gmail.com Thu Jul 27 20:54:21 2023 From: johnsomor at gmail.com (Michael Johnson) Date: Thu, 27 Jul 2023 13:54:21 -0700 Subject: [requirements] Blockdiag replacement (not working with Pillow 10) In-Reply-To: References: Message-ID: I contributed a partial fix to blockdiag back in February, but it has had no traction. Designate only has a few diagrams using blockdiag, so our plan is to switch to simple graphviz replacements and remove the requirement on blockdiag. Michael On Thu, Jul 27, 2023 at 8:42?AM wrote: > > we hit a similar issue with sphinxcontrib.seqdiag > https://github.com/openstack/nova-specs/commit/00347ca9b7ea4e52b1833340c08869b72eedbb61 > also caused by https://github.com/blockdiag/blockdiag > > we ended up rendering the images to SVGs statically and then removed the usage of > sphinxcontrib.seqdiag and included the svg images directly instead. > > that is proably what will have to happen with > https://github.com/blockdiag/sphinxcontrib-blockdiag too > > > On Thu, 2023-07-27 at 16:21 +0200, Karolina Kula wrote: > > Hello, > > > > There are projects[1] which are depending on > > python-sphinxcontrib-blockdiag[2], which depend on blockdiag [3]. > > Blockdiag project has no recent changes and seems to be unmaintained. > > Current version of blockdiag is not compatible with Pillow 10 [4] > > which is included in current in-flight requirements update [5]. > > > > I was trying to contact the maintainer, but with no success. > > What'd be the best way to proceed? Are there any replacements for this project? > > > > [1] https://codesearch.opendev.org/?q=blockdiag&i=nope&literal=nope&files=requirements.txt&excludeFiles=&repos= > > [2] https://github.com/blockdiag/sphinxcontrib-blockdiag > > [3] https://github.com/blockdiag/blockdiag > > [4] https://github.com/blockdiag/blockdiag/pull/174 > > [5] https://review.opendev.org/c/openstack/requirements/+/884564 > > > > > > Regards, > > Karolina Kula > > > > > > From gthiemonge at redhat.com Fri Jul 28 08:41:13 2023 From: gthiemonge at redhat.com (Gregory Thiemonge) Date: Fri, 28 Jul 2023 10:41:13 +0200 Subject: [Octavia] Weekly meetings cancelled Message-ID: Hi, As discussed during the weekly meeting, the next 2 Octavia meetings are cancelled. The next meeting will be on Aug 16. Thanks Greg -------------- next part -------------- An HTML attachment was scrubbed... URL: From liam.young at canonical.com Fri Jul 28 13:02:01 2023 From: liam.young at canonical.com (Liam Young) Date: Fri, 28 Jul 2023 14:02:01 +0100 Subject: [charms][sunbeam] Proposing Guillaume Boutry for charms-core In-Reply-To: References: Message-ID: Hi, I have reviewed Guillaumes work (and had my work reviewed by him). In both cases he has shown excellent technical knowledge and attention to detail. I am +1 on him becoming a member of the charms-core team. Thanks Liam On Tue, Jul 18, 2023 at 3:21?PM James Page wrote: > Hi All > > Guillaume has been working on the Sunbeam Charms for the last few months > and has proven an able reviewer and contributor to the charms; we need to > dis-entangle the sunbeam project charms from the main openstack charms set > but until that work is completed I'd like to propose Guillaume as a member > of the charms-core team. > > Cheers > > James > -------------- next part -------------- An HTML attachment was scrubbed... URL: From N.Feldt at mittwald.de Fri Jul 28 14:06:32 2023 From: N.Feldt at mittwald.de (Noah Elias Feldt) Date: Fri, 28 Jul 2023 14:06:32 +0000 Subject: AW: [cloudkitty] question to the scope_key in cloudkitty In-Reply-To: References: , Message-ID: Hello Pierre, Currently we use a workaround by simply rewriting the labels via Prometheus. But I could still commit the change, which would also be an advantage for us. I will take care of it in the next days. Regards, Noah Feldt _____ Mittwald CM Service GmbH & Co. KG K?nigsberger Stra?e 4-6 32339 Espelkamp Tel.: 05772 / 293-100 Fax: 05772 / 293-333 support at mittwald.de https://www.mittwald.de Gesch?ftsf?hrer: Robert Meyer, Florian J?rgens USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplement?rin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen Informationen zur Datenverarbeitung im Rahmen unserer Gesch?ftst?tigkeit gem?? Art. 13-14 DSGVO sind unter www.mittwald.de/ds abrufbar. ________________________________________ Von: Pierre Riteau Gesendet: Montag, 17. Juli 2023 22:10:30 An: Noah Elias Feldt Cc: openstack-discuss at lists.openstack.org; Rafael Weing?rtner Betreff: Re: [cloudkitty] question to the scope_key in cloudkitty Hello Noah, Are you still interested in contributing your code change? It would be really useful when using metrics from different Prometheus exporters which may not be using the same labels. The contributor docs are here: https://docs.openstack.org/cloudkitty/latest/contributor/contributing.html Thanks, Pierre Riteau (priteau) On Mon, 20 Feb 2023 at 18:55, Rafael Weing?rtner > wrote: Hello Noah, Even though we do not see a use case for the feature you described, you seem to have a solid use case. Other approaches could have been used to "rename"/"map" tenant_id to project_id, such as with Dynamic pollsters in Ceilometer. However, the argument you present seems solid to me. I would say that if you open a patch, we would happily review it :). AS long as things are backwards compatible, tested, and well documented, we would for sure merge it. On Mon, Feb 20, 2023 at 10:39 AM Noah Elias Feldt > wrote: Hello! I am currently trying to get CloudKitty to work with the Prometheus Collector and PyScripts. I noticed that the scope_key for Prometheus can only be changed once in the global configuration file. This was a problem for me because we need different scope_keys for different metrics. Many of the metrics still need the scope key "tenant_id" and some others "project_id". I have found in the documentation of CloudKitty no way to change this for each metric, so I have internally changed something in the source code and added the function to set the scope_key for each metric individually. I just wanted to ask if you would be interested if I submit the small change upstream? Thank you. Regards Noah Feldt -- Rafael Weing?rtner From sbauza at redhat.com Fri Jul 28 15:06:45 2023 From: sbauza at redhat.com (Sylvain Bauza) Date: Fri, 28 Jul 2023 17:06:45 +0200 Subject: [nova] Nova meetings on Aug 1 and Aug 8 are CANCELLED Message-ID: As said above, we don't have quorum. Gibi will be chairing on Aug 15 1600UTC. As a reminder, any person is welcome (even if you are not a contributor) during our IRC meetings. You can find all our weekly minutes and our connection details in https://meetings.opendev.org/#Nova_Team_Meeting Thanks, -Sylvain -------------- next part -------------- An HTML attachment was scrubbed... URL: From james.page at canonical.com Fri Jul 28 15:12:00 2023 From: james.page at canonical.com (James Page) Date: Fri, 28 Jul 2023 16:12:00 +0100 Subject: [charms][sunbeam] Proposing Guillaume Boutry for charms-core In-Reply-To: References: Message-ID: Thanks Liam Congrat Guillaume! I've added you to the charms-core team in Gerrit. On Fri, Jul 28, 2023 at 2:02?PM Liam Young wrote: > Hi, > > I have reviewed Guillaumes work (and had my work reviewed by him). In both > cases he has shown excellent technical knowledge and attention to detail. I > am +1 on him becoming a member of the charms-core team. > > Thanks > Liam > > On Tue, Jul 18, 2023 at 3:21?PM James Page > wrote: > >> Hi All >> >> Guillaume has been working on the Sunbeam Charms for the last few months >> and has proven an able reviewer and contributor to the charms; we need to >> dis-entangle the sunbeam project charms from the main openstack charms set >> but until that work is completed I'd like to propose Guillaume as a member >> of the charms-core team. >> >> Cheers >> >> James >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.boutry at canonical.com Fri Jul 28 18:07:15 2023 From: guillaume.boutry at canonical.com (Guillaume Boutry) Date: Fri, 28 Jul 2023 20:07:15 +0200 Subject: [charms][sunbeam] Proposing Guillaume Boutry for charms-core In-Reply-To: References: Message-ID: Thank you, happy to continue working on Sunbeam, I'll do my best on contributing to the project, Have a great week end, Guillaume On Fri, Jul 28, 2023 at 5:12?PM James Page wrote: > Thanks Liam > > Congrat Guillaume! I've added you to the charms-core team in Gerrit. > > On Fri, Jul 28, 2023 at 2:02?PM Liam Young > wrote: > >> Hi, >> >> I have reviewed Guillaumes work (and had my work reviewed by him). In >> both cases he has shown excellent technical knowledge and attention to >> detail. I am +1 on him becoming a member of the charms-core team. >> >> Thanks >> Liam >> >> On Tue, Jul 18, 2023 at 3:21?PM James Page >> wrote: >> >>> Hi All >>> >>> Guillaume has been working on the Sunbeam Charms for the last few months >>> and has proven an able reviewer and contributor to the charms; we need to >>> dis-entangle the sunbeam project charms from the main openstack charms set >>> but until that work is completed I'd like to propose Guillaume as a member >>> of the charms-core team. >>> >>> Cheers >>> >>> James >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From gouthampravi at gmail.com Sat Jul 29 00:04:04 2023 From: gouthampravi at gmail.com (Goutham Pacha Ravi) Date: Fri, 28 Jul 2023 17:04:04 -0700 Subject: [manila] Collaborative Review session on resource locks Message-ID: Hello Zorillas and other interested stackers, We're hosting a collab-review session on the changes pertaining to resource locks [1]. These changes are being pursued for the 2023.2/Bobcat release. The collab-review meeting will be at 1530 UTC on Thursday, Aug 3rd 2023 (gcal [2]). The meeting link will be published on the etherpad closer to the event, and shared during the IRC meeting that begins at 1500 UTC on the same day. [3] A reminder for the unfamiliar, a "collab-review" is a regular activity that the team does where patch authors provide a walkthrough of the code changes on a virtual community meeting and answer questions real time. This has helped us speed up reviews in past cycles. While the intention is to brainstorm synchronously, we will reflect all feedback - big and small, on Gerrit for posterity. In addition, the meeting call will be recorded, and the recording will be made available on the community's Youtube channel. [4] Hoping to see you there! Goutham [1] https://etherpad.opendev.org/p/manila-allow-locking-shares-against-deletion [2] https://red.ht/manila-bobcat-collab-review [3] https://wiki.openstack.org/wiki/Manila/Meetings#Next_Meeting [4] https://www.youtube.com/@openstackmanila From viswanath.kvg at gmail.com Sat Jul 29 08:37:58 2023 From: viswanath.kvg at gmail.com (Viswanath Ediga) Date: Sat, 29 Jul 2023 14:07:58 +0530 Subject: [openstack-helm] [ceph deployment failed] error:- MountVolume.SetUp failed for volume "ceph-client-admin-keyring" secret "ceph-client-admin-keyring" not found Message-ID: Team, I am trying to deploy openstack using OSH. I am following documentation https://docs.openstack.org/openstack-helm/latest/install/multinode.html I am getting error while "Deploying ceph" phase. container creation is failing. Below is the error that I see in POD description Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 24m (x73 over 3h7m) kubelet Unable to attach or mount volumes: unmounted volumes=[ceph-client-admin-keyring], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition Warning FailedMount 4m18s (x99 over 3h9m) kubelet MountVolume.SetUp failed for volume "ceph-client-admin-keyring" : secret "ceph-client-admin-keyring" not found Regards, Any help is appreciated. Kasi Viswanath -------------- next part -------------- An HTML attachment was scrubbed... URL: From satish.txt at gmail.com Sat Jul 29 10:55:46 2023 From: satish.txt at gmail.com (Satish Patel) Date: Sat, 29 Jul 2023 06:55:46 -0400 Subject: [openstack][neutron[nova][kolla-ansible]instance cannot ping after live migrate In-Reply-To: References: Message-ID: <29366023-82DE-4810-A87C-410A6EC475DF@gmail.com> An HTML attachment was scrubbed... URL: From nguyenhuukhoinw at gmail.com Sat Jul 29 14:29:41 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Sat, 29 Jul 2023 21:29:41 +0700 Subject: [openstack][neutron[nova][kolla-ansible]instance cannot ping after live migrate In-Reply-To: <29366023-82DE-4810-A87C-410A6EC475DF@gmail.com> References: <29366023-82DE-4810-A87C-410A6EC475DF@gmail.com> Message-ID: Hello. I just known about ops firewall last week. I am going to compare between them. Could you share some experience about why ovs firewall driver over iptables. Thank you. Nguyen Huu Khoi On Sat, Jul 29, 2023 at 5:55?PM Satish Patel wrote: > Why are you not using openvswitch flow based firewall instead of > Linuxbridge which will add hops in packet path. > > Sent from my iPhone > > On Jul 27, 2023, at 12:25 PM, Nguy?n H?u Kh?i > wrote: > > ? > Hello. > I figured out that my rabbitmq queues are corrupt so neutron port cannot > upgrade security rules. I need delete queues so I can migrate without > problem. > > Thank you so much for replying to me. > > On Thu, Jul 27, 2023, 8:11 AM Nguy?n H?u Kh?i > wrote: > >> Hello. >> >> When my instances was migrated to other computes. I check on dest host >> and I see that >> >> -A neutron-openvswi-i41ec1d15-e -d x.x.x.x(my instance ip)/32 -p udp -m >> udp --sport 67 --dport 68 -j RETURN missing and my instance cannot get IP. >> I must restart neutron_openvswitch_agent then this rule appears and I can >> touch the instance via network. >> >> I use openswitch and provider networks. This problem has happened this >> week. after the system was upgraded from xena to yoga and I enabled quorum >> queue. >> >> >> >> Nguyen Huu Khoi >> >> >> On Wed, Jul 26, 2023 at 5:28?PM Nguy?n H?u Kh?i < >> nguyenhuukhoinw at gmail.com> wrote: >> >>> Because I dont see any error logs. Althought, i set debug log to on. >>> >>> Your advices are very helpful to me. I will try to dig deeply. I am lost >>> so some suggests are the best way for me to continue. :) >>> >>> On Wed, Jul 26, 2023, 4:39 PM wrote: >>> >>>> On Wed, 2023-07-26 at 07:49 +0700, Nguy?n H?u Kh?i wrote: >>>> > Hello guys. >>>> > >>>> > I am using openstack yoga with kolla ansible. >>>> without logs of some kind i dont think anyoen will be able to hlep you >>>> with this. >>>> you have one issue with the config which i noted inline but that should >>>> not break live migration. >>>> but it would allow it to proceed when otherwise it would have failed. >>>> and it woudl allow this issue to happen instead of the vm goign to >>>> error ro the migration >>>> being aborted in pre live migrate. >>>> > >>>> > When I migrate: >>>> > >>>> > instance1 from host A to host B after that I cannot ping this >>>> > instance(telnet also). I must restart neutron_openvswitch_agent or >>>> move >>>> > this instance back to host B then this problem has gone. >>>> > >>>> > this is my settings: >>>> > >>>> > ----------------- neutron.conf ----------------- >>>> > [nova] >>>> > live_migration_events = True >>>> > ------------------------------------------------ >>>> > >>>> > ----------------- nova.conf ----------------- >>>> > [DEFAULT] >>>> > vif_plugging_timeout = 600 >>>> > vif_plugging_is_fatal = False >>>> you should never run with this set to false in production. >>>> it will break nova ability to detect if netroking is configured >>>> when booting or migrating a vm. >>>> we honestly should have remove this when we removed nova-networks >>>> > debug = True >>>> > >>>> > [compute] >>>> > live_migration_wait_for_vif_plug = True >>>> > >>>> > [workarounds] >>>> > enable_qemu_monitor_announce_self = True >>>> > >>>> > ----------------- openvswitch_agent.ini----------------- >>>> > [securitygroup] >>>> > firewall_driver = openvswitch >>>> > [ovs] >>>> > openflow_processed_per_port = true >>>> > >>>> > I check nova, neutron, ops logs but they are ok. >>>> > >>>> > Thank you. >>>> > >>>> > >>>> > Nguyen Huu Khoi >>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From murilo at evocorp.com.br Sat Jul 29 15:08:51 2023 From: murilo at evocorp.com.br (Murilo Morais) Date: Sat, 29 Jul 2023 12:08:51 -0300 Subject: [OSA] Failover at the storage level Message-ID: Good morning everybody! I'm trying to raise a scenario with redundancy at the storage level. I'm using CEPH with RBD-mirror. Cinder has a pretty clear documentation about it, but I didn't find anything about Glance and the "nova_libvirt_images_rbd_pool" and the "cinder_service_backup_driver". What would be the best approach? Only use Cinder with Mirror? If by chance the "nova_libvirt_images_rbd_pool" dies, if you are using config-drive the disk will die, correct? In this case, would it be better to leave local? I can't see what would be the best scenario for Failover cases with OSA. Thanks in advance! -------------- next part -------------- An HTML attachment was scrubbed... URL: From noonedeadpunk at gmail.com Sat Jul 29 16:27:25 2023 From: noonedeadpunk at gmail.com (Dmitriy Rabotyagov) Date: Sat, 29 Jul 2023 18:27:25 +0200 Subject: [OSA] Failover at the storage level In-Reply-To: References: Message-ID: I would say that quite little to do OSA itself, but how to design services themselves. Using config_overrides in OpenStack-Ansible allows you to define any service configuration you want. So you can leverage nova_nova_conf_overrides for adding any arbitrary configuration to nova.conf and cinder_cinder_conf_overrides for cinder.conf. Regarding redundancy - I can't really advice as I'm not really happy with any of approaches available due to one or another reason. We ended up with having disaster recovery instead implemented with cinder-backup to RGW (S3) storage in the different region. As it also supports incremental backups, which is quite neat On Sat, Jul 29, 2023, 17:12 Murilo Morais wrote: > Good morning everybody! > > I'm trying to raise a scenario with redundancy at the storage level. > I'm using CEPH with RBD-mirror. > > Cinder has a pretty clear documentation about it, but I didn't find > anything about Glance and the "nova_libvirt_images_rbd_pool" and the > "cinder_service_backup_driver". > > What would be the best approach? > Only use Cinder with Mirror? > If by chance the "nova_libvirt_images_rbd_pool" dies, if you are using > config-drive the disk will die, correct? In this case, would it be better > to leave local? > > I can't see what would be the best scenario for Failover cases with OSA. > > Thanks in advance! > -------------- next part -------------- An HTML attachment was scrubbed... URL: From satish.txt at gmail.com Sun Jul 30 01:26:28 2023 From: satish.txt at gmail.com (Satish Patel) Date: Sat, 29 Jul 2023 21:26:28 -0400 Subject: [openstack][neutron[nova][kolla-ansible]instance cannot ping after live migrate In-Reply-To: References: <29366023-82DE-4810-A87C-410A6EC475DF@gmail.com> Message-ID: iptables + linux bridge integration with OVS was very old and OVS ACL was not mature enough in earlier days. But nowadays OVN supports OVS base ACL and that means it's much more stable. On Sat, Jul 29, 2023 at 10:29?AM Nguy?n H?u Kh?i wrote: > Hello. > I just known about ops firewall last week. I am going to compare > between them. > Could you share some experience about why ovs firewall driver over > iptables. > Thank you. > Nguyen Huu Khoi > > > On Sat, Jul 29, 2023 at 5:55?PM Satish Patel wrote: > >> Why are you not using openvswitch flow based firewall instead of >> Linuxbridge which will add hops in packet path. >> >> Sent from my iPhone >> >> On Jul 27, 2023, at 12:25 PM, Nguy?n H?u Kh?i >> wrote: >> >> ? >> Hello. >> I figured out that my rabbitmq queues are corrupt so neutron port cannot >> upgrade security rules. I need delete queues so I can migrate without >> problem. >> >> Thank you so much for replying to me. >> >> On Thu, Jul 27, 2023, 8:11 AM Nguy?n H?u Kh?i >> wrote: >> >>> Hello. >>> >>> When my instances was migrated to other computes. I check on dest host >>> and I see that >>> >>> -A neutron-openvswi-i41ec1d15-e -d x.x.x.x(my instance ip)/32 -p udp -m >>> udp --sport 67 --dport 68 -j RETURN missing and my instance cannot get IP. >>> I must restart neutron_openvswitch_agent then this rule appears and I can >>> touch the instance via network. >>> >>> I use openswitch and provider networks. This problem has happened this >>> week. after the system was upgraded from xena to yoga and I enabled quorum >>> queue. >>> >>> >>> >>> Nguyen Huu Khoi >>> >>> >>> On Wed, Jul 26, 2023 at 5:28?PM Nguy?n H?u Kh?i < >>> nguyenhuukhoinw at gmail.com> wrote: >>> >>>> Because I dont see any error logs. Althought, i set debug log to on. >>>> >>>> Your advices are very helpful to me. I will try to dig deeply. I am >>>> lost so some suggests are the best way for me to continue. :) >>>> >>>> On Wed, Jul 26, 2023, 4:39 PM wrote: >>>> >>>>> On Wed, 2023-07-26 at 07:49 +0700, Nguy?n H?u Kh?i wrote: >>>>> > Hello guys. >>>>> > >>>>> > I am using openstack yoga with kolla ansible. >>>>> without logs of some kind i dont think anyoen will be able to hlep you >>>>> with this. >>>>> you have one issue with the config which i noted inline but that >>>>> should not break live migration. >>>>> but it would allow it to proceed when otherwise it would have failed. >>>>> and it woudl allow this issue to happen instead of the vm goign to >>>>> error ro the migration >>>>> being aborted in pre live migrate. >>>>> > >>>>> > When I migrate: >>>>> > >>>>> > instance1 from host A to host B after that I cannot ping this >>>>> > instance(telnet also). I must restart neutron_openvswitch_agent or >>>>> move >>>>> > this instance back to host B then this problem has gone. >>>>> > >>>>> > this is my settings: >>>>> > >>>>> > ----------------- neutron.conf ----------------- >>>>> > [nova] >>>>> > live_migration_events = True >>>>> > ------------------------------------------------ >>>>> > >>>>> > ----------------- nova.conf ----------------- >>>>> > [DEFAULT] >>>>> > vif_plugging_timeout = 600 >>>>> > vif_plugging_is_fatal = False >>>>> you should never run with this set to false in production. >>>>> it will break nova ability to detect if netroking is configured >>>>> when booting or migrating a vm. >>>>> we honestly should have remove this when we removed nova-networks >>>>> > debug = True >>>>> > >>>>> > [compute] >>>>> > live_migration_wait_for_vif_plug = True >>>>> > >>>>> > [workarounds] >>>>> > enable_qemu_monitor_announce_self = True >>>>> > >>>>> > ----------------- openvswitch_agent.ini----------------- >>>>> > [securitygroup] >>>>> > firewall_driver = openvswitch >>>>> > [ovs] >>>>> > openflow_processed_per_port = true >>>>> > >>>>> > I check nova, neutron, ops logs but they are ok. >>>>> > >>>>> > Thank you. >>>>> > >>>>> > >>>>> > Nguyen Huu Khoi >>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From nguyenhuukhoinw at gmail.com Sun Jul 30 15:00:22 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Sun, 30 Jul 2023 22:00:22 +0700 Subject: [openstack][neutron[nova][kolla-ansible]instance cannot ping after live migrate In-Reply-To: References: <29366023-82DE-4810-A87C-410A6EC475DF@gmail.com> Message-ID: Hello. Is it ok if we use ovs with native firewall driver which I mean don't use ovn. How about migration from ovs to ovn. Nguyen Huu Khoi On Sun, Jul 30, 2023 at 8:26?AM Satish Patel wrote: > iptables + linux bridge integration with OVS was very old and OVS ACL was > not mature enough in earlier days. But nowadays OVN supports OVS base ACL > and that means it's much more stable. > > On Sat, Jul 29, 2023 at 10:29?AM Nguy?n H?u Kh?i < > nguyenhuukhoinw at gmail.com> wrote: > >> Hello. >> I just known about ops firewall last week. I am going to compare >> between them. >> Could you share some experience about why ovs firewall driver over >> iptables. >> Thank you. >> Nguyen Huu Khoi >> >> >> On Sat, Jul 29, 2023 at 5:55?PM Satish Patel >> wrote: >> >>> Why are you not using openvswitch flow based firewall instead of >>> Linuxbridge which will add hops in packet path. >>> >>> Sent from my iPhone >>> >>> On Jul 27, 2023, at 12:25 PM, Nguy?n H?u Kh?i >>> wrote: >>> >>> ? >>> Hello. >>> I figured out that my rabbitmq queues are corrupt so neutron port cannot >>> upgrade security rules. I need delete queues so I can migrate without >>> problem. >>> >>> Thank you so much for replying to me. >>> >>> On Thu, Jul 27, 2023, 8:11 AM Nguy?n H?u Kh?i >>> wrote: >>> >>>> Hello. >>>> >>>> When my instances was migrated to other computes. I check on dest host >>>> and I see that >>>> >>>> -A neutron-openvswi-i41ec1d15-e -d x.x.x.x(my instance ip)/32 -p udp -m >>>> udp --sport 67 --dport 68 -j RETURN missing and my instance cannot get IP. >>>> I must restart neutron_openvswitch_agent then this rule appears and I can >>>> touch the instance via network. >>>> >>>> I use openswitch and provider networks. This problem has happened this >>>> week. after the system was upgraded from xena to yoga and I enabled quorum >>>> queue. >>>> >>>> >>>> >>>> Nguyen Huu Khoi >>>> >>>> >>>> On Wed, Jul 26, 2023 at 5:28?PM Nguy?n H?u Kh?i < >>>> nguyenhuukhoinw at gmail.com> wrote: >>>> >>>>> Because I dont see any error logs. Althought, i set debug log to on. >>>>> >>>>> Your advices are very helpful to me. I will try to dig deeply. I am >>>>> lost so some suggests are the best way for me to continue. :) >>>>> >>>>> On Wed, Jul 26, 2023, 4:39 PM wrote: >>>>> >>>>>> On Wed, 2023-07-26 at 07:49 +0700, Nguy?n H?u Kh?i wrote: >>>>>> > Hello guys. >>>>>> > >>>>>> > I am using openstack yoga with kolla ansible. >>>>>> without logs of some kind i dont think anyoen will be able to hlep >>>>>> you with this. >>>>>> you have one issue with the config which i noted inline but that >>>>>> should not break live migration. >>>>>> but it would allow it to proceed when otherwise it would have failed. >>>>>> and it woudl allow this issue to happen instead of the vm goign to >>>>>> error ro the migration >>>>>> being aborted in pre live migrate. >>>>>> > >>>>>> > When I migrate: >>>>>> > >>>>>> > instance1 from host A to host B after that I cannot ping this >>>>>> > instance(telnet also). I must restart neutron_openvswitch_agent or >>>>>> move >>>>>> > this instance back to host B then this problem has gone. >>>>>> > >>>>>> > this is my settings: >>>>>> > >>>>>> > ----------------- neutron.conf ----------------- >>>>>> > [nova] >>>>>> > live_migration_events = True >>>>>> > ------------------------------------------------ >>>>>> > >>>>>> > ----------------- nova.conf ----------------- >>>>>> > [DEFAULT] >>>>>> > vif_plugging_timeout = 600 >>>>>> > vif_plugging_is_fatal = False >>>>>> you should never run with this set to false in production. >>>>>> it will break nova ability to detect if netroking is configured >>>>>> when booting or migrating a vm. >>>>>> we honestly should have remove this when we removed nova-networks >>>>>> > debug = True >>>>>> > >>>>>> > [compute] >>>>>> > live_migration_wait_for_vif_plug = True >>>>>> > >>>>>> > [workarounds] >>>>>> > enable_qemu_monitor_announce_self = True >>>>>> > >>>>>> > ----------------- openvswitch_agent.ini----------------- >>>>>> > [securitygroup] >>>>>> > firewall_driver = openvswitch >>>>>> > [ovs] >>>>>> > openflow_processed_per_port = true >>>>>> > >>>>>> > I check nova, neutron, ops logs but they are ok. >>>>>> > >>>>>> > Thank you. >>>>>> > >>>>>> > >>>>>> > Nguyen Huu Khoi >>>>>> >>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaplons at redhat.com Mon Jul 31 07:12:33 2023 From: skaplons at redhat.com (Slawek Kaplonski) Date: Mon, 31 Jul 2023 09:12:33 +0200 Subject: [openstack][neutron[nova][kolla-ansible]instance cannot ping after live migrate In-Reply-To: References: Message-ID: <62631251.QqMBZyFNYO@p1> Hi, Dnia niedziela, 30 lipca 2023 17:00:22 CEST Nguy?n H?u Kh?i pisze: > Hello. > Is it ok if we use ovs with native firewall driver which I mean don't use > ovn. How about migration from ovs to ovn. Regarding migration from ML2/OVS to ML2/OVN backend it's easier to do it when You are using ML2/OVS with openvswitch (native) firewall driver as in that case plugging of the VMs into br-int will be the same before and after migration. > > Nguyen Huu Khoi > > > On Sun, Jul 30, 2023 at 8:26?AM Satish Patel wrote: > > > iptables + linux bridge integration with OVS was very old and OVS ACL was > > not mature enough in earlier days. But nowadays OVN supports OVS base ACL > > and that means it's much more stable. I'm not sure but I think there are some mixed things here. Generally in Neutron we have "backends" like ML2/OVS (neutron-openvswitch-agent) or ML2/OVN (with ovn-controller running on compute nodes). There are more backends like ML2/Linuxbridge for example but lets not include them here and focus only on ML2/OVS and ML2/OVN as those were mentioned. Now, regarding firewall drivers, in ML2/OVS backend, neutron-openvswitch-agent can use one of the following firewall drivers: * iptables_hybrid - that's the one mentioned by Satish Patel as this "very old" solution. Indeed it is using linuxbridge between VM and br-int to implement iptables rule which will work on this linuxbridge for the instance, * openvswitch - this is newer firewall driver, where all SG rules are implemented on the host as OpenFlow rules in br-int. In this case VM is plugged directly to the br-int. But this isn't related to the OVN ACLs in any way. It's all implemented in the neutron-openvswitch-agent code. Details about it are in the: https://docs.openstack.org/neutron/latest/admin/config-ovsfwdriver.html In ML2/OVN backend there is only one implementation of the Security Groups and this is based on the OVN ACL mechanism. In this case of course there is also no need to use any Linuxbridge between VM and br-int so VM is plugged directly into br-int. > > > > On Sat, Jul 29, 2023 at 10:29?AM Nguy?n H?u Kh?i < > > nguyenhuukhoinw at gmail.com> wrote: > > > >> Hello. > >> I just known about ops firewall last week. I am going to compare > >> between them. > >> Could you share some experience about why ovs firewall driver over > >> iptables. > >> Thank you. > >> Nguyen Huu Khoi > >> > >> > >> On Sat, Jul 29, 2023 at 5:55?PM Satish Patel > >> wrote: > >> > >>> Why are you not using openvswitch flow based firewall instead of > >>> Linuxbridge which will add hops in packet path. > >>> > >>> Sent from my iPhone > >>> > >>> On Jul 27, 2023, at 12:25 PM, Nguy?n H?u Kh?i > >>> wrote: > >>> > >>> ? > >>> Hello. > >>> I figured out that my rabbitmq queues are corrupt so neutron port cannot > >>> upgrade security rules. I need delete queues so I can migrate without > >>> problem. > >>> > >>> Thank you so much for replying to me. > >>> > >>> On Thu, Jul 27, 2023, 8:11 AM Nguy?n H?u Kh?i > >>> wrote: > >>> > >>>> Hello. > >>>> > >>>> When my instances was migrated to other computes. I check on dest host > >>>> and I see that > >>>> > >>>> -A neutron-openvswi-i41ec1d15-e -d x.x.x.x(my instance ip)/32 -p udp -m > >>>> udp --sport 67 --dport 68 -j RETURN missing and my instance cannot get IP. > >>>> I must restart neutron_openvswitch_agent then this rule appears and I can > >>>> touch the instance via network. > >>>> > >>>> I use openswitch and provider networks. This problem has happened this > >>>> week. after the system was upgraded from xena to yoga and I enabled quorum > >>>> queue. > >>>> > >>>> > >>>> > >>>> Nguyen Huu Khoi > >>>> > >>>> > >>>> On Wed, Jul 26, 2023 at 5:28?PM Nguy?n H?u Kh?i < > >>>> nguyenhuukhoinw at gmail.com> wrote: > >>>> > >>>>> Because I dont see any error logs. Althought, i set debug log to on. > >>>>> > >>>>> Your advices are very helpful to me. I will try to dig deeply. I am > >>>>> lost so some suggests are the best way for me to continue. :) > >>>>> > >>>>> On Wed, Jul 26, 2023, 4:39 PM wrote: > >>>>> > >>>>>> On Wed, 2023-07-26 at 07:49 +0700, Nguy?n H?u Kh?i wrote: > >>>>>> > Hello guys. > >>>>>> > > >>>>>> > I am using openstack yoga with kolla ansible. > >>>>>> without logs of some kind i dont think anyoen will be able to hlep > >>>>>> you with this. > >>>>>> you have one issue with the config which i noted inline but that > >>>>>> should not break live migration. > >>>>>> but it would allow it to proceed when otherwise it would have failed. > >>>>>> and it woudl allow this issue to happen instead of the vm goign to > >>>>>> error ro the migration > >>>>>> being aborted in pre live migrate. > >>>>>> > > >>>>>> > When I migrate: > >>>>>> > > >>>>>> > instance1 from host A to host B after that I cannot ping this > >>>>>> > instance(telnet also). I must restart neutron_openvswitch_agent or > >>>>>> move > >>>>>> > this instance back to host B then this problem has gone. > >>>>>> > > >>>>>> > this is my settings: > >>>>>> > > >>>>>> > ----------------- neutron.conf ----------------- > >>>>>> > [nova] > >>>>>> > live_migration_events = True > >>>>>> > ------------------------------------------------ > >>>>>> > > >>>>>> > ----------------- nova.conf ----------------- > >>>>>> > [DEFAULT] > >>>>>> > vif_plugging_timeout = 600 > >>>>>> > vif_plugging_is_fatal = False > >>>>>> you should never run with this set to false in production. > >>>>>> it will break nova ability to detect if netroking is configured > >>>>>> when booting or migrating a vm. > >>>>>> we honestly should have remove this when we removed nova-networks > >>>>>> > debug = True > >>>>>> > > >>>>>> > [compute] > >>>>>> > live_migration_wait_for_vif_plug = True > >>>>>> > > >>>>>> > [workarounds] > >>>>>> > enable_qemu_monitor_announce_self = True > >>>>>> > > >>>>>> > ----------------- openvswitch_agent.ini----------------- > >>>>>> > [securitygroup] > >>>>>> > firewall_driver = openvswitch > >>>>>> > [ovs] > >>>>>> > openflow_processed_per_port = true > >>>>>> > > >>>>>> > I check nova, neutron, ops logs but they are ok. > >>>>>> > > >>>>>> > Thank you. > >>>>>> > > >>>>>> > > >>>>>> > Nguyen Huu Khoi > >>>>>> > >>>>>> > -- Slawek Kaplonski Principal Software Engineer Red Hat -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: This is a digitally signed message part. URL: From katonalala at gmail.com Mon Jul 31 09:37:30 2023 From: katonalala at gmail.com (Lajos Katona) Date: Mon, 31 Jul 2023 11:37:30 +0200 Subject: Tap as a service on openstack with Juju and MaaS In-Reply-To: References: Message-ID: Hi, As I know TaaS is the only port mirroring service in Openstack for OVS and with some basic functionality for sriov. However as in recent years it has a disturbed history it has no real deployment support. I would start with checking what is done by devstack ( https://opendev.org/openstack/tap-as-a-service/src/branch/master/devstack ). If you need any help I am happy to help even with reviews if you have reviews in some charms or ansible playbooks. Lajos (lajoskatona) Jean-Bernard Altidor ezt ?rta (id?pont: 2023. j?l. 27., Cs, 15:41): > Hello, > I've been struggling to configure tap as a service on openstack for a few > months now. I can't find any guide or anything about it. Can you suggest a > method of port mirroring? I've even via the ovs bridge but I really need a > way to mirror data and send it to an instance for analysis as I'm working > on an IDS. > Thanks in advance for any ideas ! > JB > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nguyenhuukhoinw at gmail.com Mon Jul 31 11:05:14 2023 From: nguyenhuukhoinw at gmail.com (=?UTF-8?B?Tmd1eeG7hW4gSOG7r3UgS2jDtGk=?=) Date: Mon, 31 Jul 2023 18:05:14 +0700 Subject: [openstack][neutron[nova][kolla-ansible]instance cannot ping after live migrate In-Reply-To: <62631251.QqMBZyFNYO@p1> References: <62631251.QqMBZyFNYO@p1> Message-ID: Dear Slawek, I see that OVN still not supported some qos features https://docs.openstack.org/neutron/latest/admin/config-qos.html Am I correct? Will we have these future in the future? Nguyen Huu Khoi On Mon, Jul 31, 2023 at 2:12?PM Slawek Kaplonski wrote: > Hi, > > Dnia niedziela, 30 lipca 2023 17:00:22 CEST Nguy?n H?u Kh?i pisze: > > > Hello. > > > Is it ok if we use ovs with native firewall driver which I mean don't use > > > ovn. How about migration from ovs to ovn. > > Regarding migration from ML2/OVS to ML2/OVN backend it's easier to do it > when You are using ML2/OVS with openvswitch (native) firewall driver as in > that case plugging of the VMs into br-int will be the same before and after > migration. > > > > > > Nguyen Huu Khoi > > > > > > > > > On Sun, Jul 30, 2023 at 8:26?AM Satish Patel > wrote: > > > > > > > iptables + linux bridge integration with OVS was very old and OVS ACL > was > > > > not mature enough in earlier days. But nowadays OVN supports OVS base > ACL > > > > and that means it's much more stable. > > I'm not sure but I think there are some mixed things here. Generally in > Neutron we have "backends" like ML2/OVS (neutron-openvswitch-agent) or > ML2/OVN (with ovn-controller running on compute nodes). There are more > backends like ML2/Linuxbridge for example but lets not include them here > and focus only on ML2/OVS and ML2/OVN as those were mentioned. > > Now, regarding firewall drivers, in ML2/OVS backend, > neutron-openvswitch-agent can use one of the following firewall drivers: > > * iptables_hybrid - that's the one mentioned by Satish Patel as this "very > old" solution. Indeed it is using linuxbridge between VM and br-int to > implement iptables rule which will work on this linuxbridge for the > instance, > > * openvswitch - this is newer firewall driver, where all SG rules are > implemented on the host as OpenFlow rules in br-int. In this case VM is > plugged directly to the br-int. But this isn't related to the OVN ACLs in > any way. It's all implemented in the neutron-openvswitch-agent code. > Details about it are in the: > https://docs.openstack.org/neutron/latest/admin/config-ovsfwdriver.html > > In ML2/OVN backend there is only one implementation of the Security Groups > and this is based on the OVN ACL mechanism. In this case of course there is > also no need to use any Linuxbridge between VM and br-int so VM is plugged > directly into br-int. > > > > > > > > On Sat, Jul 29, 2023 at 10:29?AM Nguy?n H?u Kh?i < > > > > nguyenhuukhoinw at gmail.com> wrote: > > > > > > > >> Hello. > > > >> I just known about ops firewall last week. I am going to compare > > > >> between them. > > > >> Could you share some experience about why ovs firewall driver over > > > >> iptables. > > > >> Thank you. > > > >> Nguyen Huu Khoi > > > >> > > > >> > > > >> On Sat, Jul 29, 2023 at 5:55?PM Satish Patel > > > >> wrote: > > > >> > > > >>> Why are you not using openvswitch flow based firewall instead of > > > >>> Linuxbridge which will add hops in packet path. > > > >>> > > > >>> Sent from my iPhone > > > >>> > > > >>> On Jul 27, 2023, at 12:25 PM, Nguy?n H?u Kh?i < > nguyenhuukhoinw at gmail.com> > > > >>> wrote: > > > >>> > > > >>> ? > > > >>> Hello. > > > >>> I figured out that my rabbitmq queues are corrupt so neutron port > cannot > > > >>> upgrade security rules. I need delete queues so I can migrate without > > > >>> problem. > > > >>> > > > >>> Thank you so much for replying to me. > > > >>> > > > >>> On Thu, Jul 27, 2023, 8:11 AM Nguy?n H?u Kh?i < > nguyenhuukhoinw at gmail.com> > > > >>> wrote: > > > >>> > > > >>>> Hello. > > > >>>> > > > >>>> When my instances was migrated to other computes. I check on dest > host > > > >>>> and I see that > > > >>>> > > > >>>> -A neutron-openvswi-i41ec1d15-e -d x.x.x.x(my instance ip)/32 -p > udp -m > > > >>>> udp --sport 67 --dport 68 -j RETURN missing and my instance cannot > get IP. > > > >>>> I must restart neutron_openvswitch_agent then this rule appears and > I can > > > >>>> touch the instance via network. > > > >>>> > > > >>>> I use openswitch and provider networks. This problem has happened > this > > > >>>> week. after the system was upgraded from xena to yoga and I enabled > quorum > > > >>>> queue. > > > >>>> > > > >>>> > > > >>>> > > > >>>> Nguyen Huu Khoi > > > >>>> > > > >>>> > > > >>>> On Wed, Jul 26, 2023 at 5:28?PM Nguy?n H?u Kh?i < > > > >>>> nguyenhuukhoinw at gmail.com> wrote: > > > >>>> > > > >>>>> Because I dont see any error logs. Althought, i set debug log to > on. > > > >>>>> > > > >>>>> Your advices are very helpful to me. I will try to dig deeply. I am > > > >>>>> lost so some suggests are the best way for me to continue. :) > > > >>>>> > > > >>>>> On Wed, Jul 26, 2023, 4:39 PM wrote: > > > >>>>> > > > >>>>>> On Wed, 2023-07-26 at 07:49 +0700, Nguy?n H?u Kh?i wrote: > > > >>>>>> > Hello guys. > > > >>>>>> > > > > >>>>>> > I am using openstack yoga with kolla ansible. > > > >>>>>> without logs of some kind i dont think anyoen will be able to hlep > > > >>>>>> you with this. > > > >>>>>> you have one issue with the config which i noted inline but that > > > >>>>>> should not break live migration. > > > >>>>>> but it would allow it to proceed when otherwise it would have > failed. > > > >>>>>> and it woudl allow this issue to happen instead of the vm goign to > > > >>>>>> error ro the migration > > > >>>>>> being aborted in pre live migrate. > > > >>>>>> > > > > >>>>>> > When I migrate: > > > >>>>>> > > > > >>>>>> > instance1 from host A to host B after that I cannot ping this > > > >>>>>> > instance(telnet also). I must restart neutron_openvswitch_agent > or > > > >>>>>> move > > > >>>>>> > this instance back to host B then this problem has gone. > > > >>>>>> > > > > >>>>>> > this is my settings: > > > >>>>>> > > > > >>>>>> > ----------------- neutron.conf ----------------- > > > >>>>>> > [nova] > > > >>>>>> > live_migration_events = True > > > >>>>>> > ------------------------------------------------ > > > >>>>>> > > > > >>>>>> > ----------------- nova.conf ----------------- > > > >>>>>> > [DEFAULT] > > > >>>>>> > vif_plugging_timeout = 600 > > > >>>>>> > vif_plugging_is_fatal = False > > > >>>>>> you should never run with this set to false in production. > > > >>>>>> it will break nova ability to detect if netroking is configured > > > >>>>>> when booting or migrating a vm. > > > >>>>>> we honestly should have remove this when we removed nova-networks > > > >>>>>> > debug = True > > > >>>>>> > > > > >>>>>> > [compute] > > > >>>>>> > live_migration_wait_for_vif_plug = True > > > >>>>>> > > > > >>>>>> > [workarounds] > > > >>>>>> > enable_qemu_monitor_announce_self = True > > > >>>>>> > > > > >>>>>> > ----------------- openvswitch_agent.ini----------------- > > > >>>>>> > [securitygroup] > > > >>>>>> > firewall_driver = openvswitch > > > >>>>>> > [ovs] > > > >>>>>> > openflow_processed_per_port = true > > > >>>>>> > > > > >>>>>> > I check nova, neutron, ops logs but they are ok. > > > >>>>>> > > > > >>>>>> > Thank you. > > > >>>>>> > > > > >>>>>> > > > > >>>>>> > Nguyen Huu Khoi > > > >>>>>> > > > >>>>>> > > > > > > -- > > Slawek Kaplonski > > Principal Software Engineer > > Red Hat > -------------- next part -------------- An HTML attachment was scrubbed... URL: From satish.txt at gmail.com Mon Jul 31 13:12:43 2023 From: satish.txt at gmail.com (Satish Patel) Date: Mon, 31 Jul 2023 09:12:43 -0400 Subject: [openstack][neutron[nova][kolla-ansible]instance cannot ping after live migrate In-Reply-To: <62631251.QqMBZyFNYO@p1> References: <62631251.QqMBZyFNYO@p1> Message-ID: Hi Slawek, You are suggesting not to use an OVS base native firewall? On Mon, Jul 31, 2023 at 3:12?AM Slawek Kaplonski wrote: > Hi, > > Dnia niedziela, 30 lipca 2023 17:00:22 CEST Nguy?n H?u Kh?i pisze: > > > Hello. > > > Is it ok if we use ovs with native firewall driver which I mean don't use > > > ovn. How about migration from ovs to ovn. > > Regarding migration from ML2/OVS to ML2/OVN backend it's easier to do it > when You are using ML2/OVS with openvswitch (native) firewall driver as in > that case plugging of the VMs into br-int will be the same before and after > migration. > > > > > > Nguyen Huu Khoi > > > > > > > > > On Sun, Jul 30, 2023 at 8:26?AM Satish Patel > wrote: > > > > > > > iptables + linux bridge integration with OVS was very old and OVS ACL > was > > > > not mature enough in earlier days. But nowadays OVN supports OVS base > ACL > > > > and that means it's much more stable. > > I'm not sure but I think there are some mixed things here. Generally in > Neutron we have "backends" like ML2/OVS (neutron-openvswitch-agent) or > ML2/OVN (with ovn-controller running on compute nodes). There are more > backends like ML2/Linuxbridge for example but lets not include them here > and focus only on ML2/OVS and ML2/OVN as those were mentioned. > > Now, regarding firewall drivers, in ML2/OVS backend, > neutron-openvswitch-agent can use one of the following firewall drivers: > > * iptables_hybrid - that's the one mentioned by Satish Patel as this "very > old" solution. Indeed it is using linuxbridge between VM and br-int to > implement iptables rule which will work on this linuxbridge for the > instance, > > * openvswitch - this is newer firewall driver, where all SG rules are > implemented on the host as OpenFlow rules in br-int. In this case VM is > plugged directly to the br-int. But this isn't related to the OVN ACLs in > any way. It's all implemented in the neutron-openvswitch-agent code. > Details about it are in the: > https://docs.openstack.org/neutron/latest/admin/config-ovsfwdriver.html > > In ML2/OVN backend there is only one implementation of the Security Groups > and this is based on the OVN ACL mechanism. In this case of course there is > also no need to use any Linuxbridge between VM and br-int so VM is plugged > directly into br-int. > > > > > > > > On Sat, Jul 29, 2023 at 10:29?AM Nguy?n H?u Kh?i < > > > > nguyenhuukhoinw at gmail.com> wrote: > > > > > > > >> Hello. > > > >> I just known about ops firewall last week. I am going to compare > > > >> between them. > > > >> Could you share some experience about why ovs firewall driver over > > > >> iptables. > > > >> Thank you. > > > >> Nguyen Huu Khoi > > > >> > > > >> > > > >> On Sat, Jul 29, 2023 at 5:55?PM Satish Patel > > > >> wrote: > > > >> > > > >>> Why are you not using openvswitch flow based firewall instead of > > > >>> Linuxbridge which will add hops in packet path. > > > >>> > > > >>> Sent from my iPhone > > > >>> > > > >>> On Jul 27, 2023, at 12:25 PM, Nguy?n H?u Kh?i < > nguyenhuukhoinw at gmail.com> > > > >>> wrote: > > > >>> > > > >>> ? > > > >>> Hello. > > > >>> I figured out that my rabbitmq queues are corrupt so neutron port > cannot > > > >>> upgrade security rules. I need delete queues so I can migrate without > > > >>> problem. > > > >>> > > > >>> Thank you so much for replying to me. > > > >>> > > > >>> On Thu, Jul 27, 2023, 8:11 AM Nguy?n H?u Kh?i < > nguyenhuukhoinw at gmail.com> > > > >>> wrote: > > > >>> > > > >>>> Hello. > > > >>>> > > > >>>> When my instances was migrated to other computes. I check on dest > host > > > >>>> and I see that > > > >>>> > > > >>>> -A neutron-openvswi-i41ec1d15-e -d x.x.x.x(my instance ip)/32 -p > udp -m > > > >>>> udp --sport 67 --dport 68 -j RETURN missing and my instance cannot > get IP. > > > >>>> I must restart neutron_openvswitch_agent then this rule appears and > I can > > > >>>> touch the instance via network. > > > >>>> > > > >>>> I use openswitch and provider networks. This problem has happened > this > > > >>>> week. after the system was upgraded from xena to yoga and I enabled > quorum > > > >>>> queue. > > > >>>> > > > >>>> > > > >>>> > > > >>>> Nguyen Huu Khoi > > > >>>> > > > >>>> > > > >>>> On Wed, Jul 26, 2023 at 5:28?PM Nguy?n H?u Kh?i < > > > >>>> nguyenhuukhoinw at gmail.com> wrote: > > > >>>> > > > >>>>> Because I dont see any error logs. Althought, i set debug log to > on. > > > >>>>> > > > >>>>> Your advices are very helpful to me. I will try to dig deeply. I am > > > >>>>> lost so some suggests are the best way for me to continue. :) > > > >>>>> > > > >>>>> On Wed, Jul 26, 2023, 4:39 PM wrote: > > > >>>>> > > > >>>>>> On Wed, 2023-07-26 at 07:49 +0700, Nguy?n H?u Kh?i wrote: > > > >>>>>> > Hello guys. > > > >>>>>> > > > > >>>>>> > I am using openstack yoga with kolla ansible. > > > >>>>>> without logs of some kind i dont think anyoen will be able to hlep > > > >>>>>> you with this. > > > >>>>>> you have one issue with the config which i noted inline but that > > > >>>>>> should not break live migration. > > > >>>>>> but it would allow it to proceed when otherwise it would have > failed. > > > >>>>>> and it woudl allow this issue to happen instead of the vm goign to > > > >>>>>> error ro the migration > > > >>>>>> being aborted in pre live migrate. > > > >>>>>> > > > > >>>>>> > When I migrate: > > > >>>>>> > > > > >>>>>> > instance1 from host A to host B after that I cannot ping this > > > >>>>>> > instance(telnet also). I must restart neutron_openvswitch_agent > or > > > >>>>>> move > > > >>>>>> > this instance back to host B then this problem has gone. > > > >>>>>> > > > > >>>>>> > this is my settings: > > > >>>>>> > > > > >>>>>> > ----------------- neutron.conf ----------------- > > > >>>>>> > [nova] > > > >>>>>> > live_migration_events = True > > > >>>>>> > ------------------------------------------------ > > > >>>>>> > > > > >>>>>> > ----------------- nova.conf ----------------- > > > >>>>>> > [DEFAULT] > > > >>>>>> > vif_plugging_timeout = 600 > > > >>>>>> > vif_plugging_is_fatal = False > > > >>>>>> you should never run with this set to false in production. > > > >>>>>> it will break nova ability to detect if netroking is configured > > > >>>>>> when booting or migrating a vm. > > > >>>>>> we honestly should have remove this when we removed nova-networks > > > >>>>>> > debug = True > > > >>>>>> > > > > >>>>>> > [compute] > > > >>>>>> > live_migration_wait_for_vif_plug = True > > > >>>>>> > > > > >>>>>> > [workarounds] > > > >>>>>> > enable_qemu_monitor_announce_self = True > > > >>>>>> > > > > >>>>>> > ----------------- openvswitch_agent.ini----------------- > > > >>>>>> > [securitygroup] > > > >>>>>> > firewall_driver = openvswitch > > > >>>>>> > [ovs] > > > >>>>>> > openflow_processed_per_port = true > > > >>>>>> > > > > >>>>>> > I check nova, neutron, ops logs but they are ok. > > > >>>>>> > > > > >>>>>> > Thank you. > > > >>>>>> > > > > >>>>>> > > > > >>>>>> > Nguyen Huu Khoi > > > >>>>>> > > > >>>>>> > > > > > > -- > > Slawek Kaplonski > > Principal Software Engineer > > Red Hat > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bence.romsics at gmail.com Mon Jul 31 14:30:48 2023 From: bence.romsics at gmail.com (Bence Romsics) Date: Mon, 31 Jul 2023 16:30:48 +0200 Subject: [neutron] bug deputy report for week of 2023-07-24 Message-ID: Hi All, We had the following new bugs in the last week or so: Unassigned: * https://bugs.launchpad.net/neutron/+bug/2028651 [ovn] IPv6 VIPs broken with ML2/OVN * https://bugs.launchpad.net/neutron/+bug/2028442 Support DNS for ovn_{nb,sb}_connection * https://bugs.launchpad.net/neutron/+bug/2028285 [unit test][xena+] test_port_deletion_prevention fails when runs in isolation Likely valid and high, but my dvr-fu is weak. dvr folks, please have a look: * https://bugs.launchpad.net/neutron/+bug/2028795 Restarting OVS with DVR creates a network loop fix proposed: https://review.opendev.org/c/openstack/neutron/+/889752 High: * https://bugs.launchpad.net/neutron/+bug/2028185 Large number of FIPs causes slow sync_routers response fix proposed: https://review.opendev.org/c/openstack/neutron/+/888915 * https://bugs.launchpad.net/neutron/+bug/2028846 FIP PF don't works with vlan tenant network and ovn backend fix proposed: https://review.opendev.org/c/openstack/neutron/+/889871 Medium: * https://bugs.launchpad.net/neutron/+bug/2028161 [ovn-octavia-provider] FIP not included into LogicalSwitchPortUpdate event handler method fix merged on master, backports proposed: https://review.opendev.org/q/Ibee3906e8e9575fba7811e989e3e111a026ce45b Low: * https://bugs.launchpad.net/neutron/+bug/2028724 neutron should forbid configuring agent_down_time that is known to crash due to CPython epoll limitation fix proposed: https://review.opendev.org/c/openstack/neutron/+/889373 RFE: * https://bugs.launchpad.net/neutron/+bug/2028660 [fwaas][rfe] support list type of port_range for firewall rule Incomplete: * https://bugs.launchpad.net/neutron/+bug/2028338 Router with l3_agent state unknown Invalid: * https://bugs.launchpad.net/neutron/+bug/2028544 dhcp agent binding count greather than dhcp_agents_per_network Cheers, Bence (rubasov) From ddanelski at cloudferro.com Mon Jul 31 18:11:56 2023 From: ddanelski at cloudferro.com (Dominik Danelski) Date: Mon, 31 Jul 2023 18:11:56 +0000 Subject: [nova] Scheduler Optimiser In-Reply-To: <4ca79d6072090cf0aec8ebc9145e5875a0fae699.camel@redhat.com> References: <09981137-ce97-df6b-55f9-038df7f10498@cloudferro.com> <4ca79d6072090cf0aec8ebc9145e5875a0fae699.camel@redhat.com> Message-ID: <42f2dfe3-a3c4-30eb-0301-069185bc9e6c@cloudferro.com> @Alvaro Soto The 2 main motivations are: 1. By default, the VMs are allocated evenly, so we sometimes lack a space for very large VMs. Some less balanced system with "reservations" for them like "fill this remaining space with a smaller VM only if no other option is available" could be designed. 2. Some patterns like one customer ordering dozens of identical VMs in a row can be observed. Maybe knowledge about them could be somehow used to handle them better. Currently, there are no very concrete ideas on how to follow, besides what I wrote above. We expect to tinker with the settings a bit at first with these ideas in mind and by iterative research hopefully some more optimal settings or filters could be created. @Sean Thank you very much; that is a lot of knowledge, which we will definitely use as a starting point. It's hard for me to comment more about that at this point, but I'm grateful and will keep these hints in mind when we get to the experimentation phase. Kind regards Dominik Danelski On 4.07.2023 13:08, smooney at redhat.com wrote: > not to bias either approach but just to ensure you both understand the current > recommendation. when configuring the scheduler there are some rules fo tumb that shoudl be followed. > > first if it can be done by both a scheduler filter and placement then placement should be faster. > when specifing the filters operators a encouraged to order them following a psudo id3 algorithm approch > in other words always put the filters that elimiate the most host first i.e aggreate* filters then simple > host specific filters (num instances) then complex host spefific filters liek the numa toplogy filter. > finally if a filter is not useful for your cloud(i.e. you have not pci devices for passthough) you can remove it from > the enabled filters list > > for the weighers all weigher are enabled by default and all enabled filters run on each host that passed > the filtes. so there is no ordering to consider with weighers but if you dont need them for your cloud just > like the filters you can remove them form the enabled set. > > for placement there are only 2 things you can really tweak, the max number of results and randomisation of allcoation > candiates. > https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.max_placement_results > https://docs.openstack.org/placement/latest/configuration/config.html#placement.randomize_allocation_candidates > > i know cern set the max_placement_results very low. i.e <20 the default is 1000 but most dont modify this. > i generally think settign randomize_allocation_candidates is good espcially if you reduce the max_placement_results > > as noted there are quite a few flags to delegate filters to placemetn lik > https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.query_placement_for_routed_network_aggregates > https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.limit_tenants_to_placement_aggregate > https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.placement_aggregate_required_for_tenants > https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.query_placement_for_availability_zone > https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.query_placement_for_image_type_support > https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.image_metadata_prefilter > > > these often replace filteres like the az filter with a more effeicnt placement query so they should generally be prefer > however they dotn always have exactly the same behavior so sometimes you cant just swap form one to anohter if you > rely on the semantics of the filter that are changed. > > there are some options like > https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.host_subset_size > https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.shuffle_best_same_weighed_hosts > > that allow you to add a little bit of additional randomness to the host selection which may help with spreading/packing > or overall better utilisation of host but it should not really affect performance of the schduelr. > i generally recommend setting shuffle_best_same_weighed_hosts=true but leaving host_subset_size at its default of 1 > that way you only get randomness if there are multiple equally good options. > > by default the scheduler should be determinist today if you do not adjust either host_subset_size or > shuffle_best_same_weighed_hosts > > i don't know if that will help either of your research but those are the current best pratices for cofniguring the > scheduler. > > regards > sean. > > On Mon, 2023-07-03 at 14:08 -0600, Alvaro Soto wrote: >> Nice idea, but my main question is, how do you plan to beat the ones >> implemented currently? >> >> I'm working a little researching a little with some techniques to try to >> beat the random resource allocation schedulers. >> >> Can you share more about your research and/or implementation idea? >> >> Cheers. >> >> --- >> Alvaro Soto. >> >> Note: My work hours may not be your work hours. Please do not feel the need >> to respond during a time that is not convenient for you. >> ---------------------------------------------------------- >> Great people talk about ideas, >> ordinary people talk about things, >> small people talk... about other people. >> >> On Mon, Jul 3, 2023, 2:00 PM Dominik Danelski >> wrote: >> >>> Hello, >>> >>> >>> I would like to introduce you to the tool developed under the working >>> title "Scheduler Optimiser". It is meant to test the effectiveness of >>> different Scheduler configurations, both weights and filters on a given >>> list of VM orders and in a semi-realistic infrastructure. >>> >>> My company - CloudFerro - has been preparing in-house for the last few >>> months and foresees publishing the project as FOSS once it reaches the >>> MVP stage. To make the final result more useful to the community and >>> speed up the development (and release), I humbly ask for your expertise: >>> Are you aware of previous similar efforts? Do you notice some flaws in >>> the current approach? What, in your opinion, are more important aspects >>> of the infrastructure behaviour, and what can be relatively safely >>> ignored in terms of the effect on Scheduler results/allocation? >>> >>> >>> Project objectives: >>> >>> ? * Use Devstack (or another OpenStack deployer) with a real Scheduler >>> ??? to replay a list of compute VM orders, either real from one's >>> ??? infrastructure or artificially created. >>> ? * Assess the effectiveness of the scheduling in various terms like: >>> ??? "How many machines of a given type can still be allocated at the >>> ??? moment?" using plug-in "success meters". In a strict sense, the >>> ??? project does not simulate THE Scheduler but interacts with it. >>> ? * Use fake-virt to emulate huge architectures on a relatively tiny >>> ??? test bench. >>> ? * Have as little as possible, and ideally no changes to the Devstack's >>> ??? code that could not be included in the upstream repository. The >>> ??? usage should be as simple as: 1. Install Devstack. 2. Configure >>> ??? Devstack's cluster with its infrastructure information like flavours >>> ??? and hosts. 3. Configure Scheduler for a new test case. 4. Replay VM >>> ??? orders. 5. Repeat steps 3 and 4 to find better Scheduler settings. >>> ? * Facilitate creating a minimal required setup of the test bench. Not >>> ??? by replacing standard Devstack scripts, but mainly through tooling >>> ??? for quick rebuilding data like flavours, infrastructure state, and >>> ??? other factors relevant to the simulation. >>> >>> >>> Outside of the scope: >>> >>> ? * Running continuous analysis on the production environment, even if >>> ??? some plug-ins could be extracted for this purpose. >>> ? * Retaining information about users and projects when replaying orders. >>> ? * (Probably / low priority) replaying actions other than VM >>> ??? creation/deletion as they form a minority of operations and ignoring >>> ??? them should not have a distinct effect on the comparison experiments. >>> >>> >>> Current state: >>> >>> ??? Implemented: >>> >>> ? * Recreating flavours from JSON file exported via OpenStack CLI. >>> ? * Replaying a list of orders in the form of (creation_date, >>> ??? termination_date, resource_id (optional), flavor_id) with basic >>> ??? flavour properties like VCPU, RAM, and DISK GB. The orders are >>> ??? replayed consecutively. >>> ? * Plug-in success-rater mechanism which runs rater classes (returning >>> ??? quantified success measure) after each VM add/delete action, retains >>> ??? their intermediate history and "total success" - how it is defined >>> ??? is implementation dependent. First classes interacting with >>> ??? Placement like: "How many VMs of flavours x (with basic parameters >>> ??? for now) can fit in the cluster?" or "How many hosts are empty?". >>> >>> >>> Missing: >>> >>> ? * Recreating hosts, note the fake-virt remark from "Risks and >>> Challenges". >>> ? * Tools facilitating Scheduler configuration. >>> ? * Creating VMs with more parameters like VGPU, traits, and aggregates. >>> ? * (Lower priority) saving the intermediate state of the cluster during >>> ??? simulation i.e. allocations to analyse it without rerunning the >>> ??? experiment. Currently, only the quantified meters are saved. >>> ? * Gently failing and saving all information in case of resource >>> ??? depletion: close to completion, handling one exception type in upper >>> ??? layers is needed. >>> ? * More success meters. >>> >>> >>> Risks and Challenges: >>> >>> ? * Currently, the tool replays actions one by one, it waits for each >>> ??? creation and deletion to be complete before running success raters >>> ??? and taking another order. Thus, the order of actions is important, >>> ??? but not their absolute time and temporal density. This might skip >>> ??? some side-effects of a realistic execution. >>> ? * Similarly, to the above, fake-virt provides simple classes that will >>> ??? not reproduce some behaviours of real-world hypervisors. An explicit >>> ??? Scheduler avoids hosts that had recently failed to allocate a VM, >>> ??? but most likely fake-virt will not mock such behaviour. >>> ? * Fake-virt should reproduce a real diverse infrastructure instead of >>> ??? x copies of the same flavour. This might be the only, but very >>> ??? important change to the OpenStack codebase. If successful, it could >>> ??? benefit other projects and tests as well. >>> >>> >>> Even though the list of missing features is seemingly larger, the most >>> important parts of the program are already there, so we hope to finish >>> the MVP development in a relatively short amount of time. We are going >>> to publish it as FOSS in either case, but as mentioned your observations >>> would be very much welcome at this stage. I am also open to answering >>> more questions about the project. >>> >>> >>> Kind regards >>> >>> Dominik Danelski >>> >>> From allison at openinfra.dev Mon Jul 31 19:17:07 2023 From: allison at openinfra.dev (Allison Price) Date: Mon, 31 Jul 2023 14:17:07 -0500 Subject: [ptls][tc] OpenStack User Survey Updates Message-ID: <8B38E573-BF3C-4816-B07F-48CEA3645256@openinfra.dev> Hi Everyone, Like Helena mentioned last week, we are closing the 2023 OpenStack User Survey in a few weeks and will then open the 2024 OpenStack User Survey. At this time, we want to offer the project teams and TC the opportunity to update your project specific questions that appear at the end of the survey. As a reminder, for the project-specific questions, these appear if a survey taker selects your project in their deployment and TC questions appear to all survey takers. If you and your team would like to update the question, please let me know by Friday, August 18. I know that this is a holiday time for many, so if any team needs some extra time, just let me know. I am also able to share the existing questions with any team that needs a refresher on what is currently in the survey. In the meantime, please continue to promote openstack.org/usersurvey to anyone (and everyone!) you know who is running OpenStack or planning to in the future. We want to get as much feedback as we possibly can. Cheers, Allison -------------- next part -------------- An HTML attachment was scrubbed... URL: From altidorjb at gmail.com Mon Jul 31 22:34:20 2023 From: altidorjb at gmail.com (Jean-Bernard Altidor) Date: Mon, 31 Jul 2023 18:34:20 -0400 Subject: Tap as a service on openstack with Juju and MaaS In-Reply-To: References: Message-ID: Hello Lajos, Thanks for the quick reply. As I couldn't make it work with juju and MaaS, I did a test with DevStack and still my Tap Services and flows stay down. I followed this : https://opendev.org/openstack/tap-as-a-service/src/branch/master/devstack and read your doc that's in review : https://review.opendev.org/c/openstack/tap-as-a-service/+/828382. I also checked that the service plugin was actually added in the neutron.conf. JB On Mon, Jul 31, 2023 at 5:37?AM Lajos Katona wrote: > Hi, > As I know TaaS is the only port mirroring service in Openstack for OVS and > with some basic functionality for sriov. > However as in recent years it has a disturbed history it has no real > deployment support. > I would start with checking what is done by devstack ( > https://opendev.org/openstack/tap-as-a-service/src/branch/master/devstack > ). > If you need any help I am happy to help even with reviews if you have > reviews in some charms or ansible playbooks. > > Lajos (lajoskatona) > > Jean-Bernard Altidor ezt ?rta (id?pont: 2023. j?l. > 27., Cs, 15:41): > >> Hello, >> I've been struggling to configure tap as a service on openstack for a few >> months now. I can't find any guide or anything about it. Can you suggest a >> method of port mirroring? I've even via the ovs bridge but I really need a >> way to mirror data and send it to an instance for analysis as I'm working >> on an IDS. >> Thanks in advance for any ideas ! >> JB >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: