[neutron] unmanaged router resources - OVN interconnect
Rodolfo Alonso Hernandez
ralonsoh at redhat.com
Mon Jul 17 14:30:07 UTC 2023
Ok, I'll ask again for the tool/script/manual used to make the IC
connection. I will also ask again for the NB-IC database resources and the
NB resources needed, and how are related.
If you want Neutron to handle these resources, then you don't need anything
special: you can add an external network, handled only by one project.
Inside this project you can create the needed resources: networks, routers,
etc. This project/user (controlled by the sysadmin), won't create any VM on
these networks/routers. Then you can use these resources to connect the TS.
On Mon, Jul 17, 2023 at 3:10 PM Roberto Bartzen Acosta <
roberto.acosta at luizalabs.com> wrote:
> Hi Felix and Rodolfo,
>
>
> Em seg., 17 de jul. de 2023 às 05:39, Rodolfo Alonso Hernandez <
> ralonsoh at redhat.com> escreveu:
>
>> Hello Felix:
>>
>> That's the point: the launchpad bug describes the registers created by
>> the IC configuration but these registers do not have any identification
>> that informs about their relationship with the IC TS.
>>
>> Rather than modifying the Neutron resources (a network is a generic
>> resource that is used by multiple backends), I would try to create a NB-IC
>> resource map and the corresponding NB registers (NAT, LRP and LRSR). If you
>> can link a NB-IC TS with the corresponding NB resources, then the problem
>> is fixed because knowing them we can explicitly skip them from the DB sync
>> tool.
>>
>> If that is not the case, then please present an alternative with the
>> suggested idea of creating an external network type.
>>
>> BTW, I don't understand why you first say that "created by tools not
>> necessarily controlled by us we need to avoid changing them", but then you
>> are talking about refactoring these tools. Are we talking about the tools
>> that create these IC configurations? Can you share them?
>>
>>
> I think we are talking about two different cases.
>
> 1 - This proposed RFE intends to solve the problem of removing external
> resources and that is enough for the OVN-IC to work in generic scenarios:
> multi tenancy, single external provider, multiple external providers, etc.
> because the OVN-IC will not relate to Neutron's network topology or
> capabilities.
>
> 2- Felix' suggestion is related to integrating the TS to a network on
> Neutron (some kind of mapping), and in this case, we have a hybrid solution
> with the TS managed by an external tool, and the management of the NAT, LRP
> / TS link with the tenant being done directly on Neutron. For this
> proposal, I believe that the port filter would not be necessary because
> Neutron would know the resource mappings in the OVN-NB, but it seems to me
> a more complex solution.
>
> The question about case 2 is related to which level we are talking about
> modifying the generic network to map the TS. If it is a network provider
> level, the discussion is one - this case would solve the
> schedule/lrp-set-gateway-chassis problem as it would be a gateway port.
> but if it is a tenant' network level linked to a router / no gateway port
> (the discussion would be another) - does not solve the gateway port
> problem, but it seems closer to what the OVN-IC proposes to do at the
> network L3 interconnect level).
>
> BTW: in case the TS is connected to an external network, we would need an
> additional router for external/internet connectivity not related to the
> OVN-IC, or maybe create a router with multiple gateway ports (that doesn't
> exist yet)...
>
> Regards,
> Roberto
>
>
>
>> Regards.
>>
>> On Mon, Jul 17, 2023 at 8:43 AM Felix Huettner
>> <felix.huettner at mail.schwarz> wrote:
>>
>>> Hi everyone,
>>>
>>> On Thu, Jul 13, 2023 at 07:29:51PM -0300, Roberto Bartzen Acosta wrote:
>>> > Hi Rodolfo,
>>> >
>>> > Em qui., 13 de jul. de 2023 às 13:45, Rodolfo Alonso Hernandez <
>>> > ralonsoh at redhat.com> escreveu:
>>> >
>>> > > Hello Roberto:
>>> > >
>>> > > I think you have provided a very detailed explanation of the issue
>>> but you
>>> > > still didn't present any possible solution. I would ask you at least
>>> for a
>>> > > proposal and for specific details of what is failing in the OVN sync
>>> tool.
>>> > >
>>> > > For example, if you add a TS between two clouds, you'll need to
>>> create (1)
>>> > > routers (NAT registers), (2) router ports (LRP) and (3) some static
>>> routes
>>> > > (LRSR). All these elements are monitored by the DB sync tool and
>>> will fail
>>> > > because the counterparts in the Neutron DB don't exist. I guess that
>>> you
>>> > > have some script or tool or at least a document to manually add these
>>> > > resources; sharing it could help to find a solution.
>>> > >
>>> > > If these elements are manually created during the deployment phase,
>>> you
>>> > > can also control the information provided. I'm thinking about the
>>> > > "external_ids" field; we can push some defined constant that will
>>> make
>>> > > these registers transparent to the OVN DB sync tool.
>>>
>>> As these resources are also created by tools not necessarily controlled
>>> by us we need to avoid changing them.
>>> I would therefor propose to not add a "ignore" constant to the
>>> "external_ids", to to rather check for the existence of the neutron keys
>>> in "external_ids" (i think thats also what is described in the bug
>>> below).
>>>
>>>
> Thanks for the contribution Felix. The idea is not to add (or at least not
> expect it to be externally added) any identification in the external_ids,
> Neutron should only filter for DB consistency validation what it knows and
> has created by Neuron -> Neutron in the external ids key.
>
>
>> > >
>>> > > Please check this idea and if it is feasible. In that case, please
>>> add a
>>> > > topic to the Neutron drivers meeting agenda [1] to discuss it.
>>> > >
>>> >
>>> > That sounds good, thanks.
>>> > https://bugs.launchpad.net/neutron/+bug/2027742
>>>
>>> From my perspective there is also an additional step after the above
>>> REF. Currently it just makes it possible to use ovn-ic without neutron
>>> killing it.
>>> However if we could get neutron to treat a Transit Switch as a external
>>> network we would skip the need to implement the NAT, LRP and LRSR logic
>>> in custom tooling and could use the existing neutron code. This would
>>> probably require creating a custom type of provider network and then
>>> specifying the transit switch name in the provider options.
>>>
>>> Is that something that from your perspective we should take a look at
>>> right away, or rather first focus und neutron not burning the transit
>>> switch.
>>>
>>> Regards
>>> Felix
>>>
>>> >
>>> > Regards,
>>> > Roberto
>>> >
>>> >
>>> > >
>>> > > Regards.
>>> > >
>>> > > [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers
>>> > >
>>> > >
>>> > > On Wed, Jul 12, 2023 at 2:12 PM Roberto Bartzen Acosta <
>>> > > roberto.acosta at luizalabs.com> wrote:
>>> > >
>>> > >>
>>> > >>
>>> > >> Em qua., 12 de jul. de 2023 às 09:05, Roberto Bartzen Acosta <
>>> > >> roberto.acosta at luizalabs.com> escreveu:
>>> > >>
>>> > >>> Hi Rodolfo,
>>> > >>>
>>> > >>> Thanks for the feedback.
>>> > >>>
>>> > >>> I liked the idea of having a different network type only to hold
>>> the
>>> > >>> Transit Switches and thus the LRP but it depends on some factors
>>> like
>>> > >>> multi-tenancy.
>>> > >>> From the OVN perspective, the TS is global and will only be linked
>>> to a
>>> > >>> tenant when it is plugged into the tenant router port. My concern
>>> here is
>>> > >>> that if Neutron manages the TS, it will need to assume the dynamic
>>> > >>> characteristics of this TS function, ovn-ic creates and removes
>>> the TS in
>>> > >>> NB_Global, in addition to plugging and removing ports in this
>>> switch
>>> > >>> according to the network topology. Additionally, Neutron would
>>> still need
>>> > >>> to handle the learned remote routes (which are listed as static
>>> routes from
>>> > >>> the tenant router perspective).
>>> > >>>
>>> > >> sorry: NB_Global -> Northbound Database.
>>> > >>
>>> > >>
>>> > >>> This is an open discussion, Felix can add ideas here.
>>> > >>>
>>> > >>> In general, it seems to me that we have no alternatives for this
>>> type of
>>> > >>> solution other than OVN-IC (note that ovn-bgp-agent does not learn
>>> remote
>>> > >>> routes at the SDN level / OVN).
>>> > >>> OpenStack seems to be designed to run like a "bubble" and this
>>> traffic
>>> > >>> from one VM to another always needs to be routed at the FIP level
>>> and
>>> > >>> external routing layers.
>>> > >>>
>>> > >>> Regards,
>>> > >>> Roberto
>>> > >>>
>>> > >>> Em qua., 12 de jul. de 2023 às 05:11, Rodolfo Alonso Hernandez <
>>> > >>> ralonsoh at redhat.com> escreveu:
>>> > >>>
>>> > >>>> Hello Roberto:
>>> > >>>>
>>> > >>>> We talked about this possible RFE during the PTG [1] but I don't
>>> > >>>> remember having any proposal. Actually what I remember (and was
>>> written in
>>> > >>>> the etherpad) is that you were going to present an RFE. Can you
>>> explain it
>>> > >>>> briefly?
>>> > >>>>
>>> > >>>> We also talked about the idea, proposed by Felix in the mailing
>>> list,
>>> > >>>> of having a different network type only to hold the Transit
>>> Switches and
>>> > >>>> thus the LRP. If you have a proposal, please present it.
>>> > >>>>
>>> > >>>> Regards.
>>> > >>>>
>>> > >>>> [1]https://etherpad.opendev.org/p/neutron-bobcat-ptg#L506
>>> > >>>>
>>> > >>>> On Tue, Jul 11, 2023 at 10:50 PM Roberto Bartzen Acosta <
>>> > >>>> roberto.acosta at luizalabs.com> wrote:
>>> > >>>>
>>> > >>>>> Hello Neutron folks,
>>> > >>>>>
>>> > >>>>> Regarding the conversation started in March [1] about the use of
>>> OVN
>>> > >>>>> interconnect with Neutron, I would like to evolve the discussion
>>> in
>>> > >>>>> relation to resources allocated by OVN-IC and which are not
>>> managed by
>>> > >>>>> Neutron. In the March PTG [2], the problem related to the
>>> db_sync tool was
>>> > >>>>> presented, and a proposed solution in which Neutron does not
>>> manage these
>>> > >>>>> resources.
>>> > >>>>>
>>> > >>>>> After discussing some architectural designs with Felix/StackIT,
>>> it
>>> > >>>>> seems to make sense to come up with a proposal to change the
>>> mech_driver to
>>> > >>>>> validate the external_ids key and not remove resources present
>>> in the OVN
>>> > >>>>> backend without Neutron "signature".
>>> > >>>>>
>>> > >>>>> The discussion here is more complex than it seems, because
>>> > >>>>> architecturally Neutron does not allow us to interconnect
>>> workloads in
>>> > >>>>> multiple AZs (with different OpenStacks), but this could be a
>>> requirement
>>> > >>>>> for a high availability cloud solution (Cloud Region).
>>> Additionally, this
>>> > >>>>> OVN-IC solution allows interconnecting other cloud backends that
>>> use OVN,
>>> > >>>>> in the case of kubernetes with ovn-kube.
>>> > >>>>>
>>> > >>>>> We tested an OVN interconnect integrated with 3 OpenStack
>>> > >>>>> installations and it worked very well !!! considering direct L3
>>> traffic at
>>> > >>>>> the router level between different network infrastructures.
>>> > >>>>>
>>> > >>>>> SYNC_REPAIR - problem
>>> > >>>>>
>>> > >>>>> * Static Routes (learned OVN-IC routes)
>>> > >>>>> * Router Port -> Transit Switches
>>> > >>>>>
>>> > >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae
>>> > >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING
>>> > >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync
>>> > >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port
>>> found in
>>> > >>>>> OVN but not in Neutron, port_id=rt2-admin-tenant1
>>> > >>>>> Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae
>>> > >>>>> neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING
>>> > >>>>> neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync
>>> > >>>>> [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router
>>> > >>>>> 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes
>>> [{'destination': '
>>> > >>>>> 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '
>>> > >>>>> 10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not
>>> in
>>> > >>>>> Neutron
>>> > >>>>>
>>> > >>>>>
>>> > >>>>> Any suggestions on how to resolve this db_sync issue? since all
>>> other
>>> > >>>>> Neutron modules did not present any problem with OVN-IC (as far
>>> as I
>>> > >>>>> tested).
>>> > >>>>> I remember Terry was keen to suggest some things to help. I
>>> believe
>>> > >>>>> that before proposing some complex mechanism to solve this
>>> simple problem,
>>> > >>>>> I would like to hear the community opinion. In our use case, a
>>> bit change
>>> > >>>>> in db_sync with filter by neutron key in external_ids would be
>>> enough.
>>> > >>>>>
>>> > >>>>> Regards,
>>> > >>>>> Roberto
>>> > >>>>>
>>> > >>>>>
>>> > >>>>>
>>> > >>>>> [1]
>>> > >>>>>
>>> https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html
>>> > >>>>> [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg
>>> > >>>>>
>>> > >>>>>
>>> > >>>>>
>>> > >>>>> Additional logs:
>>> > >>>>>
>>> > >>>>> OpenStack 1
>>> > >>>>>
>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~#
>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~#
>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~#
>>> ovn-nbctl
>>> > >>>>> lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498
>>> > >>>>> IPv4 Routes
>>> > >>>>> Route Table <main>:
>>> > >>>>> 20.0.1.0/24 169.254.200.2 dst-ip
>>> (learned)
>>> > >>>>> 20.0.2.0/24 169.254.200.3 dst-ip
>>> (learned)
>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip
>>> > >>>>>
>>> > >>>>> IPv6 Routes
>>> > >>>>> Route Table <main>:
>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip
>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~#
>>> ovn-nbctl
>>> > >>>>> lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07
>>> > >>>>> IPv4 Routes
>>> > >>>>> Route Table <main>:
>>> > >>>>> 10.0.1.0/24 169.254.100.2 dst-ip
>>> (learned)
>>> > >>>>> 10.0.2.0/24 169.254.100.3 dst-ip
>>> (learned)
>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip
>>> > >>>>>
>>> > >>>>> IPv6 Routes
>>> > >>>>> Route Table <main>:
>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip
>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f931b37c:~#
>>> > >>>>>
>>> > >>>>>
>>> > >>>>> OpenStack 2
>>> > >>>>>
>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~#
>>> ovn-nbctl
>>> > >>>>> lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc
>>> > >>>>> IPv4 Routes
>>> > >>>>> Route Table <main>:
>>> > >>>>> 20.0.0.0/24 169.254.200.1 dst-ip
>>> (learned)
>>> > >>>>> 20.0.2.0/24 169.254.200.3 dst-ip
>>> (learned)
>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip
>>> > >>>>>
>>> > >>>>> IPv6 Routes
>>> > >>>>> Route Table <main>:
>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip
>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-30f7e935:~#
>>> ovn-nbctl
>>> > >>>>> lr-route-list ce45f681-6454-43fe-974f-81344bb8113a
>>> > >>>>> IPv4 Routes
>>> > >>>>> Route Table <main>:
>>> > >>>>> 10.0.0.0/24 169.254.100.1 dst-ip
>>> (learned)
>>> > >>>>> 10.0.2.0/24 169.254.100.3 dst-ip
>>> (learned)
>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip
>>> > >>>>>
>>> > >>>>> IPv6 Routes
>>> > >>>>> Route Table <main>:
>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip
>>> > >>>>>
>>> > >>>>>
>>> > >>>>>
>>> > >>>>> OpenStack 3
>>> > >>>>>
>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~#
>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~#
>>> ovn-nbctl
>>> > >>>>> lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3
>>> > >>>>> IPv4 Routes
>>> > >>>>> Route Table <main>:
>>> > >>>>> 20.0.0.0/24 169.254.200.1 dst-ip
>>> (learned)
>>> > >>>>> 20.0.1.0/24 169.254.200.2 dst-ip
>>> (learned)
>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip
>>> > >>>>>
>>> > >>>>> IPv6 Routes
>>> > >>>>> Route Table <main>:
>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip
>>> > >>>>> root at os-infra-1-neutron-ovn-northd-container-f237db97:~#
>>> ovn-nbctl
>>> > >>>>> lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0
>>> > >>>>> IPv4 Routes
>>> > >>>>> Route Table <main>:
>>> > >>>>> 10.0.0.0/24 169.254.100.1 dst-ip
>>> (learned)
>>> > >>>>> 10.0.1.0/24 169.254.100.2 dst-ip
>>> (learned)
>>> > >>>>> 0.0.0.0/0 200.200.200.1 dst-ip
>>> > >>>>>
>>> > >>>>> IPv6 Routes
>>> > >>>>> Route Table <main>:
>>> > >>>>> ::/0 fc00:ca5a:ca5a:8000:: dst-ip
>>> > >>>>>
>>> > >>>>>
>>> > >>>>> OVN-IC Global database
>>> > >>>>>
>>> > >>>>> root at ovn-global-db1:~# ovn-ic-sbctl show
>>> > >>>>> availability-zone osp1
>>> > >>>>> gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5
>>> > >>>>> hostname: osp1-gwnode1
>>> > >>>>> type: geneve
>>> > >>>>> ip: 192.168.200.28
>>> > >>>>> port admin-rt1-tenant1
>>> > >>>>> transit switch: admin-tenant1
>>> > >>>>> address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24
>>> fe80::1/64"]
>>> > >>>>> port admin-rt1-tenant1_1
>>> > >>>>> transit switch: admin-tenant1_1
>>> > >>>>> address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"]
>>> > >>>>> availability-zone osp2
>>> > >>>>> gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8
>>> > >>>>> hostname: osp2-gwnode1
>>> > >>>>> type: geneve
>>> > >>>>> ip: 192.168.200.128
>>> > >>>>> port admin-rt2-tenant1
>>> > >>>>> transit switch: admin-tenant1
>>> > >>>>> address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24
>>> fe80::2/64"]
>>> > >>>>> port admin-rt2-tenant1_1
>>> > >>>>> transit switch: admin-tenant1_1
>>> > >>>>> address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"]
>>> > >>>>> availability-zone osp3
>>> > >>>>> gateway 97595af9-7896-40d0-a883-beadbff1aa5b
>>> > >>>>> hostname: osp3-gwnode1
>>> > >>>>> type: geneve
>>> > >>>>> ip: 192.168.200.228
>>> > >>>>> port admin-rt3-tenant1
>>> > >>>>> transit switch: admin-tenant1
>>> > >>>>> address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24
>>> fe80::3/64"]
>>> > >>>>> port admin-rt3-tenant1_1
>>> > >>>>> transit switch: admin-tenant1_1
>>> > >>>>> address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"]
>>> > >>>>>
>>> > >>>>>
>>> > >>>>>
>>> > >>>>>
>>> > >>>>>
>>> > >>>>>
>>> > >>>>>
>>> > >>>>> *‘Esta mensagem é direcionada apenas para os endereços
>>> constantes no
>>> > >>>>> cabeçalho inicial. Se você não está listado nos endereços
>>> constantes no
>>> > >>>>> cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo
>>> dessa
>>> > >>>>> mensagem e cuja cópia, encaminhamento e/ou execução das ações
>>> citadas estão
>>> > >>>>> imediatamente anuladas e proibidas’.*
>>> > >>>>>
>>> > >>>>> *‘Apesar do Magazine Luiza tomar todas as precauções razoáveis
>>> para
>>> > >>>>> assegurar que nenhum vírus esteja presente nesse e-mail, a
>>> empresa não
>>> > >>>>> poderá aceitar a responsabilidade por quaisquer perdas ou danos
>>> causados
>>> > >>>>> por esse e-mail ou por seus anexos’.*
>>> > >>>>>
>>> > >>>>
>>> > >>
>>> > >> *‘Esta mensagem é direcionada apenas para os endereços constantes no
>>> > >> cabeçalho inicial. Se você não está listado nos endereços
>>> constantes no
>>> > >> cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo
>>> dessa
>>> > >> mensagem e cuja cópia, encaminhamento e/ou execução das ações
>>> citadas estão
>>> > >> imediatamente anuladas e proibidas’.*
>>> > >>
>>> > >> *‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para
>>> > >> assegurar que nenhum vírus esteja presente nesse e-mail, a empresa
>>> não
>>> > >> poderá aceitar a responsabilidade por quaisquer perdas ou danos
>>> causados
>>> > >> por esse e-mail ou por seus anexos’.*
>>> > >>
>>> > >
>>> >
>>> > --
>>> >
>>> >
>>> >
>>> >
>>> > _‘Esta mensagem é direcionada apenas para os endereços constantes no
>>> > cabeçalho inicial. Se você não está listado nos endereços constantes no
>>> > cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa
>>> > mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas
>>> estão
>>> > imediatamente anuladas e proibidas’._
>>> >
>>> >
>>> > * **‘Apesar do Magazine Luiza tomar
>>> > todas as precauções razoáveis para assegurar que nenhum vírus esteja
>>> > presente nesse e-mail, a empresa não poderá aceitar a responsabilidade
>>> por
>>> > quaisquer perdas ou danos causados por esse e-mail ou por seus
>>> anexos’.*
>>> >
>>> >
>>> >
>>> Diese E Mail enthält möglicherweise vertrauliche Inhalte und ist nur für
>>> die Verwertung durch den vorgesehenen Empfänger bestimmt.
>>> Sollten Sie nicht der vorgesehene Empfänger sein, setzen Sie den
>>> Absender bitte unverzüglich in Kenntnis und löschen diese E Mail.
>>>
>>> Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarz
>>> >.
>>>
>>>
>>> This e-mail may contain confidential content and is intended only for
>>> the specified recipient/s.
>>> If you are not the intended recipient, please inform the sender
>>> immediately and delete this e-mail.
>>>
>>> Information on data protection can be found here<
>>> https://www.datenschutz.schwarz>.
>>>
>>>
>
> *‘Esta mensagem é direcionada apenas para os endereços constantes no
> cabeçalho inicial. Se você não está listado nos endereços constantes no
> cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa
> mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão
> imediatamente anuladas e proibidas’.*
>
> *‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para
> assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não
> poderá aceitar a responsabilidade por quaisquer perdas ou danos causados
> por esse e-mail ou por seus anexos’.*
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230717/93f32577/attachment-0001.htm>
More information about the openstack-discuss
mailing list