ovn-bgp-agent installation issue

Satish Patel satish.txt at gmail.com
Tue Aug 23 15:37:38 UTC 2022


Hi Luis,

/cc - openstack discuss mailing list

Thank you so much for clearing my doubts. I have a counter question.

 BGP Mode:
 - In bgp mode where i should run ovn-bgp-agent ?  Network node or Compute
node?

EVPN Mode:
 - In evpn mode where i should run ovn-bgp-agent?
 - In my lab i am using your heck "How to use it without BGPVPN" to create
vni mapping, do you think because of that i am not seeing vni pushed out to
FRR automatically?
https://ltomasbo.wordpress.com/2021/06/25/openstack-networking-with-evpn/
 - In your demo link I can see you have bridge two different deployment
clouds using the same VNI, In this scenario we need a router (or
vrf) somewhere correct to bridge to different subnet IPs, correct? or is
this L2 stretch?

I am going to open a new thread to discuss issues related to
expose_tenant_networks=True flag issue.

On Tue, Aug 23, 2022 at 3:14 AM Luis Tomas Bolivar <ltomasbo at redhat.com>
wrote:

> Hi Satish! See inline
>
> On Mon, Aug 22, 2022 at 5:22 PM Satish Patel <satish.txt at gmail.com> wrote:
>
>> Hi Luis,
>>
>> Welcome back from your vacation, sorry i didn't know you are on vacation.
>> Hope you had a wonderful time.
>>
>> How do i start conversion on opendev thread? is there a mailing list or
>> are you saying open thread here -
>> https://storyboard.openstack.org/#!/project/x/ovn-bgp-agent
>>
>
> Actually, you did it right (sending it to openstack-discuss), but it seems
> at some point we stop adding it on our replies... probably my fault due to
> replying on the phone while on vacation.
>
> For the next issue we can open a different thread and I'll try not to drop
> the openstack-discuss list! xD
>
> It would be great though if you can sent a follow up to the previous
> thread with the final outcome (fixed by updating, or adding config X, ..),
> so that community can see the project is alive, and other people facing
> similar issues can try your solution/config.
>

>> I have an update for you, after upgrading the OVN/OVS version to the
>> latest and that resolved lots of issues. As you mentioned earlier running
>> the latest code is very important to get proper functionality.
>>
>
> Awesome! Yeah, it is a new project, so functionality/fixes are being added
> regularly, therefore using the latest version is the best approach. The
> plan is to soon create a more stable version/release, once we have enough
> testing coverage and main functionality covered
>
>>
>> You are saying in BGP mode only FIP will get exposed but not the tenant
>> VM ( What is this flag for expose_tenant_network=True ?)
>>
>
> Main idea was:
> - BGP mode for exposing VMs/LBs on provider networks, or with FIPs attached
> - EVPN mode for tenant networks
>
> To give you some more context, we started with the BGP mode, and did the
> expose_tenant_network to expose also tenant IPs. However, the BGP mode is
> lacking an API to decide what to expose. So we moved to the EVPN mode (with
> networking-bgpvpn as the API) to expose the tenant networks.
>
> That said, you are right, and the expose_tenant_network flag is intended
> for the BGP mode, to expose all tenant networks (so you need to ensure no
> overlapping CIDRs). And, if that is not working (it may be, as there is no
> testing coverage for it), it is something we can definitely look at and
> fix. So, feel free to open a new thread on openstack-discuss, or bug in
> storyboard (whatever works for you better), and I'll try to fix asap.
>
>
>>
>> You are saying in EVPN mode only tenant VM ip will get exposed but not
>> FIP but in this design what is the use of getting tenant VM exposed but you
>> can't expose FIP then how does external people access VMs?  ( I am totally
>> confused in EVPN design and use case because your tenant VM is already
>> talking to each other using geneve tunnel so why do we need EVPN/L2VPN to
>> expose them?)
>>
>
> EVPN mode is indeed only for tenant VMs (and loadbalancer). Idea for this
> mode is to connect (N/S traffic, as you mention E/W is still using the
> normal geneve path) tenant networks between different OpenStack clusters.
> EVPN will create a vxlan tunnel connecting them. So, the vni/vxlan id
> selected (with networking-bgpvpn) is the vxlan tunnel encap id being used
> to connect both tenant networks.
>
> So, the idea behind the EVPN mode is not to make your VMs in a tenant
> network publicly accessible, but being able to access them from a different
> tenant network in a different cloud (it actually does not need to be an
> OpenStack cloud, anything connected to the same vxlan id. Perhaps this is a
> bit more clear in this demo:
> https://ltomasbo.wordpress.com/2021/10/01/ovn-bgp-agent-interconnecting-kubernetes-pods-and-openstack-tenant-vms-with-evpn/
>
>
>>
>>  This is current status of my LAB
>>
>> ovn-bgp-agent in BGP Mode:
>> I am successfully able to expose FIP and provider IPs (as you mentioned)
>> but when i use expose_tenant_network=True in config then getting error and
>> vm tenant ips not getting exposed to bgp so i am not sure what is the use
>> case of that flag.
>>
>
> Yep, most probably we broke something here due to lack of upstream testing
> for this flag. I'll check asap and try to fix it
>
>>
>> ovn-bgp-agent in EVPN Mode:
>> In this mode everything is working and tenant vm also gets exposed but
>> when I attach FIP to vm those FIP ip are not getting exposed in BGP (as you
>> mentioned in your reply).
>>
> Yes, EVPN mode is not intended for FIPs or VMs on provider networks. Only
> for tenants
>
>
>> But i am seeing one bug here where vni config not getting inserted in FRR
>> https://opendev.org/x/ovn-bgp-agent/src/branch/master/ovn_bgp_agent/drivers/openstack/utils/frr.py#L26
>>
>>
>> Who will trigger that code and when will that code get triggered to
>> configure FRR for vrf/vni ?
>>
>
> This is being added by networking-bgpvpn. You can see some details about
> the integration into my blogpost (
> https://ltomasbo.wordpress.com/2021/06/25/openstack-networking-with-evpn/)
> or in the upstream documentation:
> https://opendev.org/x/ovn-bgp-agent/src/branch/master/doc/source/contributor/evpn_mode_design.rst
>
> Note though, to make networking-bgpvpn to work with this integration, this
> patch is needed (which btw I should include into the documentation):
> https://review.opendev.org/c/openstack/networking-bgpvpn/+/803161
>
> Cheers,
> Luis
>
>
>>
>>
>>
>>
>>
>> On Mon, Aug 22, 2022 at 5:44 AM Luis Tomas Bolivar <ltomasbo at redhat.com>
>> wrote:
>>
>>> Hi Satish,
>>>
>>> Sorry I was on vacation. Trying to get through my inbox.
>>>
>>> I'm not sure what is the current status, note there is a difference
>>> between EVPN and BGP mode. BGP mode is for FIPs and VMs/LBs on the provider
>>> network or with FIPs, while EVPN is for VMs on the tenant networks, without
>>> FIPs. Right now you cannot mix them and need to choose either EVPN or BGP
>>> mode. I'm planning to give it a try to make it multidriver, but I haven't
>>> even started yet.
>>>
>>> BTW, as you did with the initial email, perhaps it is worth to have
>>> conversations on the opendev thread, so that anyone else facing the same
>>> issues can get some hints (or even provide feedback), that was one of the
>>> main ideas about moving it in there. To be able to have better support.
>>>
>>> As for the questions:
>>> # # on rack-2-host-1
>>> # when i created vm1 which endup on rack-2-host-1 but it doesn't expose
>>> the vm ip address. Is that normal behavior?
>>>
>>> This should be exposed in the node with cr-lrp port, i.e., rack1-host2,
>>> and it should be as simple as adding the IP to the lo-2001 dummy device
>>>
>>> # When I attach a floating ip to vm2 then why does my floating ip
>>> address not get exposed in BGP?
>>> This is what I mentioned before, you cannot merge BGP and EVPN mode.
>>>
>>> On Mon, Aug 8, 2022 at 1:41 PM Satish Patel <satish.txt at gmail.com>
>>> wrote:
>>>
>>>> Hi Luis,
>>>>
>>>> Sorry for bugging. I’m almost there now. Do you have any thought of
>>>> following question
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Aug 5, 2022, at 1:16 AM, Satish Patel <satish.txt at gmail.com> wrote:
>>>>
>>>> 
>>>> Good morning Luis,
>>>>
>>>> Quick question, I have following deployment as per your lab
>>>>
>>>> rack-1-host-1 (controller)
>>>> rack-1-host-2 (compute1 - This is hosting cr-lrp ports, inshort router)
>>>> rack-2-host-1 (compute2)
>>>>
>>>> I have created two vms
>>>>
>>>> vagrant at rack-1-host-1:~$ nova list
>>>> nova CLI is deprecated and will be a removed in a future release
>>>>
>>>> +--------------------------------------+------+--------+------------+-------------+--------------------------------------+
>>>> | ID                                   | Name | Status | Task State |
>>>> Power State | Networks                             |
>>>>
>>>> +--------------------------------------+------+--------+------------+-------------+--------------------------------------+
>>>> | aecb4f10-c46f-4551-b112-44e4dc007e88 | vm1  | ACTIVE | -          |
>>>> Running     | private-test=10.0.0.105              |
>>>> | ceae14b9-70c2-4dbc-8071-0d64d9a0ca84 | vm2  | ACTIVE | -          |
>>>> Running     | private-test=10.0.0.86, 172.16.1.200 |
>>>>
>>>> +--------------------------------------+------+--------+------------+-------------+--------------------------------------+
>>>>
>>>> # on rack-1-host-2
>>>> when i spun up vm2 which endup on rack-1-host-2 hence it created
>>>> vrf-2001 on dummy lo-2001 interface and exposed vm2 ip address
>>>> 10.0.0.86/32
>>>>
>>>> 96: vrf-2001: <NOARP,MASTER,UP,LOWER_UP> mtu 65575 qdisc noqueue state
>>>> UP group default qlen 1000
>>>>     link/ether 22:cc:25:b3:7b:96 brd ff:ff:ff:ff:ff:ff
>>>> 97: br-2001: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
>>>> master vrf-2001 state UP group default qlen 1000
>>>>     link/ether 0a:c3:23:7a:8f:0c brd ff:ff:ff:ff:ff:ff
>>>>     inet6 fe80::851:67ff:fe64:b2c3/64 scope link
>>>>        valid_lft forever preferred_lft forever
>>>> 98: vxlan-2001: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>> noqueue master br-2001 state UNKNOWN group default qlen 1000
>>>>     link/ether 0a:c3:23:7a:8f:0c brd ff:ff:ff:ff:ff:ff
>>>>     inet6 fe80::8c3:23ff:fe7a:8f0c/64 scope link
>>>>        valid_lft forever preferred_lft forever
>>>> 99: lo-2001: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue
>>>> master vrf-2001 state UNKNOWN group default qlen 1000
>>>>     link/ether d6:60:da:91:2e:6d brd ff:ff:ff:ff:ff:ff
>>>>     inet 10.0.0.86/32 scope global lo-2001
>>>>        valid_lft forever preferred_lft forever
>>>>     inet6 fe80::d460:daff:fe91:2e6d/64 scope link
>>>>        valid_lft forever preferred_lft forever
>>>>
>>>>
>>>> # on rack-2-host-1
>>>> when i created vm1 which endup on rack-2-host-1 but it doesn't expose
>>>> the vm ip address. Is that normal behavior?
>>>>
>>>> When I attach a floating ip to vm2 then why does my floating ip address
>>>> not get exposed in BGP?
>>>>
>>>> Thank you in advance
>>>>
>>>>
>>>> On Thu, Aug 4, 2022 at 2:30 PM Satish Patel <satish.txt at gmail.com>
>>>> wrote:
>>>>
>>>>> Update: Good news, I found what was wrong.
>>>>>
>>>>> After adding AS it works. In your doc you only added VNI but look like
>>>>> it is required to add BGP AS.  Or may be your doc is little older and new
>>>>> code required AS.
>>>>>
>>>>> vagrant at rack-1-host-1:~$ sudo ovn-nbctl set logical_switch_port
>>>>> c32dcd90-7820-44bd-894f-416e44b36aa0 external_ids:"neutron_bgpvpn\:as"=64999
>>>>> vagrant at rack-1-host-1:~$ sudo ovn-nbctl set logical_switch_port
>>>>> f55c1d1e-4b5b-4d8c-b922-3ad4a9700c81 external_ids:"neutron_bgpvpn\:as"=64999
>>>>>
>>>>> Now i can see it created VRF and exposed VM tenant ip in lo-2001
>>>>>
>>>>> 73: vrf-2001: <NOARP,MASTER,UP,LOWER_UP> mtu 65575 qdisc noqueue state
>>>>> UP group default qlen 1000
>>>>>     link/ether b2:29:65:3b:ac:db brd ff:ff:ff:ff:ff:ff
>>>>> 74: br-2001: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
>>>>> master vrf-2001 state UP group default qlen 1000
>>>>>     link/ether d6:0d:61:2d:b7:29 brd ff:ff:ff:ff:ff:ff
>>>>>     inet6 fe80::6830:47ff:febc:b10b/64 scope link
>>>>>        valid_lft forever preferred_lft forever
>>>>> 75: vxlan-2001: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>>>> noqueue master br-2001 state UNKNOWN group default qlen 1000
>>>>>     link/ether d6:0d:61:2d:b7:29 brd ff:ff:ff:ff:ff:ff
>>>>>     inet6 fe80::d40d:61ff:fe2d:b729/64 scope link
>>>>>        valid_lft forever preferred_lft forever
>>>>> 76: lo-2001: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue
>>>>> master vrf-2001 state UNKNOWN group default qlen 1000
>>>>>     link/ether fe:a1:a0:87:76:7c brd ff:ff:ff:ff:ff:ff
>>>>>     inet 10.0.0.83/32 scope global lo-2001
>>>>>        valid_lft forever preferred_lft forever
>>>>>     inet6 fe80::fca1:a0ff:fe87:767c/64 scope link
>>>>>
>>>>>
>>>>> I am continuing doing testing and seeing if I hit any other bug..
>>>>>
>>>>> On Thu, Aug 4, 2022 at 11:58 AM Satish Patel <satish.txt at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Luis,
>>>>>>
>>>>>> I am following your doc, tell me if that doc is outdated or not
>>>>>> https://ltomasbo.wordpress.com/2021/06/25/openstack-networking-with-evpn/
>>>>>>
>>>>>> I have upgraded pyroute2 to 0.7.2 but still getting the same error,
>>>>>> if you look at carefully following logs you will see an agent saying I
>>>>>> can't find VNI but it's there, i can see in ovn-nbctl list
>>>>>> logical_switch_port.
>>>>>>
>>>>>> pyroute2                     0.7.2
>>>>>> pyroute2.core                0.6.13
>>>>>> pyroute2.ethtool             0.6.13
>>>>>> pyroute2.ipdb                0.6.13
>>>>>> pyroute2.ipset               0.6.13
>>>>>> pyroute2.ndb                 0.6.13
>>>>>> pyroute2.nftables            0.6.13
>>>>>> pyroute2.nslink              0.6.13
>>>>>>
>>>>>>
>>>>>> ovn-bgp-agent logs
>>>>>>
>>>>>> 2022-08-04 15:52:52.898 396714 DEBUG
>>>>>> ovn_bgp_agent.drivers.openstack.utils.ovn [-] Either "neutron_bgpvpn:vni"
>>>>>> or "neutron_bgpvpn:as" were not found or have an invalid value in the port
>>>>>> f55c1d1e-4b5b-4d8c-b922-3ad4a9700c81 external_ids {'neutron:cidrs': '
>>>>>> 172.16.1.132/24', 'neutron:device_id':
>>>>>> '43b2c756-d92c-4fe5-b17b-32d5ab9b1f37', 'neutron:device_owner':
>>>>>> 'network:router_gateway', 'neutron:network_name':
>>>>>> 'neutron-0d82e6b0-bf9f-484d-85bd-0ba38aab508d', 'neutron:port_name': '',
>>>>>> 'neutron:project_id': '', 'neutron:revision_number': '4',
>>>>>> 'neutron:security_group_ids': '', 'neutron_bgpvpn:vni': '2001'}
>>>>>> get_evpn_info
>>>>>> /usr/local/lib/python3.8/dist-packages/ovn_bgp_agent/drivers/openstack/utils/ovn.py:251
>>>>>>
>>>>>> 2022-08-04 15:52:52.899 396714 DEBUG
>>>>>> ovn_bgp_agent.drivers.openstack.ovn_evpn_driver [-] No EVPN information for
>>>>>> CR-LRP Port with IPs ['172.16.1.132/24']. Not exposing it.
>>>>>> _expose_ip
>>>>>> /usr/local/lib/python3.8/dist-packages/ovn_bgp_agent/drivers/openstack/ovn_evpn_driver.py:220
>>>>>>
>>>>>>
>>>>>> In OVN i can see vni number
>>>>>>
>>>>>> _uuid               : 3473d0ce-1348-4423-a2dc-6d4df4c06f74
>>>>>> addresses           : [router]
>>>>>> dhcpv4_options      : []
>>>>>> dhcpv6_options      : []
>>>>>> dynamic_addresses   : []
>>>>>> enabled             : true
>>>>>> external_ids        : {"neutron:cidrs"="172.16.1.132/24",
>>>>>> "neutron:device_id"="43b2c756-d92c-4fe5-b17b-32d5ab9b1f37",
>>>>>> "neutron:device_owner"="network:router_gateway",
>>>>>> "neutron:network_name"=neutron-0d82e6b0-bf9f-484d-85bd-0ba38aab508d,
>>>>>> "neutron:port_name"="", "neutron:project_id"="",
>>>>>> "neutron:revision_number"="4", "neutron:security_group_ids"="",
>>>>>> "neutron_bgpvpn:vni"="2001"}
>>>>>> ha_chassis_group    : []
>>>>>> name                : "f55c1d1e-4b5b-4d8c-b922-3ad4a9700c81"
>>>>>> options             : {exclude-lb-vips-from-garp="true",
>>>>>> nat-addresses=router, router-port=lrp-f55c1d1e-4b5b-4d8c-b922-3ad4a9700c81}
>>>>>> parent_name         : []
>>>>>> port_security       : []
>>>>>> tag                 : []
>>>>>> tag_request         : []
>>>>>> type                : router
>>>>>> up                  : true
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 4, 2022 at 11:39 AM Satish Patel <satish.txt at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Luis,
>>>>>>>
>>>>>>> This is what I have installed on the compute and controller nodes.
>>>>>>>
>>>>>>> pyroute2               0.6.13
>>>>>>> pyroute2.core          0.6.13
>>>>>>> pyroute2.ethtool       0.6.13
>>>>>>> pyroute2.ipdb          0.6.13
>>>>>>> pyroute2.ipset         0.6.13
>>>>>>> pyroute2.ndb           0.6.13
>>>>>>> pyroute2.nftables      0.6.13
>>>>>>> pyroute2.nslink        0.6.13
>>>>>>>
>>>>>>> On Wed, Aug 3, 2022 at 10:10 AM Luis Tomas Bolivar <
>>>>>>> ltomasbo at redhat.com> wrote:
>>>>>>>
>>>>>>>> What version of pyroute2 are you using? Maybe it is related to some
>>>>>>>> bug in there in an old version (I hit quite a few). You can try to upgrade
>>>>>>>> that. Also, as it is on the resync, you can also disable that by setting a
>>>>>>>> very long time in there.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wednesday, August 3, 2022, Satish Patel <satish.txt at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Any thoughts Luis. Thanks
>>>>>>>>>
>>>>>>>>> Sent from my iPhone
>>>>>>>>>
>>>>>>>>> On Jul 27, 2022, at 11:53 AM, Satish Patel <satish.txt at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> 
>>>>>>>>> Luis,
>>>>>>>>>
>>>>>>>>> If you look at this logs -
>>>>>>>>> https://paste.opendev.org/show/buRbY415guvHFUtSapFK/
>>>>>>>>>
>>>>>>>>> I am able to expose the tenant ip when I set  "expose_tenant_networks=True"
>>>>>>>>> in the ovn-bgp-agent.conf file. But interestingly it exposes ip for the
>>>>>>>>> first time but when i delete vm and re-create new vm then its throws
>>>>>>>>> following error which you can see full track in above link
>>>>>>>>>
>>>>>>>>> ERROR oslo_service.periodic_task [-] Error during BGPAgent.sync: KeyError: 'object does not exists'
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have just 1x controller and 1x compute machine at present. As
>>>>>>>>> you said something is broken when setting up "expose_tenant_networks=True"
>>>>>>>>> Let me know if you need more info or logs etc. i will continue to poke and
>>>>>>>>> see if anything else I can find.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jul 27, 2022 at 10:35 AM Luis Tomas Bolivar <
>>>>>>>>> ltomasbo at redhat.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Satish,sorry for the delay in replies,  I'm on vacations and
>>>>>>>>>> have limited connectivity.
>>>>>>>>>>
>>>>>>>>>> See soon comments/replies inline
>>>>>>>>>>
>>>>>>>>>> On Wed, Jul 27, 2022 at 4:58 AM Satish Patel <
>>>>>>>>>> satish.txt at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Luis,
>>>>>>>>>>>
>>>>>>>>>>> Just checking incase you missed my last email. If you are on
>>>>>>>>>>> vacation then ignore it :) Thank you.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Jul 23, 2022 at 12:27 AM Satish Patel <
>>>>>>>>>>> satish.txt at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Luis,
>>>>>>>>>>>>
>>>>>>>>>>>> As you suggested, checkout 5 commits back to avoid the
>>>>>>>>>>>> Load_Balancer issue that made good progress.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Great to hear!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> But now i am seeing very odd behavior where floating IP can get
>>>>>>>>>>>> exposed to BGP but when i am trying to expose VM Tenant IP then I get a
>>>>>>>>>>>> strange error. Here is the full logs output;
>>>>>>>>>>>> https://paste.opendev.org/show/buRbY415guvHFUtSapFK/
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> I was using the evpn mode for the tenant networks, so it has been
>>>>>>>>>> a while since I tested the vm tenant ip with plain bgp. Perhaps we broke it
>>>>>>>>>> with some of the new features/reshapes.
>>>>>>>>>>
>>>>>>>>>> Regarding the error, two things:
>>>>>>>>>> - Do you have several hosts or just one? The VM IP should be
>>>>>>>>>> exposed in the host holding the ovn cr-lrp port (ovn router gateway port
>>>>>>>>>> connecting the router to the provider network).
>>>>>>>>>> - The error there seems to be related to the re-sync task. I've
>>>>>>>>>> hit some issues with pyroute2 in the past, please be sure to use a recent
>>>>>>>>>> enough version (minimum supported is 0.6.4)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jul 22, 2022 at 8:55 AM Satish Patel <
>>>>>>>>>>>> satish.txt at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Luis,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you for reply,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Let me tell you that your blog is wonderful. I have used your
>>>>>>>>>>>>> method to install frr without any docker container.
>>>>>>>>>>>>>
>>>>>>>>>>>>> What is the workaround here? Can I tell ovn-bgp-agent to not
>>>>>>>>>>>>> look for container of frr?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>> You just need to point where the frr socket is on the host.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>> I have notice one more thing that it’s trying to run “copy
>>>>>>>>>>>>> /tmp/blah running-config” but that command isn’t supported.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have tried to run copy command manually on vtysh shell to
>>>>>>>>>>>>> see what options are available for copy command but there is only one
>>>>>>>>>>>>> command available with copy which is “copy running-config startup-config”
>>>>>>>>>>>>>  do you think I have wrong version of frr running? I’m running 7.2.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>> I think the minimum version I tried was either 7.4 or 7.5.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>> Please let me know if any workaround here. Thank you
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>> No easy workaround for this, as it is relying on that way to
>>>>>>>>>> reconfigure the frr.conf
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Luis
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Jul 22, 2022, at 2:41 AM, Luis Tomas Bolivar <
>>>>>>>>>>>>> ltomasbo at redhat.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Satish,
>>>>>>>>>>>>>
>>>>>>>>>>>>> The one to use should be https://opendev.org/x/ovn-bgp-agent.
>>>>>>>>>>>>> The one on my personal github repo was the initial PoC for it. But the
>>>>>>>>>>>>> opendev one is the upstream effort to develop it, and is the one being
>>>>>>>>>>>>> maintained/updated.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Looking at your second logs, it seems you are missing FRR (and
>>>>>>>>>>>>> its shell, vtysh) in the node.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Actually, thinking about this:
>>>>>>>>>>>>> "Unexpected error while running command.
>>>>>>>>>>>>> Command: /usr/bin/vtysh --vty_socket /run/frr/ -c copy
>>>>>>>>>>>>> /tmp/tmpiz5s_wvs running-config"
>>>>>>>>>>>>>
>>>>>>>>>>>>> The ovn-bgp-agent has been developed with "deploying on
>>>>>>>>>>>>> containers" in mind, meaning it is assuming there is a frr container
>>>>>>>>>>>>> running, and the container running the agent is trying to connect to the
>>>>>>>>>>>>> same socket so that it can run the vtysh commands. Perhaps in your case the
>>>>>>>>>>>>> frr socket is in a different location than /run/frr/
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jul 22, 2022 at 6:27 AM Satish Patel <
>>>>>>>>>>>>> satish.txt at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Folks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am trying to create lab of of ovn-bgp-agent using this blog
>>>>>>>>>>>>>> https://ltomasbo.wordpress.com/2021/02/04/ovn-bgp-agent-testing-setup/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So far everything went well but I'm stuck at the bgp-agent
>>>>>>>>>>>>>> installation and I encounter following error when running bgp-agent.
>>>>>>>>>>>>>> Any suggestions?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> root at rack-1-host-2:/home/vagrant/bgp-agent# bgp-agent
>>>>>>>>>>>>>> 2022-07-22 04:02:39.123 111551 INFO bgp_agent.config [-]
>>>>>>>>>>>>>> Logging enabled!
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 CRITICAL bgp-agent [-]
>>>>>>>>>>>>>> Unhandled error: AssertionError
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent Traceback
>>>>>>>>>>>>>> (most recent call last):
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent   File
>>>>>>>>>>>>>> "/usr/local/bin/bgp-agent", line 10, in <module>
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent
>>>>>>>>>>>>>> sys.exit(start())
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent   File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/bgp_agent/agent.py", line 76, in
>>>>>>>>>>>>>> start
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent
>>>>>>>>>>>>>> bgp_agent_launcher = service.launch(config.CONF, BGPAgent())
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent   File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/bgp_agent/agent.py", line 44, in
>>>>>>>>>>>>>> __init__
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent
>>>>>>>>>>>>>> self.agent_driver = driver_api.AgentDriverBase.get_instance(
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent   File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/bgp_agent/platform/driver_api.py",
>>>>>>>>>>>>>> line 25, in get_instance
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent
>>>>>>>>>>>>>> agent_driver = stevedore_driver.DriverManager(
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent   File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/stevedore/driver.py", line 54, in
>>>>>>>>>>>>>> __init__
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent
>>>>>>>>>>>>>> super(DriverManager, self).__init__(
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent   File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/stevedore/named.py", line 78, in
>>>>>>>>>>>>>> __init__
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent     extensions
>>>>>>>>>>>>>> = self._load_plugins(invoke_on_load,
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent   File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/stevedore/extension.py", line 221,
>>>>>>>>>>>>>> in _load_plugins
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent     ext =
>>>>>>>>>>>>>> self._load_one_plugin(ep,
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent   File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/stevedore/named.py", line 156, in
>>>>>>>>>>>>>> _load_one_plugin
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent     return
>>>>>>>>>>>>>> super(NamedExtensionManager, self)._load_one_plugin(
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent   File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/stevedore/extension.py", line 257,
>>>>>>>>>>>>>> in _load_one_plugin
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent     obj =
>>>>>>>>>>>>>> plugin(*invoke_args, **invoke_kwds)
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent   File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/bgp_agent/platform/osp/ovn_bgp_driver.py",
>>>>>>>>>>>>>> line 64, in __init__
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent
>>>>>>>>>>>>>> self._sb_idl = ovn.OvnSbIdl(
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent   File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/bgp_agent/platform/osp/utils/ovn.py",
>>>>>>>>>>>>>> line 62, in __init__
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent
>>>>>>>>>>>>>> super(OvnSbIdl, self).__init__(
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent   File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/bgp_agent/platform/osp/utils/ovn.py",
>>>>>>>>>>>>>> line 31, in __init__
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent
>>>>>>>>>>>>>> super(OvnIdl, self).__init__(remote, schema)
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent   File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/ovs/db/idl.py", line 283, in
>>>>>>>>>>>>>> __init__
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent     schema =
>>>>>>>>>>>>>> schema_helper.get_idl_schema()
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent   File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/ovs/db/idl.py", line 2323, in
>>>>>>>>>>>>>> get_idl_schema
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent
>>>>>>>>>>>>>> self._keep_table_columns(schema, table, columns))
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent   File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/ovs/db/idl.py", line 2330, in
>>>>>>>>>>>>>> _keep_table_columns
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent     assert
>>>>>>>>>>>>>> table_name in schema.tables
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent AssertionError
>>>>>>>>>>>>>> 2022-07-22 04:02:39.475 111551 ERROR bgp-agent
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> After googling I found one more agent at
>>>>>>>>>>>>>> https://opendev.org/x/ovn-bgp-agent and its also throwing an
>>>>>>>>>>>>>> error. Which agent should I be using?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> root at rack-1-host-2:~# ovn-bgp-agent
>>>>>>>>>>>>>> 2022-07-22 04:04:36.780 111761 INFO ovn_bgp_agent.config [-]
>>>>>>>>>>>>>> Logging enabled!
>>>>>>>>>>>>>> 2022-07-22 04:04:37.247 111761 INFO ovn_bgp_agent.agent [-]
>>>>>>>>>>>>>> Service 'BGPAgent' stopped
>>>>>>>>>>>>>> 2022-07-22 04:04:37.248 111761 INFO ovn_bgp_agent.agent [-]
>>>>>>>>>>>>>> Service 'BGPAgent' starting
>>>>>>>>>>>>>> 2022-07-22 04:04:37.248 111761 INFO
>>>>>>>>>>>>>> ovn_bgp_agent.drivers.openstack.utils.frr [-] Add VRF leak for VRF
>>>>>>>>>>>>>> ovn-bgp-vrf on router bgp 64999
>>>>>>>>>>>>>> 2022-07-22 04:04:37.248 111761 INFO oslo.privsep.daemon [-]
>>>>>>>>>>>>>> Running privsep helper: ['sudo', 'privsep-helper', '--privsep_context',
>>>>>>>>>>>>>> 'ovn_bgp_agent.privileged.vtysh_cmd', '--privsep_sock_path',
>>>>>>>>>>>>>> '/tmp/tmp4cie9eiz/privsep.sock']
>>>>>>>>>>>>>> 2022-07-22 04:04:37.687 111761 INFO oslo.privsep.daemon [-]
>>>>>>>>>>>>>> Spawned new privsep daemon via rootwrap
>>>>>>>>>>>>>> 2022-07-22 04:04:37.598 111769 INFO oslo.privsep.daemon [-]
>>>>>>>>>>>>>> privsep daemon starting
>>>>>>>>>>>>>> 2022-07-22 04:04:37.613 111769 INFO oslo.privsep.daemon [-]
>>>>>>>>>>>>>> privsep process running with uid/gid: 0/0
>>>>>>>>>>>>>> 2022-07-22 04:04:37.617 111769 INFO oslo.privsep.daemon [-]
>>>>>>>>>>>>>> privsep process running with capabilities (eff/prm/inh):
>>>>>>>>>>>>>> CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_NET_ADMIN|CAP_SYS_ADMIN/none
>>>>>>>>>>>>>> 2022-07-22 04:04:37.617 111769 INFO oslo.privsep.daemon [-]
>>>>>>>>>>>>>> privsep daemon running as pid 111769
>>>>>>>>>>>>>> 2022-07-22 04:04:37.987 111769 ERROR
>>>>>>>>>>>>>> ovn_bgp_agent.privileged.vtysh [-] Unable to execute vtysh with
>>>>>>>>>>>>>> ['/usr/bin/vtysh', '--vty_socket', '/run/frr/', '-c', 'copy
>>>>>>>>>>>>>> /tmp/tmpiz5s_wvs running-config']. Exception: Unexpected error while
>>>>>>>>>>>>>> running command.
>>>>>>>>>>>>>> Command: /usr/bin/vtysh --vty_socket /run/frr/ -c copy
>>>>>>>>>>>>>> /tmp/tmpiz5s_wvs running-config
>>>>>>>>>>>>>> Exit code: 1
>>>>>>>>>>>>>> Stdout: '% Unknown command: copy /tmp/tmpiz5s_wvs
>>>>>>>>>>>>>> running-config\n'
>>>>>>>>>>>>>> Stderr: ''
>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>   File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/ovn_bgp_agent/privileged/vtysh.py",
>>>>>>>>>>>>>> line 30, in run_vtysh_config
>>>>>>>>>>>>>>     return processutils.execute(*full_args)
>>>>>>>>>>>>>>   File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py",
>>>>>>>>>>>>>> line 438, in execute
>>>>>>>>>>>>>>     raise ProcessExecutionError(exit_code=_returncode,
>>>>>>>>>>>>>> oslo_concurrency.processutils.ProcessExecutionError:
>>>>>>>>>>>>>> Unexpected error while running command.
>>>>>>>>>>>>>> Command: /usr/bin/vtysh --vty_socket /run/frr/ -c copy
>>>>>>>>>>>>>> /tmp/tmpiz5s_wvs running-config
>>>>>>>>>>>>>> Exit code: 1
>>>>>>>>>>>>>> Stdout: '% Unknown command: copy /tmp/tmpiz5s_wvs
>>>>>>>>>>>>>> running-config\n'
>>>>>>>>>>>>>> Stderr: ''
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service [-]
>>>>>>>>>>>>>> Error starting thread.:
>>>>>>>>>>>>>> oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while
>>>>>>>>>>>>>> running command.
>>>>>>>>>>>>>> Command: /usr/bin/vtysh --vty_socket /run/frr/ -c copy
>>>>>>>>>>>>>> /tmp/tmpiz5s_wvs running-config
>>>>>>>>>>>>>> Exit code: 1
>>>>>>>>>>>>>> Stdout: '% Unknown command: copy /tmp/tmpiz5s_wvs
>>>>>>>>>>>>>> running-config\n'
>>>>>>>>>>>>>> Stderr: ''
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> File "/usr/local/lib/python3.8/dist-packages/oslo_service/service.py", line
>>>>>>>>>>>>>> 806, in run_service
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> service.start()
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> File "/usr/local/lib/python3.8/dist-packages/ovn_bgp_agent/agent.py", line
>>>>>>>>>>>>>> 50, in start
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> self.agent_driver.start()
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/ovn_bgp_agent/drivers/openstack/ovn_bgp_driver.py",
>>>>>>>>>>>>>> line 73, in start
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> frr.vrf_leak(constants.OVN_BGP_VRF, CONF.bgp_AS, CONF.bgp_router_id)
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/ovn_bgp_agent/drivers/openstack/utils/frr.py",
>>>>>>>>>>>>>> line 110, in vrf_leak
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> _run_vtysh_config_with_tempfile(vrf_config)
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> File
>>>>>>>>>>>>>> "/usr/local/lib/python3.8/dist-packages/ovn_bgp_agent/drivers/openstack/utils/frr.py",
>>>>>>>>>>>>>> line 93, in _run_vtysh_config_with_tempfile
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> ovn_bgp_agent.privileged.vtysh.run_vtysh_config(f.name)
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/priv_context.py",
>>>>>>>>>>>>>> line 271, in _wrap
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> return self.channel.remote_call(name, args, kwargs,
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> File "/usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py", line
>>>>>>>>>>>>>> 215, in remote_call
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> raise exc_type(*result[2])
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while
>>>>>>>>>>>>>> running command.
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> Command: /usr/bin/vtysh --vty_socket /run/frr/ -c copy /tmp/tmpiz5s_wvs
>>>>>>>>>>>>>> running-config
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> Exit code: 1
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> Stdout: '% Unknown command: copy /tmp/tmpiz5s_wvs running-config\n'
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> Stderr: ''
>>>>>>>>>>>>>> 2022-07-22 04:04:37.990 111761 ERROR oslo_service.service
>>>>>>>>>>>>>> 2022-07-22 04:04:37.993 111761 INFO ovn_bgp_agent.agent [-]
>>>>>>>>>>>>>> Service 'BGPAgent' stopping
>>>>>>>>>>>>>> 2022-07-22 04:04:37.994 111761 INFO ovn_bgp_agent.agent [-]
>>>>>>>>>>>>>> Service 'BGPAgent' stopped
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> LUIS TOMÁS BOLÍVAR
>>>>>>>>>>>>> Principal Software Engineer
>>>>>>>>>>>>> Red Hat
>>>>>>>>>>>>> Madrid, Spain
>>>>>>>>>>>>> ltomasbo at redhat.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> LUIS TOMÁS BOLÍVAR
>>>>>>>>>> Principal Software Engineer
>>>>>>>>>> Red Hat
>>>>>>>>>> Madrid, Spain
>>>>>>>>>> ltomasbo at redhat.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> LUIS TOMÁS BOLÍVAR
>>>>>>>> Principal Software Engineer
>>>>>>>> Red Hat
>>>>>>>> Madrid, Spain
>>>>>>>> ltomasbo at redhat.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>
>>> --
>>> LUIS TOMÁS BOLÍVAR
>>> Principal Software Engineer
>>> Red Hat
>>> Madrid, Spain
>>> ltomasbo at redhat.com
>>>
>>>
>>
>
> --
> LUIS TOMÁS BOLÍVAR
> Principal Software Engineer
> Red Hat
> Madrid, Spain
> ltomasbo at redhat.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20220823/c3d61aa5/attachment-0001.htm>


More information about the openstack-discuss mailing list