Re: [Octavia] Please help with deployment of octavia unbound lb port when creating LB

16 Sep 2020


      Hello,
thanks for your quick help. When I create the listener, the pool and add the members to it the loadbalancer works indeed.
Thank you
Greets Stefan Bujack
----- Original Message -----
From: "Michael Johnson" johnsomor@gmail.com
To: "Stefan Bujack" stefan.bujack@desy.de
Cc: "openstack-discuss" openstack-discuss@lists.openstack.org
Sent: Wednesday, 16 September, 2020 22:50:32
Subject: Re: [Octavia] Please help with deployment of octavia unbound lb port when creating LB
Hi Stefan,
The ports look ok in your output.
The VIP is configured as an "allowed address pair" in neutron to allow
failovers. "allowed address pair" ports in neutron is how you can have
a secondary IP on a port.
Each load balancer (In standalone topology) will show two ports in
neutron. A "base" port, which is a normal neutron port, and a VRRP/VIP
port which is the "allowed address pair" port.
In the output above, your base port is:
| 8baf7abb-fa03-446b-8ca2-6d026cce75d6 |
octavia-lb-vrrp-e50c5b05-69eb-45c4-a670-dc34331443f5 |
fa:16:3e:1b:c1:7d | ip_address='131.169.46.40',
subnet_id='f2a2d8d2-363e-45e7-80f8-f751a24eed8c'   | ACTIVE |
And your VRRP/VIP port is:
| bae4ffe6-a1dc-4a8a-9b1e-cc727a4b763b |
octavia-lb-cd3b28f4-62f6-48e0-bc3a-b52fcb36e073      |
fa:16:3e:bb:0f:f3 | ip_address='131.169.46.214',
subnet_id='f2a2d8d2-363e-45e7-80f8-f751a24eed8c'  | DOWN   |
If you do a "openstack port show 8baf7abb-fa03-446b-8ca2-6d026cce75d6"
(the base port) you will see at the top the allowed address pairs
configuration that points to the other port. The allowed address pairs
port will never show as ACTIVE as it is not a "real" neutron port.
Octavia also manages the security groups for you, so I don't think
security groups are likely an issue here.
I see on the load balancer output that you do not have a listener
configured on the load balancer. The VIP port will not respond to any
requests until a listener has been configured (The listener defines
the TCP/UDP port to accept connections on). This is also why the load
balancer is reporting operating_status as OFFLINE.
If you create an HTTP listener on port 80, once the load balancer
becomes ACTIVE, you should be able to curl to the VIP and get back an
HTTP 503 response. This is because there is no pool or members
configured to service the request.
Let me know if that doesn't solve your issue and we can debug it further.
Michael
On Wed, Sep 16, 2020 at 11:00 AM Bujack, Stefan stefan.bujack@desy.de wrote:
...
Hello,
I am a little lost here. Hopefully some of you nice people could help me with this issue please.
We have an Openstack Ussuri deployment on Ubuntu 20.04.
Our network is configured in an "Open vSwitch: High availability using VRRP" way.
I have gone through the official Install and configure procedure on "https://docs.openstack.org/octavia/ussuri/install/install-ubuntu.html"
We have one public network.
When I want to "Deploy a basic HTTP load balancer" like described in the official documentation "https://docs.openstack.org/octavia/ussuri/user/guides/basic-cookbook.html"
I see a problem with the created lb port. The port is down and unbound and the VIP is not reachable.
root@keystone04:~# openstack loadbalancer create --name lb1 --vip-subnet-id DESY-VLAN-46
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| availability_zone   | None                                 |
| created_at          | 2020-09-16T17:19:37                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | cd3b28f4-62f6-48e0-bc3a-b52fcb36e073 |
| listeners           |                                      |
| name                | lb1                                  |
| operating_status    | OFFLINE                              |
| pools               |                                      |
| project_id          | 0c6318a1c2414c9f805059788db47bb6     |
| provider            | amphora                              |
| provisioning_status | PENDING_CREATE                       |
| updated_at          | None                                 |
| vip_address         | 131.169.46.214                       |
| vip_network_id      | 94b6986f-7035-4b35-bee9-739451fa1871 |
| vip_port_id         | bae4ffe6-a1dc-4a8a-9b1e-cc727a4b763b |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | f2a2d8d2-363e-45e7-80f8-f751a24eed8c |
+---------------------+--------------------------------------+
root@keystone04:~# openstack loadbalancer show cd3b28f4-62f6-48e0-bc3a-b52fcb36e073
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| availability_zone   | None                                 |
| created_at          | 2020-09-16T17:19:37                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | cd3b28f4-62f6-48e0-bc3a-b52fcb36e073 |
| listeners           |                                      |
| name                | lb1                                  |
| operating_status    | OFFLINE                              |
| pools               |                                      |
| project_id          | 0c6318a1c2414c9f805059788db47bb6     |
| provider            | amphora                              |
| provisioning_status | ACTIVE                               |
| updated_at          | 2020-09-16T17:20:22                  |
| vip_address         | 131.169.46.214                       |
| vip_network_id      | 94b6986f-7035-4b35-bee9-739451fa1871 |
| vip_port_id         | bae4ffe6-a1dc-4a8a-9b1e-cc727a4b763b |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | f2a2d8d2-363e-45e7-80f8-f751a24eed8c |
+---------------------+--------------------------------------+
root@keystone04:~# openstack port list
+--------------------------------------+------------------------------------------------------+-------------------+--------------------------------------------------------------------------------+--------+
| ID                                   | Name                                                 | MAC Address       | Fixed IP Addresses                                                             | Status |
+--------------------------------------+------------------------------------------------------+-------------------+--------------------------------------------------------------------------------+--------+
| 020210e3-546a-4372-a91b-cc3e7a5cbab0 | HA port tenant 0c6318a1c2414c9f805059788db47bb6      | fa:16:3e:0b:d4:a9 | ip_address='169.254.192.26', subnet_id='4de6a91e-bb53-4869-976b-67815769bb12'  | ACTIVE |
| 20fe9c50-6c89-4ebd-bbfa-25bdf0e716fd |                                                      | fa:16:3e:f5:c3:a4 | ip_address='131.169.46.201', subnet_id='f2a2d8d2-363e-45e7-80f8-f751a24eed8c'  | N/A    |
| 2ae5a87f-803a-4e1d-9e7c-e874f200a3f4 |                                                      | fa:16:3e:57:57:ef | ip_address='131.169.46.31', subnet_id='f2a2d8d2-363e-45e7-80f8-f751a24eed8c'   | ACTIVE |
| 6948989b-40e8-40fe-9216-16f82d8071cd |                                                      | fa:16:3e:8b:59:0c | ip_address='172.16.1.1', subnet_id='2ed9de2d-ea68-4f25-a925-fdfe6c4d5fd8'      | ACTIVE |
| 784fa499-2f64-4026-a26b-732acd2f328c |                                                      | fa:16:3e:57:ec:23 | ip_address='131.169.46.128', subnet_id='f2a2d8d2-363e-45e7-80f8-f751a24eed8c'  | ACTIVE |
| 8baf7abb-fa03-446b-8ca2-6d026cce75d6 | octavia-lb-vrrp-e50c5b05-69eb-45c4-a670-dc34331443f5 | fa:16:3e:1b:c1:7d | ip_address='131.169.46.40', subnet_id='f2a2d8d2-363e-45e7-80f8-f751a24eed8c'   | ACTIVE |
| 8fa76adf-0a4b-400d-ae29-874cbd055f88 |                                                      | fa:16:3e:3f:92:14 | ip_address='172.16.0.100', subnet_id='5443e5a0-996f-465c-acb8-14128f423b1d'    | ACTIVE |
| 906f5713-c2b6-4d05-8c89-b084e09c744c |                                                      | fa:16:3e:ba:d7:74 | ip_address='172.16.1.112', subnet_id='2ed9de2d-ea68-4f25-a925-fdfe6c4d5fd8'    | ACTIVE |
| a08d5c5f-dacb-4a96-b0f4-7e1a3fd1c536 |                                                      | fa:16:3e:86:f9:d6 | ip_address='172.16.0.219', subnet_id='5443e5a0-996f-465c-acb8-14128f423b1d'    | ACTIVE |
| b5ad6738-8805-4f20-8084-a94ffacfff89 |                                                      | fa:16:3e:00:80:79 | ip_address='131.169.46.60', subnet_id='f2a2d8d2-363e-45e7-80f8-f751a24eed8c'   | ACTIVE |
| bae4ffe6-a1dc-4a8a-9b1e-cc727a4b763b | octavia-lb-cd3b28f4-62f6-48e0-bc3a-b52fcb36e073      | fa:16:3e:bb:0f:f3 | ip_address='131.169.46.214', subnet_id='f2a2d8d2-363e-45e7-80f8-f751a24eed8c'  | DOWN   |
| bf1476d0-0327-4c4f-8b79-d767c8a7dba5 |                                                      | fa:16:3e:24:79:cb | ip_address='131.169.46.126', subnet_id='f2a2d8d2-363e-45e7-80f8-f751a24eed8c'  | ACTIVE |
| c15b142f-c06c-426a-83db-46e98e4839d6 |                                                      | fa:16:3e:c7:60:d1 | ip_address='172.16.1.141', subnet_id='2ed9de2d-ea68-4f25-a925-fdfe6c4d5fd8'    | ACTIVE |
| cb75004a-aa57-4250-93be-1bb03bdc2a1b |                                                      | fa:16:3e:7e:9c:9f | ip_address='131.169.46.84', subnet_id='f2a2d8d2-363e-45e7-80f8-f751a24eed8c'   | ACTIVE |
| dc956e9a-a905-417a-b234-14782bf182d3 | HA port tenant 0c6318a1c2414c9f805059788db47bb6      | fa:16:3e:40:87:e3 | ip_address='169.254.194.172', subnet_id='4de6a91e-bb53-4869-976b-67815769bb12' | ACTIVE |
| dd48e315-2cb1-4716-8bc5-e892a948cb5f |                                                      | fa:16:3e:b0:4a:eb | ip_address='172.16.1.2', subnet_id='2ed9de2d-ea68-4f25-a925-fdfe6c4d5fd8'      | ACTIVE |
| e25ee538-7938-4992-a4f7-51f35f6831b5 | octavia-health-manager-listen-port                   | fa:16:3e:5c:b3:2f | ip_address='172.16.0.2', subnet_id='5443e5a0-996f-465c-acb8-14128f423b1d'      | ACTIVE |
| e91a5135-b076-4043-add4-21073109a730 |                                                      | fa:16:3e:4d:b8:56 | ip_address='131.169.46.102', subnet_id='f2a2d8d2-363e-45e7-80f8-f751a24eed8c'  | ACTIVE |
+--------------------------------------+------------------------------------------------------+-------------------+--------------------------------------------------------------------------------+--------+
root@keystone04:~# openstack port show bae4ffe6-a1dc-4a8a-9b1e-cc727a4b763b
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| Field                   | Value                                                                                                                                   |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up          | DOWN                                                                                                                                    |
| allowed_address_pairs   |                                                                                                                                         |
| binding_host_id         |                                                                                                                                         |
| binding_profile         |                                                                                                                                         |
| binding_vif_details     |                                                                                                                                         |
| binding_vif_type        | unbound                                                                                                                                 |
| binding_vnic_type       | normal                                                                                                                                  |
| created_at              | 2020-09-16T17:19:37Z                                                                                                                    |
| data_plane_status       | None                                                                                                                                    |
| description             |                                                                                                                                         |
| device_id               | lb-cd3b28f4-62f6-48e0-bc3a-b52fcb36e073                                                                                                 |
| device_owner            | Octavia                                                                                                                                 |
| dns_assignment          | None                                                                                                                                    |
| dns_domain              | None                                                                                                                                    |
| dns_name                | None                                                                                                                                    |
| extra_dhcp_opts         |                                                                                                                                         |
| fixed_ips               | ip_address='131.169.46.214', subnet_id='f2a2d8d2-363e-45e7-80f8-f751a24eed8c'                                                           |
| id                      | bae4ffe6-a1dc-4a8a-9b1e-cc727a4b763b                                                                                                    |
| ip_allocation           | None                                                                                                                                    |
| location                | cloud='', project.domain_id=, project.domain_name=, project.id='0c6318a1c2414c9f805059788db47bb6', project.name=, region_name='', zone= |
| mac_address             | fa:16:3e:bb:0f:f3                                                                                                                       |
| name                    | octavia-lb-cd3b28f4-62f6-48e0-bc3a-b52fcb36e073                                                                                         |
| network_id              | 94b6986f-7035-4b35-bee9-739451fa1871                                                                                                    |
| port_security_enabled   | True                                                                                                                                    |
| project_id              | 0c6318a1c2414c9f805059788db47bb6                                                                                                        |
| propagate_uplink_status | None                                                                                                                                    |
| qos_network_policy_id   | None                                                                                                                                    |
| qos_policy_id           | None                                                                                                                                    |
| resource_request        | None                                                                                                                                    |
| revision_number         | 2                                                                                                                                       |
| security_group_ids      | 0964090c-0299-401a-9156-bafbb040e345                                                                                                    |
| status                  | DOWN                                                                                                                                    |
| tags                    |                                                                                                                                         |
| trunk_details           | None                                                                                                                                    |
| updated_at              | 2020-09-16T17:19:39Z                                                                                                                    |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+
I also keep getting this error on the octavia node:
Sep 16 19:41:46 octavia04.desy.de octavia-health-manager[3009]: 2020-09-16 19:41:46.217 3009 WARNING octavia.amphorae.drivers.health.heartbeat_udp [-] Health Manager experienced an exception processing a heartbeat message from ('172.16.0.219', 8660). Ignoring this packet. Exception: 'NoneType' object has no attribute 'encode'
My security groups look like this:
root@octavia04:~# openstack security group list
+--------------------------------------+-----------------------------------------+------------------------+----------------------------------+------+
| ID                                   | Name                                    | Description            | Project                          | Tags |
+--------------------------------------+-----------------------------------------+------------------------+----------------------------------+------+
| 0964090c-0299-401a-9156-bafbb040e345 | lb-cd3b28f4-62f6-48e0-bc3a-b52fcb36e073 |                        | f89517ee676f4618bd55849477442aca | []   |
| 0cda6134-0574-430b-9250-f71b81587a53 | default                                 | Default security group |                                  | []   |
| 2236e82c-13fe-42e3-9fcf-bea43917f231 | lb-mgmt-sec-grp                         | lb-mgmt-sec-grp        | f89517ee676f4618bd55849477442aca | []   |
| 85ab9c91-9241-4ab4-ad01-368518ab1a51 | default                                 | Default security group | 35609e3390ce45be83a31cac47057efb | []   |
| e4f59cd4-75c6-4abf-9ab6-b97b4ae199b4 | lb-health-mgr-sec-grp                   | lb-health-mgr-sec-grp  | f89517ee676f4618bd55849477442aca | []   |
| ef91fcfb-fe20-4d45-bfe8-dfb7375462a3 | default                                 | Default security group | f89517ee676f4618bd55849477442aca | []   |
| efff8138-bffd-4e96-8318-2b13b4294f0b | default                                 | Default security group | 0c6318a1c2414c9f805059788db47bb6 | []   |
+--------------------------------------+-----------------------------------------+------------------------+----------------------------------+------+
root@octavia04:~# openstack security group rule list e4f59cd4-75c6-4abf-9ab6-b97b4ae199b4
+--------------------------------------+-------------+-----------+-----------+------------+-----------------------+
| ID                                   | IP Protocol | Ethertype | IP Range  | Port Range | Remote Security Group |
+--------------------------------------+-------------+-----------+-----------+------------+-----------------------+
| 20ef3407-0df0-4dcc-96cc-2693b9cdc6aa | udp         | IPv4      | 0.0.0.0/0 | 5555:5555  | None                  |
| 3e9feb44-c548-4889-aa30-1792ea89d675 | None        | IPv4      | 0.0.0.0/0 |            | None                  |
| 6cfd295f-6544-4bb6-bb51-00960e4753bb | None        | IPv6      | ::/0      |            | None                  |
+--------------------------------------+-------------+-----------+-----------+------------+-----------------------+
root@octavia04:~# openstack security group rule list 2236e82c-13fe-42e3-9fcf-bea43917f231
+--------------------------------------+-------------+-----------+-----------+------------+-----------------------+
| ID                                   | IP Protocol | Ethertype | IP Range  | Port Range | Remote Security Group |
+--------------------------------------+-------------+-----------+-----------+------------+-----------------------+
| 29e20b2b-6626-48c4-a06c-85d9dd6e6d61 | tcp         | IPv4      | 0.0.0.0/0 | 22:22      | None                  |
| 419ab26c-9cdf-4fda-bec3-95501f6bfa7d | icmp        | IPv4      | 0.0.0.0/0 |            | None                  |
| a4c70060-3580-46a6-8735-bca7046298f1 | None        | IPv6      | ::/0      |            | None                  |
| b1122fa8-1699-434f-b810-36abc0ea4ab8 | tcp         | IPv4      | 0.0.0.0/0 | 9443:9443  | None                  |
| cdc91572-afa9-4401-9212-a46414ea01ae | None        | IPv4      | 0.0.0.0/0 |            | None                  |
+--------------------------------------+-------------+-----------+-----------+------------+-----------------------+
root@octavia04:~# openstack security group rule list 0964090c-0299-401a-9156-bafbb040e345
+--------------------------------------+-------------+-----------+-----------+------------+-----------------------+
| ID                                   | IP Protocol | Ethertype | IP Range  | Port Range | Remote Security Group |
+--------------------------------------+-------------+-----------+-----------+------------+-----------------------+
| 07529aae-7732-409f-af37-c9b5287bbb16 | None        | IPv6      | ::/0      |            | None                  |
| 35701c1b-f739-4a44-a8c6-1d8f9ca82a7e | None        | IPv4      | 0.0.0.0/0 |            | None                  |
+--------------------------------------+-------------+-----------+-----------+------------+-----------------------+
My network agents lokk like this
root@keystone04:~# openstack network agent list
+--------------------------------------+--------------------+-------------------------+-------------------+-------+-------+---------------------------+
| ID                                   | Agent Type         | Host                    | Availability Zone | Alive | State | Binary                    |
+--------------------------------------+--------------------+-------------------------+-------------------+-------+-------+---------------------------+
| 0b3fd449-c123-4d82-994e-adf4aa588292 | Open vSwitch agent | neutron04-node1.desy.de | None              | :-)   | UP    | neutron-openvswitch-agent |
| 195b08ff-0b89-48d8-9ada-b59b5ff2b8ab | Open vSwitch agent | openstack04.desy.de     | None              | :-)   | UP    | neutron-openvswitch-agent |
| 3346b86a-80f9-4397-8f55-9d1ff28285dd | L3 agent           | neutron04-node1.desy.de | nova              | :-)   | UP    | neutron-l3-agent          |
| 36547753-59d7-4184-9a76-5317abf9a3aa | DHCP agent         | openstack04.desy.de     | nova              | :-)   | UP    | neutron-dhcp-agent        |
| 56ae1056-72b6-4a65-8bab-7f837c264777 | Metadata agent     | openstack04.desy.de     | None              | :-)   | UP    | neutron-metadata-agent    |
| 6678b278-6acb-439a-92a8-e2c7f932607c | L3 agent           | octavia04.desy.de       | nova              | :-)   | UP    | neutron-l3-agent          |
| 6681247b-3633-45cd-9017-e548fbd13e73 | Open vSwitch agent | neutron04.desy.de       | None              | :-)   | UP    | neutron-openvswitch-agent |
| 6d4ed4ed-5a8f-42ee-9052-ff9279a9dada | L3 agent           | openstack04.desy.de     | nova              | :-)   | UP    | neutron-l3-agent          |
| 8254d653-aff1-40e3-ade6-890d0a6b0617 | L3 agent           | neutron04.desy.de       | nova              | :-)   | UP    | neutron-l3-agent          |
| c4ce7df7-a682-4e2d-b841-73577f0abe80 | Open vSwitch agent | octavia04.desy.de       | None              | :-)   | UP    | neutron-openvswitch-agent |
+--------------------------------------+--------------------+-------------------------+-------------------+-------+-------+---------------------------+
Thanks in advance,
Stefan Bujack