Open Stack

Hi Michael,

We have checked as per your advice. Please find the below details
1)
[health_manager]
bind_port = 5555
bind_ip = 0.0.0.0
controller_ip_port_list = 172.16.0.2:5555

2)

# tcpdump -n -vv  -i  o-hm0
tcpdump: listening on o-hm0, link-type EN10MB (Ethernet), capture size
262144 bytes
10:37:37.440219 IP (tos 0x0, ttl 64, id 12636, offset 0, flags [DF], proto
UDP (17), length 319)
    172.16.1.220.59727 > 172.16.0.2.5555: [bad udp cksum 0x5b3b -> 0xa82a!]
UDP, length 291
10:37:47.495440 IP (tos 0x0, ttl 64, id 13942, offset 0, flags [DF], proto
UDP (17), length 319)
    172.16.1.220.59727 > 172.16.0.2.5555: [bad udp cksum 0x5b3b -> 0x7078!]
UDP, length 291
10:37:57.754072 IP (tos 0x0, ttl 64, id 15228, offset 0, flags [DF], proto
UDP (17), length 319)
    172.16.1.220.59727 > 172.16.0.2.5555: [bad udp cksum 0x5b3b -> 0x3088!]
UDP, length 291
10:38:07.814541 IP (tos 0x0, ttl 64, id 16645, offset 0, flags [DF], proto
UDP (17), length 318)
    172.16.1.220.59727 > 172.16.0.2.5555: [bad udp cksum 0x5b3a -> 0xc7e8!]
UDP, length 290


3 & 4 )

Enabled debug, No errors.

# tail -f /var/log/octavia/octavia-health-manager.log
2021-10-26 10:33:05.659 1277703 WARNING
octavia.amphorae.drivers.health.heartbeat_udp [-] Health Manager
experienced an exception processing a heartbeat message from
('172.16.1.220', 59727). Ignoring this packet. Exception: 'NoneType' object
has no attribute 'encode'
2021-10-26 10:33:05.744 1277704 DEBUG futurist.periodics [-] Submitting
periodic callback
'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
_process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:08.744 1277704 DEBUG futurist.periodics [-] Submitting
periodic callback
'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
_process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:11.744 1277704 DEBUG futurist.periodics [-] Submitting
periodic callback
'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
_process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:14.744 1277704 DEBUG futurist.periodics [-] Submitting
periodic callback
'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
_process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:15.717 1277703 DEBUG
octavia.amphorae.drivers.health.heartbeat_udp [-] *Received packet from*
('172.16.1.220', 59727) dorecv
/usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:189
2021-10-26 10:33:15.717 1277703 WARNING
octavia.amphorae.drivers.health.heartbeat_udp [-] Health Manager
experienced an exception processing a heartbeat message from
('172.16.1.220', 59727). Ignoring this packet. Exception: 'NoneType' object
has no attribute 'encode'
2021-10-26 10:33:17.744 1277704 DEBUG futurist.periodics [-] Submitting
periodic callback
'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
_process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:20.744 1277704 DEBUG futurist.periodics [-] Submitting
periodic callback
'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
_process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:23.744 1277704 DEBUG futurist.periodics [-] Submitting
periodic callback
'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
_process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:25.799 1277703 DEBUG
octavia.amphorae.drivers.health.heartbeat_udp [-] Received packet from
('172.16.1.220', 59727) dorecv
/usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:189
2021-10-26 10:33:25.799 1277703 WARNING
octavia.amphorae.drivers.health.heartbeat_udp [-] Health Manager
experienced an exception processing a heartbeat message from
('172.16.1.220', 59727). Ignoring this packet. Exception: 'NoneType' object
has no attribute 'encode'
2021-10-26 10:33:26.744 1277704 DEBUG futurist.periodics [-] Submitting
periodic callback
'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
_process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:29.744 1277704 DEBUG futurist.periodics [-] Submitting
periodic callback
'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
_process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:32.744 1277704 DEBUG futurist.periodics [-] Submitting
periodic callback
'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
_process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:35.744 1277704 DEBUG futurist.periodics [-] Submitting
periodic callback
'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
_process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:35.842 1277703 DEBUG
octavia.amphorae.drivers.health.heartbeat_udp [-] Received packet from
('172.16.1.220', 59727) dorecv
/usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:189
2021-10-26 10:33:35.843 1277703 WARNING
octavia.amphorae.drivers.health.heartbeat_udp [-] Health Manager
experienced an exception processing a heartbeat message from
('172.16.1.220', 59727). Ignoring this packet. Exception: 'NoneType' object
has no attribute 'encode'

Regards,
Munna


On Mon, Oct 25, 2021 at 10:23 PM Michael Johnson <johnsomor at gmail.com>
wrote:

> Hi Munna,
>
> I am guessing you are seeing the operating status offline?
>
> This is commonly caused by the amphora being unable to reach the
> health manager process.
>
> Another symptom of this is the statistics for the load balancer will
> not increase.
>
> Some things to check:
> 1. Is your controller IP and port list correct?
>
> https://docs.openstack.org/octavia/latest/configuration/configref.html#health_manager.controller_ip_port_list
> 2. Are you seeing the heartbeat packets arrive on the network
> interface on your health manager instance?
> 3. Is the health manager log reporting any issues, such as an
> incorrect heartbeat key?
> 4. If you enable debug logging on the health manager, do you see log
> messages indicating the health manager has received heartbeat packets
> from the amphora? "Received packet from"
>
> Michael
>
> On Mon, Oct 25, 2021 at 5:30 AM Md. Hejbul Tawhid MUNNA
> <munnaeebd at gmail.com> wrote:
> >
> > Hi,
> >
> > We have installed openstack ussuri version from ubuntu universe
> repository.
> >
> > We have installed octavia 6.2.0 version.
> >
> > after creating loadbalancer , listener and pool all are offline. but the
> LB operation is working as expected. changing the pool member is also
> working.
> >
> >
> > octavia is installed in compute node. 5555 is listening and allowed in
> iptables (iptables -I INPUT -i o-hm0 -p udp --dport 5555 -j ACCEPT).
> >
> > amphora to octavia-worker(172.16.0.2) is reachable.
> >
> > Any idea to troubleshoot this issue
> >
> >
> > Please find the log from octavia-worker
> >
> >
> >
> >
> //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
> > 2021-10-25 18:15:13.482 1192307 WARNING
> octavia.amphorae.drivers.haproxy.rest_api_driver
> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
> Retrying.: requests.exceptions.ConnectionError:
> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
> with url: // (Caused by
> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
> 0x7f540e192be0>: Failed to establish a new connection: [Errno 111]
> Connection refused'))
> > 2021-10-25 18:15:18.490 1192307 WARNING
> octavia.amphorae.drivers.haproxy.rest_api_driver
> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
> Retrying.: requests.exceptions.ConnectionError:
> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
> with url: // (Caused by
> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
> 0x7f540c0ddd00>: Failed to establish a new connection: [Errno 111]
> Connection refused'))
> > 2021-10-25 18:15:23.507 1192307 WARNING
> octavia.amphorae.drivers.haproxy.rest_api_driver
> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
> Retrying.: requests.exceptions.ConnectionError:
> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
> with url: // (Caused by
> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
> 0x7f540c0ddd90>: Failed to establish a new connection: [Errno 111]
> Connection refused'))
> > 2021-10-25 18:15:28.511 1192307 WARNING
> octavia.amphorae.drivers.haproxy.rest_api_driver
> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
> Retrying.: requests.exceptions.ConnectionError:
> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
> with url: // (Caused by
> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
> 0x7f540c0ddbb0>: Failed to establish a new connection: [Errno 111]
> Connection refused'))
> > 2021-10-25 18:15:33.521 1192307 WARNING
> octavia.amphorae.drivers.haproxy.rest_api_driver
> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
> Retrying.: requests.exceptions.ConnectionError:
> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
> with url: // (Caused by
> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
> 0x7f540c064190>: Failed to establish a new connection: [Errno 111]
> Connection refused'))
> > 2021-10-25 18:15:38.529 1192307 WARNING
> octavia.amphorae.drivers.haproxy.rest_api_driver
> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
> Retrying.: requests.exceptions.ConnectionError:
> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
> with url: // (Caused by
> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
> 0x7f540d111790>: Failed to establish a new connection: [Errno 111]
> Connection refused'))
> > 2021-10-25 18:15:43.540 1192307 WARNING
> octavia.amphorae.drivers.haproxy.rest_api_driver
> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
> Retrying.: requests.exceptions.ConnectionError:
> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
> with url: // (Caused by
> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
> 0x7f53ec7a3670>: Failed to establish a new connection: [Errno 111]
> Connection refused'))
> > 2021-10-25 18:15:48.549 1192307 WARNING
> octavia.amphorae.drivers.haproxy.rest_api_driver
> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
> Retrying.: requests.exceptions.ConnectionError:
> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
> with url: // (Caused by
> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
> 0x7f53ec74d0d0>: Failed to establish a new connection: [Errno 111]
> Connection refused'))
> > 2021-10-25 18:15:53.554 1192307 WARNING
> octavia.amphorae.drivers.haproxy.rest_api_driver
> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
> Retrying.: requests.exceptions.ConnectionError:
> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
> with url: // (Caused by
> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
> 0x7f53ec75a7f0>: Failed to establish a new connection: [Errno 111]
> Connection refused'))
> > 2021-10-25 18:15:58.561 1192307 WARNING
> octavia.amphorae.drivers.haproxy.rest_api_driver
> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
> Retrying.: requests.exceptions.ConnectionError:
> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
> with url: // (Caused by
> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
> 0x7f53ec75a6a0>: Failed to establish a new connection: [Errno 111]
> Connection refused'))
> > 2021-10-25 18:16:04.707 1192307 INFO
> octavia.controller.worker.v1.tasks.database_tasks
> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Mark ALLOCATED in DB for amphora:
> c661b828-1690-4866-8152-f745c43e0977 with compute id
> c9133819-b8e0-42d6-9544-bf83e3ad4b3f for load balancer:
> c0bd3e21-6983-40c9-8713-859194496b37
> > 2021-10-25 18:16:40.660 1192307 INFO
> octavia.controller.worker.v1.tasks.database_tasks
> [req-78e78f29-3bdb-4b12-ae76-b8daa4926c09 -
> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Mark ACTIVE in DB for load balancer
> id: c0bd3e21-6983-40c9-8713-859194496b37
> > 2021-10-25 18:16:44.317 1192307 INFO
> octavia.controller.queue.v1.endpoints [-] Creating listener
> 'cc45192d-de70-4d59-857b-ac23c4fc8d07'...
> > 2021-10-25 18:16:44.325 1192307 WARNING
> octavia.controller.worker.v1.controller_worker [-] Failed to fetch listener
> cc45192d-de70-4d59-857b-ac23c4fc8d07 from DB. Retrying for up to 60 seconds.
> > 2021-10-25 18:17:35.375 1192307 INFO
> octavia.controller.queue.v1.endpoints [-] Creating pool
> '9d23855f-d849-4ad9-9de1-66ab5cd268eb'...
> > 2021-10-25 18:17:35.382 1192307 WARNING
> octavia.controller.worker.v1.controller_worker [-] Failed to fetch pool
> 9d23855f-d849-4ad9-9de1-66ab5cd268eb from DB. Retrying for up to 60 seconds.
> > 2021-10-25 18:18:02.814 1192307 INFO
> octavia.controller.queue.v1.endpoints [-] Creating member
> '29bb41e5-457c-43ba-9149-5af55e73fe38'...
> > 2021-10-25 18:18:02.825 1192307 WARNING
> octavia.controller.worker.v1.controller_worker [-] Failed to fetch member
> 29bb41e5-457c-43ba-9149-5af55e73fe38 from DB. Retrying for up to 60 seconds.
> >
> > Regards,
> > Munna
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20211026/e1463ae9/attachment-0001.htm>