Hi Michael,

We have checked as per your advice. Please find the below details
1)
[health_manager]
bind_port = 5555
bind_ip = 0.0.0.0
controller_ip_port_list = 172.16.0.2:5555

2) 

# tcpdump -n -vv  -i  o-hm0
tcpdump: listening on o-hm0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:37:37.440219 IP (tos 0x0, ttl 64, id 12636, offset 0, flags [DF], proto UDP (17), length 319)
    172.16.1.220.59727 > 172.16.0.2.5555: [bad udp cksum 0x5b3b -> 0xa82a!] UDP, length 291
10:37:47.495440 IP (tos 0x0, ttl 64, id 13942, offset 0, flags [DF], proto UDP (17), length 319)
    172.16.1.220.59727 > 172.16.0.2.5555: [bad udp cksum 0x5b3b -> 0x7078!] UDP, length 291
10:37:57.754072 IP (tos 0x0, ttl 64, id 15228, offset 0, flags [DF], proto UDP (17), length 319)
    172.16.1.220.59727 > 172.16.0.2.5555: [bad udp cksum 0x5b3b -> 0x3088!] UDP, length 291
10:38:07.814541 IP (tos 0x0, ttl 64, id 16645, offset 0, flags [DF], proto UDP (17), length 318)
    172.16.1.220.59727 > 172.16.0.2.5555: [bad udp cksum 0x5b3a -> 0xc7e8!] UDP, length 290


3 & 4 )

Enabled debug, No errors. 

# tail -f /var/log/octavia/octavia-health-manager.log
2021-10-26 10:33:05.659 1277703 WARNING octavia.amphorae.drivers.health.heartbeat_udp [-] Health Manager experienced an exception processing a heartbeat message from ('172.16.1.220', 59727). Ignoring this packet. Exception: 'NoneType' object has no attribute 'encode'
2021-10-26 10:33:05.744 1277704 DEBUG futurist.periodics [-] Submitting periodic callback 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check' _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:08.744 1277704 DEBUG futurist.periodics [-] Submitting periodic callback 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check' _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:11.744 1277704 DEBUG futurist.periodics [-] Submitting periodic callback 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check' _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:14.744 1277704 DEBUG futurist.periodics [-] Submitting periodic callback 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check' _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:15.717 1277703 DEBUG octavia.amphorae.drivers.health.heartbeat_udp [-] Received packet from ('172.16.1.220', 59727) dorecv /usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:189
2021-10-26 10:33:15.717 1277703 WARNING octavia.amphorae.drivers.health.heartbeat_udp [-] Health Manager experienced an exception processing a heartbeat message from ('172.16.1.220', 59727). Ignoring this packet. Exception: 'NoneType' object has no attribute 'encode'
2021-10-26 10:33:17.744 1277704 DEBUG futurist.periodics [-] Submitting periodic callback 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check' _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:20.744 1277704 DEBUG futurist.periodics [-] Submitting periodic callback 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check' _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:23.744 1277704 DEBUG futurist.periodics [-] Submitting periodic callback 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check' _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:25.799 1277703 DEBUG octavia.amphorae.drivers.health.heartbeat_udp [-] Received packet from ('172.16.1.220', 59727) dorecv /usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:189
2021-10-26 10:33:25.799 1277703 WARNING octavia.amphorae.drivers.health.heartbeat_udp [-] Health Manager experienced an exception processing a heartbeat message from ('172.16.1.220', 59727). Ignoring this packet. Exception: 'NoneType' object has no attribute 'encode'
2021-10-26 10:33:26.744 1277704 DEBUG futurist.periodics [-] Submitting periodic callback 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check' _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:29.744 1277704 DEBUG futurist.periodics [-] Submitting periodic callback 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check' _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:32.744 1277704 DEBUG futurist.periodics [-] Submitting periodic callback 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check' _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:35.744 1277704 DEBUG futurist.periodics [-] Submitting periodic callback 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check' _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
2021-10-26 10:33:35.842 1277703 DEBUG octavia.amphorae.drivers.health.heartbeat_udp [-] Received packet from ('172.16.1.220', 59727) dorecv /usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:189
2021-10-26 10:33:35.843 1277703 WARNING octavia.amphorae.drivers.health.heartbeat_udp [-] Health Manager experienced an exception processing a heartbeat message from ('172.16.1.220', 59727). Ignoring this packet. Exception: 'NoneType' object has no attribute 'encode'

Regards,
Munna


On Mon, Oct 25, 2021 at 10:23 PM Michael Johnson <johnsomor@gmail.com> wrote:
Hi Munna,

I am guessing you are seeing the operating status offline?

This is commonly caused by the amphora being unable to reach the
health manager process.

Another symptom of this is the statistics for the load balancer will
not increase.

Some things to check:
1. Is your controller IP and port list correct?
https://docs.openstack.org/octavia/latest/configuration/configref.html#health_manager.controller_ip_port_list
2. Are you seeing the heartbeat packets arrive on the network
interface on your health manager instance?
3. Is the health manager log reporting any issues, such as an
incorrect heartbeat key?
4. If you enable debug logging on the health manager, do you see log
messages indicating the health manager has received heartbeat packets
from the amphora? "Received packet from"

Michael

On Mon, Oct 25, 2021 at 5:30 AM Md. Hejbul Tawhid MUNNA
<munnaeebd@gmail.com> wrote:
>
> Hi,
>
> We have installed openstack ussuri version from ubuntu universe repository.
>
> We have installed octavia 6.2.0 version.
>
> after creating loadbalancer , listener and pool all are offline. but the LB operation is working as expected. changing the pool member is also working.
>
>
> octavia is installed in compute node. 5555 is listening and allowed in iptables (iptables -I INPUT -i o-hm0 -p udp --dport 5555 -j ACCEPT).
>
> amphora to octavia-worker(172.16.0.2) is reachable.
>
> Any idea to troubleshoot this issue
>
>
> Please find the log from octavia-worker
>
>
>
> //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
> 2021-10-25 18:15:13.482 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f540e192be0>: Failed to establish a new connection: [Errno 111] Connection refused'))
> 2021-10-25 18:15:18.490 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f540c0ddd00>: Failed to establish a new connection: [Errno 111] Connection refused'))
> 2021-10-25 18:15:23.507 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f540c0ddd90>: Failed to establish a new connection: [Errno 111] Connection refused'))
> 2021-10-25 18:15:28.511 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f540c0ddbb0>: Failed to establish a new connection: [Errno 111] Connection refused'))
> 2021-10-25 18:15:33.521 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f540c064190>: Failed to establish a new connection: [Errno 111] Connection refused'))
> 2021-10-25 18:15:38.529 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f540d111790>: Failed to establish a new connection: [Errno 111] Connection refused'))
> 2021-10-25 18:15:43.540 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f53ec7a3670>: Failed to establish a new connection: [Errno 111] Connection refused'))
> 2021-10-25 18:15:48.549 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f53ec74d0d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
> 2021-10-25 18:15:53.554 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f53ec75a7f0>: Failed to establish a new connection: [Errno 111] Connection refused'))
> 2021-10-25 18:15:58.561 1192307 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f53ec75a6a0>: Failed to establish a new connection: [Errno 111] Connection refused'))
> 2021-10-25 18:16:04.707 1192307 INFO octavia.controller.worker.v1.tasks.database_tasks [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Mark ALLOCATED in DB for amphora: c661b828-1690-4866-8152-f745c43e0977 with compute id c9133819-b8e0-42d6-9544-bf83e3ad4b3f for load balancer: c0bd3e21-6983-40c9-8713-859194496b37
> 2021-10-25 18:16:40.660 1192307 INFO octavia.controller.worker.v1.tasks.database_tasks [req-78e78f29-3bdb-4b12-ae76-b8daa4926c09 - 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Mark ACTIVE in DB for load balancer id: c0bd3e21-6983-40c9-8713-859194496b37
> 2021-10-25 18:16:44.317 1192307 INFO octavia.controller.queue.v1.endpoints [-] Creating listener 'cc45192d-de70-4d59-857b-ac23c4fc8d07'...
> 2021-10-25 18:16:44.325 1192307 WARNING octavia.controller.worker.v1.controller_worker [-] Failed to fetch listener cc45192d-de70-4d59-857b-ac23c4fc8d07 from DB. Retrying for up to 60 seconds.
> 2021-10-25 18:17:35.375 1192307 INFO octavia.controller.queue.v1.endpoints [-] Creating pool '9d23855f-d849-4ad9-9de1-66ab5cd268eb'...
> 2021-10-25 18:17:35.382 1192307 WARNING octavia.controller.worker.v1.controller_worker [-] Failed to fetch pool 9d23855f-d849-4ad9-9de1-66ab5cd268eb from DB. Retrying for up to 60 seconds.
> 2021-10-25 18:18:02.814 1192307 INFO octavia.controller.queue.v1.endpoints [-] Creating member '29bb41e5-457c-43ba-9149-5af55e73fe38'...
> 2021-10-25 18:18:02.825 1192307 WARNING octavia.controller.worker.v1.controller_worker [-] Failed to fetch member 29bb41e5-457c-43ba-9149-5af55e73fe38 from DB. Retrying for up to 60 seconds.
>
> Regards,
> Munna