Open Stack

Hi,

The health-manager receives the messages but cannot decrypt them. There's
one configuration setting that is missing in the doc, there's an open
review to add it:

https://review.opendev.org/c/openstack/octavia/+/784022/1/doc/source/install/install-ubuntu.rst

The [health_manager]/heartbeat_key is a parameter that is used to encrypt
the heartbeat messages sent by the amphora, and there's a known issue that
occurs when the key is empty, so it should be required.

Greg

On Tue, Oct 26, 2021 at 6:48 AM Md. Hejbul Tawhid MUNNA <munnaeebd at gmail.com>
wrote:

> Hi Michael,
>
> We have checked as per your advice. Please find the below details
> 1)
> [health_manager]
> bind_port = 5555
> bind_ip = 0.0.0.0
> controller_ip_port_list = 172.16.0.2:5555
>
> 2)
>
> # tcpdump -n -vv  -i  o-hm0
> tcpdump: listening on o-hm0, link-type EN10MB (Ethernet), capture size
> 262144 bytes
> 10:37:37.440219 IP (tos 0x0, ttl 64, id 12636, offset 0, flags [DF], proto
> UDP (17), length 319)
>     172.16.1.220.59727 > 172.16.0.2.5555: [bad udp cksum 0x5b3b ->
> 0xa82a!] UDP, length 291
> 10:37:47.495440 IP (tos 0x0, ttl 64, id 13942, offset 0, flags [DF], proto
> UDP (17), length 319)
>     172.16.1.220.59727 > 172.16.0.2.5555: [bad udp cksum 0x5b3b ->
> 0x7078!] UDP, length 291
> 10:37:57.754072 IP (tos 0x0, ttl 64, id 15228, offset 0, flags [DF], proto
> UDP (17), length 319)
>     172.16.1.220.59727 > 172.16.0.2.5555: [bad udp cksum 0x5b3b ->
> 0x3088!] UDP, length 291
> 10:38:07.814541 IP (tos 0x0, ttl 64, id 16645, offset 0, flags [DF], proto
> UDP (17), length 318)
>     172.16.1.220.59727 > 172.16.0.2.5555: [bad udp cksum 0x5b3a ->
> 0xc7e8!] UDP, length 290
>
>
> 3 & 4 )
>
> Enabled debug, No errors.
>
> # tail -f /var/log/octavia/octavia-health-manager.log
> 2021-10-26 10:33:05.659 1277703 WARNING
> octavia.amphorae.drivers.health.heartbeat_udp [-] Health Manager
> experienced an exception processing a heartbeat message from
> ('172.16.1.220', 59727). Ignoring this packet. Exception: 'NoneType' object
> has no attribute 'encode'
> 2021-10-26 10:33:05.744 1277704 DEBUG futurist.periodics [-] Submitting
> periodic callback
> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
> 2021-10-26 10:33:08.744 1277704 DEBUG futurist.periodics [-] Submitting
> periodic callback
> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
> 2021-10-26 10:33:11.744 1277704 DEBUG futurist.periodics [-] Submitting
> periodic callback
> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
> 2021-10-26 10:33:14.744 1277704 DEBUG futurist.periodics [-] Submitting
> periodic callback
> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
> 2021-10-26 10:33:15.717 1277703 DEBUG
> octavia.amphorae.drivers.health.heartbeat_udp [-] *Received packet from*
> ('172.16.1.220', 59727) dorecv
> /usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:189
> 2021-10-26 10:33:15.717 1277703 WARNING
> octavia.amphorae.drivers.health.heartbeat_udp [-] Health Manager
> experienced an exception processing a heartbeat message from
> ('172.16.1.220', 59727). Ignoring this packet. Exception: 'NoneType' object
> has no attribute 'encode'
> 2021-10-26 10:33:17.744 1277704 DEBUG futurist.periodics [-] Submitting
> periodic callback
> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
> 2021-10-26 10:33:20.744 1277704 DEBUG futurist.periodics [-] Submitting
> periodic callback
> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
> 2021-10-26 10:33:23.744 1277704 DEBUG futurist.periodics [-] Submitting
> periodic callback
> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
> 2021-10-26 10:33:25.799 1277703 DEBUG
> octavia.amphorae.drivers.health.heartbeat_udp [-] Received packet from
> ('172.16.1.220', 59727) dorecv
> /usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:189
> 2021-10-26 10:33:25.799 1277703 WARNING
> octavia.amphorae.drivers.health.heartbeat_udp [-] Health Manager
> experienced an exception processing a heartbeat message from
> ('172.16.1.220', 59727). Ignoring this packet. Exception: 'NoneType' object
> has no attribute 'encode'
> 2021-10-26 10:33:26.744 1277704 DEBUG futurist.periodics [-] Submitting
> periodic callback
> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
> 2021-10-26 10:33:29.744 1277704 DEBUG futurist.periodics [-] Submitting
> periodic callback
> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
> 2021-10-26 10:33:32.744 1277704 DEBUG futurist.periodics [-] Submitting
> periodic callback
> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
> 2021-10-26 10:33:35.744 1277704 DEBUG futurist.periodics [-] Submitting
> periodic callback
> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
> 2021-10-26 10:33:35.842 1277703 DEBUG
> octavia.amphorae.drivers.health.heartbeat_udp [-] Received packet from
> ('172.16.1.220', 59727) dorecv
> /usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:189
> 2021-10-26 10:33:35.843 1277703 WARNING
> octavia.amphorae.drivers.health.heartbeat_udp [-] Health Manager
> experienced an exception processing a heartbeat message from
> ('172.16.1.220', 59727). Ignoring this packet. Exception: 'NoneType' object
> has no attribute 'encode'
>
> Regards,
> Munna
>
>
> On Mon, Oct 25, 2021 at 10:23 PM Michael Johnson <johnsomor at gmail.com>
> wrote:
>
>> Hi Munna,
>>
>> I am guessing you are seeing the operating status offline?
>>
>> This is commonly caused by the amphora being unable to reach the
>> health manager process.
>>
>> Another symptom of this is the statistics for the load balancer will
>> not increase.
>>
>> Some things to check:
>> 1. Is your controller IP and port list correct?
>>
>> https://docs.openstack.org/octavia/latest/configuration/configref.html#health_manager.controller_ip_port_list
>> 2. Are you seeing the heartbeat packets arrive on the network
>> interface on your health manager instance?
>> 3. Is the health manager log reporting any issues, such as an
>> incorrect heartbeat key?
>> 4. If you enable debug logging on the health manager, do you see log
>> messages indicating the health manager has received heartbeat packets
>> from the amphora? "Received packet from"
>>
>> Michael
>>
>> On Mon, Oct 25, 2021 at 5:30 AM Md. Hejbul Tawhid MUNNA
>> <munnaeebd at gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > We have installed openstack ussuri version from ubuntu universe
>> repository.
>> >
>> > We have installed octavia 6.2.0 version.
>> >
>> > after creating loadbalancer , listener and pool all are offline. but
>> the LB operation is working as expected. changing the pool member is also
>> working.
>> >
>> >
>> > octavia is installed in compute node. 5555 is listening and allowed in
>> iptables (iptables -I INPUT -i o-hm0 -p udp --dport 5555 -j ACCEPT).
>> >
>> > amphora to octavia-worker(172.16.0.2) is reachable.
>> >
>> > Any idea to troubleshoot this issue
>> >
>> >
>> > Please find the log from octavia-worker
>> >
>> >
>> >
>> >
>> //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
>> > 2021-10-25 18:15:13.482 1192307 WARNING
>> octavia.amphorae.drivers.haproxy.rest_api_driver
>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>> Retrying.: requests.exceptions.ConnectionError:
>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>> with url: // (Caused by
>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>> 0x7f540e192be0>: Failed to establish a new connection: [Errno 111]
>> Connection refused'))
>> > 2021-10-25 18:15:18.490 1192307 WARNING
>> octavia.amphorae.drivers.haproxy.rest_api_driver
>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>> Retrying.: requests.exceptions.ConnectionError:
>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>> with url: // (Caused by
>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>> 0x7f540c0ddd00>: Failed to establish a new connection: [Errno 111]
>> Connection refused'))
>> > 2021-10-25 18:15:23.507 1192307 WARNING
>> octavia.amphorae.drivers.haproxy.rest_api_driver
>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>> Retrying.: requests.exceptions.ConnectionError:
>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>> with url: // (Caused by
>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>> 0x7f540c0ddd90>: Failed to establish a new connection: [Errno 111]
>> Connection refused'))
>> > 2021-10-25 18:15:28.511 1192307 WARNING
>> octavia.amphorae.drivers.haproxy.rest_api_driver
>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>> Retrying.: requests.exceptions.ConnectionError:
>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>> with url: // (Caused by
>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>> 0x7f540c0ddbb0>: Failed to establish a new connection: [Errno 111]
>> Connection refused'))
>> > 2021-10-25 18:15:33.521 1192307 WARNING
>> octavia.amphorae.drivers.haproxy.rest_api_driver
>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>> Retrying.: requests.exceptions.ConnectionError:
>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>> with url: // (Caused by
>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>> 0x7f540c064190>: Failed to establish a new connection: [Errno 111]
>> Connection refused'))
>> > 2021-10-25 18:15:38.529 1192307 WARNING
>> octavia.amphorae.drivers.haproxy.rest_api_driver
>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>> Retrying.: requests.exceptions.ConnectionError:
>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>> with url: // (Caused by
>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>> 0x7f540d111790>: Failed to establish a new connection: [Errno 111]
>> Connection refused'))
>> > 2021-10-25 18:15:43.540 1192307 WARNING
>> octavia.amphorae.drivers.haproxy.rest_api_driver
>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>> Retrying.: requests.exceptions.ConnectionError:
>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>> with url: // (Caused by
>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>> 0x7f53ec7a3670>: Failed to establish a new connection: [Errno 111]
>> Connection refused'))
>> > 2021-10-25 18:15:48.549 1192307 WARNING
>> octavia.amphorae.drivers.haproxy.rest_api_driver
>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>> Retrying.: requests.exceptions.ConnectionError:
>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>> with url: // (Caused by
>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>> 0x7f53ec74d0d0>: Failed to establish a new connection: [Errno 111]
>> Connection refused'))
>> > 2021-10-25 18:15:53.554 1192307 WARNING
>> octavia.amphorae.drivers.haproxy.rest_api_driver
>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>> Retrying.: requests.exceptions.ConnectionError:
>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>> with url: // (Caused by
>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>> 0x7f53ec75a7f0>: Failed to establish a new connection: [Errno 111]
>> Connection refused'))
>> > 2021-10-25 18:15:58.561 1192307 WARNING
>> octavia.amphorae.drivers.haproxy.rest_api_driver
>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>> Retrying.: requests.exceptions.ConnectionError:
>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>> with url: // (Caused by
>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>> 0x7f53ec75a6a0>: Failed to establish a new connection: [Errno 111]
>> Connection refused'))
>> > 2021-10-25 18:16:04.707 1192307 INFO
>> octavia.controller.worker.v1.tasks.database_tasks
>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Mark ALLOCATED in DB for amphora:
>> c661b828-1690-4866-8152-f745c43e0977 with compute id
>> c9133819-b8e0-42d6-9544-bf83e3ad4b3f for load balancer:
>> c0bd3e21-6983-40c9-8713-859194496b37
>> > 2021-10-25 18:16:40.660 1192307 INFO
>> octavia.controller.worker.v1.tasks.database_tasks
>> [req-78e78f29-3bdb-4b12-ae76-b8daa4926c09 -
>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Mark ACTIVE in DB for load balancer
>> id: c0bd3e21-6983-40c9-8713-859194496b37
>> > 2021-10-25 18:16:44.317 1192307 INFO
>> octavia.controller.queue.v1.endpoints [-] Creating listener
>> 'cc45192d-de70-4d59-857b-ac23c4fc8d07'...
>> > 2021-10-25 18:16:44.325 1192307 WARNING
>> octavia.controller.worker.v1.controller_worker [-] Failed to fetch listener
>> cc45192d-de70-4d59-857b-ac23c4fc8d07 from DB. Retrying for up to 60 seconds.
>> > 2021-10-25 18:17:35.375 1192307 INFO
>> octavia.controller.queue.v1.endpoints [-] Creating pool
>> '9d23855f-d849-4ad9-9de1-66ab5cd268eb'...
>> > 2021-10-25 18:17:35.382 1192307 WARNING
>> octavia.controller.worker.v1.controller_worker [-] Failed to fetch pool
>> 9d23855f-d849-4ad9-9de1-66ab5cd268eb from DB. Retrying for up to 60 seconds.
>> > 2021-10-25 18:18:02.814 1192307 INFO
>> octavia.controller.queue.v1.endpoints [-] Creating member
>> '29bb41e5-457c-43ba-9149-5af55e73fe38'...
>> > 2021-10-25 18:18:02.825 1192307 WARNING
>> octavia.controller.worker.v1.controller_worker [-] Failed to fetch member
>> 29bb41e5-457c-43ba-9149-5af55e73fe38 from DB. Retrying for up to 60 seconds.
>> >
>> > Regards,
>> > Munna
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20211026/6509fef4/attachment-0001.htm>