Open Stack

Dear Gregory,

It's working now.

Thank you so much for your assistance.

Regards,
Munna

On Tue, Oct 26, 2021 at 12:45 PM Gregory Thiemonge <gthiemonge at redhat.com>
wrote:

> Hi,
>
> The health-manager receives the messages but cannot decrypt them. There's
> one configuration setting that is missing in the doc, there's an open
> review to add it:
>
>
> https://review.opendev.org/c/openstack/octavia/+/784022/1/doc/source/install/install-ubuntu.rst
>
> The [health_manager]/heartbeat_key is a parameter that is used to encrypt
> the heartbeat messages sent by the amphora, and there's a known issue that
> occurs when the key is empty, so it should be required.
>
> Greg
>
> On Tue, Oct 26, 2021 at 6:48 AM Md. Hejbul Tawhid MUNNA <
> munnaeebd at gmail.com> wrote:
>
>> Hi Michael,
>>
>> We have checked as per your advice. Please find the below details
>> 1)
>> [health_manager]
>> bind_port = 5555
>> bind_ip = 0.0.0.0
>> controller_ip_port_list = 172.16.0.2:5555
>>
>> 2)
>>
>> # tcpdump -n -vv  -i  o-hm0
>> tcpdump: listening on o-hm0, link-type EN10MB (Ethernet), capture size
>> 262144 bytes
>> 10:37:37.440219 IP (tos 0x0, ttl 64, id 12636, offset 0, flags [DF],
>> proto UDP (17), length 319)
>>     172.16.1.220.59727 > 172.16.0.2.5555: [bad udp cksum 0x5b3b ->
>> 0xa82a!] UDP, length 291
>> 10:37:47.495440 IP (tos 0x0, ttl 64, id 13942, offset 0, flags [DF],
>> proto UDP (17), length 319)
>>     172.16.1.220.59727 > 172.16.0.2.5555: [bad udp cksum 0x5b3b ->
>> 0x7078!] UDP, length 291
>> 10:37:57.754072 IP (tos 0x0, ttl 64, id 15228, offset 0, flags [DF],
>> proto UDP (17), length 319)
>>     172.16.1.220.59727 > 172.16.0.2.5555: [bad udp cksum 0x5b3b ->
>> 0x3088!] UDP, length 291
>> 10:38:07.814541 IP (tos 0x0, ttl 64, id 16645, offset 0, flags [DF],
>> proto UDP (17), length 318)
>>     172.16.1.220.59727 > 172.16.0.2.5555: [bad udp cksum 0x5b3a ->
>> 0xc7e8!] UDP, length 290
>>
>>
>> 3 & 4 )
>>
>> Enabled debug, No errors.
>>
>> # tail -f /var/log/octavia/octavia-health-manager.log
>> 2021-10-26 10:33:05.659 1277703 WARNING
>> octavia.amphorae.drivers.health.heartbeat_udp [-] Health Manager
>> experienced an exception processing a heartbeat message from
>> ('172.16.1.220', 59727). Ignoring this packet. Exception: 'NoneType' object
>> has no attribute 'encode'
>> 2021-10-26 10:33:05.744 1277704 DEBUG futurist.periodics [-] Submitting
>> periodic callback
>> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
>> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
>> 2021-10-26 10:33:08.744 1277704 DEBUG futurist.periodics [-] Submitting
>> periodic callback
>> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
>> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
>> 2021-10-26 10:33:11.744 1277704 DEBUG futurist.periodics [-] Submitting
>> periodic callback
>> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
>> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
>> 2021-10-26 10:33:14.744 1277704 DEBUG futurist.periodics [-] Submitting
>> periodic callback
>> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
>> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
>> 2021-10-26 10:33:15.717 1277703 DEBUG
>> octavia.amphorae.drivers.health.heartbeat_udp [-] *Received packet from*
>> ('172.16.1.220', 59727) dorecv
>> /usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:189
>> 2021-10-26 10:33:15.717 1277703 WARNING
>> octavia.amphorae.drivers.health.heartbeat_udp [-] Health Manager
>> experienced an exception processing a heartbeat message from
>> ('172.16.1.220', 59727). Ignoring this packet. Exception: 'NoneType' object
>> has no attribute 'encode'
>> 2021-10-26 10:33:17.744 1277704 DEBUG futurist.periodics [-] Submitting
>> periodic callback
>> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
>> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
>> 2021-10-26 10:33:20.744 1277704 DEBUG futurist.periodics [-] Submitting
>> periodic callback
>> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
>> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
>> 2021-10-26 10:33:23.744 1277704 DEBUG futurist.periodics [-] Submitting
>> periodic callback
>> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
>> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
>> 2021-10-26 10:33:25.799 1277703 DEBUG
>> octavia.amphorae.drivers.health.heartbeat_udp [-] Received packet from
>> ('172.16.1.220', 59727) dorecv
>> /usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:189
>> 2021-10-26 10:33:25.799 1277703 WARNING
>> octavia.amphorae.drivers.health.heartbeat_udp [-] Health Manager
>> experienced an exception processing a heartbeat message from
>> ('172.16.1.220', 59727). Ignoring this packet. Exception: 'NoneType' object
>> has no attribute 'encode'
>> 2021-10-26 10:33:26.744 1277704 DEBUG futurist.periodics [-] Submitting
>> periodic callback
>> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
>> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
>> 2021-10-26 10:33:29.744 1277704 DEBUG futurist.periodics [-] Submitting
>> periodic callback
>> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
>> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
>> 2021-10-26 10:33:32.744 1277704 DEBUG futurist.periodics [-] Submitting
>> periodic callback
>> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
>> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
>> 2021-10-26 10:33:35.744 1277704 DEBUG futurist.periodics [-] Submitting
>> periodic callback
>> 'octavia.cmd.health_manager.hm_health_check.<locals>.periodic_health_check'
>> _process_scheduled /usr/lib/python3/dist-packages/futurist/periodics.py:641
>> 2021-10-26 10:33:35.842 1277703 DEBUG
>> octavia.amphorae.drivers.health.heartbeat_udp [-] Received packet from
>> ('172.16.1.220', 59727) dorecv
>> /usr/lib/python3/dist-packages/octavia/amphorae/drivers/health/heartbeat_udp.py:189
>> 2021-10-26 10:33:35.843 1277703 WARNING
>> octavia.amphorae.drivers.health.heartbeat_udp [-] Health Manager
>> experienced an exception processing a heartbeat message from
>> ('172.16.1.220', 59727). Ignoring this packet. Exception: 'NoneType' object
>> has no attribute 'encode'
>>
>> Regards,
>> Munna
>>
>>
>> On Mon, Oct 25, 2021 at 10:23 PM Michael Johnson <johnsomor at gmail.com>
>> wrote:
>>
>>> Hi Munna,
>>>
>>> I am guessing you are seeing the operating status offline?
>>>
>>> This is commonly caused by the amphora being unable to reach the
>>> health manager process.
>>>
>>> Another symptom of this is the statistics for the load balancer will
>>> not increase.
>>>
>>> Some things to check:
>>> 1. Is your controller IP and port list correct?
>>>
>>> https://docs.openstack.org/octavia/latest/configuration/configref.html#health_manager.controller_ip_port_list
>>> 2. Are you seeing the heartbeat packets arrive on the network
>>> interface on your health manager instance?
>>> 3. Is the health manager log reporting any issues, such as an
>>> incorrect heartbeat key?
>>> 4. If you enable debug logging on the health manager, do you see log
>>> messages indicating the health manager has received heartbeat packets
>>> from the amphora? "Received packet from"
>>>
>>> Michael
>>>
>>> On Mon, Oct 25, 2021 at 5:30 AM Md. Hejbul Tawhid MUNNA
>>> <munnaeebd at gmail.com> wrote:
>>> >
>>> > Hi,
>>> >
>>> > We have installed openstack ussuri version from ubuntu universe
>>> repository.
>>> >
>>> > We have installed octavia 6.2.0 version.
>>> >
>>> > after creating loadbalancer , listener and pool all are offline. but
>>> the LB operation is working as expected. changing the pool member is also
>>> working.
>>> >
>>> >
>>> > octavia is installed in compute node. 5555 is listening and allowed in
>>> iptables (iptables -I INPUT -i o-hm0 -p udp --dport 5555 -j ACCEPT).
>>> >
>>> > amphora to octavia-worker(172.16.0.2) is reachable.
>>> >
>>> > Any idea to troubleshoot this issue
>>> >
>>> >
>>> > Please find the log from octavia-worker
>>> >
>>> >
>>> >
>>> >
>>> //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
>>> > 2021-10-25 18:15:13.482 1192307 WARNING
>>> octavia.amphorae.drivers.haproxy.rest_api_driver
>>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>>> Retrying.: requests.exceptions.ConnectionError:
>>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>>> with url: // (Caused by
>>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>>> 0x7f540e192be0>: Failed to establish a new connection: [Errno 111]
>>> Connection refused'))
>>> > 2021-10-25 18:15:18.490 1192307 WARNING
>>> octavia.amphorae.drivers.haproxy.rest_api_driver
>>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>>> Retrying.: requests.exceptions.ConnectionError:
>>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>>> with url: // (Caused by
>>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>>> 0x7f540c0ddd00>: Failed to establish a new connection: [Errno 111]
>>> Connection refused'))
>>> > 2021-10-25 18:15:23.507 1192307 WARNING
>>> octavia.amphorae.drivers.haproxy.rest_api_driver
>>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>>> Retrying.: requests.exceptions.ConnectionError:
>>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>>> with url: // (Caused by
>>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>>> 0x7f540c0ddd90>: Failed to establish a new connection: [Errno 111]
>>> Connection refused'))
>>> > 2021-10-25 18:15:28.511 1192307 WARNING
>>> octavia.amphorae.drivers.haproxy.rest_api_driver
>>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>>> Retrying.: requests.exceptions.ConnectionError:
>>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>>> with url: // (Caused by
>>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>>> 0x7f540c0ddbb0>: Failed to establish a new connection: [Errno 111]
>>> Connection refused'))
>>> > 2021-10-25 18:15:33.521 1192307 WARNING
>>> octavia.amphorae.drivers.haproxy.rest_api_driver
>>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>>> Retrying.: requests.exceptions.ConnectionError:
>>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>>> with url: // (Caused by
>>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>>> 0x7f540c064190>: Failed to establish a new connection: [Errno 111]
>>> Connection refused'))
>>> > 2021-10-25 18:15:38.529 1192307 WARNING
>>> octavia.amphorae.drivers.haproxy.rest_api_driver
>>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>>> Retrying.: requests.exceptions.ConnectionError:
>>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>>> with url: // (Caused by
>>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>>> 0x7f540d111790>: Failed to establish a new connection: [Errno 111]
>>> Connection refused'))
>>> > 2021-10-25 18:15:43.540 1192307 WARNING
>>> octavia.amphorae.drivers.haproxy.rest_api_driver
>>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>>> Retrying.: requests.exceptions.ConnectionError:
>>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>>> with url: // (Caused by
>>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>>> 0x7f53ec7a3670>: Failed to establish a new connection: [Errno 111]
>>> Connection refused'))
>>> > 2021-10-25 18:15:48.549 1192307 WARNING
>>> octavia.amphorae.drivers.haproxy.rest_api_driver
>>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>>> Retrying.: requests.exceptions.ConnectionError:
>>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>>> with url: // (Caused by
>>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>>> 0x7f53ec74d0d0>: Failed to establish a new connection: [Errno 111]
>>> Connection refused'))
>>> > 2021-10-25 18:15:53.554 1192307 WARNING
>>> octavia.amphorae.drivers.haproxy.rest_api_driver
>>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>>> Retrying.: requests.exceptions.ConnectionError:
>>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>>> with url: // (Caused by
>>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>>> 0x7f53ec75a7f0>: Failed to establish a new connection: [Errno 111]
>>> Connection refused'))
>>> > 2021-10-25 18:15:58.561 1192307 WARNING
>>> octavia.amphorae.drivers.haproxy.rest_api_driver
>>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Could not connect to instance.
>>> Retrying.: requests.exceptions.ConnectionError:
>>> HTTPSConnectionPool(host='172.16.1.220', port=9443): Max retries exceeded
>>> with url: // (Caused by
>>> NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at
>>> 0x7f53ec75a6a0>: Failed to establish a new connection: [Errno 111]
>>> Connection refused'))
>>> > 2021-10-25 18:16:04.707 1192307 INFO
>>> octavia.controller.worker.v1.tasks.database_tasks
>>> [req-806e5e6b-46b0-4a52-900f-7c2d22d4442d -
>>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Mark ALLOCATED in DB for amphora:
>>> c661b828-1690-4866-8152-f745c43e0977 with compute id
>>> c9133819-b8e0-42d6-9544-bf83e3ad4b3f for load balancer:
>>> c0bd3e21-6983-40c9-8713-859194496b37
>>> > 2021-10-25 18:16:40.660 1192307 INFO
>>> octavia.controller.worker.v1.tasks.database_tasks
>>> [req-78e78f29-3bdb-4b12-ae76-b8daa4926c09 -
>>> 9a817a70161d45fd9f0b5fe2cad30f5c - - -] Mark ACTIVE in DB for load balancer
>>> id: c0bd3e21-6983-40c9-8713-859194496b37
>>> > 2021-10-25 18:16:44.317 1192307 INFO
>>> octavia.controller.queue.v1.endpoints [-] Creating listener
>>> 'cc45192d-de70-4d59-857b-ac23c4fc8d07'...
>>> > 2021-10-25 18:16:44.325 1192307 WARNING
>>> octavia.controller.worker.v1.controller_worker [-] Failed to fetch listener
>>> cc45192d-de70-4d59-857b-ac23c4fc8d07 from DB. Retrying for up to 60 seconds.
>>> > 2021-10-25 18:17:35.375 1192307 INFO
>>> octavia.controller.queue.v1.endpoints [-] Creating pool
>>> '9d23855f-d849-4ad9-9de1-66ab5cd268eb'...
>>> > 2021-10-25 18:17:35.382 1192307 WARNING
>>> octavia.controller.worker.v1.controller_worker [-] Failed to fetch pool
>>> 9d23855f-d849-4ad9-9de1-66ab5cd268eb from DB. Retrying for up to 60 seconds.
>>> > 2021-10-25 18:18:02.814 1192307 INFO
>>> octavia.controller.queue.v1.endpoints [-] Creating member
>>> '29bb41e5-457c-43ba-9149-5af55e73fe38'...
>>> > 2021-10-25 18:18:02.825 1192307 WARNING
>>> octavia.controller.worker.v1.controller_worker [-] Failed to fetch member
>>> 29bb41e5-457c-43ba-9149-5af55e73fe38 from DB. Retrying for up to 60 seconds.
>>> >
>>> > Regards,
>>> > Munna
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20211026/48d5a3ba/attachment-0001.htm>