[octavia] victoria - loadbalancer works but its operational status is offline
Hello, I have stable/victoria baremetal openstack with octavia installed on centos8 using openvswitch mechanism driver: octavia api on controller, health-manager,housekeeping,worker on 3 compute/network nodes. Official docs include only ubuntu with linuxbridge mechanism but I used https://github.com/prastamaha/openstack-octavia as a reference to get it working on centos8 with ovs. I will push those docs instructions for centos8 soon: https://github.com/openstack/octavia/tree/master/doc/source/install. I created basic http scenario using https://docs.openstack.org/octavia/victoria/user/guides/basic-cookbook.html#.... Loadbalancer works but its operational status is offline (openstack_loadbalancer_outputs.txt). On all octavia workers I see the same warning message in health_manager.log: Health Manager experienced an exception processing a heartbeat message from ('172.31.255.233', 1907). Ignoring this packet. Exception: 'NoneType' object has no attribute 'encode' I've searched for related active bug but all I found is this not related in my opinion: https://storyboard.openstack.org/#!/story/2008615 I'm attaching all info I've gathered: - octavia.conf and health_manager debug logs (octavia_config_and_health_manager_logs.txt) - tcpdump from amphora VM (tcpdump_from_amphora_vm.txt) - tcpdump from octavia worker (tcpdump_from_octavia_worker.txt) - debug amphora-agent.log from amphora VM (amphora-agent.log) Can you point me to the right direction what I have missed? Thanks! Piotr Mossakowski https://github.com/moss2k13
Hi, Most of the OFFLINE operational status issues are caused by communication problems between the amphorae and the Octavia health-manager. In your case, the "Ignoring this packet. Exception: 'NoneType' object has no attribute 'encode'" log message shows that the health-manager receives the heartbeat packets from the amphorae but it is unable to decode them. Those packets are encrypted JSON messages and it seems that the key ([health_manager].heartbeat_key see https://docs.openstack.org/octavia/latest/configuration/configref.html#healt...) used to encrypt those messages is not defined in your configuration file. So I would suggest configuring it and restarting the Octavia services, then you can re-create or failover the load balancers (you cannot change this parameter in a running load balancer). Gregory On Sun, Mar 21, 2021 at 6:17 PM <apps@mossakowski.ch> wrote:
Hello, I have stable/victoria baremetal openstack with octavia installed on centos8 using openvswitch mechanism driver: octavia api on controller, health-manager,housekeeping,worker on 3 compute/network nodes. Official docs include only ubuntu with linuxbridge mechanism but I used https://github.com/prastamaha/openstack-octavia as a reference to get it working on centos8 with ovs. I will push those docs instructions for centos8 soon: https://github.com/openstack/octavia/tree/master/doc/source/install. I created basic http scenario using https://docs.openstack.org/octavia/victoria/user/guides/basic-cookbook.html#... . Loadbalancer works but its operational status is offline (openstack_loadbalancer_outputs.txt). On all octavia workers I see the same warning message in health_manager.log: Health Manager experienced an exception processing a heartbeat message from ('172.31.255.233', 1907). Ignoring this packet. Exception: 'NoneType' object has no attribute 'encode' I've searched for related active bug but all I found is this not related in my opinion: https://storyboard.openstack.org/#!/story/2008615 I'm attaching all info I've gathered:
- octavia.conf and health_manager debug logs (octavia_config_and_health_manager_logs.txt) - tcpdump from amphora VM (tcpdump_from_amphora_vm.txt) - tcpdump from octavia worker (tcpdump_from_octavia_worker.txt) - debug amphora-agent.log from amphora VM (amphora-agent.log)
Can you point me to the right direction what I have missed? Thanks! Piotr Mossakowski https://github.com/moss2k13
Yes, that was it: missing \[health\_manager\].heartbeat\_key in octavia.conf It is not present in openstack victoria octavia docs, I'll push it together with my installation guide for centos8. Thanks for your accurate hint Gregory. It is always crucial to ask the right guy:) Regards, Piotr Mossakowski Sent from ProtonMail mobile \-------- Original Message -------- On 22 Mar 2021, 09:10, Gregory Thiemonge < gthiemonge@redhat.com> wrote:
Hi,
Most of the OFFLINE operational status issues are caused by communication problems between the amphorae and the Octavia health-manager.
In your case, the "Ignoring this packet. Exception: 'NoneType' object has no attribute 'encode'" log message shows that the health-manager receives the heartbeat packets from the amphorae but it is unable to decode them. Those packets are encrypted JSON messages and it seems that the key (\[health\_manager\].heartbeat\_key see [https://docs.openstack.org/octavia/latest/configuration/configref.html\#health-manager][https_docs.openstack.org_octavia_latest_configuration_configref.html_health-manager]) used to encrypt those messages is not defined in your configuration file. So I would suggest configuring it and restarting the Octavia services, then you can re-create or failover the load balancers (you cannot change this parameter in a running load balancer).
Gregory
On Sun, Mar 21, 2021 at 6:17 PM <[apps@mossakowski.ch][apps_mossakowski.ch]> wrote:
Hello,
I have stable/victoria baremetal openstack with octavia installed on centos8 using openvswitch mechanism driver: octavia api on controller, health-manager,housekeeping,worker on 3 compute/network nodes.
Official docs include only ubuntu with linuxbridge mechanism but I used https://github.com/prastamaha/openstack-octavia as a reference to get it working on centos8 with ovs.
I will push those docs instructions for centos8 soon: https://github.com/openstack/octavia/tree/master/doc/source/install.
I created basic http scenario using [https://docs.openstack.org/octavia/victoria/user/guides/basic-cookbook.html\#deploy-a-basic-http-load-balancer][https_docs.openstack.org_octavia_victoria_user_guides_basic-cookbook.html_deploy-a-basic-http-load-balancer].
Loadbalancer works but its operational status is offline (openstack\_loadbalancer\_outputs.txt).
On all octavia workers I see the same warning message in health\_manager.log:
Health Manager experienced an exception processing a heartbeat message from ('172.31.255.233', 1907). Ignoring this packet. Exception: 'NoneType' object has no attribute 'encode'
I've searched for related active bug but all I found is this not related in my opinion: [https://storyboard.openstack.org/\#!/story/2008615][https_storyboard.openstack.org_story_2008615]
I'm attaching all info I've gathered:
* octavia.conf and health\_manager debug logs (octavia\_config\_and\_health\_manager\_logs.txt)
* tcpdump from amphora VM (tcpdump\_from\_amphora\_vm.txt)
* tcpdump from octavia worker (tcpdump\_from\_octavia\_worker.txt)
* debug amphora-agent.log from amphora VM (amphora-agent.log)
Can you point me to the right direction what I have missed?
Thanks!
Piotr Mossakowski
[https_docs.openstack.org_octavia_latest_configuration_configref.html_health-manager]: https://docs.openstack.org/octavia/latest/configuration/configref.html#healt... [apps_mossakowski.ch]: mailto:apps@mossakowski.ch [https_docs.openstack.org_octavia_victoria_user_guides_basic-cookbook.html_deploy-a-basic-http-load-balancer]: https://docs.openstack.org/octavia/victoria/user/guides/basic-cookbook.html#... [https_storyboard.openstack.org_story_2008615]: https://storyboard.openstack.org/#!/story/2008615
HI Gregory, I have the same issue as described here by Piotr and on checking the config I also noticed that the key is missing from the config file. What is the heartbeat_key string though, following the docs to install and configure, where is this key located? On 28/03/2021 20:32, apps@mossakowski.ch wrote:
Yes, that was it: missing [health_manager].heartbeat_key in octavia.conf It is not present in openstack victoria octavia docs, I'll push it together with my installation guide for centos8. Thanks for your accurate hint Gregory. It is always crucial to ask the right guy:)
Regards, Piotr Mossakowski Sent from ProtonMail mobile
-------- Original Message -------- On 22 Mar 2021, 09:10, Gregory Thiemonge < gthiemonge@redhat.com> wrote:
Hi,
Most of the OFFLINE operational status issues are caused by communication problems between the amphorae and the Octavia health-manager.
In your case, the "Ignoring this packet. Exception: 'NoneType' object has no attribute 'encode'" log message shows that the health-manager receives the heartbeat packets from the amphorae but it is unable to decode them. Those packets are encrypted JSON messages and it seems that the key ([health_manager].heartbeat_key see https://docs.openstack.org/octavia/latest/configuration/configref.html#healt... <https://docs.openstack.org/octavia/latest/configuration/configref.html#health-manager>) used to encrypt those messages is not defined in your configuration file. So I would suggest configuring it and restarting the Octavia services, then you can re-create or failover the load balancers (you cannot change this parameter in a running load balancer).
Gregory
On Sun, Mar 21, 2021 at 6:17 PM <apps@mossakowski.ch <mailto:apps@mossakowski.ch>> wrote:
Hello, I have stable/victoria baremetal openstack with octavia installed on centos8 using openvswitch mechanism driver: octavia api on controller, health-manager,housekeeping,worker on 3 compute/network nodes. Official docs include only ubuntu with linuxbridge mechanism but I used https://github.com/prastamaha/openstack-octavia <https://github.com/prastamaha/openstack-octavia> as a reference to get it working on centos8 with ovs. I will push those docs instructions for centos8 soon: https://github.com/openstack/octavia/tree/master/doc/source/install <https://github.com/openstack/octavia/tree/master/doc/source/install>.
I created basic http scenario using https://docs.openstack.org/octavia/victoria/user/guides/basic-cookbook.html#... <https://docs.openstack.org/octavia/victoria/user/guides/basic-cookbook.html#deploy-a-basic-http-load-balancer>. Loadbalancer works but its operational status is offline (openstack_loadbalancer_outputs.txt). On all octavia workers I see the same warning message in health_manager.log: Health Manager experienced an exception processing a heartbeat message from ('172.31.255.233', 1907). Ignoring this packet. Exception: 'NoneType' object has no attribute 'encode' I've searched for related active bug but all I found is this not related in my opinion: https://storyboard.openstack.org/#!/story/2008615 <https://storyboard.openstack.org/#!/story/2008615> I'm attaching all info I've gathered:
* octavia.conf and health_manager debug logs (octavia_config_and_health_manager_logs.txt) * tcpdump from amphora VM (tcpdump_from_amphora_vm.txt) * tcpdump from octavia worker (tcpdump_from_octavia_worker.txt) * debug amphora-agent.log from amphora VM (amphora-agent.log)
Can you point me to the right direction what I have missed? Thanks! Piotr Mossakowski https://github.com/moss2k13 <https://github.com/moss2k13>
It is documented here: https://docs.openstack.org/octavia/latest/configuration/configref.html#healt... It is also listed in the quick start guide here: https://docs.openstack.org/octavia/latest/contributor/guides/dev-quick-start... Your deployment tool should be configuring that for you. Michael On Thu, May 13, 2021 at 6:50 AM Luke Camilleri <luke.camilleri@zylacomputing.com> wrote:
HI Gregory, I have the same issue as described here by Piotr and on checking the config I also noticed that the key is missing from the config file.
What is the heartbeat_key string though, following the docs to install and configure, where is this key located?
On 28/03/2021 20:32, apps@mossakowski.ch wrote:
Yes, that was it: missing [health_manager].heartbeat_key in octavia.conf It is not present in openstack victoria octavia docs, I'll push it together with my installation guide for centos8. Thanks for your accurate hint Gregory. It is always crucial to ask the right guy:)
Regards, Piotr Mossakowski Sent from ProtonMail mobile
-------- Original Message -------- On 22 Mar 2021, 09:10, Gregory Thiemonge < gthiemonge@redhat.com> wrote:
Hi,
Most of the OFFLINE operational status issues are caused by communication problems between the amphorae and the Octavia health-manager.
In your case, the "Ignoring this packet. Exception: 'NoneType' object has no attribute 'encode'" log message shows that the health-manager receives the heartbeat packets from the amphorae but it is unable to decode them. Those packets are encrypted JSON messages and it seems that the key ([health_manager].heartbeat_key see https://docs.openstack.org/octavia/latest/configuration/configref.html#healt...) used to encrypt those messages is not defined in your configuration file. So I would suggest configuring it and restarting the Octavia services, then you can re-create or failover the load balancers (you cannot change this parameter in a running load balancer).
Gregory
On Sun, Mar 21, 2021 at 6:17 PM <apps@mossakowski.ch> wrote:
Hello, I have stable/victoria baremetal openstack with octavia installed on centos8 using openvswitch mechanism driver: octavia api on controller, health-manager,housekeeping,worker on 3 compute/network nodes. Official docs include only ubuntu with linuxbridge mechanism but I used https://github.com/prastamaha/openstack-octavia as a reference to get it working on centos8 with ovs. I will push those docs instructions for centos8 soon: https://github.com/openstack/octavia/tree/master/doc/source/install. I created basic http scenario using https://docs.openstack.org/octavia/victoria/user/guides/basic-cookbook.html#.... Loadbalancer works but its operational status is offline (openstack_loadbalancer_outputs.txt). On all octavia workers I see the same warning message in health_manager.log: Health Manager experienced an exception processing a heartbeat message from ('172.31.255.233', 1907). Ignoring this packet. Exception: 'NoneType' object has no attribute 'encode' I've searched for related active bug but all I found is this not related in my opinion: https://storyboard.openstack.org/#!/story/2008615 I'm attaching all info I've gathered:
octavia.conf and health_manager debug logs (octavia_config_and_health_manager_logs.txt) tcpdump from amphora VM (tcpdump_from_amphora_vm.txt) tcpdump from octavia worker (tcpdump_from_octavia_worker.txt) debug amphora-agent.log from amphora VM (amphora-agent.log)
Can you point me to the right direction what I have missed? Thanks! Piotr Mossakowski https://github.com/moss2k13
Actually I figured it out a few minutes after I had sent the message :-) Seems like it is just a string of characters that need to be the same between the [health-manager] section in the amphora-agent on the amphora instance and the corresponding section in octavia.conf on the controller node. This should be sorted now, thanks On 13/05/2021 18:37, Michael Johnson wrote:
It is documented here: https://docs.openstack.org/octavia/latest/configuration/configref.html#healt... It is also listed in the quick start guide here: https://docs.openstack.org/octavia/latest/contributor/guides/dev-quick-start...
Your deployment tool should be configuring that for you.
Michael
On Thu, May 13, 2021 at 6:50 AM Luke Camilleri <luke.camilleri@zylacomputing.com> wrote:
HI Gregory, I have the same issue as described here by Piotr and on checking the config I also noticed that the key is missing from the config file.
What is the heartbeat_key string though, following the docs to install and configure, where is this key located?
On 28/03/2021 20:32, apps@mossakowski.ch wrote:
Yes, that was it: missing [health_manager].heartbeat_key in octavia.conf It is not present in openstack victoria octavia docs, I'll push it together with my installation guide for centos8. Thanks for your accurate hint Gregory. It is always crucial to ask the right guy:)
Regards, Piotr Mossakowski Sent from ProtonMail mobile
-------- Original Message -------- On 22 Mar 2021, 09:10, Gregory Thiemonge < gthiemonge@redhat.com> wrote:
Hi,
Most of the OFFLINE operational status issues are caused by communication problems between the amphorae and the Octavia health-manager.
In your case, the "Ignoring this packet. Exception: 'NoneType' object has no attribute 'encode'" log message shows that the health-manager receives the heartbeat packets from the amphorae but it is unable to decode them. Those packets are encrypted JSON messages and it seems that the key ([health_manager].heartbeat_key see https://docs.openstack.org/octavia/latest/configuration/configref.html#healt...) used to encrypt those messages is not defined in your configuration file. So I would suggest configuring it and restarting the Octavia services, then you can re-create or failover the load balancers (you cannot change this parameter in a running load balancer).
Gregory
On Sun, Mar 21, 2021 at 6:17 PM <apps@mossakowski.ch> wrote:
Hello, I have stable/victoria baremetal openstack with octavia installed on centos8 using openvswitch mechanism driver: octavia api on controller, health-manager,housekeeping,worker on 3 compute/network nodes. Official docs include only ubuntu with linuxbridge mechanism but I used https://github.com/prastamaha/openstack-octavia as a reference to get it working on centos8 with ovs. I will push those docs instructions for centos8 soon: https://github.com/openstack/octavia/tree/master/doc/source/install. I created basic http scenario using https://docs.openstack.org/octavia/victoria/user/guides/basic-cookbook.html#.... Loadbalancer works but its operational status is offline (openstack_loadbalancer_outputs.txt). On all octavia workers I see the same warning message in health_manager.log: Health Manager experienced an exception processing a heartbeat message from ('172.31.255.233', 1907). Ignoring this packet. Exception: 'NoneType' object has no attribute 'encode' I've searched for related active bug but all I found is this not related in my opinion: https://storyboard.openstack.org/#!/story/2008615 I'm attaching all info I've gathered:
octavia.conf and health_manager debug logs (octavia_config_and_health_manager_logs.txt) tcpdump from amphora VM (tcpdump_from_amphora_vm.txt) tcpdump from octavia worker (tcpdump_from_octavia_worker.txt) debug amphora-agent.log from amphora VM (amphora-agent.log)
Can you point me to the right direction what I have missed? Thanks! Piotr Mossakowski https://github.com/moss2k13
participants (4)
-
apps＠mossakowski.ch
-
Gregory Thiemonge
-
Luke Camilleri
-
Michael Johnson