[Octavia] Amphora build issues
Hi, It seems that every build I have attempted of an amphora fails in some way. I have tried CentOS 7, Ubuntu Bionic, Xenial, and Trusty. Note that we are running Stein. I will concentrate on Ubuntu issues for now. I first create a fresh VM that is used to install the diskimage-create tool, then run (after sudo'ing to root): apt update apt -y upgrade apt-get -y install qemu qemu-system-common uuid-runtime curl kpartx git jq python-pip debootstrap libguestfs-tools pip install 'networkx==2.2' pip install argparse Babel dib-utils PyYAML git clone -b stable/stein https://github.com/openstack/octavia.git git clone https://git.openstack.org/openstack/diskimage-builder.git cd diskimage-builder pip install -r requirements.txt cd ../octavia/diskimage-create/ pip install -r requirements.txt # And finally, I run the diskimage-create script, specifying the image's OS, so ONE of these, depending on the OS: ./diskimage-create.sh -d bionic # or to use Xenial: ./diskimage-create.sh -d xenial # Note that when selecting Trusty, diskimage-create.sh error's, and so never finishes successfully. # Somewhat expected since it is quite old and unsupported. ./diskimage-create.sh -d trusty The amphorae launch when creating a load balancer, but the amphora agent fails to start, and thus is not responsive on TCP Port 9443. The log from inside the amphora is below. Has anyone successfully created an image? Am I missing something? Thanks! Eric Amphora agent fails to start inside amphora - this is logged when running the agent from the command line: 2019-10-16 03:41:04.835 1119 INFO octavia.common.config [-] /usr/local/bin/amphora-agent version 5.1.0.dev20 2019-10-16 03:41:04.835 1119 DEBUG octavia.common.config [-] command line: /usr/local/bin/amphora-agent --config-file /etc/octavia/amphora-agent.conf setup_logging /opt/amphora-agent-venv/lib/python3.5/site-packages/octavia/common/confi g.py:779 2019-10-16 03:41:05.036 1124 INFO octavia.amphorae.backends.health_daemon.health_daemon [-] Health Manager Sender starting. 2019-10-16 03:41:05.084 1119 CRITICAL octavia [-] Unhandled error: FileNotFoundError: [Errno 2] No such file or directory 2019-10-16 03:41:05.084 1119 ERROR octavia Traceback (most recent call last): 2019-10-16 03:41:05.084 1119 ERROR octavia File "/usr/local/bin/amphora-agent", line 8, in <module> 2019-10-16 03:41:05.084 1119 ERROR octavia sys.exit(main()) 2019-10-16 03:41:05.084 1119 ERROR octavia File "/opt/amphora-agent-venv/lib/python3.5/site-packages/octavia/cmd/agent.p y", line 89, in main 2019-10-16 03:41:05.084 1119 ERROR octavia AmphoraAgent(server_instance.app, options).run() 2019-10-16 03:41:05.084 1119 ERROR octavia File "/opt/amphora-agent-venv/lib/python3.5/site-packages/gunicorn/app/base.p y", line 72, in run 2019-10-16 03:41:05.084 1119 ERROR octavia Arbiter(self).run() 2019-10-16 03:41:05.084 1119 ERROR octavia File "/opt/amphora-agent-venv/lib/python3.5/site-packages/gunicorn/arbiter.py ", line 60, in __init__ 2019-10-16 03:41:05.084 1119 ERROR octavia self.setup(app) 2019-10-16 03:41:05.084 1119 ERROR octavia File "/opt/amphora-agent-venv/lib/python3.5/site-packages/gunicorn/arbiter.py ", line 95, in setup 2019-10-16 03:41:05.084 1119 ERROR octavia self.log = self.cfg.logger_class(app.cfg) 2019-10-16 03:41:05.084 1119 ERROR octavia File "/opt/amphora-agent-venv/lib/python3.5/site-packages/gunicorn/glogging.p y", line 200, in __init__ 2019-10-16 03:41:05.084 1119 ERROR octavia self.setup(cfg) 2019-10-16 03:41:05.084 1119 ERROR octavia File "/opt/amphora-agent-venv/lib/python3.5/site-packages/gunicorn/glogging.p y", line 227, in setup 2019-10-16 03:41:05.084 1119 ERROR octavia self.error_log, cfg, self.syslog_fmt, "error" 2019-10-16 03:41:05.084 1119 ERROR octavia File "/opt/amphora-agent-venv/lib/python3.5/site-packages/gunicorn/glogging.p y", line 449, in _set_syslog_handler 2019-10-16 03:41:05.084 1119 ERROR octavia facility=facility, socktype=socktype) 2019-10-16 03:41:05.084 1119 ERROR octavia File "/usr/lib/python3.5/logging/handlers.py", line 806, in __init__ 2019-10-16 03:41:05.084 1119 ERROR octavia self._connect_unixsocket(address) 2019-10-16 03:41:05.084 1119 ERROR octavia File "/usr/lib/python3.5/logging/handlers.py", line 823, in _connect_unixsocket 2019-10-16 03:41:05.084 1119 ERROR octavia self.socket.connect(address) 2019-10-16 03:41:05.084 1119 ERROR octavia FileNotFoundError: [Errno 2] No such file or directory
Just an update on this. It appears that the diskimage-create script is pulling the master version of the Octavia amphora agent, instead of the Stein branch. I took a closer look at the first error line: 2019-10-16 19:47:46.389 1160 ERROR octavia File "/opt/amphora-agent-venv/lib/python3.5/site-packages/octavia/cmd/agent.p y", line 89, in main 2019-10-16 19:47:46.389 1160 ERROR octavia AmphoraAgent(server_instance.app, options).run() and it references line 89, which doesn't exist in agent.py except in the master branch. I had cloned the Stein branch of Octavia here: git clone -b stable/stein https://github.com/openstack/octavia.git I will keep looking... Eric
Hi Eric, You are correct, diskimage-builder defaults to pulling the master version of the amphora agent. You want to set the following variables: export DIB_REPOREF_amphora_agent=stable/stein Then run the diskimage-create script. See the guide for more information: https://docs.openstack.org/octavia/latest/admin/amphora-image-build.html#env... Michael On Wed, Oct 16, 2019 at 1:09 PM Eric K. Miller <emiller@genesishosting.com> wrote:
Just an update on this.
It appears that the diskimage-create script is pulling the master version of the Octavia amphora agent, instead of the Stein branch.
I took a closer look at the first error line:
2019-10-16 19:47:46.389 1160 ERROR octavia File "/opt/amphora-agent-venv/lib/python3.5/site-packages/octavia/cmd/agent.p y", line 89, in main 2019-10-16 19:47:46.389 1160 ERROR octavia AmphoraAgent(server_instance.app, options).run()
and it references line 89, which doesn't exist in agent.py except in the master branch.
I had cloned the Stein branch of Octavia here: git clone -b stable/stein https://github.com/openstack/octavia.git
I will keep looking...
Eric
Thank you Michael! After doing this, a different problem occurs, which is logged in the octavia_worker.log. This also happened with all of my tests with CentOS amphorae. See the log snippet below. Note that creation of the load balancer fails with a status of ERROR and amphorae are deleted right after, so I wasn't able to login to the amphorae. Also note that this was deployed with Kolla Ansible 8.0.2, in case that helps. Eric 2019-10-16 16:27:49.399 23 DEBUG octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask' (0af6f3fc-83d0-4093-b5e4-9cb955bf3397) transitioned into state 'REVERTING' from state 'SUCCESS' _task_receiver /var/lib/kolla/venv/lib/python2.7/site-packages/taskflow/listeners/logging.py:194 2019-10-16 16:27:49.404 23 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask' (0af6f3fc-83d0-4093-b5e4-9cb955bf3397) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' 2019-10-16 16:27:49.415 23 WARNING octavia.controller.worker.controller_worker [-] Flow 'octavia-create-loadbalancer-flow' (337640bf-0f7c-4c9e-b903-994f2c5827dd) transitioned into state 'REVERTED' from state 'RUNNING' 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server [-] Exception during message handling: WrappedFailure: WrappedFailure: [Failure: octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not Found, Failure: octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not Found] 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/octavia/controller/queue/endpoint.py", line 45, in create_load_balancer 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server self.worker.create_load_balancer(load_balancer_id, flavor) 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/tenacity/__init__.py", line 292, in wrapped_f 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server return self.call(f, *args, **kw) 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/tenacity/__init__.py", line 358, in call 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server do = self.iter(retry_state=retry_state) 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/tenacity/__init__.py", line 319, in iter 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server return fut.result() 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/concurrent/futures/_base.py", line 455, in result 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server return self.__get_result() 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/tenacity/__init__.py", line 361, in call 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/octavia/controller/worker/controller_worker.py", line 343, in create_load_balancer 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server create_lb_tf.run() 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/taskflow/types/failure.py", line 341, in reraise_if_any 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server raise exc.WrappedFailure(failures) 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server WrappedFailure: WrappedFailure: [Failure: octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not Found, Failure: octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not Found] 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server ~
Looking at the error, it appears it can't find the exceptions.py script, but it appears to be in the octavia worker container: 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server WrappedFailure: WrappedFailure: [Failure: octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not Found, Failure: octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not Found] (octavia-worker)[root@controller001 haproxy]# cd /var/lib/kolla/venv/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy (octavia-worker)[root@controller001 haproxy]# ls -al total 84 drwxr-xr-x. 2 root root 186 Sep 29 08:12 . drwxr-xr-x. 6 root root 156 Sep 29 08:12 .. -rw-r--r--. 1 root root 3400 Sep 29 08:12 data_models.py -rw-r--r--. 1 root root 4797 Sep 29 08:12 data_models.pyc -rw-r--r--. 1 root root 2298 Sep 29 08:12 exceptions.py -rw-r--r--. 1 root root 3326 Sep 29 08:12 exceptions.pyc -rw-r--r--. 1 root root 572 Sep 29 08:12 __init__.py -rw-r--r--. 1 root root 163 Sep 29 08:12 __init__.pyc -rw-r--r--. 1 root root 26572 Sep 29 08:12 rest_api_driver.py -rw-r--r--. 1 root root 25269 Sep 29 08:12 rest_api_driver.pyc Eric
This is caused by the controller version being older than the image version. Our upgrade strategy requires the control plane be updated before the image. A few lines above in the log you will see it is attempting to connect to /0.5 and not finding it. You have two options: 1. update the controllers to use the latest stable/stein verison 2. build a new image, but limit it to an older version of stein (such as commit 15358a71e4aeffee8a3283ed08137e3b6daab52e) We would recommend you run the latest minor release of stable/stein. (4.1.0 is the latest) https://releases.openstack.org/stein/index.html#octavia I'm not sure why kolla would install an old release. I'm not very familiar with it. Michael On Wed, Oct 16, 2019 at 2:43 PM Eric K. Miller <emiller@genesishosting.com> wrote:
Looking at the error, it appears it can't find the exceptions.py script, but it appears to be in the octavia worker container:
2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server WrappedFailure: WrappedFailure: [Failure: octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not Found, Failure: octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not Found]
(octavia-worker)[root@controller001 haproxy]# cd /var/lib/kolla/venv/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy (octavia-worker)[root@controller001 haproxy]# ls -al total 84 drwxr-xr-x. 2 root root 186 Sep 29 08:12 . drwxr-xr-x. 6 root root 156 Sep 29 08:12 .. -rw-r--r--. 1 root root 3400 Sep 29 08:12 data_models.py -rw-r--r--. 1 root root 4797 Sep 29 08:12 data_models.pyc -rw-r--r--. 1 root root 2298 Sep 29 08:12 exceptions.py -rw-r--r--. 1 root root 3326 Sep 29 08:12 exceptions.pyc -rw-r--r--. 1 root root 572 Sep 29 08:12 __init__.py -rw-r--r--. 1 root root 163 Sep 29 08:12 __init__.pyc -rw-r--r--. 1 root root 26572 Sep 29 08:12 rest_api_driver.py -rw-r--r--. 1 root root 25269 Sep 29 08:12 rest_api_driver.pyc
Eric
Thanks Michael! I will configure Kolla Ansible to pull the latest Stein release of the controller components and rebuild/install the containers and get back to you. Much appreciated for the assistance. Eric
Success! The current Kolla Ansible release simply has the 4.0.1 version specified in kolla-build.conf, which can be easily updated to 4.1.0. So, I adjusted the kolla-build.conf file, re-built the Octavia containers, deleted the containers from the controllers, re-deployed Octavia with Kolla Ansible, and tested load balancer creation, and everything succeeded. Again, much appreciated for the assistance. Eric
-----Original Message----- From: Michael Johnson [mailto:johnsomor@gmail.com] Sent: Wednesday, October 16, 2019 5:36 PM To: Eric K. Miller Cc: openstack-discuss Subject: Re: [Octavia] Amphora build issues
This is caused by the controller version being older than the image version. Our upgrade strategy requires the control plane be updated before the image. A few lines above in the log you will see it is attempting to connect to /0.5 and not finding it.
You have two options: 1. update the controllers to use the latest stable/stein verison 2. build a new image, but limit it to an older version of stein (such as commit 15358a71e4aeffee8a3283ed08137e3b6daab52e)
We would recommend you run the latest minor release of stable/stein. (4.1.0 is the latest) https://releases.openstack.org/stein/index.html#octavia
I'm not sure why kolla would install an old release. I'm not very familiar with it.
Michael
On Wed, Oct 16, 2019 at 2:43 PM Eric K. Miller <emiller@genesishosting.com> wrote:
Looking at the error, it appears it can't find the exceptions.py script, but it
appears to be in the octavia worker container:
2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server
WrappedFailure: WrappedFailure: [Failure: octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not Found, Failure: octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not Found]
(octavia-worker)[root@controller001 haproxy]# cd
/var/lib/kolla/venv/lib/python2.7/site- packages/octavia/amphorae/drivers/haproxy
(octavia-worker)[root@controller001 haproxy]# ls -al total 84 drwxr-xr-x. 2 root root 186 Sep 29 08:12 . drwxr-xr-x. 6 root root 156 Sep 29 08:12 .. -rw-r--r--. 1 root root 3400 Sep 29 08:12 data_models.py -rw-r--r--. 1 root root 4797 Sep 29 08:12 data_models.pyc -rw-r--r--. 1 root root 2298 Sep 29 08:12 exceptions.py -rw-r--r--. 1 root root 3326 Sep 29 08:12 exceptions.pyc -rw-r--r--. 1 root root 572 Sep 29 08:12 __init__.py -rw-r--r--. 1 root root 163 Sep 29 08:12 __init__.pyc -rw-r--r--. 1 root root 26572 Sep 29 08:12 rest_api_driver.py -rw-r--r--. 1 root root 25269 Sep 29 08:12 rest_api_driver.pyc
Eric
Hi Eric, Octavia has recently been bumped up to 4.1.0 [1] This applies to the cases when you are either using our (upstream, in-registry) images or kolla from git. Released kolla is behind. @kolla Stable branches are really stable so I am not sure whether we should not just point people willing to build their own images to use the version from git rather than PyPI, especially since our images are built from branch, not release. [1] https://review.opendev.org/688426 Kind regards, Radek czw., 17 paź 2019 o 07:23 Eric K. Miller <emiller@genesishosting.com> napisał(a):
Success!
The current Kolla Ansible release simply has the 4.0.1 version specified in kolla-build.conf, which can be easily updated to 4.1.0.
So, I adjusted the kolla-build.conf file, re-built the Octavia containers, deleted the containers from the controllers, re-deployed Octavia with Kolla Ansible, and tested load balancer creation, and everything succeeded.
Again, much appreciated for the assistance.
Eric
-----Original Message----- From: Michael Johnson [mailto:johnsomor@gmail.com] Sent: Wednesday, October 16, 2019 5:36 PM To: Eric K. Miller Cc: openstack-discuss Subject: Re: [Octavia] Amphora build issues
This is caused by the controller version being older than the image version. Our upgrade strategy requires the control plane be updated before the image. A few lines above in the log you will see it is attempting to connect to /0.5 and not finding it.
You have two options: 1. update the controllers to use the latest stable/stein verison 2. build a new image, but limit it to an older version of stein (such as commit 15358a71e4aeffee8a3283ed08137e3b6daab52e)
We would recommend you run the latest minor release of stable/stein. (4.1.0 is the latest) https://releases.openstack.org/stein/index.html#octavia
I'm not sure why kolla would install an old release. I'm not very familiar with it.
Michael
On Wed, Oct 16, 2019 at 2:43 PM Eric K. Miller <emiller@genesishosting.com> wrote:
Looking at the error, it appears it can't find the exceptions.py
script, but it appears to be in the octavia worker container:
2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server
WrappedFailure: WrappedFailure: [Failure: octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not Found, Failure: octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not Found]
(octavia-worker)[root@controller001 haproxy]# cd
/var/lib/kolla/venv/lib/python2.7/site- packages/octavia/amphorae/drivers/haproxy
(octavia-worker)[root@controller001 haproxy]# ls -al total 84 drwxr-xr-x. 2 root root 186 Sep 29 08:12 . drwxr-xr-x. 6 root root 156 Sep 29 08:12 .. -rw-r--r--. 1 root root 3400 Sep 29 08:12 data_models.py -rw-r--r--. 1 root root 4797 Sep 29 08:12 data_models.pyc -rw-r--r--. 1 root root 2298 Sep 29 08:12 exceptions.py -rw-r--r--. 1 root root 3326 Sep 29 08:12 exceptions.pyc -rw-r--r--. 1 root root 572 Sep 29 08:12 __init__.py -rw-r--r--. 1 root root 163 Sep 29 08:12 __init__.pyc -rw-r--r--. 1 root root 26572 Sep 29 08:12 rest_api_driver.py -rw-r--r--. 1 root root 25269 Sep 29 08:12 rest_api_driver.pyc
Eric
Hi Radek, In case this was useful, Kolla Ansible pulls the files (when using the "source" option) from: http://tarballs.openstack.org/octavia/ So, I just adjusted the version it pulled for Octavia. Eric From: Radosław Piliszek [mailto:radoslaw.piliszek@gmail.com] Sent: Thursday, October 17, 2019 1:15 AM To: Eric K. Miller Cc: Michael Johnson; openstack-discuss Subject: Re: [Octavia][Kolla] Amphora build issues Hi Eric, Octavia has recently been bumped up to 4.1.0 [1] This applies to the cases when you are either using our (upstream, in-registry) images or kolla from git. Released kolla is behind. @kolla Stable branches are really stable so I am not sure whether we should not just point people willing to build their own images to use the version from git rather than PyPI, especially since our images are built from branch, not release. [1] https://review.opendev.org/688426 Kind regards, Radek czw., 17 paź 2019 o 07:23 Eric K. Miller <emiller@genesishosting.com> napisał(a): Success! The current Kolla Ansible release simply has the 4.0.1 version specified in kolla-build.conf, which can be easily updated to 4.1.0. So, I adjusted the kolla-build.conf file, re-built the Octavia containers, deleted the containers from the controllers, re-deployed Octavia with Kolla Ansible, and tested load balancer creation, and everything succeeded. Again, much appreciated for the assistance. Eric > -----Original Message----- > From: Michael Johnson [mailto:johnsomor@gmail.com] > Sent: Wednesday, October 16, 2019 5:36 PM > To: Eric K. Miller > Cc: openstack-discuss > Subject: Re: [Octavia] Amphora build issues > > This is caused by the controller version being older than the image > version. Our upgrade strategy requires the control plane be updated > before the image. > A few lines above in the log you will see it is attempting to connect > to /0.5 and not finding it. > > You have two options: > 1. update the controllers to use the latest stable/stein verison > 2. build a new image, but limit it to an older version of stein (such > as commit 15358a71e4aeffee8a3283ed08137e3b6daab52e) > > We would recommend you run the latest minor release of stable/stein. > (4.1.0 is the latest) > https://releases.openstack.org/stein/index.html#octavia > > I'm not sure why kolla would install an old release. I'm not very > familiar with it. > > Michael > > On Wed, Oct 16, 2019 at 2:43 PM Eric K. Miller > <emiller@genesishosting.com> wrote: > > > > Looking at the error, it appears it can't find the exceptions.py script, but it > appears to be in the octavia worker container: > > > > 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server > WrappedFailure: WrappedFailure: [Failure: > octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not Found, > Failure: octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not > Found] > > > > (octavia-worker)[root@controller001 haproxy]# cd > /var/lib/kolla/venv/lib/python2.7/site- > packages/octavia/amphorae/drivers/haproxy > > (octavia-worker)[root@controller001 haproxy]# ls -al > > total 84 > > drwxr-xr-x. 2 root root 186 Sep 29 08:12 . > > drwxr-xr-x. 6 root root 156 Sep 29 08:12 .. > > -rw-r--r--. 1 root root 3400 Sep 29 08:12 data_models.py > > -rw-r--r--. 1 root root 4797 Sep 29 08:12 data_models.pyc > > -rw-r--r--. 1 root root 2298 Sep 29 08:12 exceptions.py > > -rw-r--r--. 1 root root 3326 Sep 29 08:12 exceptions.pyc > > -rw-r--r--. 1 root root 572 Sep 29 08:12 __init__.py > > -rw-r--r--. 1 root root 163 Sep 29 08:12 __init__.pyc > > -rw-r--r--. 1 root root 26572 Sep 29 08:12 rest_api_driver.py > > -rw-r--r--. 1 root root 25269 Sep 29 08:12 rest_api_driver.pyc > > > > Eric
Correction - I meant "Kolla" (not Kolla Ansible) Eric From: Eric K. Miller Sent: Thursday, October 17, 2019 1:18 AM To: 'Radosław Piliszek' Cc: Michael Johnson; openstack-discuss Subject: RE: [Octavia][Kolla] Amphora build issues Hi Radek, In case this was useful, Kolla Ansible pulls the files (when using the "source" option) from: http://tarballs.openstack.org/octavia/ So, I just adjusted the version it pulled for Octavia. Eric From: Radosław Piliszek [mailto:radoslaw.piliszek@gmail.com] Sent: Thursday, October 17, 2019 1:15 AM To: Eric K. Miller Cc: Michael Johnson; openstack-discuss Subject: Re: [Octavia][Kolla] Amphora build issues Hi Eric, Octavia has recently been bumped up to 4.1.0 [1] This applies to the cases when you are either using our (upstream, in-registry) images or kolla from git. Released kolla is behind. @kolla Stable branches are really stable so I am not sure whether we should not just point people willing to build their own images to use the version from git rather than PyPI, especially since our images are built from branch, not release. [1] https://review.opendev.org/688426 Kind regards, Radek czw., 17 paź 2019 o 07:23 Eric K. Miller <emiller@genesishosting.com> napisał(a): Success! The current Kolla Ansible release simply has the 4.0.1 version specified in kolla-build.conf, which can be easily updated to 4.1.0. So, I adjusted the kolla-build.conf file, re-built the Octavia containers, deleted the containers from the controllers, re-deployed Octavia with Kolla Ansible, and tested load balancer creation, and everything succeeded. Again, much appreciated for the assistance. Eric > -----Original Message----- > From: Michael Johnson [mailto:johnsomor@gmail.com] > Sent: Wednesday, October 16, 2019 5:36 PM > To: Eric K. Miller > Cc: openstack-discuss > Subject: Re: [Octavia] Amphora build issues > > This is caused by the controller version being older than the image > version. Our upgrade strategy requires the control plane be updated > before the image. > A few lines above in the log you will see it is attempting to connect > to /0.5 and not finding it. > > You have two options: > 1. update the controllers to use the latest stable/stein verison > 2. build a new image, but limit it to an older version of stein (such > as commit 15358a71e4aeffee8a3283ed08137e3b6daab52e) > > We would recommend you run the latest minor release of stable/stein. > (4.1.0 is the latest) > https://releases.openstack.org/stein/index.html#octavia > > I'm not sure why kolla would install an old release. I'm not very > familiar with it. > > Michael > > On Wed, Oct 16, 2019 at 2:43 PM Eric K. Miller > <emiller@genesishosting.com> wrote: > > > > Looking at the error, it appears it can't find the exceptions.py script, but it > appears to be in the octavia worker container: > > > > 2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server > WrappedFailure: WrappedFailure: [Failure: > octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not Found, > Failure: octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not > Found] > > > > (octavia-worker)[root@controller001 haproxy]# cd > /var/lib/kolla/venv/lib/python2.7/site- > packages/octavia/amphorae/drivers/haproxy > > (octavia-worker)[root@controller001 haproxy]# ls -al > > total 84 > > drwxr-xr-x. 2 root root 186 Sep 29 08:12 . > > drwxr-xr-x. 6 root root 156 Sep 29 08:12 .. > > -rw-r--r--. 1 root root 3400 Sep 29 08:12 data_models.py > > -rw-r--r--. 1 root root 4797 Sep 29 08:12 data_models.pyc > > -rw-r--r--. 1 root root 2298 Sep 29 08:12 exceptions.py > > -rw-r--r--. 1 root root 3326 Sep 29 08:12 exceptions.pyc > > -rw-r--r--. 1 root root 572 Sep 29 08:12 __init__.py > > -rw-r--r--. 1 root root 163 Sep 29 08:12 __init__.pyc > > -rw-r--r--. 1 root root 26572 Sep 29 08:12 rest_api_driver.py > > -rw-r--r--. 1 root root 25269 Sep 29 08:12 rest_api_driver.pyc > > > > Eric
Hi Eric, that's exactly what the change I mentioned has done. :-) Kind regards, Radek czw., 17 paź 2019 o 08:20 Eric K. Miller <emiller@genesishosting.com> napisał(a):
Correction - I meant "Kolla" (not Kolla Ansible)
Eric
*From:* Eric K. Miller *Sent:* Thursday, October 17, 2019 1:18 AM *To:* 'Radosław Piliszek' *Cc:* Michael Johnson; openstack-discuss *Subject:* RE: [Octavia][Kolla] Amphora build issues
Hi Radek,
In case this was useful, Kolla Ansible pulls the files (when using the "source" option) from:
http://tarballs.openstack.org/octavia/
So, I just adjusted the version it pulled for Octavia.
Eric
*From:* Radosław Piliszek [mailto:radoslaw.piliszek@gmail.com] *Sent:* Thursday, October 17, 2019 1:15 AM *To:* Eric K. Miller *Cc:* Michael Johnson; openstack-discuss *Subject:* Re: [Octavia][Kolla] Amphora build issues
Hi Eric,
Octavia has recently been bumped up to 4.1.0 [1]
This applies to the cases when you are either using our (upstream, in-registry) images or kolla from git.
Released kolla is behind.
@kolla
Stable branches are really stable so I am not sure whether we should not just point people willing to build their own images to use the version from git rather than PyPI, especially since our images are built from branch, not release.
[1] https://review.opendev.org/688426
Kind regards,
Radek
czw., 17 paź 2019 o 07:23 Eric K. Miller <emiller@genesishosting.com> napisał(a):
Success!
The current Kolla Ansible release simply has the 4.0.1 version specified in kolla-build.conf, which can be easily updated to 4.1.0.
So, I adjusted the kolla-build.conf file, re-built the Octavia containers, deleted the containers from the controllers, re-deployed Octavia with Kolla Ansible, and tested load balancer creation, and everything succeeded.
Again, much appreciated for the assistance.
Eric
-----Original Message----- From: Michael Johnson [mailto:johnsomor@gmail.com] Sent: Wednesday, October 16, 2019 5:36 PM To: Eric K. Miller Cc: openstack-discuss Subject: Re: [Octavia] Amphora build issues
This is caused by the controller version being older than the image version. Our upgrade strategy requires the control plane be updated before the image. A few lines above in the log you will see it is attempting to connect to /0.5 and not finding it.
You have two options: 1. update the controllers to use the latest stable/stein verison 2. build a new image, but limit it to an older version of stein (such as commit 15358a71e4aeffee8a3283ed08137e3b6daab52e)
We would recommend you run the latest minor release of stable/stein. (4.1.0 is the latest) https://releases.openstack.org/stein/index.html#octavia
I'm not sure why kolla would install an old release. I'm not very familiar with it.
Michael
On Wed, Oct 16, 2019 at 2:43 PM Eric K. Miller <emiller@genesishosting.com> wrote:
Looking at the error, it appears it can't find the exceptions.py
script, but it appears to be in the octavia worker container:
2019-10-16 16:27:49.416 23 ERROR oslo_messaging.rpc.server
WrappedFailure: WrappedFailure: [Failure: octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not Found, Failure: octavia.amphorae.drivers.haproxy.exceptions.NotFound: Not Found]
(octavia-worker)[root@controller001 haproxy]# cd
/var/lib/kolla/venv/lib/python2.7/site- packages/octavia/amphorae/drivers/haproxy
(octavia-worker)[root@controller001 haproxy]# ls -al total 84 drwxr-xr-x. 2 root root 186 Sep 29 08:12 . drwxr-xr-x. 6 root root 156 Sep 29 08:12 .. -rw-r--r--. 1 root root 3400 Sep 29 08:12 data_models.py -rw-r--r--. 1 root root 4797 Sep 29 08:12 data_models.pyc -rw-r--r--. 1 root root 2298 Sep 29 08:12 exceptions.py -rw-r--r--. 1 root root 3326 Sep 29 08:12 exceptions.pyc -rw-r--r--. 1 root root 572 Sep 29 08:12 __init__.py -rw-r--r--. 1 root root 163 Sep 29 08:12 __init__.pyc -rw-r--r--. 1 root root 26572 Sep 29 08:12 rest_api_driver.py -rw-r--r--. 1 root root 25269 Sep 29 08:12 rest_api_driver.pyc
Eric
participants (3)
-
Eric K. Miller
-
Michael Johnson
-
Radosław Piliszek