Hi Pierre, thanks for replying to my message. To install kayobe I followed the documentation which summarise: installing a few system packages and setting up the kayobe virtual environment and then pulling the correct kayobe git version for the openstack to be installed. After configuring the yaml files I have run these commands: - kayobe control host bootstrap - kayobe overcloud host configure -> this one is failing with /usr/libexec/platform-python: not found After reading your message on the weekend I concluded that maybe I had done something wrong. Today, I re-pulled the kayobe wallaby git and manually transferred the configuration over to the new directory structure on the ansible host and set up again as per the guide but the same issue is seen. What I ended up doing to try and resolve was finding where this "platform-python" is coming from. It is coming from the virtual environment which is being set up during the kayobe ansible host bootstrap. Initially, I found the base.yml and it looks like it tries to match what the host is. I noticed that there is no ubuntu 20 listed there so I created it however it did not resolve the issue. So then I tried systematically replacing this reference in the other files found in the same location "venvs\kayobe\share\kayobe\ansible". The file I changed which allowed it to progress is "kayobe-target-venv.yml" But unfortunately it fails a bit further on, failing to find an selinux package [1] Seeing as the error is mentioning selinux (a RedHat security feature not installed on ubuntu) could the root cause issue be that kayobe is not matching the host as ubuntu? I did already set in kayobe that I am using ubuntu OS distribution within globals.yml [2]. Are there any extra steps that I need to complete that maybe are not listed in the documentation / guide? [1] TASK [MichaelRigart.interfaces : Debian | install current/latest network package - Pastebin.com <https://pastebin.com/GDizpJRg> [2] ---# Kayobe global configuration.######################################### - Pastebin.com <https://pastebin.com/1BvNQSkK> Regards, Tony Pearce On Fri, 11 Jun 2021 at 21:05, Pierre Riteau <pierre@stackhpc.com> wrote:

Hi Tony,

Kayobe doesn't use platform-python anymore, on both stable/wallaby and stable/victoria: https://review.opendev.org/q/I0d477325e0edd13d1aba211c13dc2e8b7a9b4c98

Can you double-check what version you are using, and share how you installed it? Note that only stable/wallaby supports Ubuntu 20 hosts.

Best wishes, Pierre

On Fri, 11 Jun 2021 at 13:20, Tony Pearce <tonyppe@gmail.com> wrote:

...

I'm trying to run "kayobe overcloud host configure" against an ubuntu 20

machine to deploy Wallaby. I'm getting an error that python is not found during the host configure part.

...

PLAY [Verify that the Kayobe Ansible user account is accessible] TASK [Verify that a command can be executed]

fatal: [juc-ucsb-5-p]: FAILED! => {"changed": false, "module_stderr":

"/bin/sh: 1: /usr/libexec/platform-python: not found\n", "module_stdout": "", "msg": "The module failed to execute correctly, you probably need to set the interpreter.\nSee stdout/stderr for the exact error", "rc": 127}

...

Python3 is installed on the host. When searching where this

platform-python is coming from it returns the kolla-ansible virtual envs:

...

$ grep -rni -e "platform-python"

venvs/kolla-ansible/lib/python3.8/site-packages/ansible/config/base.yml:1450: '8': /usr/libexec/platform-python

...

venvs/kolla-ansible/lib/python3.8/site-packages/ansible/config/base.yml:1470: - /usr/libexec/platform-python

...

I had a look through the deployment guide for Kayobe Wallaby and didnt

see a note about changing this.

...

Do I need to do further steps to support the ubuntu overcloud host? I

have already set (as per the doc):

...

os_distribution: ubuntu os_release: focal

Regards,

Tony Pearce

Hi Mark,

I followed this guide to do a "git clone" specifying the branch "-b" to
"stable/wallaby" [1]. What additional steps do I need to do to get the
latest commits?

[1] OpenStack Docs: Overcloud
<https://docs.openstack.org/kayobe/wallaby/configuration/scenarios/all-in-one/overcloud.html>

Kind regards,

Tony Pearce


On Mon, 14 Jun 2021 at 16:10, Mark Goddard <mark@stackhpc.com> wrote:

> On Mon, 14 Jun 2021 at 07:21, Tony Pearce <tonyppe@gmail.com> wrote:
> >
> > Hi Pierre, thanks for replying to my message.
> >
> > To install kayobe I followed the documentation which summarise:
> installing a few system packages and setting up the kayobe virtual
> environment and then pulling the correct kayobe git version for the
> openstack to be installed. After configuring the yaml files I have run
> these commands:
> >
> > - kayobe control host bootstrap
> > - kayobe overcloud host configure -> this one is failing with
> /usr/libexec/platform-python: not found
> >
> > After reading your message on the weekend I concluded that maybe I had
> done something wrong. Today, I re-pulled the kayobe wallaby git and
> manually transferred the configuration over to the new directory structure
> on the ansible host and set up again as per the guide but the same issue is
> seen.
> >
> > What I ended up doing to try and resolve was finding where this
> "platform-python" is coming from. It is coming from the virtual environment
> which is being set up during the kayobe ansible host bootstrap. Initially,
> I found the base.yml and it looks like it tries to match what the host is.
> I noticed that there is no ubuntu 20 listed there so I created it however
> it did not resolve the issue.
> >
> > So then I tried systematically replacing this reference in the other
> files found in the same location "venvs\kayobe\share\kayobe\ansible". The
> file I changed which allowed it to progress is "kayobe-target-venv.yml"
> >
> > But unfortunately it fails a bit further on, failing to find an selinux
> package [1]
> >
> > Seeing as the error is mentioning selinux (a RedHat security feature not
> installed on ubuntu) could the root cause issue be that kayobe is not
> matching the host as ubuntu? I did already set in kayobe that I am using
> ubuntu OS distribution within globals.yml [2].
> >
> > Are there any extra steps that I need to complete that maybe are not
> listed in the documentation / guide?
> >
> > [1] TASK [MichaelRigart.interfaces : Debian | install current/latest
> network package - Pastebin.com
> > [2] ---# Kayobe global
> configuration.######################################### - Pastebin.com
>
> Hi Tony,
>
> That's definitely not a recent Wallaby checkout you're using. Ubuntu
> no longer uses that MichaelRigart.interfaces role. Check that you have
> recent commits. Here is the most recent on stable/wallaby:
> 13169077aaec0f7a28ae1f15b419dafc2456faf7.
>
> Mark
>
> >
> > Regards,
> >
> > Tony Pearce
> >
> >
> >
> > On Fri, 11 Jun 2021 at 21:05, Pierre Riteau <pierre@stackhpc.com> wrote:
> >>
> >> Hi Tony,
> >>
> >> Kayobe doesn't use platform-python anymore, on both stable/wallaby and
> >> stable/victoria:
> >> https://review.opendev.org/q/I0d477325e0edd13d1aba211c13dc2e8b7a9b4c98
> >>
> >> Can you double-check what version you are using, and share how you
> >> installed it? Note that only stable/wallaby supports Ubuntu 20 hosts.
> >>
> >> Best wishes,
> >> Pierre
> >>
> >> On Fri, 11 Jun 2021 at 13:20, Tony Pearce <tonyppe@gmail.com> wrote:
> >> >
> >> > I'm trying to run "kayobe overcloud host configure" against an ubuntu
> 20 machine to deploy Wallaby. I'm getting an error that python is not found
> during the host configure part.
> >> >
> >> > PLAY [Verify that the Kayobe Ansible user account is accessible]
> >> > TASK [Verify that a command can be executed]
> >> >
> >> > fatal: [juc-ucsb-5-p]: FAILED! => {"changed": false, "module_stderr":
> "/bin/sh: 1: /usr/libexec/platform-python: not found\n", "module_stdout":
> "", "msg": "The module failed to execute correctly, you probably need to
> set the interpreter.\nSee stdout/stderr for the exact error", "rc": 127}
> >> >
> >> > Python3 is installed on the host. When searching where this
> platform-python is coming from it returns the kolla-ansible virtual envs:
> >> >
> >> > $ grep -rni -e "platform-python"
> >> >
> venvs/kolla-ansible/lib/python3.8/site-packages/ansible/config/base.yml:1450:
>     '8': /usr/libexec/platform-python
> >> >
> venvs/kolla-ansible/lib/python3.8/site-packages/ansible/config/base.yml:1470:
> - /usr/libexec/platform-python
> >> >
> >> > I had a look through the deployment guide for Kayobe Wallaby and
> didnt see a note about changing this.
> >> >
> >> > Do I need to do further steps to support the ubuntu overcloud host? I
> have already set (as per the doc):
> >> >
> >> > os_distribution: ubuntu
> >> > os_release: focal
> >> >
> >> > Regards,
> >> >
> >> > Tony Pearce
> >> >
>

Hi Mark,

I had never used the "pip install ." method. Maybe a miscomprehension on my
side, from the documentation [1] there are three ways to install kayobe. I
had opted for the first way which is "pip install kayobe" since January
2020. The understanding was as conveyed in the doc "Installing from PyPI
ensures the use of well used and tested software".

I have since followed your steps in your mail which is the installation
from source. I had new problems:

*During ansible bootstrap:*
During ansible host bootstrap it errors out and says the kolla_ansible is
not found and needs to be installed in the same virtual environment. In all
previous times, I had understood that kolla ansible is installed by kayobe
at this point. I eventually done "pip install kolla-ansible" and it seemed
to take care of that and allowed me to move on to "host configure"

*During host configure:*
I was able to get past the previous python issue but then it failed on the
network due to a "duplicate bond name", though this config was deployed
successfully in Train. I dont think I really need a bond at this point so I
deleted the bond and the host configure is now successful. (fyi this is an
all-in-one host.)

*During kayobe service deploy:*
This then fails with "no module named docker" on the host. To troubleshoot
this I logged into the host and activated the kayobe virtual env
(/opt/kayobe/venvs/kayobe/bin/activate) and then "pip install docker". It
was already installed. Eventually, I issued "pip install --ignore-installed
docker" within these three (environment) locations which resolved this and
allowed the kayobe command to complete successfully and progress further:
- /opt/kayobe/venvs/kayobe/
- /opt/kayobe/venvs/kolla-ansible/
- native on the host after deactivating the venv.

Now the blocker is the following failure;

TASK [nova-cell : Waiting for nova-compute services to register themselves]
**********************************************************************************************
FAILED - RETRYING: Waiting for nova-compute services to register themselves
(20 retries left).
FAILED - RETRYING: Waiting for nova-compute services to register themselves
(19 retries left).

I haven't seen this one before but previously I had seen something similar
with mariadb because the API dns was not available. What I have been using
here is a /etc/hosts entry for this. I checked that this entry is available
on the host and in the nova containers. I decided to reboot the host anyway
(previously resolved similar mariadb issue) to restart the containers just
in case the dns was not available in one of them and I missed it.

Unfortunately I now have two additional issues which are hard blockers:
1. The network is no longer working on the host after reboot, so I am
unable to ssh
2. The user password has been changed by kayobe, so I am unable to login
using the console

Due to the above, I am unable to login to the host to investigate or
remediate. Previously when this happened with centos I could use the root
user to log in. This time around as it's ubuntu I do not have a root user.
The user I am using for both "kolla_ansible_user" and "kayobe_ansible_user"
is the same - is this causing a problem with Victoria and Wallaby? I had
this user password change issue beginning with Victoria.

So at this point I need to re-install the host and go back to the host
configure before service deploy.

*Summary*
Any guidance is well appreciated as I'm at a loss at this point. Last week
I had a working Openstack Train deployment in a single host. "Kayobe"
stopped working (maybe because I had previously always used pip install
kayobe).

I would like to deploy Wallaby, should I be able to successfully do this
today or should I be using Victoria at the moment (or even, Train)?

[1] OpenStack Docs: Installation
<https://docs.openstack.org/kayobe/wallaby/installation.html>

Regards,

Tony Pearce


On Mon, 14 Jun 2021 at 18:36, Mark Goddard <mark@stackhpc.com> wrote:

> On Mon, 14 Jun 2021 at 09:40, Tony Pearce <tonyppe@gmail.com> wrote:
> >
> > Hi Mark,
> >
> > I followed this guide to do a "git clone" specifying the branch "-b" to
> "stable/wallaby" [1]. What additional steps do I need to do to get the
> latest commits?
>
> That should be sufficient. When you install it via pip, note that 'pip
> install kayobe' will still pull from PyPI, even if there is a local
> kayobe directory. Use ./kayobe, or 'pip install .'  if in the same
> directory.
>
> Mark
> >
> > [1] OpenStack Docs: Overcloud
> >
> > Kind regards,
> >
> > Tony Pearce
> >
> >
> > On Mon, 14 Jun 2021 at 16:10, Mark Goddard <mark@stackhpc.com> wrote:
> >>
> >> On Mon, 14 Jun 2021 at 07:21, Tony Pearce <tonyppe@gmail.com> wrote:
> >> >
> >> > Hi Pierre, thanks for replying to my message.
> >> >
> >> > To install kayobe I followed the documentation which summarise:
> installing a few system packages and setting up the kayobe virtual
> environment and then pulling the correct kayobe git version for the
> openstack to be installed. After configuring the yaml files I have run
> these commands:
> >> >
> >> > - kayobe control host bootstrap
> >> > - kayobe overcloud host configure -> this one is failing with
> /usr/libexec/platform-python: not found
> >> >
> >> > After reading your message on the weekend I concluded that maybe I
> had done something wrong. Today, I re-pulled the kayobe wallaby git and
> manually transferred the configuration over to the new directory structure
> on the ansible host and set up again as per the guide but the same issue is
> seen.
> >> >
> >> > What I ended up doing to try and resolve was finding where this
> "platform-python" is coming from. It is coming from the virtual environment
> which is being set up during the kayobe ansible host bootstrap. Initially,
> I found the base.yml and it looks like it tries to match what the host is.
> I noticed that there is no ubuntu 20 listed there so I created it however
> it did not resolve the issue.
> >> >
> >> > So then I tried systematically replacing this reference in the other
> files found in the same location "venvs\kayobe\share\kayobe\ansible". The
> file I changed which allowed it to progress is "kayobe-target-venv.yml"
> >> >
> >> > But unfortunately it fails a bit further on, failing to find an
> selinux package [1]
> >> >
> >> > Seeing as the error is mentioning selinux (a RedHat security feature
> not installed on ubuntu) could the root cause issue be that kayobe is not
> matching the host as ubuntu? I did already set in kayobe that I am using
> ubuntu OS distribution within globals.yml [2].
> >> >
> >> > Are there any extra steps that I need to complete that maybe are not
> listed in the documentation / guide?
> >> >
> >> > [1] TASK [MichaelRigart.interfaces : Debian | install current/latest
> network package - Pastebin.com
> >> > [2] ---# Kayobe global
> configuration.######################################### - Pastebin.com
> >>
> >> Hi Tony,
> >>
> >> That's definitely not a recent Wallaby checkout you're using. Ubuntu
> >> no longer uses that MichaelRigart.interfaces role. Check that you have
> >> recent commits. Here is the most recent on stable/wallaby:
> >> 13169077aaec0f7a28ae1f15b419dafc2456faf7.
> >>
> >> Mark
> >>
> >> >
> >> > Regards,
> >> >
> >> > Tony Pearce
> >> >
> >> >
> >> >
> >> > On Fri, 11 Jun 2021 at 21:05, Pierre Riteau <pierre@stackhpc.com>
> wrote:
> >> >>
> >> >> Hi Tony,
> >> >>
> >> >> Kayobe doesn't use platform-python anymore, on both stable/wallaby
> and
> >> >> stable/victoria:
> >> >>
> https://review.opendev.org/q/I0d477325e0edd13d1aba211c13dc2e8b7a9b4c98
> >> >>
> >> >> Can you double-check what version you are using, and share how you
> >> >> installed it? Note that only stable/wallaby supports Ubuntu 20 hosts.
> >> >>
> >> >> Best wishes,
> >> >> Pierre
> >> >>
> >> >> On Fri, 11 Jun 2021 at 13:20, Tony Pearce <tonyppe@gmail.com> wrote:
> >> >> >
> >> >> > I'm trying to run "kayobe overcloud host configure" against an
> ubuntu 20 machine to deploy Wallaby. I'm getting an error that python is
> not found during the host configure part.
> >> >> >
> >> >> > PLAY [Verify that the Kayobe Ansible user account is accessible]
> >> >> > TASK [Verify that a command can be executed]
> >> >> >
> >> >> > fatal: [juc-ucsb-5-p]: FAILED! => {"changed": false,
> "module_stderr": "/bin/sh: 1: /usr/libexec/platform-python: not found\n",
> "module_stdout": "", "msg": "The module failed to execute correctly, you
> probably need to set the interpreter.\nSee stdout/stderr for the exact
> error", "rc": 127}
> >> >> >
> >> >> > Python3 is installed on the host. When searching where this
> platform-python is coming from it returns the kolla-ansible virtual envs:
> >> >> >
> >> >> > $ grep -rni -e "platform-python"
> >> >> >
> venvs/kolla-ansible/lib/python3.8/site-packages/ansible/config/base.yml:1450:
>     '8': /usr/libexec/platform-python
> >> >> >
> venvs/kolla-ansible/lib/python3.8/site-packages/ansible/config/base.yml:1470:
> - /usr/libexec/platform-python
> >> >> >
> >> >> > I had a look through the deployment guide for Kayobe Wallaby and
> didnt see a note about changing this.
> >> >> >
> >> >> > Do I need to do further steps to support the ubuntu overcloud
> host? I have already set (as per the doc):
> >> >> >
> >> >> > os_distribution: ubuntu
> >> >> > os_release: focal
> >> >> >
> >> >> > Regards,
> >> >> >
> >> >> > Tony Pearce
> >> >> >
>

Hi Mark, thanks again for your reply and direction on this.

Regarding the need to do a pip install within the venv, I re-installed the
host and re-installed kayobe and this issue is no longer present as of
yesterday afternoon.

Regarding the "Waiting for nova-compute services to register themselves" -
I tailed *.log in the kolla/nova log dir and to be honest there wasnt much
being logged at that time and no errors, either.

Regarding Wallaby hasnt yet been released, I went back to Victoria on
Centos 8.4 this morning, using the "pip install ." (installation from
source). I will use this install from source going forward to make sure I
am getting the latest kayobe patches. I had two blockers during "host
configure":

First was that python was not found on the host:

TASK [Verify that a command can be executed]
*****************************************************************************************************************************
fatal: [juc-ucsb-5-p]: FAILED! => {"changed": false, "module_stderr":
"/bin/sh: /usr/bin/python3: No such file or directory\n", "module_stdout":
"", "msg": "The module failed to execute correctly, you probably need to
set the interpreter.\nSee stdout/stderr for the exact error", "rc": 127}

So I manually installed python to see what would happen. I was able to get
past that point but then it fails again:

TASK [Ensure the Python virtualenv package is installed]
*****************************************************************************************************************
[WARNING]: Updating cache and auto-installing missing dependency:
python3-apt
fatal: [juc-ucsb-5-p]: FAILED! => {"changed": false, "cmd": "apt-get
update", "msg": "[Errno 2] No such file or directory: b'apt-get':
b'apt-get'", "rc": 2}

PLAY RECAP
***************************************************************************************************************************************************************
juc-ucsb-5-p               : ok=7    changed=0    unreachable=0
failed=1    skipped=10   rescued=0    ignored=0

The above log suggests that the CentOS 8.4 host is being matched as ubuntu.
As an FYI, what I mean when I say "went back to Victoria" is that I
followed the initial kayobe install steps again with new top directory
structure and then manually transferred over the kayobe config to the new
empty config which is downloaded during the git clone. "manually
transferred" means I opened each yml config file individually and copied
over *only* the variables that have been configured within the file, making
sure to leave defaults as a prefernce.

After the above issue I decided to go back to Train and have been able to
successfully deploy Openstack using kayobe Train today - no blocking
issues, first time.

*Summary*
I really appreciate your time to help me with this, so thanks again. I
Still think kayobe is the best way to deploy and manage openstack and I
have been "telling my friends" also :) Now I am back to this point I just
have some minor things to work through before I can start using it.

Kind regards,

Tony Pearce


On Tue, 15 Jun 2021 at 16:02, Mark Goddard <mark@stackhpc.com> wrote:

> On Tue, 15 Jun 2021 at 06:51, Tony Pearce <tonyppe@gmail.com> wrote:
> >
> > Hi Mark,
> >
> > I had never used the "pip install ." method. Maybe a miscomprehension on
> my side, from the documentation [1] there are three ways to install kayobe.
> I had opted for the first way which is "pip install kayobe" since January
> 2020. The understanding was as conveyed in the doc "Installing from PyPI
> ensures the use of well used and tested software".
>
> That is true, but since Wallaby has not been released for Kayobe yet,
> it is not on PyPI. If you do install from PyPI, I would advise using a
> version constraint to ensure you get the release series you need.
>
> >
> > I have since followed your steps in your mail which is the installation
> from source. I had new problems:
> >
> > During ansible bootstrap:
> > During ansible host bootstrap it errors out and says the kolla_ansible
> is not found and needs to be installed in the same virtual environment. In
> all previous times, I had understood that kolla ansible is installed by
> kayobe at this point. I eventually done "pip install kolla-ansible" and it
> seemed to take care of that and allowed me to move on to "host configure"
>
> Kolla Ansible should be installed automatically during 'kayobe control
> host bootstrap', in a separate virtualenv from Kayobe. You should not
> need to install it manually, and I would again advise against doing so
> without version constraints.
>
> >
> > During host configure:
> > I was able to get past the previous python issue but then it failed on
> the network due to a "duplicate bond name", though this config was deployed
> successfully in Train. I dont think I really need a bond at this point so I
> deleted the bond and the host configure is now successful. (fyi this is an
> all-in-one host.)
> >
> > During kayobe service deploy:
> > This then fails with "no module named docker" on the host. To
> troubleshoot this I logged into the host and activated the kayobe virtual
> env (/opt/kayobe/venvs/kayobe/bin/activate) and then "pip install docker".
> It was already installed. Eventually, I issued "pip install
> --ignore-installed docker" within these three (environment) locations which
> resolved this and allowed the kayobe command to complete successfully and
> progress further:
> > - /opt/kayobe/venvs/kayobe/
> > - /opt/kayobe/venvs/kolla-ansible/
> > - native on the host after deactivating the venv.
> >
> > Now the blocker is the following failure;
> >
> > TASK [nova-cell : Waiting for nova-compute services to register
> themselves]
> **********************************************************************************************
> > FAILED - RETRYING: Waiting for nova-compute services to register
> themselves (20 retries left).
> > FAILED - RETRYING: Waiting for nova-compute services to register
> themselves (19 retries left).
> >
> > I haven't seen this one before but previously I had seen something
> similar with mariadb because the API dns was not available. What I have
> been using here is a /etc/hosts entry for this. I checked that this entry
> is available on the host and in the nova containers. I decided to reboot
> the host anyway (previously resolved similar mariadb issue) to restart the
> containers just in case the dns was not available in one of them and I
> missed it.
>
> I'd check the nova compute logs here, to find why they are not
> registering themselves.
> >
> > Unfortunately I now have two additional issues which are hard blockers:
> > 1. The network is no longer working on the host after reboot, so I am
> unable to ssh
> > 2. The user password has been changed by kayobe, so I am unable to login
> using the console
> >
> > Due to the above, I am unable to login to the host to investigate or
> remediate. Previously when this happened with centos I could use the root
> user to log in. This time around as it's ubuntu I do not have a root user.
> > The user I am using for both "kolla_ansible_user" and
> "kayobe_ansible_user" is the same - is this causing a problem with Victoria
> and Wallaby? I had this user password change issue beginning with Victoria.
> >
> > So at this point I need to re-install the host and go back to the host
> configure before service deploy.
> >
> > Summary
> > Any guidance is well appreciated as I'm at a loss at this point. Last
> week I had a working Openstack Train deployment in a single host. "Kayobe"
> stopped working (maybe because I had previously always used pip install
> kayobe).
> >
> > I would like to deploy Wallaby, should I be able to successfully do this
> today or should I be using Victoria at the moment (or even, Train)?
>
> We are very close to release of Wallaby, and I expect that it should
> generally work, but Ubuntu is a new distro for Kayobe, and Wallaby is
> a new release. There may be teething problems, so if you're looking
> for something more stable then I'd suggest CentOS & Victoria.
>
> >
> > [1] OpenStack Docs: Installation
> >
> > Regards,
> >
> > Tony Pearce
> >
> >
> > On Mon, 14 Jun 2021 at 18:36, Mark Goddard <mark@stackhpc.com> wrote:
> >>
> >> On Mon, 14 Jun 2021 at 09:40, Tony Pearce <tonyppe@gmail.com> wrote:
> >> >
> >> > Hi Mark,
> >> >
> >> > I followed this guide to do a "git clone" specifying the branch "-b"
> to "stable/wallaby" [1]. What additional steps do I need to do to get the
> latest commits?
> >>
> >> That should be sufficient. When you install it via pip, note that 'pip
> >> install kayobe' will still pull from PyPI, even if there is a local
> >> kayobe directory. Use ./kayobe, or 'pip install .'  if in the same
> >> directory.
> >>
> >> Mark
> >> >
> >> > [1] OpenStack Docs: Overcloud
> >> >
> >> > Kind regards,
> >> >
> >> > Tony Pearce
> >> >
> >> >
> >> > On Mon, 14 Jun 2021 at 16:10, Mark Goddard <mark@stackhpc.com> wrote:
> >> >>
> >> >> On Mon, 14 Jun 2021 at 07:21, Tony Pearce <tonyppe@gmail.com> wrote:
> >> >> >
> >> >> > Hi Pierre, thanks for replying to my message.
> >> >> >
> >> >> > To install kayobe I followed the documentation which summarise:
> installing a few system packages and setting up the kayobe virtual
> environment and then pulling the correct kayobe git version for the
> openstack to be installed. After configuring the yaml files I have run
> these commands:
> >> >> >
> >> >> > - kayobe control host bootstrap
> >> >> > - kayobe overcloud host configure -> this one is failing with
> /usr/libexec/platform-python: not found
> >> >> >
> >> >> > After reading your message on the weekend I concluded that maybe I
> had done something wrong. Today, I re-pulled the kayobe wallaby git and
> manually transferred the configuration over to the new directory structure
> on the ansible host and set up again as per the guide but the same issue is
> seen.
> >> >> >
> >> >> > What I ended up doing to try and resolve was finding where this
> "platform-python" is coming from. It is coming from the virtual environment
> which is being set up during the kayobe ansible host bootstrap. Initially,
> I found the base.yml and it looks like it tries to match what the host is.
> I noticed that there is no ubuntu 20 listed there so I created it however
> it did not resolve the issue.
> >> >> >
> >> >> > So then I tried systematically replacing this reference in the
> other files found in the same location "venvs\kayobe\share\kayobe\ansible".
> The file I changed which allowed it to progress is "kayobe-target-venv.yml"
> >> >> >
> >> >> > But unfortunately it fails a bit further on, failing to find an
> selinux package [1]
> >> >> >
> >> >> > Seeing as the error is mentioning selinux (a RedHat security
> feature not installed on ubuntu) could the root cause issue be that kayobe
> is not matching the host as ubuntu? I did already set in kayobe that I am
> using ubuntu OS distribution within globals.yml [2].
> >> >> >
> >> >> > Are there any extra steps that I need to complete that maybe are
> not listed in the documentation / guide?
> >> >> >
> >> >> > [1] TASK [MichaelRigart.interfaces : Debian | install
> current/latest network package - Pastebin.com
> >> >> > [2] ---# Kayobe global
> configuration.######################################### - Pastebin.com
> >> >>
> >> >> Hi Tony,
> >> >>
> >> >> That's definitely not a recent Wallaby checkout you're using. Ubuntu
> >> >> no longer uses that MichaelRigart.interfaces role. Check that you
> have
> >> >> recent commits. Here is the most recent on stable/wallaby:
> >> >> 13169077aaec0f7a28ae1f15b419dafc2456faf7.
> >> >>
> >> >> Mark
> >> >>
> >> >> >
> >> >> > Regards,
> >> >> >
> >> >> > Tony Pearce
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Fri, 11 Jun 2021 at 21:05, Pierre Riteau <pierre@stackhpc.com>
> wrote:
> >> >> >>
> >> >> >> Hi Tony,
> >> >> >>
> >> >> >> Kayobe doesn't use platform-python anymore, on both
> stable/wallaby and
> >> >> >> stable/victoria:
> >> >> >>
> https://review.opendev.org/q/I0d477325e0edd13d1aba211c13dc2e8b7a9b4c98
> >> >> >>
> >> >> >> Can you double-check what version you are using, and share how you
> >> >> >> installed it? Note that only stable/wallaby supports Ubuntu 20
> hosts.
> >> >> >>
> >> >> >> Best wishes,
> >> >> >> Pierre
> >> >> >>
> >> >> >> On Fri, 11 Jun 2021 at 13:20, Tony Pearce <tonyppe@gmail.com>
> wrote:
> >> >> >> >
> >> >> >> > I'm trying to run "kayobe overcloud host configure" against an
> ubuntu 20 machine to deploy Wallaby. I'm getting an error that python is
> not found during the host configure part.
> >> >> >> >
> >> >> >> > PLAY [Verify that the Kayobe Ansible user account is accessible]
> >> >> >> > TASK [Verify that a command can be executed]
> >> >> >> >
> >> >> >> > fatal: [juc-ucsb-5-p]: FAILED! => {"changed": false,
> "module_stderr": "/bin/sh: 1: /usr/libexec/platform-python: not found\n",
> "module_stdout": "", "msg": "The module failed to execute correctly, you
> probably need to set the interpreter.\nSee stdout/stderr for the exact
> error", "rc": 127}
> >> >> >> >
> >> >> >> > Python3 is installed on the host. When searching where this
> platform-python is coming from it returns the kolla-ansible virtual envs:
> >> >> >> >
> >> >> >> > $ grep -rni -e "platform-python"
> >> >> >> >
> venvs/kolla-ansible/lib/python3.8/site-packages/ansible/config/base.yml:1450:
>     '8': /usr/libexec/platform-python
> >> >> >> >
> venvs/kolla-ansible/lib/python3.8/site-packages/ansible/config/base.yml:1470:
> - /usr/libexec/platform-python
> >> >> >> >
> >> >> >> > I had a look through the deployment guide for Kayobe Wallaby
> and didnt see a note about changing this.
> >> >> >> >
> >> >> >> > Do I need to do further steps to support the ubuntu overcloud
> host? I have already set (as per the doc):
> >> >> >> >
> >> >> >> > os_distribution: ubuntu
> >> >> >> > os_release: focal
> >> >> >> >
> >> >> >> > Regards,
> >> >> >> >
> >> >> >> > Tony Pearce
> >> >> >> >
>

Possibly fact caching could have been the issue. The Victoria guide
describes to configure fact caching so it was being used by my ACH. I did
not think of this at the time of the issue, sorry about that.

I have a new question about SSL cert, so will send a new mail for the
topic.

Thank you again,

Kind regards,

Tony Pearce



On Wed, 16 Jun 2021 at 16:09, Mark Goddard <mark@stackhpc.com> wrote:

> On Wed, 16 Jun 2021 at 07:09, Tony Pearce <tonyppe@gmail.com> wrote:
> >
> > Hi Mark, thanks again for your reply and direction on this.
> >
> > Regarding the need to do a pip install within the venv, I re-installed
> the host and re-installed kayobe and this issue is no longer present as of
> yesterday afternoon.
> >
> > Regarding the "Waiting for nova-compute services to register themselves"
> - I tailed *.log in the kolla/nova log dir and to be honest there wasnt
> much being logged at that time and no errors, either.
> >
> > Regarding Wallaby hasnt yet been released, I went back to Victoria on
> Centos 8.4 this morning, using the "pip install ." (installation from
> source). I will use this install from source going forward to make sure I
> am getting the latest kayobe patches. I had two blockers during "host
> configure":
> >
> > First was that python was not found on the host:
> >
> > TASK [Verify that a command can be executed]
> *****************************************************************************************************************************
> > fatal: [juc-ucsb-5-p]: FAILED! => {"changed": false, "module_stderr":
> "/bin/sh: /usr/bin/python3: No such file or directory\n", "module_stdout":
> "", "msg": "The module failed to execute correctly, you probably need to
> set the interpreter.\nSee stdout/stderr for the exact error", "rc": 127}
>
> In stable/victoria there is a task called 'Ensure python is installed'
> which should ensure that Python 3 is installed. Did that task run?
>
> >
> > So I manually installed python to see what would happen. I was able to
> get past that point but then it fails again:
> >
> > TASK [Ensure the Python virtualenv package is installed]
> *****************************************************************************************************************
> > [WARNING]: Updating cache and auto-installing missing dependency:
> python3-apt
> > fatal: [juc-ucsb-5-p]: FAILED! => {"changed": false, "cmd": "apt-get
> update", "msg": "[Errno 2] No such file or directory: b'apt-get':
> b'apt-get'", "rc": 2}
> >
> > PLAY RECAP
> ***************************************************************************************************************************************************************
> > juc-ucsb-5-p               : ok=7    changed=0    unreachable=0
> failed=1    skipped=10   rescued=0    ignored=0
> >
> > The above log suggests that the CentOS 8.4 host is being matched as
> ubuntu. As an FYI, what I mean when I say "went back to Victoria" is that I
> followed the initial kayobe install steps again with new top directory
> structure and then manually transferred over the kayobe config to the new
> empty config which is downloaded during the git clone. "manually
> transferred" means I opened each yml config file individually and copied
> over only the variables that have been configured within the file, making
> sure to leave defaults as a prefernce.
>
> Something odd is clearly going on here. We use the package module,
> which uses ansible's ansible_pkg_mgr fact to dispatch the correct
> packaging module. Perhaps you have fact caching enabled, and have
> stale facts from when the host was running Ubuntu?
>
> >
> > After the above issue I decided to go back to Train and have been able
> to successfully deploy Openstack using kayobe Train today - no blocking
> issues, first time.
> >
> > Summary
> > I really appreciate your time to help me with this, so thanks again. I
> Still think kayobe is the best way to deploy and manage openstack and I
> have been "telling my friends" also :) Now I am back to this point I just
> have some minor things to work through before I can start using it.
> >
> > Kind regards,
> >
> > Tony Pearce
> >
> >
> > On Tue, 15 Jun 2021 at 16:02, Mark Goddard <mark@stackhpc.com> wrote:
> >>
> >> On Tue, 15 Jun 2021 at 06:51, Tony Pearce <tonyppe@gmail.com> wrote:
> >> >
> >> > Hi Mark,
> >> >
> >> > I had never used the "pip install ." method. Maybe a miscomprehension
> on my side, from the documentation [1] there are three ways to install
> kayobe. I had opted for the first way which is "pip install kayobe" since
> January 2020. The understanding was as conveyed in the doc "Installing from
> PyPI ensures the use of well used and tested software".
> >>
> >> That is true, but since Wallaby has not been released for Kayobe yet,
> >> it is not on PyPI. If you do install from PyPI, I would advise using a
> >> version constraint to ensure you get the release series you need.
> >>
> >> >
> >> > I have since followed your steps in your mail which is the
> installation from source. I had new problems:
> >> >
> >> > During ansible bootstrap:
> >> > During ansible host bootstrap it errors out and says the
> kolla_ansible is not found and needs to be installed in the same virtual
> environment. In all previous times, I had understood that kolla ansible is
> installed by kayobe at this point. I eventually done "pip install
> kolla-ansible" and it seemed to take care of that and allowed me to move on
> to "host configure"
> >>
> >> Kolla Ansible should be installed automatically during 'kayobe control
> >> host bootstrap', in a separate virtualenv from Kayobe. You should not
> >> need to install it manually, and I would again advise against doing so
> >> without version constraints.
> >>
> >> >
> >> > During host configure:
> >> > I was able to get past the previous python issue but then it failed
> on the network due to a "duplicate bond name", though this config was
> deployed successfully in Train. I dont think I really need a bond at this
> point so I deleted the bond and the host configure is now successful. (fyi
> this is an all-in-one host.)
> >> >
> >> > During kayobe service deploy:
> >> > This then fails with "no module named docker" on the host. To
> troubleshoot this I logged into the host and activated the kayobe virtual
> env (/opt/kayobe/venvs/kayobe/bin/activate) and then "pip install docker".
> It was already installed. Eventually, I issued "pip install
> --ignore-installed docker" within these three (environment) locations which
> resolved this and allowed the kayobe command to complete successfully and
> progress further:
> >> > - /opt/kayobe/venvs/kayobe/
> >> > - /opt/kayobe/venvs/kolla-ansible/
> >> > - native on the host after deactivating the venv.
> >> >
> >> > Now the blocker is the following failure;
> >> >
> >> > TASK [nova-cell : Waiting for nova-compute services to register
> themselves]
> **********************************************************************************************
> >> > FAILED - RETRYING: Waiting for nova-compute services to register
> themselves (20 retries left).
> >> > FAILED - RETRYING: Waiting for nova-compute services to register
> themselves (19 retries left).
> >> >
> >> > I haven't seen this one before but previously I had seen something
> similar with mariadb because the API dns was not available. What I have
> been using here is a /etc/hosts entry for this. I checked that this entry
> is available on the host and in the nova containers. I decided to reboot
> the host anyway (previously resolved similar mariadb issue) to restart the
> containers just in case the dns was not available in one of them and I
> missed it.
> >>
> >> I'd check the nova compute logs here, to find why they are not
> >> registering themselves.
> >> >
> >> > Unfortunately I now have two additional issues which are hard
> blockers:
> >> > 1. The network is no longer working on the host after reboot, so I am
> unable to ssh
> >> > 2. The user password has been changed by kayobe, so I am unable to
> login using the console
> >> >
> >> > Due to the above, I am unable to login to the host to investigate or
> remediate. Previously when this happened with centos I could use the root
> user to log in. This time around as it's ubuntu I do not have a root user.
> >> > The user I am using for both "kolla_ansible_user" and
> "kayobe_ansible_user" is the same - is this causing a problem with Victoria
> and Wallaby? I had this user password change issue beginning with Victoria.
> >> >
> >> > So at this point I need to re-install the host and go back to the
> host configure before service deploy.
> >> >
> >> > Summary
> >> > Any guidance is well appreciated as I'm at a loss at this point. Last
> week I had a working Openstack Train deployment in a single host. "Kayobe"
> stopped working (maybe because I had previously always used pip install
> kayobe).
> >> >
> >> > I would like to deploy Wallaby, should I be able to successfully do
> this today or should I be using Victoria at the moment (or even, Train)?
> >>
> >> We are very close to release of Wallaby, and I expect that it should
> >> generally work, but Ubuntu is a new distro for Kayobe, and Wallaby is
> >> a new release. There may be teething problems, so if you're looking
> >> for something more stable then I'd suggest CentOS & Victoria.
> >>
> >> >
> >> > [1] OpenStack Docs: Installation
> >> >
> >> > Regards,
> >> >
> >> > Tony Pearce
> >> >
> >> >
> >> > On Mon, 14 Jun 2021 at 18:36, Mark Goddard <mark@stackhpc.com> wrote:
> >> >>
> >> >> On Mon, 14 Jun 2021 at 09:40, Tony Pearce <tonyppe@gmail.com> wrote:
> >> >> >
> >> >> > Hi Mark,
> >> >> >
> >> >> > I followed this guide to do a "git clone" specifying the branch
> "-b" to "stable/wallaby" [1]. What additional steps do I need to do to get
> the latest commits?
> >> >>
> >> >> That should be sufficient. When you install it via pip, note that
> 'pip
> >> >> install kayobe' will still pull from PyPI, even if there is a local
> >> >> kayobe directory. Use ./kayobe, or 'pip install .'  if in the same
> >> >> directory.
> >> >>
> >> >> Mark
> >> >> >
> >> >> > [1] OpenStack Docs: Overcloud
> >> >> >
> >> >> > Kind regards,
> >> >> >
> >> >> > Tony Pearce
> >> >> >
> >> >> >
> >> >> > On Mon, 14 Jun 2021 at 16:10, Mark Goddard <mark@stackhpc.com>
> wrote:
> >> >> >>
> >> >> >> On Mon, 14 Jun 2021 at 07:21, Tony Pearce <tonyppe@gmail.com>
> wrote:
> >> >> >> >
> >> >> >> > Hi Pierre, thanks for replying to my message.
> >> >> >> >
> >> >> >> > To install kayobe I followed the documentation which summarise:
> installing a few system packages and setting up the kayobe virtual
> environment and then pulling the correct kayobe git version for the
> openstack to be installed. After configuring the yaml files I have run
> these commands:
> >> >> >> >
> >> >> >> > - kayobe control host bootstrap
> >> >> >> > - kayobe overcloud host configure -> this one is failing with
> /usr/libexec/platform-python: not found
> >> >> >> >
> >> >> >> > After reading your message on the weekend I concluded that
> maybe I had done something wrong. Today, I re-pulled the kayobe wallaby git
> and manually transferred the configuration over to the new directory
> structure on the ansible host and set up again as per the guide but the
> same issue is seen.
> >> >> >> >
> >> >> >> > What I ended up doing to try and resolve was finding where this
> "platform-python" is coming from. It is coming from the virtual environment
> which is being set up during the kayobe ansible host bootstrap. Initially,
> I found the base.yml and it looks like it tries to match what the host is.
> I noticed that there is no ubuntu 20 listed there so I created it however
> it did not resolve the issue.
> >> >> >> >
> >> >> >> > So then I tried systematically replacing this reference in the
> other files found in the same location "venvs\kayobe\share\kayobe\ansible".
> The file I changed which allowed it to progress is "kayobe-target-venv.yml"
> >> >> >> >
> >> >> >> > But unfortunately it fails a bit further on, failing to find an
> selinux package [1]
> >> >> >> >
> >> >> >> > Seeing as the error is mentioning selinux (a RedHat security
> feature not installed on ubuntu) could the root cause issue be that kayobe
> is not matching the host as ubuntu? I did already set in kayobe that I am
> using ubuntu OS distribution within globals.yml [2].
> >> >> >> >
> >> >> >> > Are there any extra steps that I need to complete that maybe
> are not listed in the documentation / guide?
> >> >> >> >
> >> >> >> > [1] TASK [MichaelRigart.interfaces : Debian | install
> current/latest network package - Pastebin.com
> >> >> >> > [2] ---# Kayobe global
> configuration.######################################### - Pastebin.com
> >> >> >>
> >> >> >> Hi Tony,
> >> >> >>
> >> >> >> That's definitely not a recent Wallaby checkout you're using.
> Ubuntu
> >> >> >> no longer uses that MichaelRigart.interfaces role. Check that you
> have
> >> >> >> recent commits. Here is the most recent on stable/wallaby:
> >> >> >> 13169077aaec0f7a28ae1f15b419dafc2456faf7.
> >> >> >>
> >> >> >> Mark
> >> >> >>
> >> >> >> >
> >> >> >> > Regards,
> >> >> >> >
> >> >> >> > Tony Pearce
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On Fri, 11 Jun 2021 at 21:05, Pierre Riteau <
> pierre@stackhpc.com> wrote:
> >> >> >> >>
> >> >> >> >> Hi Tony,
> >> >> >> >>
> >> >> >> >> Kayobe doesn't use platform-python anymore, on both
> stable/wallaby and
> >> >> >> >> stable/victoria:
> >> >> >> >>
> https://review.opendev.org/q/I0d477325e0edd13d1aba211c13dc2e8b7a9b4c98
> >> >> >> >>
> >> >> >> >> Can you double-check what version you are using, and share how
> you
> >> >> >> >> installed it? Note that only stable/wallaby supports Ubuntu 20
> hosts.
> >> >> >> >>
> >> >> >> >> Best wishes,
> >> >> >> >> Pierre
> >> >> >> >>
> >> >> >> >> On Fri, 11 Jun 2021 at 13:20, Tony Pearce <tonyppe@gmail.com>
> wrote:
> >> >> >> >> >
> >> >> >> >> > I'm trying to run "kayobe overcloud host configure" against
> an ubuntu 20 machine to deploy Wallaby. I'm getting an error that python is
> not found during the host configure part.
> >> >> >> >> >
> >> >> >> >> > PLAY [Verify that the Kayobe Ansible user account is
> accessible]
> >> >> >> >> > TASK [Verify that a command can be executed]
> >> >> >> >> >
> >> >> >> >> > fatal: [juc-ucsb-5-p]: FAILED! => {"changed": false,
> "module_stderr": "/bin/sh: 1: /usr/libexec/platform-python: not found\n",
> "module_stdout": "", "msg": "The module failed to execute correctly, you
> probably need to set the interpreter.\nSee stdout/stderr for the exact
> error", "rc": 127}
> >> >> >> >> >
> >> >> >> >> > Python3 is installed on the host. When searching where this
> platform-python is coming from it returns the kolla-ansible virtual envs:
> >> >> >> >> >
> >> >> >> >> > $ grep -rni -e "platform-python"
> >> >> >> >> >
> venvs/kolla-ansible/lib/python3.8/site-packages/ansible/config/base.yml:1450:
>     '8': /usr/libexec/platform-python
> >> >> >> >> >
> venvs/kolla-ansible/lib/python3.8/site-packages/ansible/config/base.yml:1470:
> - /usr/libexec/platform-python
> >> >> >> >> >
> >> >> >> >> > I had a look through the deployment guide for Kayobe Wallaby
> and didnt see a note about changing this.
> >> >> >> >> >
> >> >> >> >> > Do I need to do further steps to support the ubuntu
> overcloud host? I have already set (as per the doc):
> >> >> >> >> >
> >> >> >> >> > os_distribution: ubuntu
> >> >> >> >> > os_release: focal
> >> >> >> >> >
> >> >> >> >> > Regards,
> >> >> >> >> >
> >> >> >> >> > Tony Pearce
> >> >> >> >> >
>

Hi, me again :) I tested this again Friday and today (Monday) using a centos Ansible Control Host as well as different installation methods of the openstack host (such as minimal OS install and "server with gui"). Essentially, the deployment of Openstack Victora fails during "kayobe overcloud service deploy" because of the: TASK [openvswitch : Ensuring OVS bridge is properly setup] . I investigated this, comparing it with a Train version. On Victoria, the host is missing: - ifcfg-p-bond0-ovs - ifcfg-p-bond0-phy And these are not visible in the bridge config as seen with "ovs-vsctl show". I tried to manually add the ifcfg and add to the bridge but I inadvertently created a bridging loop. Are you guys aware of this? I am not sure what else I can do to try and either help the kayobe/kolla-ansible teams or; resolve this to allow a successful Victoria install - please let me know? Regards, Tony Pearce On Thu, 17 Jun 2021 at 16:13, Tony Pearce <tonyppe@gmail.com> wrote:

Hi Mark,

I made some time to test this again today with Victoria on a different ACH. During host configure, it fails not finding python:

TASK [Verify that a command can be executed] ********************************************************************************************************************************** fatal: [juc-ucsb-5-p]: FAILED! => {"changed": false, "module_stderr": "Shared connection to 192.168.29.235 closed.\r\n", "module_stdout": "/bin/sh: /usr/bin/python3: No such file or directory\r\n", "msg": "The module failed to execute correctly, you probably need to set the interpreter.\nSee stdout/stderr for the exact error", "rc": 127}

PLAY RECAP ******************************************************************************************************************************************************************** juc-ucsb-5-p : ok=4 changed=1 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0

The task you mentioned previously, was ran but was not run against the host because no hosts matched:

PLAY [Ensure python is installed] ********************************************************************************************************************************************* skipping: no hosts matched

I looked at `venvs/kayobe/share/kayobe/ansible/kayobe-ansible-user.yml` and a comment in there says it's only run if the kayobe user account is inaccessible. In my deployment I have "#kayobe_ansible_user:" which is not defined by me. Previously, I defined it as my management user and it caused an issue with the password. So I'm unsure why this is an issue.

To work around, I manually installed python and the host configure was successful this time around. I tried this twice and same experience both times.

Then later, during service deploy it fails here:

RUNNING HANDLER [common : Restart fluentd container] ************************************************************************************************************************** fatal: [juc-ucsb-5-p]: FAILED! => {"changed": true, "msg": "'Traceback (most recent call last):\\n File \"/opt/kayobe/venvs/kolla-ansible/lib/python3.6/site-packages/docker/api/client.py\", line 259, in _raise_for_status\\n response.raise_for_status()\\n File \"/opt/kayobe/venvs/kolla-ansible/lib/python3.6/site-packages/requests/models.py\", line 941, in raise_for_status\\n raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/containers/fluentd/start\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \"/tmp/ansible_kolla_docker_payload_34omrn2y/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 1131, in main\\n File \"/tmp/ansible_kolla_docker_payload_34omrn2y/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 785, in recreate_or_restart_container\\n File \"/tmp/ansible_kolla_docker_payload_34omrn2y/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 817, in start_container\\n File \"/opt/kayobe/venvs/kolla-ansible/lib/python3.6/site-packages/docker/utils/decorators.py\", line 19, in wrapped\\n return f(self, resource_id, *args, **kwargs)\\n File \"/opt/kayobe/venvs/kolla-ansible/lib/python3.6/site-packages/docker/api/container.py\", line 1108, in start\\n self._raise_for_status(res)\\n File \"/opt/kayobe/venvs/kolla-ansible/lib/python3.6/site-packages/docker/api/client.py\", line 261, in _raise_for_status\\n raise create_api_error_from_http_exception(e)\\n File \"/opt/kayobe/venvs/kolla-ansible/lib/python3.6/site-packages/docker/errors.py\", line 31, in create_api_error_from_http_exception\\n raise cls(e, response=response, explanation=explanation)\\ndocker.errors.APIError: 500 Server Error: Internal Server Error (\"error while creating mount source path \\'/etc/localtime\\': mkdir /etc/lo

The error says that the file exists. So the first time I just renamed the symlink file and then this was successful in terms of allowing the deploy process to proceed past this point of failure. The 2nd time around, the rename was not good enough because there's a check to make sure that the file is present there. So the 2nd time around I issued "touch /etc/localtime" after renaming the existing and then this passed.

Lastly, the deploy fails with a blocking action that I cannot resolve myself:

TASK [openvswitch : Ensuring OVS bridge is properly setup] ******************************************************************************************************************** changed: [juc-ucsb-5-p] => (item=['enp6s0-ovs', 'enp6s0'])

This step breaks networking on the host. Looking at the openvswitchdb, I think this could be something similar to the issue seen before with Wallaby. The first time I tried this was with enp6s0 configured as a bond0 as desired. I then tried without a bond0 and both times got the same result. If I reboot the host then I can get successful ping replies for a short while before they stop again. Same experience as previous. I believe the pings stop when the bridge config is applied from the container shortly after host boot up. Ovs-vsctl show output: [1]

I took a look at the logs [2] but to me I dont see anything alarming or that could point to the issue. I've previously tried turning off IPv6 and this did not have success in this part, although the log message about IPv6 went away.

I tried removing the physical interface from the bridge "ovs-vsctl del-port..." and as soon as I do this, I can ping the host once again. Once I re-add the port back to the bridge, I can no longer connect to the host. There's no errors from ovs-vsctl at this point, either.

[1] ovs-vsctl output screenshot <https://pasteboard.co/K6ZCxDr.png> [2] ovs logs screenshot <https://pasteboard.co/K6ZEcdI.png>

BTW I am trimming the rest of the mail off because it exceeds 40kb size for the group.

Kind regards,

Tony Pearce

...

...

I see. I probably should have explained better, I tried resolving this by using a console connection to the host - this is outside of "kayobe" commands, because the host network gets broken during "kayobe service deploy" which causes the deployment to fail. I tried manually creating the interfaces and adding the ports. In short, the "host configure" gets completed without issue (apart from the localtime and the python3 issue already mentioned that I am able to work around). After the host is configured, I run "service deploy" which halts when the host is no longer reachable over IP, which is the task where it tries to check that the bridge is set up. I have tried over the past week or so to do enough testing to either confirm or rule out a local issue on my side, having tried multiples of systems and getting the same result. I think there may be a bug with kayobe / kolla-ansible which is causing the deployment failure. Are you aware of anything else that I could try to be certain? Kind regards, Tony Pearce On Mon, 21 Jun 2021 at 16:36, Mark Goddard <mark@stackhpc.com> wrote:

On Mon, 21 Jun 2021 at 06:51, Tony Pearce <tonyppe@gmail.com> wrote:

...

Hi, me again :)

I tested this again Friday and today (Monday) using a centos Ansible

Control Host as well as different installation methods of the openstack host (such as minimal OS install and "server with gui"). Essentially, the deployment of Openstack Victora fails during "kayobe overcloud service deploy" because of the: TASK [openvswitch : Ensuring OVS bridge is properly setup] .

...

I investigated this, comparing it with a Train version. On Victoria, the

host is missing:

...

- ifcfg-p-bond0-ovs - ifcfg-p-bond0-phy

And these are not visible in the bridge config as seen with "ovs-vsctl show". I tried to manually add the ifcfg and add to the bridge but I inadvertently created a bridging loop.

Are you guys aware of this? I am not sure what else I can do to try and either help the kayobe/kolla-ansible teams or; resolve this to allow a successful Victoria install - please let me know?

One relevant thing that changed between Train and Victoria is that Kayobe supports plugging a non-bridge interface directly into OVS, without the veth pairs. So if your bond0 interface is not a bridge (I assume it's a bond), then you would no longer get the veth links. I'm not sure how it would have worked without a bridge previously though. Mark

...

Regards,

Tony Pearce

On Thu, 17 Jun 2021 at 16:13, Tony Pearce <tonyppe@gmail.com> wrote:

...

Hi Mark,

I made some time to test this again today with Victoria on a different

ACH. During host configure, it fails not finding python:

...

...

TASK [Verify that a command can be executed]

---

...

...

fatal: [juc-ucsb-5-p]: FAILED! => {"changed": false, "module_stderr": "Shared connection to 192.168.29.235 closed.\r\n", "module_stdout": "/bin/sh: /usr/bin/python3: No such file or directory\r\n", "msg": "The module failed to execute correctly, you probably need to set the interpreter.\nSee stdout/stderr for the exact error", "rc": 127}

PLAY RECAP

---

...

...

juc-ucsb-5-p : ok=4 changed=1 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0

The task you mentioned previously, was ran but was not run against the host because no hosts matched:

PLAY [Ensure python is installed]

---

...

...

skipping: no hosts matched

I looked at `venvs/kayobe/share/kayobe/ansible/kayobe-ansible-user.yml` and a comment in there says it's only run if the kayobe user account is inaccessible. In my deployment I have "#kayobe_ansible_user:" which is not defined by me. Previously, I defined it as my management user and it caused an issue with the password. So I'm unsure why this is an issue.

To work around, I manually installed python and the host configure was successful this time around. I tried this twice and same experience both times.

Then later, during service deploy it fails here:

RUNNING HANDLER [common : Restart fluentd container]

---

...

...

fatal: [juc-ucsb-5-p]: FAILED! => {"changed": true, "msg": "'Traceback (most recent call last):\\n File \"/opt/kayobe/venvs/kolla-ansible/lib/python3.6/site-packages/docker/api/client.py\", line 259, in _raise_for_status\\n response.raise_for_status()\\n File \"/opt/kayobe/venvs/kolla-ansible/lib/python3.6/site-packages/requests/models.py\", line 941, in raise_for_status\\n raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/containers/fluentd/start\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \"/tmp/ansible_kolla_docker_payload_34omrn2y/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 1131, in main\\n File \"/tmp/ansible_kolla_docker_payload_34omrn2y/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 785, in recreate_or_restart_container\\n File \"/tmp/ansible_kolla_docker_payload_34omrn2y/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 817, in start_container\\n File \"/opt/kayobe/venvs/kolla-ansible/lib/python3.6/site-packages/docker/utils/decorators.py\", line 19, in wrapped\\n return f(self, resource_id, *args, **kwargs)\\n File \"/opt/kayobe/venvs/kolla-ansible/lib/python3.6/site-packages/docker/api/container.py\", line 1108, in start\\n self._raise_for_status(res)\\n File \"/opt/kayobe/venvs/kolla-ansible/lib/python3.6/site-packages/docker/api/client.py\", line 261, in _raise_for_status\\n raise create_api_error_from_http_exception(e)\\n File \"/opt/kayobe/venvs/kolla-ansible/lib/python3.6/site-packages/docker/errors.py\", line 31, in create_api_error_from_http_exception\\n raise cls(e, response=response, explanation=explanation)\\ndocker.errors.APIError: 500 Server Error: Internal Server Error (\"error while creating mount source path \\'/etc/localtime\\': mkdir /etc/lo

The error says that the file exists. So the first time I just renamed the symlink file and then this was successful in terms of allowing the deploy process to proceed past this point of failure. The 2nd time around, the rename was not good enough because there's a check to make sure that the file is present there. So the 2nd time around I issued "touch /etc/localtime" after renaming the existing and then this passed.

Lastly, the deploy fails with a blocking action that I cannot resolve myself:

TASK [openvswitch : Ensuring OVS bridge is properly setup]

---

...

...

changed: [juc-ucsb-5-p] => (item=['enp6s0-ovs', 'enp6s0'])

This step breaks networking on the host. Looking at the openvswitchdb, I think this could be something similar to the issue seen before with Wallaby. The first time I tried this was with enp6s0 configured as a bond0 as desired. I then tried without a bond0 and both times got the same result. If I reboot the host then I can get successful ping replies for a short while before they stop again. Same experience as previous. I believe the pings stop when the bridge config is applied from the container shortly after host boot up. Ovs-vsctl show output: [1]

I took a look at the logs [2] but to me I dont see anything alarming or that could point to the issue. I've previously tried turning off IPv6 and this did not have success in this part, although the log message about IPv6 went away.

I tried removing the physical interface from the bridge "ovs-vsctl del-port..." and as soon as I do this, I can ping the host once again. Once I re-add the port back to the bridge, I can no longer connect to the host. There's no errors from ovs-vsctl at this point, either.

[1] ovs-vsctl output screenshot [2] ovs logs screenshot

BTW I am trimming the rest of the mail off because it exceeds 40kb size for the group.

Kind regards,

Tony Pearce

...

...