Re: Wallaby install via kayobe onto ubuntu 20 all in one host

Possibly fact caching could have been the issue. The Victoria guide
describes to configure fact caching so it was being used by my ACH. I did
not think of this at the time of the issue, sorry about that.

I have a new question about SSL cert, so will send a new mail for the
topic.

Thank you again,

Kind regards,

Tony Pearce



On Wed, 16 Jun 2021 at 16:09, Mark Goddard <mark@stackhpc.com> wrote:

> On Wed, 16 Jun 2021 at 07:09, Tony Pearce <tonyppe@gmail.com> wrote:
> >
> > Hi Mark, thanks again for your reply and direction on this.
> >
> > Regarding the need to do a pip install within the venv, I re-installed
> the host and re-installed kayobe and this issue is no longer present as of
> yesterday afternoon.
> >
> > Regarding the "Waiting for nova-compute services to register themselves"
> - I tailed *.log in the kolla/nova log dir and to be honest there wasnt
> much being logged at that time and no errors, either.
> >
> > Regarding Wallaby hasnt yet been released, I went back to Victoria on
> Centos 8.4 this morning, using the "pip install ." (installation from
> source). I will use this install from source going forward to make sure I
> am getting the latest kayobe patches. I had two blockers during "host
> configure":
> >
> > First was that python was not found on the host:
> >
> > TASK [Verify that a command can be executed]
> *****************************************************************************************************************************
> > fatal: [juc-ucsb-5-p]: FAILED! => {"changed": false, "module_stderr":
> "/bin/sh: /usr/bin/python3: No such file or directory\n", "module_stdout":
> "", "msg": "The module failed to execute correctly, you probably need to
> set the interpreter.\nSee stdout/stderr for the exact error", "rc": 127}
>
> In stable/victoria there is a task called 'Ensure python is installed'
> which should ensure that Python 3 is installed. Did that task run?
>
> >
> > So I manually installed python to see what would happen. I was able to
> get past that point but then it fails again:
> >
> > TASK [Ensure the Python virtualenv package is installed]
> *****************************************************************************************************************
> > [WARNING]: Updating cache and auto-installing missing dependency:
> python3-apt
> > fatal: [juc-ucsb-5-p]: FAILED! => {"changed": false, "cmd": "apt-get
> update", "msg": "[Errno 2] No such file or directory: b'apt-get':
> b'apt-get'", "rc": 2}
> >
> > PLAY RECAP
> ***************************************************************************************************************************************************************
> > juc-ucsb-5-p               : ok=7    changed=0    unreachable=0
> failed=1    skipped=10   rescued=0    ignored=0
> >
> > The above log suggests that the CentOS 8.4 host is being matched as
> ubuntu. As an FYI, what I mean when I say "went back to Victoria" is that I
> followed the initial kayobe install steps again with new top directory
> structure and then manually transferred over the kayobe config to the new
> empty config which is downloaded during the git clone. "manually
> transferred" means I opened each yml config file individually and copied
> over only the variables that have been configured within the file, making
> sure to leave defaults as a prefernce.
>
> Something odd is clearly going on here. We use the package module,
> which uses ansible's ansible_pkg_mgr fact to dispatch the correct
> packaging module. Perhaps you have fact caching enabled, and have
> stale facts from when the host was running Ubuntu?
>
> >
> > After the above issue I decided to go back to Train and have been able
> to successfully deploy Openstack using kayobe Train today - no blocking
> issues, first time.
> >
> > Summary
> > I really appreciate your time to help me with this, so thanks again. I
> Still think kayobe is the best way to deploy and manage openstack and I
> have been "telling my friends" also :) Now I am back to this point I just
> have some minor things to work through before I can start using it.
> >
> > Kind regards,
> >
> > Tony Pearce
> >
> >
> > On Tue, 15 Jun 2021 at 16:02, Mark Goddard <mark@stackhpc.com> wrote:
> >>
> >> On Tue, 15 Jun 2021 at 06:51, Tony Pearce <tonyppe@gmail.com> wrote:
> >> >
> >> > Hi Mark,
> >> >
> >> > I had never used the "pip install ." method. Maybe a miscomprehension
> on my side, from the documentation [1] there are three ways to install
> kayobe. I had opted for the first way which is "pip install kayobe" since
> January 2020. The understanding was as conveyed in the doc "Installing from
> PyPI ensures the use of well used and tested software".
> >>
> >> That is true, but since Wallaby has not been released for Kayobe yet,
> >> it is not on PyPI. If you do install from PyPI, I would advise using a
> >> version constraint to ensure you get the release series you need.
> >>
> >> >
> >> > I have since followed your steps in your mail which is the
> installation from source. I had new problems:
> >> >
> >> > During ansible bootstrap:
> >> > During ansible host bootstrap it errors out and says the
> kolla_ansible is not found and needs to be installed in the same virtual
> environment. In all previous times, I had understood that kolla ansible is
> installed by kayobe at this point. I eventually done "pip install
> kolla-ansible" and it seemed to take care of that and allowed me to move on
> to "host configure"
> >>
> >> Kolla Ansible should be installed automatically during 'kayobe control
> >> host bootstrap', in a separate virtualenv from Kayobe. You should not
> >> need to install it manually, and I would again advise against doing so
> >> without version constraints.
> >>
> >> >
> >> > During host configure:
> >> > I was able to get past the previous python issue but then it failed
> on the network due to a "duplicate bond name", though this config was
> deployed successfully in Train. I dont think I really need a bond at this
> point so I deleted the bond and the host configure is now successful. (fyi
> this is an all-in-one host.)
> >> >
> >> > During kayobe service deploy:
> >> > This then fails with "no module named docker" on the host. To
> troubleshoot this I logged into the host and activated the kayobe virtual
> env (/opt/kayobe/venvs/kayobe/bin/activate) and then "pip install docker".
> It was already installed. Eventually, I issued "pip install
> --ignore-installed docker" within these three (environment) locations which
> resolved this and allowed the kayobe command to complete successfully and
> progress further:
> >> > - /opt/kayobe/venvs/kayobe/
> >> > - /opt/kayobe/venvs/kolla-ansible/
> >> > - native on the host after deactivating the venv.
> >> >
> >> > Now the blocker is the following failure;
> >> >
> >> > TASK [nova-cell : Waiting for nova-compute services to register
> themselves]
> **********************************************************************************************
> >> > FAILED - RETRYING: Waiting for nova-compute services to register
> themselves (20 retries left).
> >> > FAILED - RETRYING: Waiting for nova-compute services to register
> themselves (19 retries left).
> >> >
> >> > I haven't seen this one before but previously I had seen something
> similar with mariadb because the API dns was not available. What I have
> been using here is a /etc/hosts entry for this. I checked that this entry
> is available on the host and in the nova containers. I decided to reboot
> the host anyway (previously resolved similar mariadb issue) to restart the
> containers just in case the dns was not available in one of them and I
> missed it.
> >>
> >> I'd check the nova compute logs here, to find why they are not
> >> registering themselves.
> >> >
> >> > Unfortunately I now have two additional issues which are hard
> blockers:
> >> > 1. The network is no longer working on the host after reboot, so I am
> unable to ssh
> >> > 2. The user password has been changed by kayobe, so I am unable to
> login using the console
> >> >
> >> > Due to the above, I am unable to login to the host to investigate or
> remediate. Previously when this happened with centos I could use the root
> user to log in. This time around as it's ubuntu I do not have a root user.
> >> > The user I am using for both "kolla_ansible_user" and
> "kayobe_ansible_user" is the same - is this causing a problem with Victoria
> and Wallaby? I had this user password change issue beginning with Victoria.
> >> >
> >> > So at this point I need to re-install the host and go back to the
> host configure before service deploy.
> >> >
> >> > Summary
> >> > Any guidance is well appreciated as I'm at a loss at this point. Last
> week I had a working Openstack Train deployment in a single host. "Kayobe"
> stopped working (maybe because I had previously always used pip install
> kayobe).
> >> >
> >> > I would like to deploy Wallaby, should I be able to successfully do
> this today or should I be using Victoria at the moment (or even, Train)?
> >>
> >> We are very close to release of Wallaby, and I expect that it should
> >> generally work, but Ubuntu is a new distro for Kayobe, and Wallaby is
> >> a new release. There may be teething problems, so if you're looking
> >> for something more stable then I'd suggest CentOS & Victoria.
> >>
> >> >
> >> > [1] OpenStack Docs: Installation
> >> >
> >> > Regards,
> >> >
> >> > Tony Pearce
> >> >
> >> >
> >> > On Mon, 14 Jun 2021 at 18:36, Mark Goddard <mark@stackhpc.com> wrote:
> >> >>
> >> >> On Mon, 14 Jun 2021 at 09:40, Tony Pearce <tonyppe@gmail.com> wrote:
> >> >> >
> >> >> > Hi Mark,
> >> >> >
> >> >> > I followed this guide to do a "git clone" specifying the branch
> "-b" to "stable/wallaby" [1]. What additional steps do I need to do to get
> the latest commits?
> >> >>
> >> >> That should be sufficient. When you install it via pip, note that
> 'pip
> >> >> install kayobe' will still pull from PyPI, even if there is a local
> >> >> kayobe directory. Use ./kayobe, or 'pip install .'  if in the same
> >> >> directory.
> >> >>
> >> >> Mark
> >> >> >
> >> >> > [1] OpenStack Docs: Overcloud
> >> >> >
> >> >> > Kind regards,
> >> >> >
> >> >> > Tony Pearce
> >> >> >
> >> >> >
> >> >> > On Mon, 14 Jun 2021 at 16:10, Mark Goddard <mark@stackhpc.com>
> wrote:
> >> >> >>
> >> >> >> On Mon, 14 Jun 2021 at 07:21, Tony Pearce <tonyppe@gmail.com>
> wrote:
> >> >> >> >
> >> >> >> > Hi Pierre, thanks for replying to my message.
> >> >> >> >
> >> >> >> > To install kayobe I followed the documentation which summarise:
> installing a few system packages and setting up the kayobe virtual
> environment and then pulling the correct kayobe git version for the
> openstack to be installed. After configuring the yaml files I have run
> these commands:
> >> >> >> >
> >> >> >> > - kayobe control host bootstrap
> >> >> >> > - kayobe overcloud host configure -> this one is failing with
> /usr/libexec/platform-python: not found
> >> >> >> >
> >> >> >> > After reading your message on the weekend I concluded that
> maybe I had done something wrong. Today, I re-pulled the kayobe wallaby git
> and manually transferred the configuration over to the new directory
> structure on the ansible host and set up again as per the guide but the
> same issue is seen.
> >> >> >> >
> >> >> >> > What I ended up doing to try and resolve was finding where this
> "platform-python" is coming from. It is coming from the virtual environment
> which is being set up during the kayobe ansible host bootstrap. Initially,
> I found the base.yml and it looks like it tries to match what the host is.
> I noticed that there is no ubuntu 20 listed there so I created it however
> it did not resolve the issue.
> >> >> >> >
> >> >> >> > So then I tried systematically replacing this reference in the
> other files found in the same location "venvs\kayobe\share\kayobe\ansible".
> The file I changed which allowed it to progress is "kayobe-target-venv.yml"
> >> >> >> >
> >> >> >> > But unfortunately it fails a bit further on, failing to find an
> selinux package [1]
> >> >> >> >
> >> >> >> > Seeing as the error is mentioning selinux (a RedHat security
> feature not installed on ubuntu) could the root cause issue be that kayobe
> is not matching the host as ubuntu? I did already set in kayobe that I am
> using ubuntu OS distribution within globals.yml [2].
> >> >> >> >
> >> >> >> > Are there any extra steps that I need to complete that maybe
> are not listed in the documentation / guide?
> >> >> >> >
> >> >> >> > [1] TASK [MichaelRigart.interfaces : Debian | install
> current/latest network package - Pastebin.com
> >> >> >> > [2] ---# Kayobe global
> configuration.######################################### - Pastebin.com
> >> >> >>
> >> >> >> Hi Tony,
> >> >> >>
> >> >> >> That's definitely not a recent Wallaby checkout you're using.
> Ubuntu
> >> >> >> no longer uses that MichaelRigart.interfaces role. Check that you
> have
> >> >> >> recent commits. Here is the most recent on stable/wallaby:
> >> >> >> 13169077aaec0f7a28ae1f15b419dafc2456faf7.
> >> >> >>
> >> >> >> Mark
> >> >> >>
> >> >> >> >
> >> >> >> > Regards,
> >> >> >> >
> >> >> >> > Tony Pearce
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On Fri, 11 Jun 2021 at 21:05, Pierre Riteau <
> pierre@stackhpc.com> wrote:
> >> >> >> >>
> >> >> >> >> Hi Tony,
> >> >> >> >>
> >> >> >> >> Kayobe doesn't use platform-python anymore, on both
> stable/wallaby and
> >> >> >> >> stable/victoria:
> >> >> >> >>
> https://review.opendev.org/q/I0d477325e0edd13d1aba211c13dc2e8b7a9b4c98
> >> >> >> >>
> >> >> >> >> Can you double-check what version you are using, and share how
> you
> >> >> >> >> installed it? Note that only stable/wallaby supports Ubuntu 20
> hosts.
> >> >> >> >>
> >> >> >> >> Best wishes,
> >> >> >> >> Pierre
> >> >> >> >>
> >> >> >> >> On Fri, 11 Jun 2021 at 13:20, Tony Pearce <tonyppe@gmail.com>
> wrote:
> >> >> >> >> >
> >> >> >> >> > I'm trying to run "kayobe overcloud host configure" against
> an ubuntu 20 machine to deploy Wallaby. I'm getting an error that python is
> not found during the host configure part.
> >> >> >> >> >
> >> >> >> >> > PLAY [Verify that the Kayobe Ansible user account is
> accessible]
> >> >> >> >> > TASK [Verify that a command can be executed]
> >> >> >> >> >
> >> >> >> >> > fatal: [juc-ucsb-5-p]: FAILED! => {"changed": false,
> "module_stderr": "/bin/sh: 1: /usr/libexec/platform-python: not found\n",
> "module_stdout": "", "msg": "The module failed to execute correctly, you
> probably need to set the interpreter.\nSee stdout/stderr for the exact
> error", "rc": 127}
> >> >> >> >> >
> >> >> >> >> > Python3 is installed on the host. When searching where this
> platform-python is coming from it returns the kolla-ansible virtual envs:
> >> >> >> >> >
> >> >> >> >> > $ grep -rni -e "platform-python"
> >> >> >> >> >
> venvs/kolla-ansible/lib/python3.8/site-packages/ansible/config/base.yml:1450:
>     '8': /usr/libexec/platform-python
> >> >> >> >> >
> venvs/kolla-ansible/lib/python3.8/site-packages/ansible/config/base.yml:1470:
> - /usr/libexec/platform-python
> >> >> >> >> >
> >> >> >> >> > I had a look through the deployment guide for Kayobe Wallaby
> and didnt see a note about changing this.
> >> >> >> >> >
> >> >> >> >> > Do I need to do further steps to support the ubuntu
> overcloud host? I have already set (as per the doc):
> >> >> >> >> >
> >> >> >> >> > os_distribution: ubuntu
> >> >> >> >> > os_release: focal
> >> >> >> >> >
> >> >> >> >> > Regards,
> >> >> >> >> >
> >> >> >> >> > Tony Pearce
> >> >> >> >> >
>