Open Stack

On Mon, Aug 27, 2018, at 4:21 PM, Clark Boylan wrote:
> Hello everyone,
> 
> I've been trying to help mnaser bootstrap a new cloud region in 
> nodepool. Part of this requires launching a new mirror node in that 
> cloud region and I have discovered some interesting things related to 
> our migration to launch node and how they affect our ability to boot new 
> instances. I'm going to try and summarize all that I have learned as you 
> may find yourself needing to launch nodes too. Also there is plenty to 
> do to help make this better if interested :)
> 
> The first thing I ran into was that /etc/openstack/clouds.yaml is not 
> managed by ansible on bridge.openstack.org. This meant the new region 
> info was not automatically added to that file. I made a copy of the 
> clouds.yaml file in my homedir, updated it with new region, tested that 
> it worked, then updated the global file with these changes manually. 
> Pabelanger has volunteered to work on getting this managed by ansible 
> and https://review.openstack.org/#/c/593029/ seemed to be where Monty 
> had started.
> 
> While testing the clouds.yaml changes I ran into the lack of 
> openstackclient installation which had me learn that virtualenv is weird 
> on Bionic with python3.  `python3 /usr/lib/python3/dist-packages/
> virtualenv.py --python=python3 venv` is what I ended up using to create 
> a virtualenv to install openstackclient into.
> 
> With that done I ran launch-node.py and ran into a couple issues. First 
> we need volume size specification support for boot from volume hosts 
> (added in https://review.openstack.org/554313). I also updated docs to 
> be clear we run under python3 in 
> https://review.openstack.org/#/c/596859/1 and fixed a bug with unlinking 
> ansible caches at https://review.openstack.org/#/c/596873/1. Reviews 
> much appreciated.
> 
> This got the instance booting happily but then ssh wasn't working which 
> was the result of not running cloud launcher on bridge.o.o yet (and its 
> not running on old puppet master either). I went ahead and manually 
> triggered cloud launcher against both new accounts in this cloud region 
> to update ssh root keys and security groups. We will need to get this 
> running automatically on bridge.o.o.
> 
> This resulted in launch-node.py exiting successfully but then the puppet 
> cron failed to run puppet on the host to turn it into a new mirror node 
> because there is no python2 installed on Xenial by default. 
> https://review.openstack.org/#/c/596913/ is a work around for this 
> (explicitly install python2 on xenial for now, this will help transition 
> us from trusty to xenial too). But also resulted in some other attempts 
> to address this at https://review.openstack.org/#/c/596905/, 
> https://review.openstack.org/#/c/596911/1, and 
> https://review.openstack.org/#/c/596894/. Once again reviews welcome.
> 
> The last thing to note out of this is that Trusty comes with python3.4 
> and centos7 doesn't have python3 and we can't run ansible under python3 
> on these platforms. This means launch-node.py cannot currently boot 
> trusty or centos nodes. If we end up needing to boot a trusty or centos 
> node we will have to force it to use python2 on the remote instead.
> 
> TL;DR We need to:
> * Manage clouds.yaml with ansible
> * Fix a couple bugs in launch-node.py
> * Run cloud launcher on bridge.openstack.org
> * Figure out our python2 to python3 transition plan for Ansible
> 
> Hopefully this helps you out if you end up needing to launch a new node 
> while we transition to bridge.openstack.org.

One quick new observation. launch-node.py does not install puppet at all so the subsequent ansible runs on the newly launched instances will fail when attempting to stop the puppet service (and will continue on to fail to run puppet as well I think).

Clark