Open Stack

On Tue, Apr 5, 2016, at 07:13 AM, Isaac Beckman wrote:
> Hi Team,
> 
> I tried to do the following:
> 1) build an image for devstack node
> 2) upload it to glance.
> 3) use that image it in nodepool, by running nodepool image-update
> 
> 
> To build the image I execute the following:
> 
> DISTRO="ubuntu" bash tools/build-image.sh
> (As documented in [1])
> 
> When the image is created I uploaded it to glance and tried to run 
> "nodepool image-update"
> My nodepool.yaml file is very simple:
> 
> script-dir: /etc/nodepool/scripts
> elements-dir: /etc/nodepool/elements
> images-dir: /opt/nodepool_dib
> 
> cron:
>   cleanup: '*/1 * * * *'
>   check: '*/15 * * * *'
>   image-update: '14 14 * * *'
> 
> zmq-publishers:
>   - tcp://ci-jenkins-master.openstacklocal:8888
> 
> gearman-servers:
>   - host: ci-zuul.openstacklocal
> 
> labels:
>   - name: ubuntu-trusty-devstack
>     image: ubuntu-trusty-devstack
>     min-ready: 1
>     providers:
>       - name: ibm-ci
> 
> providers:
>   - name: ibm-ci
>     cloud: osic-cloud2
>     max-servers: 1
>     images:
>       - name: ubuntu-trusty-devstack
>         base-image: 'devstack-gate-new'         <----- This is the image 
> name in the cloud (glance)
>         min-ram: 4096
>         setup: prepare_node_devstack.sh
>         username: jenkins
>         private-key: /home/nodepool/.ssh/id_rsa
>         config-drive: true
> 
> targets:
>   - name: ci-jenkins-master.openstacklocal
> 
> diskimages:
>   - name: ubuntu-trusty-devstack
> 
> 
> 
> The problem is that nodepool's image-update command exits with timeout 
> while trying to ssh connect:
> 
> 2016-04-03 16:33:37,167 INFO nodepool.utils: Auth exception for 
> root at 9.xxx.xxx.xxx. Try number 4...
> 
> 
> I looked in nodepool.py [2]
> And found the following code:
> 
>     def bootstrapServer(self, server, key, use_password=False):
>         log = logging.getLogger("nodepool.image.build.%s.%s" %
>                                 (self.provider.name, self.image.name))
> 
>         ssh_kwargs = dict(log=log)
>         if not use_password:
>             ssh_kwargs['pkey'] = key
>         else:
>             ssh_kwargs['password'] = server['admin_pass']
> 
>         host = utils.ssh_connect(server['public_ip'], 'root', ssh_kwargs, 
>  
>                                  timeout=CONNECT_TIMEOUT)
>     def bootstrapServer(self, server, key, use_password=False):
>         log = logging.getLogger("nodepool.image.build.%s.%s" %
>                                 (self.provider.name, self.image.name))
> 
>         ssh_kwargs = dict(log=log)
>         if not use_password:
>             ssh_kwargs['pkey'] = key
>         else:
>             ssh_kwargs['password'] = server['admin_pass']
> 
>         host = utils.ssh_connect(server['public_ip'], 'root', ssh_kwargs,
>                                  timeout=CONNECT_TIMEOUT)
> 
>         if not host:
>             # We have connected to the node but couldn't do anything as 
> root
>             # try distro specific users, since we know ssh is up (a 
> timeout
>             # didn't occur), we can connect with a very sort timeout.
>             for username in ['ubuntu', 'fedora', 'cloud-user', 'centos',
>                              'debian']:
>                 try:
>                     host = utils.ssh_connect(server['public_ip'], 
> username,
>                                              ssh_kwargs,
>                                              timeout=10)
>                     if host:
>                         break
>                 except:
>                     continue
> 
>         if not host:
>             raise Exception("Unable to log in via SSH") 
>         if not host:
>             # We have connected to the node but couldn't do anything as 
> root
>             # try distro specific users, since we know ssh is up (a 
> timeout
>             # didn't occur), we can connect with a very sort timeout.
>             for username in ['ubuntu', 'fedora', 'cloud-user', 'centos',
>                              'debian']:
>                 try:
>                     host = utils.ssh_connect(server['public_ip'], 
> username,
>                                              ssh_kwargs,
>                                              timeout=10)
>                     if host:
>                         break
>                 except:
>                     continue
> 
>         if not host:
>             raise Exception("Unable to log in via SSH") 
> 
> 
> This code tries to use the key that was generated earlier (by paramiko)
> in 
> order to ssh_connect to root.
> The problem is that no one put that ssh key into root's authorized_keys, 
> and the connection fails.
> 
> Is it a bug? 
> 
> [1] 
> https://github.com/openstack-infra/project-config/tree/master/nodepool/elements
> 
> [2] 
> https://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/nodepool.py

The expectation is that something like cloud init or glean will use the
information provided in the config drive to configure the
authorized_keys. I do not think this is a bug in nodepool; it relies on
the cloud and images cooperating to bootstrap the process. You can check
the instances console log to see if those are running and if they have
any errors.

Clark