Open Stack

Hey Stackers,

In Fuel we use bare metal testing for deployment tests. This is essentially a core component of Fuel CI and as much as we like having it around we’d rather spend time and resources integrating with upstream instead of growing and polishing third-party testing solutions.

On one of the previous Infra team meetings we discussed the possibility of bringing testing on bare metal nodes to openstack-infra[1]. This is not a new topic, similar question was brought up by Magnum some time ago[2] and there might other times this was discussed. We use bare metal testing for Fuel, I assume that Magnum still wants to use it, TripleO would probably also fit in the picture in some way (though I’m not familiar with current scheme of TripleO CI) - hope this is enough to consider implementation of generic way to use baremetal nodes in CI.

The most obvious way to do this seems to be using existing OpenStack service for bare metal provisioning - Ironic. Ironic fits pretty well in existing Infra workflow, Ironic usage (in form of Rackspace's OnMetal) was previously discussed in Magnum thread[2] with the main technical issue being inability to use custom glance images to boot instances. AFAIK the situation didn't change much with OnMetal, but Ironic perfectly supports booting from glance images created by diskimage-builder - which is exactly the way Nodepool currently works for virtual machines.

With the work currently going on InfraCloud there's a possibility to properly design and implement bare metal testing, Zuul v3 spec[3] also brings a number of relevant changes to Nodepool. So, summing up some points of possible implementation:
* Multiple pools of bare metal nodes under Ironic management are available as a part of InfraCloud
* Ironic acts as an additional hypervisor for Nova, providing the ability to use bare metal nodes by booting an instance with a specific flavor
* Nodepool manages booting bare metal instances using the images generated with diskimage-builder and stored in Glance
* Nodepool also manages redeployment of bare metal nodes - redeploying a glance image on a bare metal node takes only a few minutes, but time may depend on a set of cleaning steps used to redeploy a node
* Bare metal instances are exposed to Jenkins (or a different worker in case of Zuul v3) by Nodepool 

I suppose there are security issues when we talk about running custom code on bare metal slaves, but I'm not sure I understand the difference from running custom code on a virtual machine if bare metal nodes are isolated, don't contain any sensitive data and follow a regular redeployment procedure.

I'd like to add that we're ready to start donating hardware from the Fuel CI pool (2 pools in different locations, to be accurate) to see this initiative taking off.

Please, share your thoughts and opinions.

[1]http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-03-29-19.03.log.html
[2]http://lists.openstack.org/pipermail/openstack-infra/2015-September/003138.html
[3]http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html
--
Igor Belikov
Fuel CI Engineer
ibelikov at mirantis.com