[openstack-dev] [ironic] using ironic as a replacement for existing datacenter baremetal provisioning

Clint Byrum clint at fewbar.com
Mon Jun 6 21:56:37 UTC 2016

Excerpts from Kris G. Lindgren's message of 2016-06-06 20:44:26 +0000:
> Hi ironic folks,
> As I'm trying to explore how GoDaddy can use ironic I've created the following in an attempt to document some of my concerns, and I'm wondering if you folks could help myself identity ongoing work to solve these (or alternatives?)

Hi Kris. I've been using Ironic in various forms for a while, and I can
answer a few of these things.

> List of concerns with ironic:
> 1.)Nova <-> ironic interactions are generally seem terrible?

I don't know if I'd call it terrible, but there's friction. Things that
are unchangable on hardware are just software configs in vms (like mac
addresses, overlays, etc), and things that make no sense in VMs are
pretty standard on servers (trunked vlans, bonding, etc).

One way we've gotten around it is by using Ironic standalone via
Bifrost[1]. This deploys Ironic in wide open auth mode on,
and includes playbooks to build config drives and deploy images in a
fairly rudimentary way without Nova.

I call this the "better than Cobbler" way of getting a toe into the
Ironic waters.

[1] https://github.com/openstack/bifrost

>   -How to accept raid config and partitioning(?) from end users? Seems to not a yet agreed upon method between nova/ironic.

AFAIK accepting it from the users just isn't solved. Administrators
do have custom ramdisks that they boot to pre-configure RAID during

>    -How to run multiple conductors/nova-computes?   Right now as far as I can tell all of ironic front-end by a single nova-compute, which I will have to manage via a cluster technology between two or mode nodes.  Because of this and the way host-agregates work I am unable to expose fault domains for ironic instances (all of ironic can only be under a single AZ (the az that is assigned to the nova-compute node)). Unless I create multiple nova-compute servers and manage multiple independent ironic setups.  This makes on-boarding/query of hardware capacity painful.

The nova-compute does almost nothing. It really just talks to the
scheduler to tell it what's going on in Ironic. If it dies, deploys
won't stop. You can run many many conductors and spread load and fault
tolerance among them easily. I think for multiple AZs though, you're
right, there's no way to expose that. Perhaps it can be done with cells,
which I think Rackspace's OnMetal uses (but I'll let them refute or
confirm that).

Seems like the virt driver could be taught to be AZ-aware and some
metadata in the server record could allow AZs to go through to Ironic.

>   - Nova appears to be forcing a we are "compute" as long as "compute" is VMs, means that we will have a baremetal flavor explosion (ie the mismatch between baremetal and VM).
>       - This is a feeling I got from the ironic-nova cross project meeting in Austin.  General exmaple goes back to raid config above. I can configure a single piece of hardware many different ways, but to fit into nova's world view I need to have many different flavors exposed to end-user.  In this way many flavors can map back to a single piece of hardware with just a lsightly different configuration applied. So how am I suppose to do a single server with 6 drives as either: Raid 1 + Raid 5, Raid 5, Raid 10, Raid 6, or JBOD.  Seems like I would need to pre-mark out servers that were going to be a specific raid level.  Which means that I need to start managing additional sub-pools of hardware to just deal with how the end users wants the raid configured, this is pretty much a non-starter for us.  I have not really heard of whats being done on this specific front.

You got that right. Perhaps people are comfortable with this limitation.
It is at least simple.

> 2.) Inspector:
>   - IPA service doesn't gather port/switching information
>   - Inspection service doesn't process port/switching information, which means that it wont add it to ironic.  Which makes managing network swinging of the host a non-starter.  As I would inspect the host – then modify the ironic record to add the details about what port/switch the server is connected to from a different source.  At that point why wouldn't I just onboard everything through the API?
>   - Doesn't grab hardware disk configurations, If the server has multiple raids (r1 + r5) only reports boot raid disk capacity.
>   - Inspection is geared towards using a different network and dnsmasq infrastructure than what is in use for ironic/neutron.  Which also means that in order to not conflict with dhcp requests for servers in ironic I need to use different networks.  Which also means I now need to handle swinging server ports between different networks.
> 3.) IPA image:
>   - Default build stuff is pinned to extremly old versions due to gate failure issues. So I can not work without a fork for onboard of servers due to the fact that IPMI modules aren't built for the kernel, so inspection can never match the node against ironic.  Seems like currently functionality here is MVP for gate to work and to deploy images.  But if you need to do firmware, bios-config, any other hardware specific features you are pretty much going to need to roll your own IPA image and IPA modules to do standard provisioning tasks.
> 4.) Conductor:
>   - Serial-over-lan consoles require a unique port on the conductor server (I have seen purposes to try and fix this?), this is painful to manage with large numbers of servers.
>   - SOL consoles aren't restarted when conductor is restarted (I think this might be fixed in newer versions of ironic?), again if end users aren't suppose to consume ironic api's directly - this is painful to handle.
>   - Its very easy to get a node to fall off the staemachine rails (reboot a server while an image is being deployed to it), the only way I have seen to be able to fix this is to update the DB directly.
>   - As far as I can tell shell-in-a- box, SOL consoles aren't support via nova – so how are end users suppose to consume the shell-in-box console?
>   - I have BMC that need specific configuration (some require SOL on com2, others on com1) this makes it pretty much impossible without per box overrides against the conductor hardcoded templates.
>   - Additionally it would be nice to default to having a provisioning kernel/image that was set as a single config option with per server overrides – rather than on each server.  If we ever change the IPA image – that means at scale we would need to update thousands of ironic nodes.
> What is ironic doing to monitor the hardware for failures?  I assume the answer here is nothing and that we will need to make sure the images that we deploy are correctly configuring the tools to monitor disk/health/psu's/ram errors, ect. ect.
> Overall the above concerns have me wondering if I am going crazy, or is ironic really not ready to take over datacenters baremetal provisioning (unless you have a very limited view of functionality that is needed in those datacenters).

You're not crazy at all. I remember having some of these discussions in
the early days when it was just 'nova baremetal' and people wanted AZs
and better serial console access. For whatever reason, it just hasn't
quite happened yet.

It's not a small job, but I don't think anything you say above is
particularly troubling. I'd certainly like better console support and AZs.

More information about the OpenStack-dev mailing list