Open Stack

Others made good points for posting this on the ML, so here it is in
full. Sorry for the markdown formatting, I just copied this from the
blog post.

// jim

Another cycle, another summit. The ironic project had ten design summit
sessions to get together and chat about some of our current and future
work. We also led a cross-project session on bare metal networking, had
a joint session with nova, and a contributor's meetup for the first half
of Friday. The following is a summary of those sessions.

# Cross-project: the future of bare-metal networking

[Etherpad](https://etherpad.openstack.org/p/newton-baremetal-networking)

This session was meant to have the Nova, Ironic, and Neutron folks get
together and figure out some of the details of the [work we're
doing](https://review.openstack.org/#/c/277853/) to decouple the
physical network infrastructure from the logical networking that users
interact with. Unfortunately, we spent most of the time explaining the
problem and the goals, and not much time actually figuring out how
things should work. We were able to decide that the trunk port work in
neutron should mostly work for us.

There was plenty of hallway chats about this throughout the week, and
from those I think we have a good idea of what needs to be done. The
spec linked above will be updated soon to clarify where we are at here.

# Nova-compatible serial and graphical consoles

[Etherpad](https://etherpad.openstack.org/p/ironic-newton-summit-console)

This session began with a number of proposals to implement serial and
graphical consoles that would work with Nova, and a goal to narrow them
down so folks can move forward with the code.

The first thing we decided is that in the short term, we want to focus
on the serial console. It's supported by almost all hardware and most
cases where someone needs a console are covered by a simple serial
console. We do want to do graphical consoles eventually, but would like
to take one thing at a time.

We then spent some time dissecting our requirements (and preferences)
for what we want an implementation to do, which are listed toward the
bottom of the etherpad.

We narrowed the serial console work down to two implementations:

* [ironic-console-server](https://review.openstack.org/#/c/306755/).
  The tl;dr here is that the conductor will shell out to a command that
  creates a listening port, and forks a process that connects to the
  console, and pipes data between the two. This command is called once
  per console session. The upside with this approach is that operators
  don't need to do much when the change is deployed.

* [ironic-ipmiproxy](https://review.openstack.org/#/c/296869/).
  This is similar to ironic-console-server, except that it runs as its
  own daemon with a small REST API for start/stop/get. It spawns a
  process for each console, which does not fork itself. The upside here
  is that it can be scaled independently, and has no implications on
  conductor failover; however it will need its own HA model as well, and
  will be more work for deployers.

It seems like most folks agree that the latter is more desirable, in
terms of scaling model and such, but we didn't quite come to consensus
during the session. We need to do that ASAP.

We also talked a bit about console logging, and the pitfalls of doing it
automatically for every instance. For example, some BMCs crash if power
status is called repeatedly with a serial-over-lan session active (this
is something to consider for regular console attach as well). We'll need
to make this operator configurable, possibly per-node, so that we aren't
automatically crashing bad BMCs for people. The nova team agreed later
that this is fine, as long as a decent error is returned in this case.

# Status and future of our gate

[Etherpad](https://etherpad.openstack.org/p/ironic-newton-summit-gate)

We discussed the current status of our gate, and the plans for Newton.

We first talked about third-party CI, and where we're at with that. Kurt
Taylor is doing the main tracking of that, and explained where and how
we're tracking it. There was a call for help for some of the missing
data, and getting all the right pages updated (stackalytics,
openstack.org marketplace, etc).

We also talked about the current changes going into our gate that we
want to push forward. Moving to tinyipa and virtualbmc (with ipmitool
drivers) are the main changes right now. We discussed the progress on
upgrade testing via grenade. There hasn't been a lot of progress made,
but some of the groundwork to make local testing easy has been done.
Later in the week, during the priorities session, we agreed that the
upgrade testing was our top priority right now, and some folks
volunteered to help move it along.

# Hardware pool management

[Etherpad](https://etherpad.openstack.org/p/ironic-newton-summit-hardware-pools)

This topic is talked about at nearly every summit, and we said that we
need to at least solve the internals this round.

We narrowed in on what we think is a good architecture for this. Given
that names are hard, we decided we would add a "thing" resource (yes,
this name will change). This is some sort of management interface for a
group of nodes, and every node will be mapped 1:1 to a "thing" by
default. Credentials can be optionally placed in the "thing" instead of
on the node, and there can be a 1:n thing:node mapping. This will allow
ironic to do group operations for hardware to support it. Of course,
there will be "thing" drivers, because every hardware does this
differently. :)

We also decided to make this an internal-only feature for now, and not
expose it to the REST API yet. It can be used as an optimization in
internal code, or to support hardware that can only be managed as a
group. We may eventually decide to expose group management features to
the REST API, but not yet.

# Driver composition

[Etherpad](https://etherpad.openstack.org/p/ironic-newton-summit-driver-composition)

This is another topic we keep discussing without coming to conclusions,
and I think we made good progress this round. Significant work went into
the spec before the summit, and folks came prepared with a solid
proposal.

We agreed on the concept of a "hardware type", which declares
compatibility between driver interface implementations. These will be
hard-coded into ironic, to what the vendor expects and the generic
interfaces that ironic provides.

We also agreed that out of tree implementations of an interface should
not be allowed to declare compatibility with in-tree vendor hardware
types. For example, one could not make a power interface out of tree
that declares compatibility with the in-tree "pizza box" hardware type.
Out of tree drivers can, however, declare their own hardware types that
may be used.

We also discussed upgrades, and using a new `hardware_type` field on the
node object, to be used for the migration path. We didn't fully come to
consensur on the upgrade path, but we're close, and the details are in
the etherpad.

# Making ops less worse

[Etherpad](https://etherpad.openstack.org/p/ironic-newton-summit-ops)

We discussed some common failure cases that operators see, and how we
can solve them in code.

We discussed flaky BMCs, which end with the node in maintenance mode,
and if Ironic can get them out of that mode automagically. We identified
the need to distinguish between maintenance set by ironic and set by
operators, and do things like attempt to connect to the BMC on a power
state request, and turn off maintenance mode if successful. JayF is
going to write a spec for this differentiation.

Folks also expressed the desire to be able to reset the BMC via APIs. We
have a BMC reset function in the vendor interface for the ipmitool
driver; dtantsur volunteered to write a spec to promote that method to
an official ManagementInterface method.

We also talked for a while about stuck states. This has been mostly
solved in code, but is still a problem for some deployers. We decided
that we should not have a "reset-state" API like nova does, but rather a
command line tool to handle this. lintan has volunteered to write a
proposal for this; I have also posted some [straw man
code](https://review.openstack.org/#/c/311273/) that someone is welcome
to take over or use.

# Anomaly detection and resolution

[Etherpad](https://etherpad.openstack.org/p/ironic-newton-summit-anomaly-detection)

This session kind of naturally flowed from the previous one, but was
more about how we can automatically detect failure cases, and
potentially automatically fix them. If we can't do these, then what do
we need to build to allow external tooling to do it?

We concluded with a few things that we can do now to get started:

* Build a node error event DB table, such that errors can be fetched via
  our API.

* Sending notifications on every state change (this is already in
  progress). Other tools can subscribe to them to watch for anomalies.

* Add a periodic task that polls BMCs for hardware event/error logs. We
  could store these or emit them as notifications.

# Ansible deploy driver

[Etherpad](https://etherpad.openstack.org/p/ironic-newton-summit-ansible-deploy)

Some folks from Mirantis presented their proposal for an ansible-based
deploy driver, and allowed us to ask questions.

The primary use case for this is to allow operators to easily change the
deploy process; the ansible playbooks for this are configuration, node
code. Some people had concerns about this, especially around
supportability and the fact that we (upstream) effectively have no
control over how a deployment works. This is analogous to allowing an
operator to modify spawn() in the nova libvirt driver. However, most
people present were okay with this.

We discussed how to build and secure ramdisks for this, and tossed some
ideas around. We didn't come to any clear consensus, though.

Last, we found that currently each node is deployed in serial. We noted
that this driver is a non-starter until it can deploy many (50?)
machines in parallel from a single conductor host. As such, we've
decided that until this is possible, the team shouldn't be spending much
review time on this.

# Live upgrades

[Etherpad](https://etherpad.openstack.org/p/ironic-newton-summit-live-upgrades)

Here, we discussed what we need to get to rolling upgrades, and what we
should be testing to confirm these work.

We noted that the requirement for the "supports rolling upgrade" tag in
the governance repo is only testing last-stable to master upgrade (and
the equivalent for changes on the stable branch. Given that we do
intermediate releases, we also want to test upgrades from last numbered
release (because that may be more recent than last stable) to master.
Last, we should run a job that upgrades ironic but does not upgrade
nova, to make sure services can be upgraded independently.

We decided that for ironic upgrades, conductor should go first, followed
by the API. This is so that the API doesn't expose functionality before
a conductor supports it.

We decided for full cloud upgrades, ironic should go before nova,
because older nova should always work with newer ironic. We should also
upgrade neutron before ironic, because ironic consumes neutron and we
don't want to depend on functionality that doesn't exist yet. There's an
action item for me to check with the Neutron folks on this, to make sure
Neutron before ironic before nova seems kosher to them.

# Inspector HA

[Etherpad](https://etherpad.openstack.org/p/ironic-newton-summit-inspector)

Milan gave a quick presentation on his proposal for an HA model for
inspector, and we discussed.

Things we agreed on:

* the general proposal

* use tooz for locking and leader election

* split it into an api and conductor service

  * conductor runs active-active

* don't split firewall and dhcp services to a separate service

Details are in the etherpad. :)

# Newton priorities

[Etherpad](https://etherpad.openstack.org/p/ironic-newton-summit-priorities)

We discussed our priorities for the Newton cycle here.

Of note, we decided that we need to get cold upgrade testing (i.e.
grenade) running ASAP. We have lots of large changes lined up that feel
like they could easily break upgrades, and want to be able to test them.
Much of the team is jumping in to help get this going.

The priorities for the cycle have been published
[here](http://specs.openstack.org/openstack/ironic-specs/priorities/newton-priorities.html).

The etherpad also lists some smaller work that we want to prioritize,
but did not publish as such. My big task for early this cycle is to
build a quick landing page that has all priorities with relevant links
to them, and these small things will be included on this page.

# Nova/Ironic cross-project

[Etherpad](https://etherpad.openstack.org/p/newton-nova-ironic)

We started this session by updating the Nova team on the status of a few
things.

We discussed the multitenant networking work, and what's left to do
there. We wondered out loud if the "routed networks" feature planned for
Nova will conflict with this work - johnthetubaguy and myself are to
investigate this further.

We talked about the multiple-compute work, and if the
generic-resource-pools work is a better route to getting there. This
discussion has continued beyond the summit and is being investigated
further.

We then talked about the future console work, and went over what we
decided in the previous session we had about that.

We discussed what nova needs from the ironic team - full tempest runs
(minus what ironic doesn't support) and faster CI runs. Surprise! We
discussed some progress and some options here.

Last, we talked for a few minutes about passing configuration from
flavors to ironic - think BIOS configuration on the fly, depending on
the flavor requested. This was obviously too big a topic to solve in a
few minutes, but we got the wheels spinning.

# Summary

All in all, it was a productive summit for the ironic team, and we have
a clear vision for the next six months.

On Mon, May 09, 2016 at 06:00:46PM -0400, Jim Rollenhagen wrote:
> Hey all,
> 
> I wrote a recap of the summit on my blog:
> http://jroll.ghost.io/newton-summit-recap/
> 
> I hope this covers everything that folks missed or couldn't remember. As
> always, questions/comments/concerns welcome.
> 
> // jim
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev