Open Stack

This is a bit delayed due to the release rush, finally getting back to
writing up my experiences at the Ops Meetup.

Nova Feedback Session
=====================

We had a double session for Feedback for Nova from Operators, raw
etherpad here - https://etherpad.openstack.org/p/NYC-ops-Nova.

The median release people were on in the room was Kilo. Some were
upgrading to Liberty, many had older than Kilo clouds. Remembering
these are the larger ops environments that are engaged enough with the
community to send people to the Ops Meetup.


Performance Bottlenecks
-----------------------

* scheduling issues with Ironic - (this is a bug we got through during
  the week after the session)
* live snapshots actually end up performance issue for people

The workarounds config group was not well known, and everyone in the
room wished we advertised that a bit more. The solution for snapshot
performance is in there.

There were also general questions about what scale cells should be
considered at.

ACTION: we should make sure workarounds are advertised better
ACTION: we should have some document about "when cells"?

Networking
----------

A number of folks in the room were still on Nova Net, and were a bit
nervous about it going away. As they are Kilo / Liberty it's still a
few upgrades before they get there, but that nervousness and concern
was definitely there.

Policy
------

How are you customizing policy? People were largely making policy
changes to protect their users that didn't really understand cloud
semantics. Turning off features that they thought would confuse them
(like pause). The large number of VM states is confusing, and not
clearly useful for end users, and they would like simplification.

Ideally policy could be set on a project by project admin, because
they would like to delegate that responsibility down.

No one was using the user_id based custom policy (yay!).

There was desire that flavors could be RBAC locked down, which was
actually being done via policy hacks right now. Providers want to
expose some flavors (especially those with aggregate affinity) to only
some projects.

People were excited about the policy in code effort, only concern was
that the defacto documentation of what you could change wouldn't be in
the sample config.

ACTION: ensure there is policy config reference now that the sample
file is empty
ACTION: flavor RBAC is a thing most of the room wanted, is there a
taker on spec / implementation?

Upgrade
-------

Everyone waits to do any optional thing until they absolutely have
to.

The Cells API db caught a bunch of people off guard because it was in
Kilo (with release note) optional. Status quo in Liberty, with no
release note about it existing, then forced in Mitaka. When an
optional component is out there make sure it continues to be talked
about in releases even when it's status did not change, or people
forget.

People were on Kilo, so EC2 out of tree didn't really have any
data. About 25% of folks users have some existing AWS tooling, that
it's good to be able to just let them use to onboard them.

The current DB online data upgrade model feels *very opaque* to
ops. They didn't realize the current model Nova was using, and didn't
feel like it was documented anywhere.

ACTION: document the DB data lifecycle better for operators
ACTION: make sure we are cautious in rewarning people about changes
they have to make (like Cells API db)

API
---

API upgrade seemed fine for folks. The only question was the new
policy names, which was taking folks a bit of time to adjust to.

No one in the room was using custom API extensions (or at least
admitted to it when I asked).

Tracking Feedback
-----------------

We talked a bit about tracking feedback. The silence on the ops list
mostly comes from people not using a particular feature, so they don't
really have an opinion.

Most ops do not have time to look at our specs. That is an unlikely
place to get feedback.

Additional Questions
--------------------

There was an ask about VM HA. I stated that was beyond scope for Nova,
plus Nova's view of the world is non authoritative enough you didn't
want it to do that anyway. I told folks that the NFV efforts were
working on this kind of thing beyond Nova, and people should team up
there.

There was an ask on status of Cinder Multi Attach. We gave them a bit
of status on where things were at.

ACTION: Cinder Multi Attach should maybe be a priority effort in the
next cycle.


Upgrade Pain Points
===================

Raw etherpad -
https://etherpad.openstack.org/p/NYC-ops-Upgrades-Pain-points

Most people are a couple of releases back (Kilo / Liberty or even
older). The only team CDing in the room was RAX, they are now 2 to 3
months behind master.

Everyone agrees upgrades are getting better with every release.

Most are taking change windows and downtime for upgrades.

Why are upgrades taking so long?
--------------------------------

About half way through this session I threw this powder keg into the
room, it generated a lot of feedback.

People are holding a lot of out of tree patches, causing
latency. These are for:

* bug fixes that get fixed on old versions of OpenStack, so upstream
  won't take them (chicken / egg problem of being on an old release)
* custom identity driver for keystone
* some new feature that customer wants, not taken upstream
* lack of time to invest working patches upstream

(Thinking out loud, I do wonder if there is a way we could close this gap)

Defcore is actually forcing upgrades, because people loose their
OpenStack trademark if they don't stay in the Defcore supported window.

Other Sessions
==============

There were a ton of other sessions there. Interesting things that I
remember from them.

In the session on deploying OpenStack in containers, there was a split
between idempotent (docker) vs. system (lxc/lxd) containers. Both were
getting used in different ways, and there was debate between the camps
as to which was most effective.

Ceilometer deployments are highly coupled to Heat, and seem to be only
used when users want Heat auto scaling.

There are noticable failures in our CLI / API paths when using UTF8
names for projects / resources. Noticed by many .eu folks. Would be
good to increase testing in areas like this.


The full list of all etherpads are here for anyone else looking to
dive in and learn more - https://etherpad.openstack.org/p/NYC-ops-meetup

	-Sean

-- 
Sean Dague
http://dague.net