Open Stack

On Fri, 17 Aug 2018, Doug Hellmann wrote:

> If we ignore the political concerns in the short term, are there
> other projects actually interested in using placement? With what
> technical caveats? Perhaps with modifications of some sort to support
> the needs of those projects?

I think ignoring the political concerns (in any term) is not
possible. We are a group of interacting humans, politics are always
present. Cordial but active debate to determine the best course of
action is warranted.

(tl;dr: Let's have existing and potential placement contributors
decide its destiny.)

Five topics I think are relevant here, in order of politics, least
to most:

1. Placement has been designed from the outset to have a hard
contract between it and the services that use it. Being embedded
and/or deeply associated with one other single service means that
that contract evolves in a way that is strongly coupled. We made
placement have an HTTP API, not use RPC, and not produce or consume
notifications because it is supposed to be bounded and independent.
Sharing code and human management doesn't enable that. As you'll
read below, placement's progress has been overly constrained by
compute.

2. There are other projects actively using placement, not merely
interested. If you search codesearch.o.o for terms like "resource
provider" you can find them. But to rattle off those that I'm aware
of (which I'm certain is an incomplete list):

* Cyborg is actively working on using placement to track FPGA
   e.g., https://review.openstack.org/#/c/577438/

* Blazar is working on using them for reservations:
   https://review.openstack.org/#/q/status:open+project:openstack/blazar+branch:master+topic:bp/placement-api

* Neutron has been reporting to placement for some time and has work
   in progress on minimum bandwidth handling with the help of
   placement:
   https://review.openstack.org/#/q/status:open+project:openstack/neutron-lib+branch:master+topic:minimum-bandwidth-allocation-placement-api

* Ironic uses resource classes to describe types of nodes

* Mogan (which may or may not be dead, not clear) was intending to
   track nodes with placement:
   http://git.openstack.org/cgit/openstack/mogan-specs/tree/specs/pike/approved/track-resources-using-placement.rst

* Zun is working to use placement for "unified resource management":
   https://blueprints.launchpad.net/zun/+spec/use-placement-resource-management

* Cinder has had discussion about using placement to overcome race
   conditions in its existing scheduling subsystem (a purpose to
   which placement was explicitly designed).

3. Placement's direction and progress is heavily curtailed by the
choices and priorities that compute wants or needs to make. That
means that for the past year or more much of the effort in placement
has been devoted to eventually satisfying NFV use cases driven by
"enhanced platform awareness" to the detriment of the simple use
case of "get me some resource providers". Compute is under a lot of
pressure in this area, and is under-resourced, so placement's
progress is delayed by being in the (necessarily) narrow engine of
compute. Similarly, computes's overall progress is delayed because a
lot of attention is devoted to placement.

I think the relevance of that latter point has been under-estimated
by the voices that are hoping to keep placement near to nova. The
concern there has been that we need to continue iterating in concert
and quickly. I disagree with that from two angles. One is that we
_will_ continue to work in concert. We are OpenStack, and presumably
all the same people working on placement now will continue to do so,
and many of those are active contributors to nova. We will work
together.

The other angle is that, actually, placement is several months ahead
of nova in terms of features and it would be to everyone's advantage if
placement, from a feature standpoint, took a time out (to extract)
while nova had a chance to catch up with fully implementing shared
providers, nested resource providers, consumer generations, resource
request groups, using the reshaper properly from the virt drivers,
having a fast forward upgrade script talking to PlacementDirect, and
other things that I'm not remembering right now. The placement side
for those things is in place. The work that it needs now is a
_diversity_ of callers (not just nova) so that the features can been
fully exercised and bugs and performance problems found.

The projects above, which might like to--and at various times have
expressed desire to do so--work on features within placement that
would benefit their projects, are forced to compete with existing
priorities to get blueprint attention. Though runways seemed to help
a bit on that front this just-ending cycle, it's simply too dense a
competitive environment for good, clean progress.

4. While extracting the placement code into another repo within the
compute umbrella might help a small amount with some of the
competition described in item 3, it would be insufficient. The same
forces would apply.

Similarly, _if_ there are factors which are preventing some people
from being willing to participate with a compute-associated project,
a repo within compute is an insufficient break.

Also, if we are going to go to the trouble of doing any kind of
disrupting transition of the placement code, we may as well take as
a big a step as possible in this one instance as these opportunities
are rare and our capacity for change is slow. I started working on
placement in early 2016, at that time we had plans to extract it to
"it's own thing". We've passed the half-way point in 2018.

5. In OpenStack we have a tradition of the contributors having a
strong degree of self-determination. If that tradition is to be
upheld, then it would make sense that the people who designed and
wrote the code that is being extracted would get to choose what
happens with it. As much as Mel's and Dan's (only picking on them
here because they are the dissenting voices that have showed up so
far) input has been extremely important and helpful in the evolution
of placement, they are not those people.

So my hope is that (in no particular order) Jay Pipes, Eric Fried,
Takashi Natsume, Tetsuro Nakamura, Matt Riedemann, Andrey Volkov,
Alex Xu, Balazs Gibizer, Ed Leafe, and any other contributor to
placement whom I'm forgetting [1] would express their preference on
what they'd like to see happen.

At the same time, if people from neutron, cinder, blazar, zun,
mogan, ironic, and cyborg could express their preferences, we can get
through this by acclaim and get on with getting things done.

Thank you.

[1] My apologies if I have left you out. It's Saturday, I'm tried
from trying to make this happen for so long, and I'm using various
forms of git blame and git log to extract names from the git history
and there's some degree of magic and guessing going on.

-- 
Chris Dent                       ٩◔̯◔۶           https://anticdent.org/
freenode: cdent                                         tw: @anticdent