Open Stack

On 20/02/19 9:46 AM, Chris Dent wrote:
> 
> It's the Campaigning slot of the TC election process, where members
> of the community (including the candidates) are encouraged to ask
> the candidates questions and witness some debate. I have some
> questions.
> 
> First off, I'd like to thank all the candidates for running and
> being willing to commit some of their time. I'd also like to that
> group as a whole for being large enough to force an election. A
> representative body that is not the result of an election would not
> be very representing nor have much of a mandate.

+1

> The questions follow. Don't feel obliged to answer all of these. The
> point here is to inspire some conversation that flows to many
> places. I hope other people will ask in the areas I've chosen to
> skip. If you have a lot to say, it might make sense to create a
> different message for each response. Beware, you might be judged on
> your email etiquette and attention to good email technique!
> 
> * How do you account for the low number of candidates? Do you
>    consider this a problem? Why or why not?

In retrospect, it appears the tradition of waiting until the last 
possible minute had a lot to do with it. But I definitely had some other 
thoughts the day before nominations closed.

Looking at some of the folks who either decided not to run or who 
delayed their decision to run, I see a lot of standard, non-alarming 
reasons. Someone who is organising a wedding, someone who just changed 
jobs, someone who has a big internal project to work on at $DAYJOB, &c.

However, those things have always happened. The pool of candidates has 
shrunk to the point where folks who are borderline on having enough time 
to run are now the difference between having an election and having 
unfilled seats. That's in part due to the lower number of folks employed 
to work on OpenStack, and to the extra workload being shouldered by 
those left.

This is a problem for OpenStack, for at least the reason you mentioned 
above: TC members don't have much of a mandate if they didn't actually 
have an election.

Shrinking the TC is one option for dealing with this, and one we should 
probably at least investigate. But we could also work harder on 
developing future leaders of the project and getting them involved in 
the work of the TC (I guess the Stewardship WG used to be a more formal 
expression of this) before they eventually win seats on the TC - in much 
the same way that projects should hope to turn contributors into core 
reviewers.

> * Compare and contrast the role of the TC now to 4 years ago. If you
>    weren't around 4 years ago, comment on the changes you've seen
>    over the time you have been around. In either case: What do you
>    think the TC role should be now?

In some ways it's very different and in some ways depressingly familiar :)

Four years ago, the TC was focused on managing explosive growth of 
contributions (particularly in the form of new projects). The TC of that 
era did a lot of work putting governance into place that would make it 
possible to deal with the flood of new projects. That involved the TC 
removing itself from the technical review of new projects, in part 
because it had failed to give at least one project *any answer at all* 
on two successive graduation applications due to a lack of time on the 
part of TC members to investigate what it was for.

Fast-forwarding to the present, the flood has slowed to a trickle and 
the governance structures are mature, but the TC is still in many ways 
very focused on 'governance' to the exclusion of helping to get the 
technical community to work in concert.

> * What, to you, is the single most important thing the OpenStack
>    community needs to do to ensure that packagers, deployers, and
>    hobbyist users of OpenStack are willing to consistently upstream
>    their fixes and have a positive experience when they do? What is
>    the TC's role in helping make that "important thing" happen?

Since you're forcing me to pick only one thing, I am going to say 
'education'.

The economics of participating in an open source project are just not 
intuitive to anyone who isn't steeped in this stuff, and most people are 
not. Most companies are vulnerable to acting especially clueless, 
because the folks making the decisions are not the same folks who would 
even have the chance to work in the community. Even most of the 
companies working on OpenStack in the early days were pretty bad at it, 
though almost all improved with practice.

Forking a rapidly-evolving project is one of the most expensive mistakes 
you'll ever make, and if people understood that I really don't believe 
they'd do it because the time-to-market gains or whatever are just not 
worth it, and soon evaporate when you're looking at the time-to-market 
for v2 of your now-unmaintainable product.

I feel like it's part of OpenStack's role in the world to explain the 
open source way - both the benefits it confers and the responsibilities 
it entails - to folks who are receptive but who were not closely 
associated enough with projects like Linux to have learned the ins and outs.

(If I get a second answer, it'd be the one I brought up at the Forum in 
Berlin: prioritising giving immediate feedback to any and all 
contributions from those folks. 
http://lists.openstack.org/pipermail/openstack-discuss/2018-November/000052.html)

> * If you had a magic wand and could inspire and make a single
>    sweeping architectural or software change across the services,
>    what would it be? For now, ignore legacy or upgrade concerns.

Oh wow you went there.

If I had to pick one and I had a magic wand, I would replace the current 
backend infrastructure (comprising MariaDB, RabbitMQ, and at least 
theoretically etcd3) with a single component, with requirements such as 
the following:

- Efficiently scalable to some defined level
- First-class support for messaging patterns (e.g. pub-sub)
- Reliable to a passes-jepsen-or-better standard
- Well-defined semantics for resolving conflicts/partitions/&c.
- Support for large numbers of geographically distributed replicas (for 
edge use cases)
- To the extent that any of this needs to be implemented in the 
OpenStack community, it only has to be implemented once

In response to the inevitable complaints:

- Yes, MySQL can be scaled using sharding. But we have 50+ projects and 
only 2 implementations of sharding. And to underscore just how difficult 
that is, both were in the same project.
- Yes, RabbitMQ supports messaging patterns, but it's not something 
we've built our services around (e.g. Kubernetes is entirely event 
driven, but in OpenStack very little data escapes the compute node 
without querying for it) - in fact we mostly use it (inappropriately) 
for RPC. And it's a separate service from the database, so you can't 
e.g. have a transaction that includes sending a message.
- RabbitMQ failed Jepsen spectacularly 
(https://aphyr.com/posts/315-call-me-maybe-rabbitmq) - if you can manage 
to keep it running at all. It's a constant source of complaints from 
OpenStack operators. Galera also failed BTW 
(https://aphyr.com/posts/327-call-me-maybe-mariadb-galera-cluster).
- As an OpenStack developer, I can't even find any documentation to tell 
me what transaction isolation level I should expect from the DB. Not 
only is it not documented anywhere, but AFAICT until very recently we 
officially supported two *different* levels and at least tacitly 
encouraged operators to effectively choose for themselves which to use. 
How can I, as an OpenStack developer, even hope to write correct code 
under those circumstances? This is not theoretical; many times I've been 
in discussions that ended with us just guessing what the database would 
do and hoping for the best. I went searching fruitlessly for docs yet 
again because I literally spent half of Wednesday trying to implement 
something without having a race condition, and whether it was possible 
or not (spoiler: it's not) depended on the transaction isolation level.
- Something something Edge.
- See the first point above.

I'm not wedded to any particular choice, but I believe such things 
exist. I'm particularly intrigued by FoundationDB, although extremely 
wary of their non-open development model for the core.

Anyhow, with that in place I would wave my magic wand and build a 
messaging library and then a user-facing API that used it. Then I'd have 
everything in OpenStack work together as an event-driven system, with 
(select) events potentially routed out to userspace and back in again if 
the user so chose.

I don't think there's a path to this happening, but I think it's a very 
valuable exercise to imagine how things could be and think about what 
we're missing out on.

>    What role should the TC have in inspiring and driving such
>    changes?

I don't think it should be the TC's job to necessarily come up with such 
ideas. Great ideas might come from anywhere. And I don't think it can be 
the TC's job to tell people what ideas to implement, because that's not 
how open source works.

Plagiarising myself (from 
https://review.openstack.org/#/c/622400/1/reference/role-of-the-tc.rst@73): 
"The ideas we need are out there in the community. The TC's job is to 
ensure that there's space for them to be heard, to amplify the best 
ones, to build consensus, to ensure that decisions get made when 
consensus is not possible (i.e. we don't default to paralysis when 
reasonable people disagree), and to be electorally accountable for 
getting it wrong."

> * What can the TC do to make sure that the community (in its many
>    dimensions) is informed of and engaged in the discussions and
>    decisions of the TC?

The TC is extremely open in how it conducts it's business. So the 
challenge, as always is in filtering the firehose.

I think the weekly TC update email from the chair is a big improvement, 
because it's significantly easier to engage with than trying to follow 
everything.

> * How do you counter people who assert the TC is not relevant?
>    (Presumably you think it is, otherwise you would not have run. If
>    you don't, why did you run?)

Trick question! I wouldn't try to counter them. If somebody believes 
that the TC is not relevant *to them* then they're almost certainly 
right. I would be interested in listening to what things _would_ be 
relevant to them and exploring whether those things should be done, 
whether they should be done by the TC, or whether the TC should delegate 
them to some other group. But let's remember that there will always be 
people who are just heads-down in their own project getting stuff done 
and never needing to worry about the TC, and that's fine. OpenStack can 
accommodate an ~unlimited number of those folks. But it can't prosper 
with only those folks. We need people who keep the lines of 
communication open between projects and shape the system as a whole. If 
the TC can't be relevant to that latter group, then we really have a 
problem.

cheers,
Zane.